MultiLabel Classification and Cross Entropy Weights

Good afternoon!
I have a model that has 6 classes on which each class has several possible labels. I wanted to ask if it is possible to give a list of weights for each label of each class.

My model:

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.model = pretrainedmodels.__dict__["resnet50"](pretrained="imagenet")
        self.fc1 = nn.Linear(2048, 3)    
        self.fc3 = nn.Linear(2048, 5)    
        self.fc4 = nn.Linear(2048, 4)    
        self.fc5 = nn.Linear(2048, 5)    
        self.fc7 = nn.Linear(2048, 2)   
        self.fc9 = nn.Linear(2048, 3)    
    def forward(self, x):
        bs, _, _, _ = x.shape
        x = self.model.features(x)
        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)
        label1 = torch.sigmoid(self.fc1(x))
        label3 = torch.sigmoid(self.fc3(x))  
        label4 = torch.sigmoid(self.fc4(x))  
        label5 = torch.sigmoid(self.fc5(x))  
        label7 = torch.sigmoid(self.fc7(x))
        label9 = self.fc9(x)
        return {'label1': label1, 'label3': label3, 'label4': label4, 'label5': label5, 'label7': label7, 'label9': label9}

I wanted to do this:

weights = [[0.3, 0.3, 0.4], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3], [0.1, 0.5, 0.4]]

class_weights = torch.FloatTensor(weights).cuda()

criterion = nn.CrossEntropyLoss()

I get this error: ValueError: expected sequence of length 3 at dim 1 (got 4)

Thank you in advance

I don’t totally understand what you’re planning to use the weights for, but your error is coming from the fact that Tensors need to have a well-defined shape, however the elements in your list have different lengths. As a quick fix you can try padding with 0.0 weights to get a consistent shape, like this:

weights = [[0.3, 0.3, 0.4, 0.0], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3, 0.0, 0.0], [0.1, 0.5, 0.4, 0.0]]

Thank you for your answer!!
I want to give a weight o each label of each class because I have an imbalance dataset.

For example, for Class1, I have label1, label2, label3. I want to give different weights to the different labels. Then I have Class2, with label1, label2, label3 and label4. Now I want to give weights to these labels of this class. This is what I want to do.

Hi Another!

Yes, you can weight your labels / classes individually.

But first, some context and terminology:

At a technical level, you are performing 6 multi-class classification
problems “in parallel.” What you call “6 classes,” I would call 6
classification problems. And what you call “several possible labels,”
I would call the separate sets of classes for those 6 classification

Your final fully-connected Linear layers make sense in this context (but
they can be combined together into a single Linear (2048, 22), which
is how I would do it).

Get rid of the sigmoid()s. (They are not appropriate for a multi-class
problem. Furthermore, CrossEntropyLoss, which you should use,
has, in effect, softmax() built into it, and that plays the role that you
might be imagining for sigmoid().)

You have 6 separate classification problems (with differing numbers of
classes), so you will want 6 separate CrossEntropyLoss loss criteria
(that you then sum together, either equally or in some weighted fashion).

However, pass in the slices of your class_weights tensor into the
constructors of your 6 loss criteria as the class weights for each of
the 6 problems separately, e.g., something like:

loss_fn1 = nn.CrossEntropyLoss (weight = class_weights[0])
loss_fn3 = nn.CrossEntropyLoss (weight = class_weights[1])
loss_fn9 = nn.CrossEntropyLoss (weight = class_weights[5])

The weight constructor argument of CrossEntropyLoss consists of
“class-weight” values that weight the classes differently in the loss
function, and is often used to compensate for unbalanced data.


K. Frank

Thank you so much for the detailed explanation! Very useful!
Cheers :smiley: