MultiLabel Classification and Cross Entropy Weights

vision
1

Good afternoon!
I have a model that has 6 classes on which each class has several possible labels. I wanted to ask if it is possible to give a list of weights for each label of each class.

My model:

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        
        self.model = pretrainedmodels.__dict__["resnet50"](pretrained="imagenet")
        
        self.fc1 = nn.Linear(2048, 3)    
        self.fc3 = nn.Linear(2048, 5)    
        self.fc4 = nn.Linear(2048, 4)    
        self.fc5 = nn.Linear(2048, 5)    
        self.fc7 = nn.Linear(2048, 2)   
        self.fc9 = nn.Linear(2048, 3)    
        
        
    def forward(self, x):
        bs, _, _, _ = x.shape
        x = self.model.features(x)
        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)
        label1 = torch.sigmoid(self.fc1(x))
        label3 = torch.sigmoid(self.fc3(x))  
        label4 = torch.sigmoid(self.fc4(x))  
        label5 = torch.sigmoid(self.fc5(x))  
        label7 = torch.sigmoid(self.fc7(x))
        label9 = self.fc9(x)
          
        return {'label1': label1, 'label3': label3, 'label4': label4, 'label5': label5, 'label7': label7, 'label9': label9}

I wanted to do this:

weights = [[0.3, 0.3, 0.4], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3], [0.1, 0.5, 0.4]]

class_weights = torch.FloatTensor(weights).cuda()

criterion = nn.CrossEntropyLoss()

I get this error: ValueError: expected sequence of length 3 at dim 1 (got 4)

Thank you in advance

2

I don’t totally understand what you’re planning to use the weights for, but your error is coming from the fact that Tensors need to have a well-defined shape, however the elements in your list have different lengths. As a quick fix you can try padding with 0.0 weights to get a consistent shape, like this:

weights = [[0.3, 0.3, 0.4, 0.0], [0.2, 0.2, 0.2, 0.4], [0.3, 0.3, 0.3, 0.1], [0.3, 0.3, 0.3, 0.1], [0.7, 0.3, 0.0, 0.0], [0.1, 0.5, 0.4, 0.0]]

3

Thank you for your answer!!
I want to give a weight o each label of each class because I have an imbalance dataset.

For example, for Class1, I have label1, label2, label3. I want to give different weights to the different labels. Then I have Class2, with label1, label2, label3 and label4. Now I want to give weights to these labels of this class. This is what I want to do.

4

Hi Another!

Yes, you can weight your labels / classes individually.

But first, some context and terminology:

At a technical level, you are performing 6 multi-class classification
problems “in parallel.” What you call “6 classes,” I would call 6
classification problems. And what you call “several possible labels,”
I would call the separate sets of classes for those 6 classification
problems.

Your final fully-connected Linear layers make sense in this context (but
they can be combined together into a single Linear (2048, 22), which
is how I would do it).

Get rid of the sigmoid()s. (They are not appropriate for a multi-class
problem. Furthermore, CrossEntropyLoss, which you should use,
has, in effect, softmax() built into it, and that plays the role that you
might be imagining for sigmoid().)

You have 6 separate classification problems (with differing numbers of
classes), so you will want 6 separate CrossEntropyLoss loss criteria
(that you then sum together, either equally or in some weighted fashion).

However, pass in the slices of your class_weights tensor into the
constructors of your 6 loss criteria as the class weights for each of
the 6 problems separately, e.g., something like:

loss_fn1 = nn.CrossEntropyLoss (weight = class_weights[0])
loss_fn3 = nn.CrossEntropyLoss (weight = class_weights[1])
...
loss_fn9 = nn.CrossEntropyLoss (weight = class_weights[5])

The weight constructor argument of CrossEntropyLoss consists of
“class-weight” values that weight the classes differently in the loss
function, and is often used to compensate for unbalanced data.

Best.

K. Frank

5

Thank you so much for the detailed explanation! Very useful!
Cheers :smiley: