Multi-Label, Multi-Class class imbalance

22

Hi… We need to take pos_Weight only from train_dataset or the whole dataset for a multilabel classifier

23

what is the relation between the weights, and the positive weights and negative weights which are computed based on the frequency of the positive labels and the negative? (in multi-label problem)

24

I’m unsure I understand the question completely as you would be using pos_weight in nn.BCEWithLogitsLoss, which is defined as nb_negative / nb_positive as described in the docs.

1 Like
25

Thanks, I get it now, i have one more question please, what is the appropriate way of calculating the weights, calculating them over each batch and passing them to the loss

loss = nn.BCEWithLogitsLoss()

for image, labels in train_loader:
     positive_weights = get_weights(labels)
     loss.pos_weight = positive_weights
....

or just one time on the entire training data then, pass them one time when we declare the loss nn.BCEWithLogitsLoss

26

I think the common approach would be to compute the pos_weight using the entire dataset.

1 Like
27

thank you very much for your help

28

Hi @ptrblck,
i still have a question. I have a sensor-based multi-label classification problem with highly unbalanced classes. Without treating the imbalance my F1-Score is going down, as my model predicts only certain labels (one of the highest occuring ones) ignoring the others. For each input (one point of a time series) always the same labels are predicted.
I tried to set the reduction to none and recalculate the loss with my class weights (calculated with sklearn.compute_class_weight), but this doesnt help. As a result, my model does not learn properly. I also tried to set my class weights in the pos_weights parameter in the BCELossWithLogits function, but it also doesn’t help the probem. The model still just predicts the same labels for each input.
Do you have any idea what the problem could be and how to tackle it?
Thank u in advance

29

Weighting the loss sounds like the right approach and you could increase the weight of tghe minority class(es) massively to see if this could force the model to change its behavior.

30

Thank you very much!
Do I need to adjust other parameters as well? As I increased the weights ( from negative weights/(positive weights*number_labels)) to (negative weights/postitive weights) my model does not learn anything anymore meaning the loss and accuracy are fluctuating all the time and my model is predicting that no label is active in the end.
Is it possible that i have to few input rows? I have ~300.000 input (sensor) rows and want to classify them to 51 (activity) labels (several labels can be active for one input row). My imbalance is very high, meaning that one label is active ~200.000 times and the label with the lowest occurence is active ~200 times.

So in general my problem is mulit-label classification for human activity recognition. Is the result of weighting the classes maybe due to inappropriate model input? For now i feed the model with single datapoints (representing one timestamp). Could segmention like sliding windows resolve the problem?

31

Hi Patrick,

For each epoch, I am training my model, and validating it at the same time (after going thru all the training samples of course), where number of samples in training is 3840 and number of samples in validation is 960. I have also put aside 1200 samples for testing the model at the end. When calculating these weights, do I use the total number of samples, ie 6000? Thank you!

32

I would use the training samples only as using information from the validation or test dataset could be seen as a data leak.

1 Like