How to decide weight parameters of nn.Cross_entropy() correctly

1

I am solving a multi-class image segmentation problem.
Classes are imbalanced.
I am trying to calibrate weight parameter of nn.Cross_entropy() using the below weight -

weights = torch.tensor([0.90,0.40,0.40,0.20])
criterion = torch.nn.CrossEntropyLoss(weights)

Above number , I used were arbitrary.
On calculating number of different classes in a single image, following it the result -
(tensor([0, 1, 2, 3]), tensor([305903, 12655, 10342, 1060]))

Generally, the above values will vary for each image but class 0 will remain in abundance(it is background), class 1 and class 2 in moderation( desired segmented classes), class 3 will be very less( it is for unknown cases.
How to choose weights values for it?

2

Hi Mr. Bing!

There is no single right answer for choosing weights to reweight an
unbalanced data set. But a good starting point is:
weight_for_class = 1 / number_of_samples_in_class.

The idea is that you might want each class to be making about the
same overall contribution to the loss function – so, more samples
in a class means you use a lower weight.

So, you might start with:

counts = torch.FloatTensor ([305903, 12655, 10342, 1060])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()

(I’m using the old pytorch version 0.3.0.)

You don’t need this, but if you want your weights to be normalized (sum
to one), the last line illustrates this.

This yields:

>>> counts

 3.0590e+05
 1.2655e+04
 1.0342e+04
 1.0600e+03
[torch.FloatTensor of size 4]

>>> weights

1.00000e-04 *
  0.0327
  0.7902
  0.9669
  9.4340
[torch.FloatTensor of size 4]

>>> weights_normalized

 0.0029
 0.0704
 0.0862
 0.8405
[torch.FloatTensor of size 4]

Good luck!

K. Frank

2 Likes
3

Hi K. Frank,

Thanks for your explanation. I would like to validate my approach of getting the counts in the sample code above and some of my concerns for volumetric CT data.

import torch
torch.manual_seed(3)

image = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelA = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelB = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)

Generating the background label from the annotated label A & B i.e 0 in pixel value where either A or B is 1 and 1 elsewhere

background = torch.zeros((labelB.shape), dtype=torch.long)

for label in [labelA, labelB]:
    background |= background.long()

background = background^1
outputs = f"Image: {image},\n background: {background}, \n labelA: {labelA}, \n labelB: {labelB}"
print(outputs)

>>> outputs
Image: tensor([[0.0043, 0.1056, 0.2858],
        [0.0270, 0.4716, 0.0601],
        [0.7719, 0.7437, 0.5944]]),
 background: tensor([[0, 1, 0],
        [0, 0, 1],
        [0, 0, 0]]), 
 labelA: tensor([[1., 0., 1.],
        [1., 1., 0.],
        [1., 1., 0.]]), 
 labelB: tensor([[0., 0., 0.],
        [1., 1., 0.],
        [0., 0., 1.]])

counts = torch.tensor([torch.sum(background), torch.sum(labelA), torch.sum(labelB)])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()

>>> counts
tensor([4., 3., 4.])
>>> weights
tensor([0.2500, 0.3333, 0.2500])
>>>weights_normalized
tensor([0.3000, 0.4000, 0.3000])

These weights can now be passed into the loss function. My concerns are:

  • I believe my approach of creating the background is not well optimized, is there an inbuilt PyTorch function that can be used or another way around it?
  • Is it computationally efficient to create the background label class (for each corresponding image and label A/B) in the data loader or separately create it out the data loader, save it (.npy), and then reference it just like the label class A & B?

Thanks for your enlightenment