# How to decide weight parameters of nn.Cross_entropy() correctly

I am solving a multi-class image segmentation problem.
Classes are imbalanced.
I am trying to calibrate weight parameter of nn.Cross_entropy() using the below weight -

``````weights = torch.tensor([0.90,0.40,0.40,0.20])
criterion = torch.nn.CrossEntropyLoss(weights)
``````

Above number , I used were arbitrary.
On calculating number of different classes in a single image, following it the result -
`(tensor([0, 1, 2, 3]), tensor([305903, 12655, 10342, 1060]))`

Generally, the above values will vary for each image but class 0 will remain in abundance(it is background), class 1 and class 2 in moderation( desired segmented classes), class 3 will be very less( it is for unknown cases.
How to choose weights values for it?

Hi Mr. Bing!

There is no single right answer for choosing weights to reweight an
unbalanced data set. But a good starting point is:
`weight_for_class = 1 / number_of_samples_in_class`.

The idea is that you might want each class to be making about the
same overall contribution to the loss function – so, more samples
in a class means you use a lower weight.

``````counts = torch.FloatTensor ([305903, 12655, 10342, 1060])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()
``````

(I’m using the old pytorch version 0.3.0.)

You don’t need this, but if you want your weights to be normalized (sum
to one), the last line illustrates this.

This yields:

``````>>> counts

3.0590e+05
1.2655e+04
1.0342e+04
1.0600e+03
[torch.FloatTensor of size 4]

>>> weights

1.00000e-04 *
0.0327
0.7902
0.9669
9.4340
[torch.FloatTensor of size 4]

>>> weights_normalized

0.0029
0.0704
0.0862
0.8405
[torch.FloatTensor of size 4]
``````

Good luck!

K. Frank

2 Likes

Hi K. Frank,

Thanks for your explanation. I would like to validate my approach of getting the `counts` in the sample code above and some of my concerns for volumetric CT data.

``````import torch
torch.manual_seed(3)

image = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelA = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
labelB = torch.randint(low=0, high=2, size=(3, 3), dtype=torch.float32)
``````

Generating the background label from the annotated label A & B i.e 0 in pixel value where either A or B is 1 and 1 elsewhere

``````background = torch.zeros((labelB.shape), dtype=torch.long)

for label in [labelA, labelB]:
background |= background.long()

background = background^1
outputs = f"Image: {image},\n background: {background}, \n labelA: {labelA}, \n labelB: {labelB}"
print(outputs)
``````
``````>>> outputs
Image: tensor([[0.0043, 0.1056, 0.2858],
[0.0270, 0.4716, 0.0601],
[0.7719, 0.7437, 0.5944]]),
background: tensor([[0, 1, 0],
[0, 0, 1],
[0, 0, 0]]),
labelA: tensor([[1., 0., 1.],
[1., 1., 0.],
[1., 1., 0.]]),
labelB: tensor([[0., 0., 0.],
[1., 1., 0.],
[0., 0., 1.]])
``````
``````counts = torch.tensor([torch.sum(background), torch.sum(labelA), torch.sum(labelB)])
weights = 1.0 / counts
weights_normalized = (1.0 / counts) / (1.0 / counts).sum()
``````
``````>>> counts
tensor([4., 3., 4.])
>>> weights
tensor([0.2500, 0.3333, 0.2500])
>>>weights_normalized
tensor([0.3000, 0.4000, 0.3000])
``````

These weights can now be passed into the loss function. My concerns are:

• I believe my approach of creating the background is not well optimized, is there an inbuilt PyTorch function that can be used or another way around it?
• Is it computationally efficient to create the background label class (for each corresponding image and label A/B) in the data loader or separately create it out the data loader, save it (.npy), and then reference it just like the label class A & B?