# Passing the weights to CrossEntropyLoss correctly

**URL:** <https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731>\
**Category:** Uncategorized\
**Created:** [March 10, 2018, 10:05pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731 "2018-03-10T22:05:18Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivan-bilan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ivan-bilan/32/3481_2.png) [@ivan-bilan](https://discuss.pytorch.org/u/ivan-bilan)\
**Post date:** [March 10, 2018, 10:05pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/1 "2018-03-10T22:05:18Z")

</div>

Hi,

I just wanted to ask how the mechanism of passing the weights to CrossEntropyLoss works. Currently, I have a list of class labels that are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

I create the loss function in the **init** and pass the weights to the loss:

```auto
weights = [0.5, 1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
class_weights = torch.FloatTensor(weights).cuda()
self.criterion = nn.CrossEntropyLoss(weight=class_weights)

```

Then in the update step, I pass the labels of my current batch to the loss function:

```auto
self.model.train()
self.optimizer.zero_grad()
logits, _ = self.model(inputs)
loss = self.criterion(logits, labels)

```

Now the labels can be something like this: [2, 4, 2, 1, 0, 4, 5]

So, since the weights are passed as a list, will they be automatically mapped to the current labels passed? like so: [2: 1.0, 4: 0.3, 2: 1.0, 1: 1.0, 0: 0.5, 4: 0.3, 5: 1.0]?

The reason I am asking this is that I would be much more comfortable passing a dictionary of weights to the loss function, instead of a list. So that I am sure the weights are mapped to the proper classes. Can someone elaborate on the way weights are passed to the loss function here?

---

<div class="post-metadata">

**Author:** ![richard](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/32/1861_2.png) [@richard](https://discuss.pytorch.org/u/richard)\
**Post date:** [March 12, 2018, 1:53pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/2 "2018-03-12T13:53:54Z")

</div>

Your interpretation is correct, that is how the class weights will work with CrossEntropyLoss.

There’s a little more detail on the docs on how this is done: [http://pytorch.org/docs/master/nn.html?highlight=nll%20loss#torch.nn.NLLLoss](http://pytorch.org/docs/master/nn.html?highlight=nll%20loss#torch.nn.NLLLoss)

---

<div class="post-metadata">

**Author:** ![amy](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@amy](https://discuss.pytorch.org/u/amy)\
**Post date:** [January 25, 2019, 6:02am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/5 "2019-01-25T06:02:44Z")

</div>

how are we getting the weights? Like if the true distribution while testing is 10:1 (for sample frequency) and in training it is 1:1 (sample frequency). What should the weights be in that case?

---

<div class="post-metadata">

**Author:** ![soltius](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soltius/32/49726_2.png) [@soltius](https://discuss.pytorch.org/u/soltius)\
**Post date:** [February 5, 2019, 9:13am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/6 "2019-02-05T09:13:10Z")

</div>

You shouldn’t have such variations in your distributions between train and test anyhow.

---

<div class="post-metadata">

**Author:** ![zezhishao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zezhishao/32/22260_2.png) [@zezhishao](https://discuss.pytorch.org/u/zezhishao)\
**Post date:** [April 8, 2019, 3:14am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/7 "2019-04-08T03:14:55Z")

</div>

Hi,  
I noticed that the `KLDivLoss` class does not have the parameter `weight`, so how can I passing the weight to it?

---

<div class="post-metadata">

**Author:** ![BikashgG](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bikashgg/32/1116_2.png) [@BikashgG](https://discuss.pytorch.org/u/BikashgG)\
**Post date:** [May 6, 2019, 11:32am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/8 "2019-05-06T11:32:57Z")

</div>

Hi, shouldn’t your weights/class\_weights vector be normalised (i.e. the total adds up to 1) before passing it to the CrossEntropyLoss function?

---

<div class="post-metadata">

**Author:** ![MariosOreo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mariosoreo/32/18330_2.png) [@MariosOreo](https://discuss.pytorch.org/u/MariosOreo)\
**Post date:** [May 7, 2019, 1:26am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/9 "2019-05-07T01:26:49Z")

</div>

I think we don’t need to normalize the weights of classes, using invert class sampling count is okay.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 7, 2019, 10:41am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/10 "2019-05-07T10:41:42Z")

</div>

Small addition to @MariosOreo’s answer:  
if your loss function uses `reduction='mean'`, the loss will be normalized by the sum of the corresponding weights for each element. If you are using `reduction='none'`, you would have to take care of the normalization yourself.  
Here is a small example:

```python
x = torch.randn(10, 5)
target = torch.randint(0, 5, (10,))

weights = torch.tensor([1., 2., 3., 4., 5.])
criterion_weighted = nn.CrossEntropyLoss(weight=weights)
loss_weighted = criterion_weighted(x, target)

criterion_weighted_manual = nn.CrossEntropyLoss(weight=weights, reduction='none')
loss_weighted_manual = criterion_weighted_manual(x, target)
loss_weighted_manual = loss_weighted_manual.sum() / weights[target].sum()

print(loss_weighted == loss_weighted_manual)

```

---

<div class="post-metadata">

**Author:** ![balloch](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/balloch/32/10942_2.png) [@balloch](https://discuss.pytorch.org/u/balloch)\
**Post date:** [May 31, 2019, 9:42pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/11 "2019-05-31T21:42:11Z")

</div>

This is actually exactly related to my problem; working in a continual learning setting, the number of classes changes through time; If I want to use weight balancing, but the weights vector changes through time, what do I do here? Do I have to reinitialize my loss for every new task to use weight balancing?

---

<div class="post-metadata">

**Author:** ![gunshi\_gupta](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gunshi_gupta/32/16217_2.png) [@gunshi\_gupta](https://discuss.pytorch.org/u/gunshi_gupta)\
**Post date:** [September 25, 2019, 8:35pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/12 "2019-09-25T20:35:53Z")

</div>

@ptrblck will there be gradients backpropp-ed through these weights if we constructed the weights from the logits themselves (as in focal losses for object detection ([https://arxiv.org/pdf/1708.02002.pdf](https://arxiv.org/pdf/1708.02002.pdf)))

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 25, 2019, 10:42pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/13 "2019-09-25T22:42:44Z")

</div>

No, as the derivative for `weights` is not implemented.  
You could check it by passing a `weights` tensor using `requires_grad=True`, which will raise the exception:

```python
RuntimeError: the derivative for 'weight' is not implemented

```

However, have a look at [this implementation](https://github.com/mbsariyildiz/focal-loss.pytorch/blob/6551bd3e433ce41020b6bc8d99221eb6cd10ae17/focalloss.py#L33) of the focal loss.

---

<div class="post-metadata">

**Author:** ![gunshi\_gupta](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gunshi_gupta/32/16217_2.png) [@gunshi\_gupta](https://discuss.pytorch.org/u/gunshi_gupta)\
**Post date:** [September 26, 2019, 12:17am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/14 "2019-09-26T00:17:00Z")

</div>

I see, thanks.  
Alternatively, even if i construct this weights matrix and matrix multiply it with the loss matrix i get back when using the loss with reduction = ‘none’, it should be equivalent right?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 26, 2019, 12:19am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/15 "2019-09-26T00:19:21Z")

</div>

I think so, but would need to see the code to be sure. 😉  
However, I’m not sure, how you are constructing the weights and if they should also be trainable or not.

---

<div class="post-metadata">

**Author:** ![gunshi\_gupta](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gunshi_gupta/32/16217_2.png) [@gunshi\_gupta](https://discuss.pytorch.org/u/gunshi_gupta)\
**Post date:** [September 26, 2019, 12:27am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/16 "2019-09-26T00:27:25Z")

</div>

In my case the final focal loss computation looks like the code below (focal loss is supposed to backprop the gradients even through the weights as i understand, since none of the repos i referenced including the one mentioned above, calls detach() on these weights for which backward() is well defined):

```
    alpha_factor = torch.ones(targets.shape).cuda() * alpha
    alpha_factor = torch.where(targets_eq_1, 1. - alpha_factor, alpha_factor)             
    focal_weight = torch.where(targets_eq_1, 1. - classification, classification)
    focal_weight = alpha_factor * torch.pow(focal_weight, gamma)
    cls_loss = F.binary_cross_entropy_with_logits(classification, targets, size_average=False, reduction = 'none')
    final_loss = focal_weight*cls_loss
```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 26, 2019, 12:31am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/17 "2019-09-26T00:31:20Z")

</div>

You might be right, but wouldn’t you recreate `focal_weight` in each iteration (based on the targets)?  
Where would the gradient in `focal_weight` be used in this case?

---

<div class="post-metadata">

**Author:** ![gunshi\_gupta](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gunshi_gupta/32/16217_2.png) [@gunshi\_gupta](https://discuss.pytorch.org/u/gunshi_gupta)\
**Post date:** [September 26, 2019, 12:44am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/18 "2019-09-26T00:44:18Z")

</div>

Yes, this focal weight would change every iteration. I myself wasn’t entirely sure if focal weights should be backpropagated into, but it seems that way from open-source implementations (the original paper doesn’t have a clarification [https://arxiv.org/abs/1708.02002](https://arxiv.org/abs/1708.02002))  
In this case, the gradients would flow from the focal weights into/through the ‘classification’ vector (which are the logits) I guess.

---

<div class="post-metadata">

**Author:** ![John\_Grabner](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_grabner/32/26165_2.png) [@John\_Grabner](https://discuss.pytorch.org/u/John_Grabner)\
**Post date:** [July 24, 2020, 9:14pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/19 "2020-07-24T21:14:59Z")

</div>

I’m missing something. Loss (Error on rare class) same as Loss (Error on common class)?

> weight\_inbalance=torch.tensor([1000.0,1.0]) # very rare first class  
> loss\_inbalance = torch.nn.CrossEntropyLoss(weight=weight\_inbalance,reduction=‘mean’)
> 
> - batch of 2 samples, toggle only first sample  
> inbal\_err\_on\_rare = loss\_inbalance(torch.tensor([[-1.,-1],[-1,1]]), torch.tensor([0,1]))  
> inbal\_err\_on\_com = loss\_inbalance(torch.tensor([[1.,1],[-1,1]]), torch.tensor([0,1]))  
> print(inbal\_err\_on\_rare, inbal\_err\_on\_com)  
> tensor(0.6926) tensor(0.6926)

The same experiment on a ballance class yeilds the same equivalence, just offset in value.

> weight\_balance=torch.tensor([1.0,1.0])  
> loss\_balance = torch.nn.CrossEntropyLoss(weight=weight\_balance,reduction=‘mean’)  
> bal\_err\_on\_not\_rare = loss\_balance(torch.tensor([[-1.,-1],[-1,1]]), torch.tensor([0,1]))  
> bal\_err\_on\_com = loss\_balance(torch.tensor([[1.,1],[-1,1]]), torch.tensor([0,1]))  
> print(bal\_err\_on\_not\_rare, bal\_err\_on\_com)  
> tensor(0.4100) tensor(0.4100)

I generated the contours, and they look the same, just shifted.

> weight\_inbalance=torch.tensor([1000.0,1.0]) # very rare first class  
> weight\_balance=torch.tensor([1.0,1.0])  
> loss\_inbalance = torch.nn.CrossEntropyLoss(weight=weight\_inbalance,reduction=‘mean’)  
> loss\_balance = torch.nn.CrossEntropyLoss(weight=weight\_balance,reduction=‘mean’)  
> X = np.arange(-1., 1.1, 0.1)  
> Y = np.arange(-1., 1.1, 0.1)  
> Z\_inbalance = np.zeros((X.shape[0],Y.shape[0]))  
> Z\_balance = np.zeros((X.shape[0],Y.shape[0]))  
> #print(X.shape, Y.shape, Z.shape)
> 
> i = -1  
> for x in X:  
> i+=1  
> j=-1  
> for y in Y:  
> j+=1  
> batch=torch.tensor([[x, y], [-1, 1]], dtype=torch.float32)
> 
> ```
> X[i], Y[j], Z_inbalance[i, j] =x,y,loss_inbalance(batch, torch.tensor([0,1]))
> X[i], Y[j], Z_balance[i, j] =x,y,loss_balance(batch, torch.tensor([0,1]))
> 
> ```
> 
> fig, ax = plt.subplots(2)
> 
> CS0 = ax[0].contour(Y,X,Z\_inbalance,cmap=cm.coolwarm)  
> ax[0].clabel(CS0, inline=1, fontsize=14)  
> #ax[0].set\_title(‘Inbalanced Classes’)  
> ax[0].set\_ylabel(‘true class is rare’)  
> #ax[0].set\_xlabel(“not selected class is Common”)
> 
> CS1 = ax[1].contour(Y,X,Z\_balance,cmap=cm.coolwarm)  
> ax[1].clabel(CS1, inline=1, fontsize=14)  
> #ax[1].set\_title(‘Balanced Classes’)  
> ax[1].set\_ylabel(‘true class is common’)  
> ax[1].set\_xlabel(“not selected class is Common”)

So either I am missing the point of weights, or made a newbie error. Either way, I would appreciate some comments.

![image](https://discuss.pytorch.org/uploads/default/original/3X/2/9/293dcdf39dc322c3db509fe4cb7e9b0e1ffa8b05.png)

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [July 25, 2020, 4:50am UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/20 "2020-07-25T04:50:32Z")

</div>

Note that in your first example both output tensors predict class1 only, so it’s expected to get the same loss value.  
You can check it via:

```python
torch.argmax(torch.tensor([[-1.,-1],[-1,1]]), 1)
> tensor([1, 1])

torch.argmax(torch.tensor([[1.,1],[-1,1]]), 1)
> tensor([1, 1])

```

I haven’t looked into the second experiment as the formatting is unclear, but you could rerun it with another reduction (e.g. `reduction='sum'`), since the default `reduction='mean'` would normalize the loss with the applied weights.

---

<div class="post-metadata">

**Author:** ![John\_Grabner](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_grabner/32/26165_2.png) [@John\_Grabner](https://discuss.pytorch.org/u/John_Grabner)\
**Post date:** [July 25, 2020, 5:31pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/21 "2020-07-25T17:31:20Z")

</div>

Thank you, again.  
Not following how the argmax relates CrossEntropyLoss. Please elaborate.

You are correct, somehow the “mean” nullifies the weight bias.

Experiment 1 shows an error on any logit when classes are balanced results in the same loss.  
Experiment 2 shows if the rare class is the true class, the losses are amplified when any error occurs.

**Experiment 1:**

- balanced 2 classes,
- batch of 2 samples,
- introduce a single logit error and observe loss for true class = 0, 0 and 1, 1

> loss = torch.nn.CrossEntropyLoss(weight=torch.tensor([1.,1.]),reduction=‘sum’)  
> print(“2 balanced class, reduction=sum”)
> 
> true\_class\_index = torch.tensor([0,0]) # so perfect prediction is [1,-1], [1,-1]  
> incorrect\_sample\_0\_class\_0 = loss(torch.tensor([[-1., -1],[1.,-1]]), true\_class\_index)  
> incorrect\_sample\_0\_class\_1 = loss(torch.tensor([[1., 1],[1.,-1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_0 = loss(torch.tensor([[1., -1],[-1.,-1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_1 = loss(torch.tensor([[1., -1],[1.,1]]), true\_class\_index)  
> print(f"all have one logit wrong {incorrect\_sample\_0\_class\_0:0.2f}, {incorrect\_sample\_0\_class\_1:0.2f}, {incorrect\_sample\_1\_class\_0:0.2f}, {incorrect\_sample\_1\_class\_1:0.2f}")
> 
> - all have one logit wrong 0.82, 0.82, 0.82, 0.82
> 
> true\_class\_index = torch.tensor([1,1]) # so perfect prediction is [1,-1], [1,-1]  
> incorrect\_sample\_0\_class\_0 = loss(torch.tensor([[1., 1],[-1.,1]]), true\_class\_index)  
> incorrect\_sample\_0\_class\_1 = loss(torch.tensor([[-1., -1],[-1.,1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_0 = loss(torch.tensor([[-1., 1],[1.,1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_1 = loss(torch.tensor([[-1., 1],[-1.,-1]]), true\_class\_index)  
> print(f"all have one logit wrong {incorrect\_sample\_0\_class\_0:0.2f}, {incorrect\_sample\_0\_class\_1:0.2f}, {incorrect\_sample\_1\_class\_0:0.2f}, {incorrect\_sample\_1\_class\_1:0.2f}")
> 
> - all have one logit wrong 0.82, 0.82, 0.82, 0.82

**Experiment 2:**

- unbalanced 2 classes,
- batch of 2 samples,
- introduce a single logit error and observe loss for true class = 0, 0 and 1, 1

> loss = torch.nn.CrossEntropyLoss(weight=torch.tensor([10000.,1.]),reduction=‘sum’)  
> print(“2 unbalanced class, reduction=sum”)
> 
> true\_class\_index = torch.tensor([0,0]) # so perfect prediction is [1,-1], [1,-1]  
> incorrect\_sample\_0\_class\_0 = loss(torch.tensor([[-1., -1],[1.,-1]]), true\_class\_index)  
> incorrect\_sample\_0\_class\_1 = loss(torch.tensor([[1., 1],[1.,-1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_0 = loss(torch.tensor([[1., -1],[-1.,-1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_1 = loss(torch.tensor([[1., -1],[1.,1]]), true\_class\_index)  
> print(f"all have one logit wrong {incorrect\_sample\_0\_class\_0:0.2f}, {incorrect\_sample\_0\_class\_1:0.2f}, {incorrect\_sample\_1\_class\_0:0.2f}, {incorrect\_sample\_1\_class\_1:0.2f}")  
> **\> - all have one logit wrong 8200.75, 8200.75, 8200.75, 8200.75**
> 
> true\_class\_index = torch.tensor([1,1]) # so perfect prediction is [1,-1], [1,-1]  
> incorrect\_sample\_0\_class\_0 = loss(torch.tensor([[1., 1],[-1.,1]]), true\_class\_index)  
> incorrect\_sample\_0\_class\_1 = loss(torch.tensor([[-1., -1],[-1.,1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_0 = loss(torch.tensor([[-1., 1],[1.,1]]), true\_class\_index)  
> incorrect\_sample\_1\_class\_1 = loss(torch.tensor([[-1., 1],[-1.,-1]]), true\_class\_index)  
> print(f"all have one logit wrong {incorrect\_sample\_0\_class\_0:0.2f}, {incorrect\_sample\_0\_class\_1:0.2f}, {incorrect\_sample\_1\_class\_0:0.2f}, {incorrect\_sample\_1\_class\_1:0.2f}")
> 
> - all have one logit wrong 0.82, 0.82, 0.82, 0.82

btw, is there a way to post code snippets and not loose indentation on this site?

---

<div class="post-metadata">

**Author:** ![John\_Grabner](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_grabner/32/26165_2.png) [@John\_Grabner](https://discuss.pytorch.org/u/John_Grabner)\
**Post date:** [July 25, 2020, 6:28pm UTC](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731/22 "2020-07-25T18:28:14Z")

</div>

An excellent summary for CrossEntropy [https://youtu.be/ErfnhcEV1O8](https://youtu.be/ErfnhcEV1O8)

[Next page](https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731.md?page=2)