# Question about weight in CrossEntropyLoss

**URL:** <https://discuss.pytorch.org/t/question-about-weight-in-crossentropyloss/95094>\
**Category:** Uncategorized\
**Created:** [September 3, 2020, 5:50pm UTC](https://discuss.pytorch.org/t/question-about-weight-in-crossentropyloss/95094 "2020-09-03T17:50:29Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [September 3, 2020, 10:41pm UTC](https://discuss.pytorch.org/t/question-about-weight-in-crossentropyloss/95094/2 "2020-09-03T22:41:15Z")

</div>

Hi Philipp!

> [@Philipp\_Singer](#):
>
> in the case of mean reduction, the loss is first scaled per sample, and then the sum is normalized by the sum of weights within the batch.

This is correct (if I understand what you are saying).

> [@](#):
>
> I wonder what is the idea behind that?

The “mean reduction” computes a (conventional) _weighted average_,  
that is, it does divide by the sum of the weights.

This makes sense to me because if, by happenstance, all of the  
samples in the batch have the same loss, `loss_all`, I would like the  
mean reduction over that batch also to give a batch mean of `loss_all`.  
The conventional weighted average does this.

These two threads (about `NLLLoss`, but its the same issue) give some  
additional words of explanation:

> [@Mean Reduction in nll\_loss](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314):
>
> According to nll\_loss documentation, for reduction parameter, " 'none' : no reduction will be applied, 'mean' : the sum of the output will be divided by the number of elements in the output, 'sum' : the output will be summed." However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate? input = torch.randn(3, 5, requires\_grad=True) ls =F.log\_softmax(input, 1…

> [@Weights in NllLoss behave unexpectedly](https://discuss.pytorch.org/t/weights-in-nllloss-behave-unexpectedly/93116):
>
> Hello, I’m having trouble understanding behaviour of class weights in CrossEntropyLoss. Specifically, when reduction=‘mean’. I test it like this: input = torch.randn(5, 2, requires\_grad=True) m = nn.LogSoftmax(dim=1) mi = m(input) target = torch.tensor([0, 0, 1, 1, 1]) w = torch.tensor([1, 100]).float() Now, without weights everything behaves reasonably, next 2 lines give the same result F.nll\_loss(mi, target, reduction=‘none’).mean() F.nll\_loss(mi, target, reduction=‘mean’) Howev…

Best.

K. Frank

---

_[View the full topic](https://discuss.pytorch.org/t/question-about-weight-in-crossentropyloss/95094)._