Question about weight in CrossEntropyLoss

[KFrank]

Hi Philipp!

@Philipp_Singer

in the case of mean reduction, the loss is first scaled per sample, and then the sum is normalized by the sum of weights within the batch.

This is correct (if I understand what you are saying).

I wonder what is the idea behind that?

The “mean reduction” computes a (conventional) weighted average,
that is, it does divide by the sum of the weights.

This makes sense to me because if, by happenstance, all of the
samples in the batch have the same loss, loss_all, I would like the
mean reduction over that batch also to give a batch mean of loss_all.
The conventional weighted average does this.

These two threads (about NLLLoss, but its the same issue) give some
additional words of explanation:

Mean Reduction in nll_loss

According to nll_loss documentation, for reduction parameter, " 'none' : no reduction will be applied, 'mean' : the sum of the output will be divided by the number of elements in the output, 'sum' : the output will be summed." However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate? input = torch.randn(3, 5, requires_grad=True) ls =F.log_softmax(input, 1…

Weights in NllLoss behave unexpectedly

Hello, I’m having trouble understanding behaviour of class weights in CrossEntropyLoss. Specifically, when reduction=‘mean’. I test it like this: input = torch.randn(5, 2, requires_grad=True) m = nn.LogSoftmax(dim=1) mi = m(input) target = torch.tensor([0, 0, 1, 1, 1]) w = torch.tensor([1, 100]).float() Now, without weights everything behaves reasonably, next 2 lines give the same result F.nll_loss(mi, target, reduction=‘none’).mean() F.nll_loss(mi, target, reduction=‘mean’) Howev…

Best.

K. Frank