# Mean Reduction in nll\_loss **URL:** \ **Category:** Uncategorized\ **Created:** [August 27, 2020, 3:09pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314 "2020-08-27T15:09:42Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![coolsuntraveler](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/32/16640_2.png) [@coolsuntraveler](https://discuss.pytorch.org/u/coolsuntraveler)\ **Post date:** [August 27, 2020, 3:09pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314/1 "2020-08-27T15:09:42Z")
According to nll\_loss documentation, for reduction parameter, " `'none'` : no reduction will be applied, `'mean'` : the sum of the output will be divided by the number of elements in the output, `'sum'` : the output will be summed." However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate? > > > input = torch.randn(3, 5, requires\_grad=True) > > > ls =F.log\_softmax(input, 1) > > > ls > > > tensor([[-2.5076, -2.2387, -0.4952, -2.2435, -2.3390], > > > [-0.9897, -1.8469, -1.6642, -1.4942, -2.8676], > > > [-0.7173, -3.8274, -2.2838, -1.0784, -3.0335]], > > > grad\_fn=) > > > F.nll\_loss(ls, torch.tensor([0,0,4]), reduction=“none”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0])) tensor([2.5076, 0.9897, 9.1005], grad\_fn=) > > > F.nll\_loss(ls, torch.tensor([0,0,4]), reduction=“sum”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0])) tensor(12.5978, grad\_fn=) > > > F.nll\_loss(ls, torch.tensor([0,0,4]), reduction=“mean”, weight=torch.tensor([1.0, 1.0, 1.0, 1.0, 3.0])) tensor(2.5196, grad\_fn=) ---
**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\ **Post date:** [August 27, 2020, 3:27pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314/2 "2020-08-27T15:27:45Z")
Hello Johnson! > [@coolsuntraveler](#): > > However, it seems “mean” is divided by the sum of the weights of each element, not number of elements in the output. Am I misunderstanding the documentation or the documentation is inaccurate? You are correct that in the weighted case, “mean” is calculated by dividing by the sum of the weights. (In other words, a _weighted average_ is being calculated.) The documentation on this is arguably inaccurate, or at least rather unclear. Anyway, the way I think of it, if I have ten samples, all with the same loss, say, 1.23, I would like my “mean” loss, weighted or not, to be this same loss value, 1.23. So dividing by the sum of the weights is, for me, the “expected behavior,” even if the documentation says otherwise. (This issue has come up before on the forum, although I don’t have any references at hand.) Best. K. Frank ---
**Author:** ![coolsuntraveler](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/32/16640_2.png) [@coolsuntraveler](https://discuss.pytorch.org/u/coolsuntraveler)\ **Post date:** [August 27, 2020, 5:37pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314/3 "2020-08-27T17:37:46Z")
> Anyway, the way I think of it, if I have ten samples, all with the same > loss, say, 1.23, I would like my “mean” loss, weighted or not, to be > this same loss value, 1.23. So dividing by the sum of the weights is, > for me, the “expected behavior,” even if the documentation says > otherwise. Hello Frank, I think the example you gave is actually the expected behavior as described in the documentation. The mean should be divided by the number of samples, not by the sum of the weights of the expected class. That is what I expected from reading the documentation. ---
**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\ **Post date:** [August 27, 2020, 6:05pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314/4 "2020-08-27T18:05:08Z")
Hello Johnson! > [@coolsuntraveler](#): > > I think the example you gave is actually the expected behavior as described in the documentation. The mean should be divided by the number of samples, not by the sum of the weights of the expected class. No, I think you have my example backwards. Let’s say that you have three classes, `0, 1, 2`. Let’s say you also have three samples, one from each class, in order. Now suppose that the (unweighted) per-sample loss happens to be the same for all three samples, `1.23`. If you pass in a weight vector (class weights) of `[1.0, 1.0, 3.0]`, then after weighting, but before taking the “mean,” you will have a summed, weighted loss of `1.0 * 1.23 + 1.0 * 1.23 + 3.0 * 1.23 = 5.0 * 1.23 = 6.15` If you divide by the number of samples, `3`, you will get `2.05`. If you divide by the sum of the weights, `5.0`, you will get `1.23`, the loss value, that in this example, happens to be the same for each of the three samples. If I “average” together a bunch of numbers that all happen to have the same value, to me it makes sense that the average should be that value. That’s what I meant by saying that this is my “expected behavior” (regardless of the documentation, which I do think is inaccurate). Best. K. Frank ---
**Author:** ![coolsuntraveler](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/coolsuntraveler/32/16640_2.png) [@coolsuntraveler](https://discuss.pytorch.org/u/coolsuntraveler)\ **Post date:** [August 28, 2020, 5:09pm UTC](https://discuss.pytorch.org/t/mean-reduction-in-nll-loss/94314/5 "2020-08-28T17:09:27Z")
Thanks for the explanation K. Frank.