# Loss reduction sum vs mean: when to use each?

**URL:** <https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641>\
**Category:** Uncategorized\
**Created:** [March 23, 2021, 4:06am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641 "2021-03-23T04:06:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pumplerod](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/32/36129_2.png) [@pumplerod](https://discuss.pytorch.org/u/pumplerod)\
**Post date:** [March 23, 2021, 4:06am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/1 "2021-03-23T04:06:00Z")

</div>

I’m rather new to pytorch (and NN architecture in general). While experimenting with my model I see that the various Loss classes for pytorch will accept a `reduction` parameter (none | sum | mean) for example. The differences are rather obvious regarding what will be returned, but I’m curious when it would be useful to use sum as opposed to mean? Does it have an effect on the backprop during training? Or am I really only choosing between a large loss value or smaller (average) loss value for aesthetic reasons for human readability?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 23, 2021, 5:19am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/2 "2021-03-23T05:19:27Z")

</div>

You would not only change the loss scale, but also the gradients:

```python
# setup
model = nn.Linear(10, 10)
x = torch.randn(10, 10)
y = torch.randn(10, 10)

# mean
criterion = nn.MSELoss(reduction='mean')
out = model(x)
loss = criterion(out, y)
loss.backward()
print(model.weight.grad.abs().sum())
> tensor(5.6143)

# sum
model.zero_grad()
criterion = nn.MSELoss(reduction='sum')
out = model(x)
loss = criterion(out, y)
loss.backward()
print(model.weight.grad.abs().sum())
> tensor(561.4255)

```

I think the disadvantage in using the `sum` reduction would also be that the loss scale (and gradients) depend on the batch size, so you would probably need to change the learning rate based on the batch size. While this is surely possibly, a `mean` reduction would not make this necessary.

On the other hand, the `none` reduction gives you the flexibility to add any custom operations to the unreduced loss and you would either have to reduce it manually or provide the gradients in the right shape when calling `backward` on the unreduced loss.

---

<div class="post-metadata">

**Author:** ![pumplerod](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/32/36129_2.png) [@pumplerod](https://discuss.pytorch.org/u/pumplerod)\
**Post date:** [March 23, 2021, 6:51am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/3 "2021-03-23T06:51:57Z")

</div>

Thank you @ptrblck

At the risk of exposing myself as a complete novice. Could you provide a layman’s understanding of what the impact of that large gradient has on the training process? Do I understand that this will essentially propagate a greater value back when adjusting the weights? And could this be balanced out with a much smaller learning rate? If my batch sizes are always the same will there be a noticeable impact on the ability to converge, because I am perhaps taking too large of a step when adjusting the weight values?

This little tidbit has really helped expose a slightly better understanding of the training process, so thank you very much for shining some light.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 23, 2021, 8:00am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/4 "2021-03-23T08:00:36Z")

</div>

> [@pumplerod](#):
>
> Do I understand that this will essentially propagate a greater value back when adjusting the weights? And could this be balanced out with a much smaller learning rate?

Yes and yes 🙂 You could certainly reduce these large gradients by adjusting the learning rate, but note that the learning rate would then depend on the batch size. I.e. if you decide to use a smaller or larger batch size you would also have to change the learning rate.  
The first figure in e.g. [this blog post](https://www.jeremyjordan.me/nn-learning-rate/) shows the effects of a large weight update on a simple use case.

> [@pumplerod](#):
>
> If my batch sizes are always the same will there be a noticeable impact on the ability to converge, because I am perhaps taking too large of a step when adjusting the weight values?

If you use the same script, adjust the learning rate, I would assume that you should be able to let the model converge to the same final result as with the mean reduction.

---

<div class="post-metadata">

**Author:** ![pumplerod](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pumplerod/32/36129_2.png) [@pumplerod](https://discuss.pytorch.org/u/pumplerod)\
**Post date:** [March 24, 2021, 1:59am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/5 "2021-03-24T01:59:40Z")

</div>

Thank you. I really appreciate your contribution to this community.

---

<div class="post-metadata">

**Author:** ![chaslie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/32.png) [@chaslie](https://discuss.pytorch.org/u/chaslie)\
**Post date:** [September 7, 2021, 8:54am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/6 "2021-09-07T08:54:21Z")

</div>

hi Ptrblck,

I have a similar issue, but when i use MSEloss(reduction=‘sum’) and MSEloss(reduction=‘mean’) I do not see the difference in the calculated loss function as the size of the batch after the first iteration eg:

```auto
criterion = torch.nn.MSELoss(reduction='sum')
criterion(out,real_data_I)
tensor(40190.8242, device='cuda:0', grad_fn=<MseLossBackward>)

criterion = torch.nn.MSELoss()
criterion(out,real_data_I)
tensor(0.2726, device='cuda:0', grad_fn=<MseLossBackward>)

```

The batchsize is 9 so i would expect the loss to be either 4,465.647 or 2.4534 depending on which is correct?

I am more suspicious of the 0.276 value as this looks too low for the start og the model, but I don’t understand why the error.

chaslie

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 7, 2021, 8:56am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/7 "2021-09-07T08:56:44Z")

</div>

I’m not sure I understand the issue correctly, but it seems you are concerned about the correctness of the reduction in the second approach? Could you post the shapes of `out` and `real_data_I` so that we could double check the calculation?

---

<div class="post-metadata">

**Author:** ![chaslie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/32.png) [@chaslie](https://discuss.pytorch.org/u/chaslie)\
**Post date:** [September 7, 2021, 8:59am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/8 "2021-09-07T08:59:47Z")

</div>

hi ptrblk,

the shapes are [9,1,128,128].

chaslie

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 7, 2021, 9:13am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/9 "2021-09-07T09:13:13Z")

</div>

Thanks! Based on this shape, the loss calculation seems to be correct using the `reduction='mean'` setting:

```python
40190.8242 / (9 * 1 * 128 * 128)
> 0.2725614705403646

```

---

<div class="post-metadata">

**Author:** ![chaslie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/32.png) [@chaslie](https://discuss.pytorch.org/u/chaslie)\
**Post date:** [September 7, 2021, 9:19am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/10 "2021-09-07T09:19:36Z")

</div>

I realised that, doh. I thought the difference was based purely on batch size and not the size of the array 🤯

---

<div class="post-metadata">

**Author:** ![chaslie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/32.png) [@chaslie](https://discuss.pytorch.org/u/chaslie)\
**Post date:** [September 7, 2021, 9:19am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/11 "2021-09-07T09:19:57Z")

</div>

thanks for taking the time out to deal with an idiot…

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 7, 2021, 9:22am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/12 "2021-09-07T09:22:43Z")

</div>

Ah, c’mon… we are all missing these “trivial” things all the time 😉

---

<div class="post-metadata">

**Author:** ![chaslie](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/32.png) [@chaslie](https://discuss.pytorch.org/u/chaslie)\
**Post date:** [September 7, 2021, 9:27am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/13 "2021-09-07T09:27:16Z")

</div>

too kind my friend, too kind 🙂

---

<div class="post-metadata">

**Author:** ![Shreeyak](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shreeyak/32/13800_2.png) [@Shreeyak](https://discuss.pytorch.org/u/Shreeyak)\
**Post date:** [December 7, 2021, 6:41pm UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/14 "2021-12-07T18:41:12Z")

</div>

We could remove the dependency on batch size by dividing the loss by batch size. Would that work?

```auto
criterion = nn.MSELoss(reduction='sum')
out = model(x)
loss = criterion(out, y)
loss = loss / x.shape[0]

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 7, 2021, 7:54pm UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/15 "2021-12-07T19:54:54Z")

</div>

Yes, you could surely divide by the batch size in case you don’t want to divide by the number of elements for specific reasons.

---

<div class="post-metadata">

**Author:** ![Shreeyak](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shreeyak/32/13800_2.png) [@Shreeyak](https://discuss.pytorch.org/u/Shreeyak)\
**Post date:** [December 10, 2021, 9:46am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/16 "2021-12-10T09:46:52Z")

</div>

Thanks! I believe that for dense predictions (such as segmentation or depth estimation), the average value per pixel would not be too meaningful, since it will result in only a small change in value because of averaging so many pixels. It would be hard to capture subtle changes in the prediction. Calculating the value across an entire image should give a much better signal.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [December 10, 2021, 9:52am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/17 "2021-12-10T09:52:29Z")

</div>

I don’t think the interesting difference is the actual range, as you could always increase or decrease the learning rate. The advantage of using the average of all elements would be to get a loss value, which would not depend on the shape (i.e. using a larger or smaller spatial size would yield approx. the same loss values assuming your model is flexible regarding the spatial shape).

---

<div class="post-metadata">

**Author:** ![Raccoon\_2](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raccoon_2/32/46091_2.png) [@Raccoon\_2](https://discuss.pytorch.org/u/Raccoon_2)\
**Post date:** [January 26, 2022, 9:59am UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/19 "2022-01-26T09:59:03Z")

</div>

Both are equal if the dimensions are constant because you can change the learning rate.  
But if the output dimensions are variable, both approaches will not be appropriate.  
Instead, you need to do this

```auto
loss = F.mse_loss(y_pred,y_true,reduction='none')
loss = (loss/torch.as_tensor(loss.size()).prod().sqrt()).sum()

```

---

<div class="post-metadata">

**Author:** ![Torcione](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/torcione/32/52091_2.png) [@Torcione](https://discuss.pytorch.org/u/Torcione)\
**Post date:** [March 16, 2022, 1:35pm UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/20 "2022-03-16T13:35:24Z")

</div>

Hi, does the `reduction = 'mean'` normalization change between different loss functions?  
For example I read [here](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) that for cross entropy loss the different normalization between `sum` and `mean` `reduction` is not fixed by the input size but just by the number of element in the batch N (if we set parameter `weight=None`).  
When in the description says:

> and N spans the minibatch dimension

Does it mean the number of images in a batch or the the number of images multiplied by their dimension?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [March 16, 2022, 4:17pm UTC](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641/21 "2022-03-16T16:17:30Z")

</div>

You’ve copied only part of the description as it mentions:

> and N spans the minibatch dimension as well as d1,…,dk​ for the K-dimensional case

which means that the `mean` reduction is dividing by the target shape. A quick test also shows this:

```python
criterion_mean = nn.CrossEntropyLoss(reduction='mean')
criterion_sum = nn.CrossEntropyLoss(reduction='sum')

output = torch.randn(2, 3, 224, 224)
target = torch.randint(0, 3, (2, 224, 224))

loss_mean = criterion_mean(output, target)
loss_sum = criterion_sum(output, target)

print(loss_mean - (loss_sum / target.nelement()))
# > tensor(0.)

```

[Next page](https://discuss.pytorch.org/t/loss-reduction-sum-vs-mean-when-to-use-each/115641.md?page=2)