# What happens when loss are negative?

**URL:** <https://discuss.pytorch.org/t/what-happens-when-loss-are-negative/47883>\
**Category:** Uncategorized\
**Created:** [June 13, 2019, 5:21pm UTC](https://discuss.pytorch.org/t/what-happens-when-loss-are-negative/47883 "2019-06-13T17:21:07Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![KFrank](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/32.png) [@KFrank](https://discuss.pytorch.org/u/KFrank)\
**Post date:** [June 13, 2019, 5:57pm UTC](https://discuss.pytorch.org/t/what-happens-when-loss-are-negative/47883/3 "2019-06-13T17:57:19Z")

</div>

Hello Brandon!

> [@ljj7975](#):
>
> Based on my understanding of back prop and gradient descent,  
> Loss is multiplied to gradient when taking a step with gradient descent.  
> So when gradient becomes negative, gradient descent takes a step in the opposite direction.

This isn’t true. All common optimization algorithms I’m aware  
of – and in particular, gradient descent – only care about the  
gradient of the loss, and not the loss itself.

Plain-vanilla gradient descent takes the following optimization  
step:

new weights = old weights - learning rate \* gradient

Could you have misread “learning rate” for “loss” at some point?

> [@](#):
>
> Such idea is well captured when implementing gradient ascent,  
> as it can simply be implemented by multiplying -1 to the loss.
> 
> Then, what happens if my loss starts from positive and goes below zero?  
> If what I said above is correct, training will go for minimization when the loss is positive and go for maximization when the loss is negative.  
> And this means that gradient descent is going for zero instead of minimum value…
> 
> Can anyone help me understand this concept?

Gradient descent (and, again, all common algorithms that I am  
aware of) seek to minimize the loss, and don’t care whether that  
minimum value is a large positive value, a value close to zero,  
exactly zero, or a large negative value. It simply seeks to drive  
the loss to a smaller (that is, algebraically more negative) value.

You could replace your loss with

modified loss = conventional loss - 2 \* Pi

and you should get the exact same training results and model  
performance (except that all values of your loss will be shifted  
down by 2 \* Pi).

It is the case that we often use loss functions that become equal  
to zero when the fit of the model to the training data is perfect,  
but the optimization algorithms don’t care about this, and they  
drive the loss function to algebraically more negative values,  
and not towards zero.

Good luck!

K. Frank

---

_[View the full topic](https://discuss.pytorch.org/t/what-happens-when-loss-are-negative/47883)._