Strange problem when manually gradient descent

If I code:

with torch.no_grad():
    w -= 0.01 * w.grad

Everything works well with no error.
However, if I code:

with torch.no_grad():
    w = w - 0.01 * w.grad

I meet the following error:

Traceback (most recent call last):
  File "/home/jihao/deep_learning/", line 34, in <module>
AttributeError: 'NoneType' object has no attribute 'zero_'

What’s the difference between the two snips of code?

w -= 0.01 * w.grad is an in-place operation, so it performs calculation on existing w and updates the value.

However, w = w - 0.01 * w.grad is not in-place operation, so it creates a new variable w, which does not have requires_grad set and so the error.

You can quickly check this by calling - print(w.requires_grad). In the first case you would get True wheres in the second case it would be False.


I understand. Thank you so much!

Could you please mark the thread as solved for the benefit of other members.

Already marked. Thank you very much for solving my first question here.

Ah great. Thanks :slight_smile: