Strange problem when manually gradient descent

autograd
1

If I code:

with torch.no_grad():
    w -= 0.01 * w.grad
    w.grad.zero_()

Everything works well with no error.
However, if I code:

with torch.no_grad():
    w = w - 0.01 * w.grad
    w.grad.zero_()

I meet the following error:

Traceback (most recent call last):
  File "/home/jihao/deep_learning/auto_gradient.py", line 34, in <module>
    w.grad.zero_()
AttributeError: 'NoneType' object has no attribute 'zero_'

What’s the difference between the two snips of code?

2

w -= 0.01 * w.grad is an in-place operation, so it performs calculation on existing w and updates the value.

However, w = w - 0.01 * w.grad is not in-place operation, so it creates a new variable w, which does not have requires_grad set and so the error.

You can quickly check this by calling - print(w.requires_grad). In the first case you would get True wheres in the second case it would be False.

3 Likes
3

I understand. Thank you so much!

4

Could you please mark the thread as solved for the benefit of other members.

5

Already marked. Thank you very much for solving my first question here.

6

Ah great. Thanks :slight_smile: