__isub__ vs = ... - ... for tensors (grads)

#1

Hi,

why does

W = W - learning_rate * W.grad
W.grad.zero_()

gives me:

AttributeError: ‘NoneType’ object has no attribute ‘zero_’

after one step, while

W -= learning_rate * W.grad
W.grad.zero_()

works as expected?

#2

Hi,

__isub__ is an inplace operation and so W remains the same object (and so it’s .grad field remains)
If you do W = ... then you create a new object and assign it to W. This new object does not have a .grad field.

1 Like