Note that there is a difference from doing
param.data -= learning_rate * param.grad.data
and
param.data = param.data - learning_rate * param.grad.data
In the first case, the operation is performed in-place, so the python object is still the same, while in the second one you create a new object.
To give an example:
a = torch.rand(3)
print(id(a))
b = a # same python object
print(id(b))
b = b - 1
print(id(b)) # object reference changed!
a -= 1 # in-place operation doesn't change object
print(id(a)) # still the same object