Problems with requires_grad

Hello! I have this simple (part of a) code:

a = torch.tensor(0.5)
a.requires_grad=True
lr = 1e-3
for i in range(20):
    yy = torch.cos(a*x) 
    loss = torch.mean((yy-y)**2)
    loss.backward()
    a = a-a.grad.data

Where x and y are fixed numbers, generated using y=(0.538*x). Basically I want to make a go from 0.5 to 0.538 using gradient descent.

The first loop works fine and a goes in the right direction, but at the second loop I am getting this error at the a = a-a.grad.data line: AttributeError: 'NoneType' object has no attribute 'data'. What am I doing wrong? What changes from the first to second iteration? Thank you!

Hi,

Few things:

  • Do not use .data, it’s not a thing anymore and causes a lot of bugs. If you want to make a weight update here, do :
with torch.no_grad():
  a -=  a.grad
  • And your real issue is that when you do a = xxx. Then you re-assign a in a differentiable manner. Meaning that your new a is not a leaf anymore. Using the code in the point above will change a inplace and it will thus remain a leaf.
  • (EDIT) Also a.grad.zero_() inside the no_grad block as @crowsonkb mentioned below

Try this instead:

a = torch.tensor(0.5)
a.requires_grad = True
lr = 1e-3
for i in range(20):
    yy = torch.cos(a*x)
    loss = torch.mean((yy-y)**2)
    loss.backward()
    with torch.no_grad():
        a -= lr * a.grad
        a.grad.zero_()

The following also works (but may be less efficient?) :

    with torch.no_grad():
        a = a - lr * a.grad
        a.requires_grad = True

The problem you are running into is that when you do a = a - a.grad you are getting a new a which does not have requires_grad set. So either you need to do in-place operations on a (preserving the flag) or manually re-enable it. Also, the gradient descent step has to be done inside a no_grad() context.

I was too slow…
But if you are interested in a nice introductory tutorial, which helped me understand the basics, take a look at this medium article:
https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e
I covers exactly the problem you were having and goes a bit more into detail about what is happening. :slight_smile: