Parameter vs tensor.requires_grad = True

Hello everyone ,
I have a question regarding the autograd system , why we have the following difference between these two pictures :

this one :

VS

this one :

1 Like

a nn.Parameters is a container aimed to create learnable tensors inside a nn.Module
That’s why it keeps tracking the grad, as any other parameter in a nn.Module would do. Besides,
You are instantiating the Parameter class with rand0.3 as argument, meanwhile in the other case the variable which has “requires_grad” is later submitted to another operation, thus, the resulting variable is not a leaf variable. Namely, w=3rand(1) is not pointing to the same object that w=rand(1). In the former case you are pointing to the result of 3*rand(1) which is a different tensor than rand(1)

import torch
w=torch.rand(5).requires_grad_()
q=3*w
w.is_leaf
Out[8]: True
q.is_leaf
Out[9]: False
5 Likes

thanks a lot you are right !

Does it mean that we have to use nn.paramter (but not just tensor with requires_grad_) when building a custom network module layer?

Yes.
That’s the difference between a buffer (a tensor that is static but not intended to change by backprop) and a parameter.

The latter is returned when you invoke model.parameters() too.

1 Like