Are the parameters of nn.Module "leaf" nodes of the Graph?

autograd
1

Hello,

when I read the docs of torch.Tensor I found that " Only leaf Tensors will have their grad populated during a call to backward(). To get grad populated for non-leaf Tensors, you can use retain_grad()."

So does that mean the parameters of nn.Module are leaf nodes too? Because, they has to hold the grad to execute the .step() method to update themselves.

Many thanks!

2

Hi,

Yes by design, all the Parameters are leaf Tensors that require gradients. So they will always have their .grad populated (unless you manually set the requires_grad field to False).

3

Thanks a lot! I also had a question about the Autograd of torch::jit::script::Module of C++ API.
I try to load a trained pytorch model into C++(let’s call this model in C++ “M”), first I do the forward process by:
out = M(input);
loss = loss_function(out, ground-truth)
then I tried to use it do backward propagation to calculate the grad of it’s input tensor “input”. However I found that the .grad of “input” is always zero. And the .grad_fn of “output” is a “nullptr”.

Any solutions for this?

Many thanks!

4

Hi,

I am not as familiar with the jit c++ API.
But did you make sure that the input require gradients? And if it is not a leaf, you can call .retain_grad() on it to ensure its .grad field is properly populated.
Also make sure that you don’t disable the autograd with a NoGradGuard.

5

Thanks for the suggestions! I’ll try them!