when I read the docs of torch.Tensor I found that " Only leaf Tensors will have their
grad populated during a call to
backward(). To get
grad populated for non-leaf Tensors, you can use
So does that mean the parameters of nn.Module are leaf nodes too? Because, they has to hold the grad to execute the .step() method to update themselves.
Yes by design, all the
Parameters are leaf Tensors that require gradients. So they will always have their .grad populated (unless you manually set the requires_grad field to False).
Thanks a lot! I also had a question about the Autograd of torch::jit::script::Module of C++ API.
I try to load a trained pytorch model into C++(let’s call this model in C++ “M”), first I do the forward process by:
out = M(input);
loss = loss_function(out, ground-truth)
then I tried to use it do backward propagation to calculate the grad of it’s input tensor “input”. However I found that the .grad of “input” is always zero. And the .grad_fn of “output” is a “nullptr”.
Any solutions for this?
I am not as familiar with the jit c++ API.
But did you make sure that the
input require gradients? And if it is not a leaf, you can call
.retain_grad() on it to ensure its
.grad field is properly populated.
Also make sure that you don’t disable the autograd with a NoGradGuard.
Thanks for the suggestions! I’ll try them!