Hi, I’m using another third-party code where the authors define a input x as nn.Parameter and computes the grad himself (without using loss.backward() for sparsity). So the code looks like
x = nn.Parameter(..., requires_grad=True)
# then come custom ops
# not loss.backward() here
sparse_grad()
optim = optim.RMSprop([x])
optim.step()
optim.zero_grad()
I checked x after the sparse_grad function, x.grad indeed gave me the correct output. And this should be correct as we are replacing torch’s backward with my own backward func. Now I want to continue the computation from the gradients of x to some other input tensor. Is it possible in torch?
You could call backward() on intermediate tensors, but since x is created as an nn.Parameter I don’t know which “other input tensor” would be connected to it. Could you describe your use case a bit more?
The tensor.backward() operation can be called without passing an explicit gradient to it as an argument if the tensor is a scalar tensor. If the tensor has more than a single element you would have to pass the gradient to backward. You could use loss = x.sum() to calculate the gradient in fc if you would like to use x.sum() as the loss.