Directly getting gradients

Is there a way for me to directly compute the gradient of a variable w.r.t. another variable, like tf.gradients()?

6 Likes

Let’s say you compute variable y as a function of variable x.

When you call y.backward(dL_dy) you’ll get the value of dL_dx in x.grad (which is dL_dy * dy_dx). If you just put a tensor full of ones instead of dL_dy you’ll get precisely the gradient you are looking for.

import torch
from torch.autograd import Variable

x = Variable(torch.ones(10), requires_grad=True)
y = x * Variable(torch.linspace(1, 10, 10), requires_grad=False)
y.backward(torch.ones(10))
print(x.grad)

produces

Variable containing:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
[torch.FloatTensor of size 10]
9 Likes

Thanks for the reply:)
I should have been more precise with my question, though.

I’m trying to look for a way to compute second-order derivatives.
In tensorflow, I would do something like:
tf.gradients(tf.gradients(y, x), x)

As far as I can tell, there’s no way of accomplishing this using y.backward(). That’s why I was wondering if there was a standalone function that differentiated.

1 Like

The second order derivative of pytorch functions are not “known” by pytorch, while the first order are coded “by hand” in the backward methods. If you want the second derivative of your whole graph, I’m afraid that you need at least to extend all the functions you use with a method similar to backward for second order derivative, by hands.

1 Like

we are working on enabling this. This will be in the next major pytorch release 2 months away. It wont be a full 2nd derivative but it will be the Hessian-vector product (whether it’s tf.gradients or in pytorch). We are tracking this feature as double-backprop / double backwards.

8 Likes

@smth has this been implemented? If yes, how can I use it?

Dear PyTorch dev,

I am a professor in one of the US Universities working on data-driven scientific computing using PyTorch now. The ability to get gradients allows for some amazing new scientific computing algorithms. You can see from this paper, and this github link (e.g., starting on line 121, “u = tf.gradients(psi, y)”), the ability to get gradients between two variables is in Tensorflow and is becoming one of the major differentiator between platforms in scientific computing. This paper is published in 2019 and has gained 168 citations, very high in the realm of scientific computing. I see many people are following suite and using this algorithm. We are now using PyTorch but might be forced to migrate to TF if such functionality is not available with PyTorch. Really hoping it could be released…

1 Like

@albanD Would the work involved in https://github.com/pytorch/pytorch/issues/30632 resolve this feature request?

Yes it will. Hopefully this will be done in the next weeks !

Is this issue has be solved?

Yes,
It was launched with v1.5

Is it working? I tried that with v1.6.

x = Variable(torch.Tensor([1, 2, 3, 4, 5, 6]), requires_grad= True)
y = Variable(torch.Tensor([2, 4, 6, 8, 10, 12]), requires_grad= False)
torch.autograd.grad(y, x)
Traceback (most recent call last):
File “”, line 1, in
File “anaconda3/envs/updated/lib/python3.8/site-packages/torch/autograd/init.py”, line 190, in grad
return Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn`

Same message with older versions.

Hi,

This is quite unrelated. The problem here is just that y does not require gradients. So no gradient can be computed.