Using grad to calculate derivative of two outputs w.r.t. the same input

[lvoet (Laurens Voet)]

Hi,

I’m trying to use the autograd to get derivatives of two output values (y_a and y_b) w.r.t. the same input vector (x). I implemented two ways of doing this, which give me different results and I don’t understand why. Only the second method gives me the results.

Method 1: applying x.requires_grad_(True) only once

x.requires_grad_(True)
y_a = … # Calculation of y_a
y_a.backward(retain_graph=True)
dy_a/dx = x.grad
y_b = … # Calculation of y_b
y_b.backward(retain_graph=True)
dy_b/dx = x.grad

Method 2: applying x.requires_grad_(True) twice

x.requires_grad_(True)
y_a = … # Calculation of y_a
y_a.backward()
dy_a/dx = x.grad
x.requires_grad_(True)
y_b = … # Calculation of y_b
y_b.backward()
dy_b/dx = x.grad

Thanks for the help!

[InnovArul (Arul)]

Make sure to zero the gradients (if x.grad: x.grad.zero_()) before calculating new y.

Some more info:

Why do we need to set the gradients manually to zero in pytorch?

y.backward() doesn’t just assign the value of y’(x) to x.grad (say y depends on x). It actually adds y’(x) to the current value of x.grad (think it as x.grad += true_gradient). In the following example, y.backward() is called 5 times, so the final value of x.grad will be 5*cos(0)=5. import torch from torch.autograd import Variable x = Variable(torch.Tensor([[0]]), requires_grad=True) for t in range(5): y = x.sin() y.backward() print(x.grad) # shows 5 Calling x.grad.data.zero_…