jsuit
January 9, 2018, 3:49am
1
Could someone please explain why x_hv and y_hv are being multiplied by 5 and 4, respectively?
return x_grad, x_grad_clone
# Accumulate in-place when create_graph is False
x_grad, x_grad_clone = compute_grad(create_graph=False)
self.assertEqual(x_grad, x_grad_clone * 2)
# Accumulate out-of-place when create_graph is False
x_grad, x_grad_clone = compute_grad(create_graph=True)
self.assertEqual(x_grad, x_grad_clone)
def test_hessian_vector(self):
x = Variable(torch.randn(2, 2), requires_grad=True)
y = Variable(torch.randn(2, 2), requires_grad=True)
z = x ** 2 + y * x + y ** 2
z.backward(torch.ones(2, 2), create_graph=True)
x_grad = 2 * x.data + y.data
y_grad = x.data + 2 * y.data
self.assertEqual(x.grad.data, x_grad)
self.assertEqual(y.grad.data, y_grad)
def test_hessian_vector(self):
x = Variable(torch.randn(2, 2), requires_grad=True)
y = Variable(torch.randn(2, 2), requires_grad=True)
z = x ** 2 + y * x + y ** 2
z.backward(torch.ones(2, 2), create_graph=True)
x_grad = 2 * x.data + y.data
y_grad = x.data + 2 * y.data
self.assertEqual(x.grad.data, x_grad)
self.assertEqual(y.grad.data, y_grad)
grad_sum = 2 * x.grad + y.grad
grad_sum.backward(torch.ones(2, 2))
x_hv = torch.ones(2, 2) * 5
y_hv = torch.ones(2, 2) * 4
self.assertEqual(x.grad.data, x_grad + x_hv)
self.assertEqual(y.grad.data, y_grad + y_hv)
Could someone please explain why x_hv and y_hv are being multiplied by 5 and 4, respectively?
Effectively, x.grad = 2 * x + y
, y.grad = x + 2 * y
.
Because of this, the amount that x
contributes to grad_sum
is 4 through the 2 * x.grad
term and 1 through the y_grad
term. Hence x_hv = 5 * torch.ones(2, 2)
.
The amount that y
contributes to grad_sum
is 2 through the 2 * x.grad
term and 2 through the y.grad
term. Hence the factor of 4.
1 Like
jsuit
January 9, 2018, 3:04pm
3
Thanks. I have another question. What is grad_sum? Is that just an arbitrarily defined function, or does that fall out of the previous work above it?
We’ve arbitrarily defined grad_sum = 2*x.grad + y.grad
1 Like