Gradients in variable obtained through narrowing

1

I have a question regarding narrowing tensors when using autograd.
If I have a variable x1 obtained by narrowing another variable x, the backward phase seems to correctly compute the gradients for x, but x1.grad is zero. Is this an expeced behaviour?

import torch
from torch.autograd import Variable

x = Variable(torch.linspace(1, 12, 12).view(3, 4), requires_grad=True)
x1 = x[:,:2]    # x1 is 3 x 2
x2 = x[:,1:]    # x2 is 3 x 3

y1 = 2 * x1
y2 = 3 * x2
y1.backward(torch.ones(3, 2))
y2.backward(torch.ones(3, 3))

print(x.grad)  # This is correct

# Variable containing:
#  2  5  3  3
#  2  5  3  3
#  2  5  3  3
# [torch.FloatTensor of size 3x4]

print(x1.grad)  # This is zero

# Variable containing:
#  0  0
#  0  0
#  0  0
# [torch.FloatTensor of size 3x2]
2

x1.grad is zero because x1 is a non-leaf variable. We dont populate the gradients of non-leaf variables. If you want access to them you’d have to use Hooks

2 Likes