After much searching, it appears that this discussion from @apaszke is relevant with the .unsqueeze(1) being critical to it working.
Full example:
import torch from torch.autograd import Variable import numpy as np x = np.array([ 0.00834103, 0.00212306, -999.0, 0.00149333, 0.00899409]) x = Variable(torch.from_numpy(x.astype('float32')),requires_grad=True) y = x[(x>-999.).unsqueeze(1)] out = y.sum() out.backward() print x.grad