I have this simple pytorch code:
x = torch.arange(3,dtype=float)
x.requires_grad_(True)
y = 3*x + x.sum()
y.backward(torch.ones(3))
x.grad
This gives me [6,6,6], but shouldn’t it be [4,4,4] ? Because if we have f(x)=3 * x0 + 3 * x1 + 3 * x2 + x0+x1+x2, partial derivatives would be 3+1=4 ?
This is a tricky one. The result of x.sum() is implicitly broadcasted to match the shape of 3*x. So y[0] = 3 * x[0] + x[0] + x[1] + x[2] = 4 * x[0] + x[1] + x[2]. If we do y[0].backward(), we would get x.grad = [4, 1, 1], and since we do y.backward(), it would be [4, 1, 1] + [1, 4, 1] + [1, 1, 4] = [6, 6, 6].