Torch.sum() result influenced by shape

1

I have a 3x2 tensor, and I do a column-sum. Then, I compare the results with a separate python sum and numpy.sum() of one of the columns. The results don’t agree.

x = torch.FloatTensor([1.1233445,1.52342352,-2.124345345,1.1233445,1.52342352,-2.124345345]).view(3,2)
y1 = torch.sum(x,dim=0,keepdim=True)*923430
y2 = sum(x[:,0])*923430
y3 = np.sum(x[:,0].numpy())*923430
print(y1[0,0]-y2,y1[0,0]-y3,y2-y3)  # expected 0; observed not 0
>>> -0.0024566650390625 -0.0024566650390625 0.0

This discrepancy doesn’t show for a 3x1 tensor:

x = torch.FloatTensor([1.1233445,1.52342352,-2.124345345]).view(3,1)
y1 = torch.sum(x)*923430
y2 = sum(x)*923430
y3 = np.sum(x.numpy())*923430
print(y1-y2,y1-y3,y2-y3)  # expected 0; observed 0
>>> 0 0.0 0

Furthermore, if I turn the 3x2 tensor into a Variable, the discrepancy again doesn’t appear (this may or may not be related to Sum of Variable not equal to sum of its data:

x = Variable(torch.FloatTensor([1.1233445,1.52342352,-2.124345345,1.1233445,1.52342352,-2.124345345])).view(3,2)
y1 = torch.sum(x,dim=0,keepdim=True)*923430
y2 = sum(x[:,0])*923430
y3 = np.sum(x.data.numpy(),axis=0)*923430
print(y1[0,0]-y2,y1[0,0]-y3[0],y2-y3[0])  # expected 0; observed 0
>>> 0 0 0

I ran the above on a Windows machine with Pytorch 0.3.0 (0.4.0 is not yet available). I used the floating point values shown because the discrepancy shows only for long decimals.

2

So the summing of the FloatTensors is of precision ~1e-6-1e-9, we are optimistic and use 1e-9. By multiplying the sum with ~1e6 you make the precision ~1e-3.

Best regards

Thomas

3

Thanks. Although, it’s still not clear to me why this depends on the shape of the tensor.

4

With precision, you can be “lucky” in that you accidentally have the exact same ops yielding the exaxt same result, but you are only “guaranteed” the limited precision. Here it might be something simple like the summation order that drives causea the sum to be different.

Best regards

Thomas