Simple math problems showing different results for torch and numpy

1

Hi,
So I was working with a code when I realized this issue:
Suppose,

import torch
import numpy as np

mu1 = torch.Tensor([0.51])
mu2 = torch.Tensor([0.52])
sigma1 = torch.Tensor([0.10])
sigma2 = torch.Tensor([0.11])
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5])

print(output)

mu1 = torch.Tensor([0.51])
mu2 = torch.Tensor([0.52])
sigma1 = torch.Tensor([0.10])
sigma2 = torch.Tensor([0.11])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)

The results are: tensor([0.0127])
0.012665540341760373, pretty good!
But if,

import torch
import numpy as np

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5])

print(output)

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)

gives results: tensor([5.9605e-08])
2.3976444829543198e-08
What is going on? Why such weird behaviour?

2

I believe that this is because numpy will do operations in float64 by default while in torch they are in float32. For example:

import torch
import numpy as np

mu1 = torch.Tensor([0.4651]).to(torch.double)
mu2 = torch.Tensor([0.4652]).to(torch.double)
sigma1 = torch.Tensor([0.7910]).to(torch.double)
sigma2 = torch.Tensor([0.7911]).to(torch.double)
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5]).to(torch.double)

print(output)

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)
print(output.dtype)

yields the following output

tensor([2.3976e-08], dtype=torch.float64)
2.3976444829543198e-08
float64

on my machine.

1 Like
3 
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

calling item() returns python’s standard float, which is double precision, thus np.log will keep processing floats as double precision
If you want to keep precision intact you can replace item() with numpy(), this will convert all tensor.float32 to np.float32

output = np.log(sigma2.numpy()/sigma1.numpy()) + (sigma1.numpy()**2 + (mu1.numpy() - mu2.numpy())**2)/(2*sigma2.numpy()**2) - 0.5
>>> output
array([5.9604645e-08], dtype=float32)
1 Like