Pytorch calculation precision

autograd
1

Hi I’m fairly new to pytorch, plz correct me if I’m wrong.

My question is, does pytorch calculates everything in float64 but saves its parameters using float32? I heard pytorch prefers float32 for its speed and memory efficiency, so I wrote my own autograd using numpy and set its dtype to float32 as well.

However, when compared the result with pytorch, it varies by a bit: (for example, the final loss comparison is: numpy autograd:0.005946858786046505 vs pytorch autograd: 0.005946869496256113).

When I used float64 in numpy, the result became almost identical to pytorch: (numpy autograd: 0.9532327802786481 vs pytorch autograd: 0.9532327802786484)

I can’t seem to find an answer elsewhere on this topic, thanks for any help!

2

No, PyTorch uses float32 by default for the parameter storage and all calculations.
If you are using mixed-precision training via torch.cuda.amp then some operations will be performed with float16 inputs/outputs, but the computation will still be performed in float32.

The difference might come from the limited floating point precision and a different order of operations between PyTorch and numpy.