Pytorch calculation precision

Hi I’m fairly new to pytorch, plz correct me if I’m wrong.

My question is, does pytorch calculates everything in float64 but saves its parameters using float32? I heard pytorch prefers float32 for its speed and memory efficiency, so I wrote my own autograd using numpy and set its dtype to float32 as well.

However, when compared the result with pytorch, it varies by a bit: (for example, the final loss comparison is: numpy autograd:0.005946858786046505 vs pytorch autograd: 0.005946869496256113).

When I used float64 in numpy, the result became almost identical to pytorch: (numpy autograd: 0.9532327802786481 vs pytorch autograd: 0.9532327802786484)

I can’t seem to find an answer elsewhere on this topic, thanks for any help!

No, PyTorch uses float32 by default for the parameter storage and all calculations.
If you are using mixed-precision training via torch.cuda.amp then some operations will be performed with float16 inputs/outputs, but the computation will still be performed in float32.

The difference might come from the limited floating point precision and a different order of operations between PyTorch and numpy.