If you need more numerical precision, you could use float64, but note that the performance would be worse.
Based on the new code snippet you are accumulating the loss so I’m not sure which loss value you are printing.
In any case, in case the rounding issue is reproducible for you, please post an executable code snippet, so that we could debug it.