Hi,
I run this example https://github.com/pytorch/examples/blob/master/mnist/main.py twice
with same seed ‘0’ but I get a different result each time. I only have one GPU (Titan x on Ubuntu 16)
e.g: the first time I got
"Test set: Average loss: 0.0513, Accuracy: 9848/10000 (98%)"
and the second time
"Test set: Average loss: 0.0509, Accuracy: 9847/10000 (98%)"
What could be the issue?
------------------- update
I fixed the issue after setting
torch.backends.cudnn.deterministic = True
However, the training time is slower.
Is there a better solution?
there isn’t a better solution. the GPU code (CuDNN library) trades off some determinism for speed.
1 Like