The code is the same, the input is the same, but why the computed loss is different?

When I use the alexnet to train the Cifar10, I use the following code to set the rand seed:

random.seed(666)
np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.cuda.manual_seed(666)
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = True

And I train it twice, and print the input and loss every iter. But I find the loss of the two different running is different, however, the input is the same. Just as the content in red box.

The code is the same, the input is the same, but why the computed loss is different?

Do someone know how it comes? Thank you very much!!

torch.backends.cudnn.benchmark should be set to False.
If that doesn’t help, you might see some floating precision issues to to atomic aperations (docs).