Strange problem with torch reproducibility

autograd
1

I have already set random seed like beblow:

import torch
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

import numpy as np
np.random.seed(0)

import random
random.seed(0)

And, in the first two epoch, it seems that the model can produce same result. However, then the result becomes different with the other. Does anyone know why?

Epoch 00000 | Loss 2.046489238739 | Accuracy 0.455997971088 
Epoch 00001 | Loss 4.619217872620 | Accuracy 0.129850367740 
Epoch 00002 | Loss 10.045362472534 | Accuracy 0.448135937104
Epoch 00003 | Loss 5.151265144348 | Accuracy 0.441795587116 
Epoch 00004 | Loss 3.293608665466 | Accuracy 0.432919097134 
Epoch 00005 | Loss 3.348810672760 | Accuracy 0.501648490997 
Epoch 00006 | Loss 2.631152153015 | Accuracy 0.417448643165 
Epoch 00007 | Loss 2.960071325302 | Accuracy 0.429368501141 
Epoch 00008 | Loss 2.069313049316 | Accuracy 0.421760081156 
Epoch 00009 | Loss 2.420869827271 | Accuracy 0.429114887142 
Epoch 00010 | Loss 2.132256269455 | Accuracy 0.406036013188 
Epoch 00011 | Loss 2.006080150604 | Accuracy 0.406036013188 
Epoch 00012 | Loss 2.006951093674 | Accuracy 0.441541973117 
Epoch 00013 | Loss 1.845007181168 | Accuracy 0.533603854933 
Epoch 00014 | Loss 1.875343561172 | Accuracy 0.518133400964 

Epoch 00000 | Loss 2.046489238739 | Accuracy 0.455997971088 
Epoch 00001 | Loss 4.619217872620 | Accuracy 0.129850367740 
Epoch 00002 | Loss 10.045364379883 | Accuracy 0.448135937104
Epoch 00003 | Loss 5.151239871979 | Accuracy 0.441795587116 
Epoch 00004 | Loss 3.293608665466 | Accuracy 0.432919097134 
Epoch 00005 | Loss 3.348819971085 | Accuracy 0.501648490997 
Epoch 00006 | Loss 2.631155490875 | Accuracy 0.417448643165 
Epoch 00007 | Loss 2.960067749023 | Accuracy 0.429368501141 
Epoch 00008 | Loss 2.069319725037 | Accuracy 0.421760081156 
Epoch 00009 | Loss 2.420872926712 | Accuracy 0.429114887142 
Epoch 00010 | Loss 2.132259845734 | Accuracy 0.406036013188 
Epoch 00011 | Loss 2.006079673767 | Accuracy 0.406036013188 
Epoch 00012 | Loss 2.006967306137 | Accuracy 0.441541973117 
Epoch 00013 | Loss 1.844992518425 | Accuracy 0.533857468932 
Epoch 00014 | Loss 1.875328063965 | Accuracy 0.518133400964
2

Hi,

Have you read the note on reproducibility? https://pytorch.org/docs/stable/notes/randomness.html
Do you use any function that is not deterministic?