Can't achieve reproducible results

Hello everyone,

I’ve been trying to train my model and achieve reproducible results, but unfortunately, I’m not able to.
I’ve taken this code from an old thread, but that still didn’t make it work:

def set_random_seed(seed_value, use_cuda=True):
np.random.seed(seed_value) # cpu vars
torch.manual_seed(seed_value) # cpu vars
random.seed(seed_value) # Python
os.environ[‘PYTHONHASHSEED’] = str(seed_value) # Python hash buildin
if use_cuda:
torch.cuda.manual_seed(seed_value)
torch.cuda.manual_seed_all(seed_value) # gpu vars
torch.backends.cudnn.deterministic = True # needed
torch.backends.cudnn.benchmark = False

I also set all num_workers=0 in the data loaders to avoid a well-known problem there.

I’ve noticed a strange thing that happens after several training iterations (2 similar runs):

there is a slight change in the calculation that leads to totally different results.
Did anyone face this problem? is it even fixable?

Thanks!

Did you check functions you used against the randomness/determinism note?
If you’re using one of the functions listed there, nondeterminism is expected.