I’ve been trying to train my model and achieve reproducible results, but unfortunately, I’m not able to.
I’ve taken this code from an old thread, but that still didn’t make it work:
def set_random_seed(seed_value, use_cuda=True):
np.random.seed(seed_value) # cpu vars
torch.manual_seed(seed_value) # cpu vars
random.seed(seed_value) # Python
os.environ[‘PYTHONHASHSEED’] = str(seed_value) # Python hash buildin
torch.cuda.manual_seed_all(seed_value) # gpu vars
torch.backends.cudnn.deterministic = True # needed
torch.backends.cudnn.benchmark = False
I also set all num_workers=0 in the data loaders to avoid a well-known problem there.
I’ve noticed a strange thing that happens after several training iterations (2 similar runs):
there is a slight change in the calculation that leads to totally different results.
Did anyone face this problem? is it even fixable?