Hey,
How can I make the model train be the same if the hyperparameters are the same?
What is the seeds code that I need to put?
Regards,
André
Take a look here.
2 Likes
Here it sets the seed for the data loader using torch.Generator(), does it make the batches equal from run to run?
The seed_worker keeps the reproducibility of what?
Yes, per the docs, you need both the generator and worker_init_fn=seed_worker to make the behavior of the DataLoader the same from run to run. This suggests that the generator is PyTorch’s RNG, and the code inside the seed_worker function is used to fetch this seed and pass it to other libraries (like numpy and random) to ensure consistency.
1 Like
To make my code give the same result run to run i am doing the following:
# Seed
seed = 42
torch.manual_seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
def seed_worker(worker_id):
np.random.seed(42)
And in the DataLoader I am doing the following:
dl = DataLoader(ds, batch_size=BatchSize, shuffle=False, num_workers= 8, pin_memory=True, drop_last=True, generator=g, worker_init_fn= seed_worker)
Despite this, my code when I run it again with the same parameters it gives different results?
Note: I initialize the LSTM cell and hidden states only in the init method of my LSTM class. So when I say I make a new run it implies creating the model object model = LSTM(args).
Another question that naturally comes is: What is the best seed number?
Try this as well:
torch.use_deterministic_algorithms(True)
Where are you running all those commands? Presumably in the main body of the program, not inside the LSTM class initialization, right?
Can you please include all the stuff inside seed_worker per the tutorial? I don’t know if np.random.seed is enough because maybe internally some of the libraries call random rather than numpy.random so you may need the random.seed(worker_seed) part as well.
Another question that naturally comes is: What is the best seed number?
This question is closely related to “what is the best lottery number to play?” In both cases, the answer is that it ought to be random and unpredictable.
Where are you running all those commands? Presumably in the main body of the program, not inside the LSTM class initialization, right?
The seeds I showed there I am running in the program body, not inside the LSTM class.
When i put the “torch.use_deterministic_algorithms(True)” i got the following error:
RuntimeError: Deterministic behavior was enabled with either torch.use_deterministic_algorithms(True)orat::Context::setDeterministicAlgorithms(true), but this operation is not deterministic because it uses CuBLAS and you have CUDA >= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility
No solutions here but I did Google that error and found several threads on it on the forum, those might be helpful.
os.environ["CUBLAS_WORKSPACE_CONFIG"]=":4096:8"
torch.use_deterministic_algorithms(True)
I am doing this at the beginning of the code. The solution i found out here. I will tell if it works or not.
Edit: It works and solves the problem!
It solves the error, however the code still not reproducible do you have more suggestions @Andrei_Cristea ?