Hi, I’m doing some experiments with pytorch and have a problem of reproducing the same value for each different runs.
I created conda env by “conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia”, and tried to fix the seeds by following 5 lines of code.
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
When I print the feature values or loss, the values are the same at first several iterations, then there occurs small differences (1e-4) which affects the final prediction values.
One strange thing is that when the same experiments (exactly the same codes) are tested in pytorch 1.12.0, I could successfully fix the seeds.
Has anyone got a problem like me? Or would there be additional nondeterministic operations (that I should fix) in pytorch 2.1.0 compared to pytorch 1.12?