Problem fixing seeds in pytorch 2.1

Hi, I’m doing some experiments with pytorch and have a problem of reproducing the same value for each different runs.

I created conda env by “conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia”, and tried to fix the seeds by following 5 lines of code.

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

When I print the feature values or loss, the values are the same at first several iterations, then there occurs small differences (1e-4) which affects the final prediction values.

One strange thing is that when the same experiments (exactly the same codes) are tested in pytorch 1.12.0, I could successfully fix the seeds.

Has anyone got a problem like me? Or would there be additional nondeterministic operations (that I should fix) in pytorch 2.1.0 compared to pytorch 1.12?

Could you post a minimal and executable code snippet reproducing the issue, please?