Hi,
I have a code base, which is stable, and where the cpu and gpu seeds are fixed.
I used to run this code on my 2080Ti gpu with a cuda 10.2, but recently I upgraded my cuda version to 11.6, and I noticed that the performance changed, on the majority of experiments it got better, but for one experiment it got worse. Note that nothing changed in the code, seed or hardware, only the version of cuda.
Is this a known issue? If so, is there any way to understand what happened?
Thank you in advance.