I recently start using a server that has cudnn.
Although the GPUs are better than what i used to hav, the same code runs slower.
How I can disable cudnn so my code does not use it?
I think cudnn is the reason behind this slowness
should i do
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False would disable cuDNN.
Could you post your results, if cuDNN slows down your code?
Thank you for your help.
Can cudnn also effect the performance?
I didnot record the exact time, but the code that used to take around 7hrs (on TITAN Xp) to finish is taking more 11 hr (rtx 2080 ti) now.
oh and also we mistakenly didnot have cudnn on the first machine at all, but we have it on the second machine (same verdsion of Cuda 9.0) same version of pytorch. now im gonna go ahead and disable it to see if any changes will happen or not
cuDNN should speed up the training time.
Also if you set
torch.backends.cudnn.benchmark = True, cuDNN will use some heuristics at the beginning of your training to figure out which algorithm will be most performant for your model architecture and input.
This might be especially helpful, if your input shapes are fixed and not changing a lot during training.
I have not enable cudnn.benchmark in my code, but i dont know if it can be enabled by default or not.
my input shapes change a lot from 600600 to 12001200. and i can see that the code suddenly become slow, i havenot noticed this kind of slowness before, so i assume this can be caused by cudnn, but im not 100% sure yet
It is weird
I disable the cudnn and not i get this error:
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1512387374934/work/torch/lib/THC/THCBlas.cu:246
If I’m not mistaken, I think your RTX2080Ti needs CUDA10 to run properly.
Could you try to create a new environment and install PyTorch with CUDA10 and try it again?
It’s strange, that the code was running in the first place, but might be worth a try.
I will give it a try but i first need to see if cuda 10 and my pytorch (0.3) are compatable
Thank you for the suggestion though.
I will give it a try