How should i disable using cudnn in my code?

1

I recently start using a server that has cudnn.
Although the GPUs are better than what i used to hav, the same code runs slower.
How I can disable cudnn so my code does not use it?
I think cudnn is the reason behind this slowness
should i do

 torch.backends.cudnn.deterministic = False
 torch.backends.cudnn.benchmark = False

?

1 Like
2

torch.backends.cudnn.enabled = False would disable cuDNN.
Could you post your results, if cuDNN slows down your code?

4 Likes
3

Thank you for your help.
Can cudnn also effect the performance?
I didnot record the exact time, but the code that used to take around 7hrs (on TITAN Xp) to finish is taking more 11 hr (rtx 2080 ti) now.
oh and also we mistakenly didnot have cudnn on the first machine at all, but we have it on the second machine (same verdsion of Cuda 9.0) same version of pytorch. now im gonna go ahead and disable it to see if any changes will happen or not

4

cuDNN should speed up the training time.
Also if you set torch.backends.cudnn.benchmark = True, cuDNN will use some heuristics at the beginning of your training to figure out which algorithm will be most performant for your model architecture and input.
This might be especially helpful, if your input shapes are fixed and not changing a lot during training.

2 Likes
5

I have not enable cudnn.benchmark in my code, but i dont know if it can be enabled by default or not.

my input shapes change a lot from 600600 to 12001200. and i can see that the code suddenly become slow, i havenot noticed this kind of slowness before, so i assume this can be caused by cudnn, but im not 100% sure yet

6

It is weird :frowning_face:
I disable the cudnn and not i get this error:
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1512387374934/work/torch/lib/THC/THCBlas.cu:246

7

If I’m not mistaken, I think your RTX2080Ti needs CUDA10 to run properly.
Could you try to create a new environment and install PyTorch with CUDA10 and try it again?
It’s strange, that the code was running in the first place, but might be worth a try.

8

I will give it a try but i first need to see if cuda 10 and my pytorch (0.3) are compatable :confused:
Thank you for the suggestion though.
I will give it a try