Ok, I just found an answer by soumith on another thread:
“if you want to use pytorch with an NVIDIA GPU, all you need to do is install pytorch binaries and start using it. We ship with everything in-built (pytorch binaries include CUDA, CuDNN, NCCL, MKL, etc.).”
so that means the whole installing CUDA and cuDNN on Ubuntu shenanigans are actually not necessary at all?! That would also explain my confusion, why I get the same time to train no matter if I install cudnn or not
sorry for the confusion