Could not load library But I'm sure that I have set the right LD_LIBRARY_PATH

The PyTorch binaries ship with their own CUDA dependencies (including cuDNN), so remove your locally installed cuDNN (temporarily) from the library path and let PyTorch load it’s own libs.
If you want to use your locally installed CUDA toolkit you could build PyTorch from source.