Could not load library libcudnn_cnn_train.so.8. But I'm sure that I have set the right LD_LIBRARY_PATH

[ptrblck]

@lycnight

Moreover, I can even see the libcudnn_cnn_train.so.8 right in the /usr/local/cuda-12.1/lib64, why can’t pytorch load it?

The PyTorch binaries ship with their own CUDA dependencies (including cuDNN), so remove your locally installed cuDNN (temporarily) from the library path and let PyTorch load it’s own libs.
If you want to use your locally installed CUDA toolkit you could build PyTorch from source.