Unable to Import PyTorch After Upgrade in Docker Environment

unbreading · February 7, 2025, 3:43pm

Unable to Import PyTorch After Upgrade in Docker Environment

opened 03:02PM - 07 Feb 25 UTC

triaged module: docker

### 🐛 Describe the bug ### Problem Description Before upgrade: <img width="8…38" alt="Image" src="https://github.com/user-attachments/assets/8950b303-510f-4136-b44d-8a3b55185941" /> After upgrade (by `pip install torch==2.4.0`) <img width="837" alt="Image" src="https://github.com/user-attachments/assets/25a81dc3-8202-4dbc-98ec-71a11c87e0cf" /> I found that libcudnn.so.9 exists in the path /usr/local/lib/python3.10/site-packages/nvidia/cudnn/lib/, but it was not being loaded. To resolve this, I modified the _load_global_deps() function in torch/__init__.py by adding line 227 and deleting lines 245-246, as shown in the attached image. After these changes, I was able to successfully import torch. <img width="935" alt="Image" src="https://github.com/user-attachments/assets/3405e706-3522-4234-958f-64e9af5ffb10" /> ### Questions - Is this the right solution? Am I missing any steps? - After upgrading PyTorch using this method, I encountered numerous 'undefined symbol' errors during the installation and import of vllm and flash-attn. Could these undefined symbol problems be related to the changes ### Versions ### Environment: Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: TencentOS Server 3.2 (Final) (x86_64) GCC version: (GCC) 8.5.0 20210514 (Tencent 8.5.0-23) Clang version: Could not collect CMake version: version 3.26.5 Libc version: glibc-2.28 Python version: 3.10.8 (main, Oct 15 2022, 14:44:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-10)] (64-bit runtime) Python platform: Linux-5.4.119-1-tlinux4-0009.3-x86_64-with-glibc2.28 Is CUDA available: True CUDA runtime version: 12.1.66 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: A100-SXM4-40GB GPU 1: A100-SXM4-40GB GPU 2: A100-SXM4-40GB GPU 3: A100-SXM4-40GB GPU 4: A100-SXM4-40GB GPU 5: A100-SXM4-40GB GPU 6: A100-SXM4-40GB GPU 7: A100-SXM4-40GB Nvidia driver version: 450.156.00 cuDNN version: Probably one of the following: /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.9.4 /usr/local/cuda-12.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.9.4 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-254 Off-line CPU(s) list: 255 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 2 NUMA node(s): 16 Vendor ID: AuthenticAMD BIOS Vendor ID: Advanced Micro Devices, Inc. CPU family: 25 Model: 1 Model name: AMD EPYC 7K83 64-Core Processor BIOS Model name: AMD EPYC 7K83 64-Core Processor Stepping: 1 CPU MHz: 3243.849 CPU max MHz: 2450.0000 CPU min MHz: 1500.0000 BogoMIPS: 4891.30 Virtualization: AMD-V L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 32768K NUMA node0 CPU(s): 0-7,128-135 NUMA node1 CPU(s): 8-15,136-143 NUMA node2 CPU(s): 16-23,144-151 NUMA node3 CPU(s): 24-31,152-159 NUMA node4 CPU(s): 32-39,160-167 NUMA node5 CPU(s): 40-47,168-175 NUMA node6 CPU(s): 48-55,176-183 NUMA node7 CPU(s): 56-63,184-191 NUMA node8 CPU(s): 64-71,192-199 NUMA node9 CPU(s): 72-79,200-207 NUMA node10 CPU(s): 80-87,208-215 NUMA node11 CPU(s): 88-95,216-223 NUMA node12 CPU(s): 96-103,224-231 NUMA node13 CPU(s): 104-111,232-239 NUMA node14 CPU(s): 112-119,240-247 NUMA node15 CPU(s): 120-127,248-254 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.8.61 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] torch==2.4.0 [pip3] torchaudio==2.2.0 [pip3] torchvision==0.17.0 [pip3] triton==3.0.0 [conda] Could not collect

onedeadmatch · February 12, 2025, 10:21pm

Tipically when libcudnn is not found it means that the path is missing in ld_library_path env variable.

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
source ~/.bashrc

Anyway, I am not sure that this will 100% solve the problem, if you need a custom container you can create one from scratch using pre-configured nvidia’s images. You can have a look here for reference: GitHub - DanielRossi1/AutoDock: The perfect environment for your AI Research Project!