Undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11

1

installed yolov6 on Ubuntu 22.04. Cuda 11.2 + python3.10

run the command : python3 tools/infer.py -weights yolov6n.pt …
Got the following message : undefined symbol

Any advice to solve this problem?

Traceback (most recent call last):
File “/home/mitchell/envs/torchgpu/lib/python3.10/site-packages/torch/init.py”, line 172, in _load_global_deps
ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
File “/usr/lib/python3.10/ctypes/init.py”, line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/mitchell/envs/torchgpu/lib/python3.10/site-packages/torch/lib/…/…/nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/mitchell/envs/YOLOv6/tools/infer.py”, line 8, in
import torch
File “/home/mitchell/envs/torchgpu/lib/python3.10/site-packages/torch/init.py”, line 217, in
_load_global_deps()
File “/home/mitchell/envs/torchgpu/lib/python3.10/site-packages/torch/init.py”, line 178, in _load_global_deps
_preload_cuda_deps()
File “/home/mitchell/envs/torchgpu/lib/python3.10/site-packages/torch/init.py”, line 158, in _preload_cuda_deps
ctypes.CDLL(cublas_path)
File “/usr/lib/python3.10/ctypes/init.py”, line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/mitchell/envs/torchgpu/lib/python3.10/site-packages/nvidia/cublas/lib/libcublas.so.11: undefined symbol: cublasLtGetStatusString, version libcublasLt.so.11

2

Could you reinstall PyTorch as it could be related to this issue which we already fixed?

3

I’ve had the same issue with PyTorch 1.13.1. What helped was removing nvidia-cublas-cu11 and installing nvidia-cublas-cu12 instead:

pip uninstall nvidia-cublas-cu11
pip install nvidia-cublas-cu12
4

The PyTorch wheels use nvidia-cublas-cu11 as an explicit dependency since they were built with the corresponding CUDA toolkit shipping cuBLAS 11. Removing it and replacing it with a newer cuBLAS version from CUDA 12 could yield undefined behavior.
Could you explain your previous setup in more detail, how you’ve installed it, etc.?