CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

10

It’s the legendary @ptrblck! Thank you a lot, truly.

I am going to post 2 different calls to python -m torch.utils.collect_env. One from my system environment using version 1.7.1 and another using torch 1.8.0 on a virtual environment. Oddly enough it seems that torch is using a different CUDA version than what’s specified in nvidia-smi! (it’s using 10.2, instead of 11.0).

Torch version 1.7.1 where the original error message DOES NOT appear:

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.7.1
[conda] Could not collect

Now, the virtualenv 1.8 version where the original error DOES happen:

Collecting environment information...
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 450.102.04
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.0
[conda] Could not collect

Did you build from source?

No, I installed via pip install torch with pip version 20.0.2 on Python version 3.8.5. The pip version inside my virtualenv is 21.0.1

I am sorry for the noob question. How might I install the 11.1 pip wheels? (assuming that 11.1 refers to the conda version)

Thanks again!
Dani

2 Likes