RuntimeError: cublas runtime error : the GPU program failed to execute at at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/THCBlas.cu:258

#1

Hi everyone, I tried to run the pointnet module on Pytorch but got the following error:
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/THC/THCBlas.cu:258

It shows the error comes from F.linear(input, self.weight, self.bias).

The whole error message is:

==============
Traceback (most recent call last):
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1741, in
main()
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1735, in main
globals = debugger.run(setup[‘file’], None, None, is_module)
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/pydevd.py”, line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File “/home/yirus/software/pycharm-community-2018.3.4/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile
exec(compile(contents+"\n", file, ‘exec’), glob, loc)
File “/home/yirus/Projects/pointnet.pytorch/train_classification.py”, line 78, in
pred, _ = classifier(points)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 100, in forward
x, trans = self.feat(x)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 100, in forward
x, trans = self.feat(x)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 73, in forward
trans = self.stn(x)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/yirus/Projects/pointnet.pytorch/pointnet.py”, line 47, in forward
x = F.relu(self.bn4(self.fc1(x)))
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 67, in forward
return F.linear(input, self.weight, self.bias)
File “/home/yirus/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1352, in linear
ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())

==============

I tried to search on the Internet but cannot find any posts that are helpful.

======================
Here is the versions installed:

GPU info: GeForce RTX 2080
$ nvidia-smi
It shows “Driver Version: 415.18, CUDA version: 10.0”

$nvcc -V
It shows “Cuda compilation tools, release 9.2, V9.2.148”

Pytorch version: 1.0.1.post2 (installed by anaconda3)
Operation system: Fedora 28
Python version: 3.7

I have a feeling that it’s related to the wrong version of CUDA or should I reinstall pytorch?

Thanks in advance!

#2

CUDA 9.2 doesn’t support turing gpus.

#3

I’m facing the same error:

RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCBlas.cu:259

also from F.linear(input, self.weight, self.bias)

However, my CUDA Version is 10.0.130

I’m using:

  • python 3.7.3
  • ubuntu 18.04.3
  • pytorch 1.1.0
  • GeForce RTX 2080Ti

Is there a version mismatch?

#4

Could you update PyTorch to the latest stable version, retry the code?
If you are still seeing this error, could you please post a reproducible code snippet?

#5

Even though my CUDA version is 10.0.130 (obtained with cat /usr/local/cuda/version.txt), my CUDA toolkit version was 9.0. After updating this to 10.1.243, it seems to be solved now. Thank you

#6

@ptrblck ,I am facing the same issue, i am using a 3090 GPU cuda version 11.4,pytorch==1.0.0

#7

PyTorch 1.0.0 doesn’t support CUDA11, which is needed for your 3090, so you would need to update to the current stable or nightly release (and select CUDA11 as the CUDA version for the binaries).

#8

Hi,inspite of upgrading torch to torch==1.9.1 and torchvision=0.10.1
i am facing this issue
Traceback (most recent call last):
File “train.py”, line 357, in
main()
File “train.py”, line 112, in main
model = Network(args.init_channels, num_domains, args.layers, criterion)
File “/voyager-volume/code_1_test/code_1_test/code_1_test/code_1_test/code_1_test/model_search.py”, line 95, in init
self._initialize_alphas()
File “/voyager-volume/code_1_test/code_1_test/code_1_test/code_1_test/code_1_test/model_search.py”, line 124, in _initialize_alphas
self.alphas_normal = Variable(1e-3*torch.randn(k, num_ops).cuda(), requires_grad=True)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

#9

This would indicate that the currently used PyTorch build doesn’t contain kernels for the right compute capability (8.6 in your case), which would happen if you’ve installed e.g. the CUDA10.2 binaries. Make sure to select the CUDA11.1 (or 11.3) runtime during the installation.

#10

NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

This is the root of the error actually,what do I need to do?

#11

You need to install the binaries with the CUDA11.1 runtime.
In the install instructions you see the CUDA selection:

image
image842×413 33.7 KB

Click on “CUDA 11.1”, select other options, copy/paste the install command, and execute it in your terminal.