CUDA 11.4 and torch version 1.11.0 not working

Hi there,

I have CUDA 11.4 and I’ve installed pytorch 1.11.0.
I thought these versions where compatible but I get this error when I run a python script : RuntimeError: CUDA error: no kernel image is available for execution on the device.

When I run torch.cuda.is_available() I get True.

Is the problem coming from the CUDA and pytorch versions I’ve installed ?

I have a NVIDIA RTX A4000.

1 Like

I guess you’ve installed the pip wheels or conda binaries, which will ship with their own CUDA runtime.
Your locally installed CUDA toolkit will be used if you build PyTorch from source or custom CUDA extensions.
Based on the error message you’ve most likely installed the binary with the CUDA 10.2 runtime while your Ampere GPU needs CUDA >= 11.

Thank you for your answer.
The CUDA Runtime Version that I’ve installed is 11.3.

If it can help, here’s what I get when I display the deviceQuery :

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA RTX A4000"
  CUDA Driver Version / Runtime Version          11.4 / 11.3
  CUDA Capability Major/Minor version number:    8.6
  Total amount of global memory:                 16095 MBytes (16876699648 bytes)
  (48) Multiprocessors, (128) CUDA Cores/MP:     6144 CUDA Cores
  GPU Max Clock rate:                            1560 MHz (1.56 GHz)
  Memory Clock rate:                             7001 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.3, NumDevs = 1, Device0 = NVIDIA RTX A4000
Result = PASS

I really don’t see where the problem is…

As described before, the binaries will use their own Runtime, which is given by:

print(torch.version.cuda)

and I guess it would return 10.2, which is incompatible with your GPU as it’s missing the desired compute capability.

In this case, select the CUDA 11 version for the pip wheel / conda binaries and reinstall it.

Hello,

torch.version.cuda returns : 11.3 .
So it’s supposed to be compatible right ?

Yes, 11.3 should work and I also just verified that the binaries with the 11.3 runtime work properly on an A4000 using:

import torch

print(torch.__version__)
print(torch.version.cuda)
print(torch.backends.cudnn.version())
print(torch.cuda.get_device_name(0))
print(torch.cuda.get_device_properties(0))

x = torch.randn(1, 3, 224, 224, device='cuda')
conv = torch.nn.Conv2d(3, 3, 3).cuda()

out = conv(x)
print(out.shape)

Output:

1.11.0+cu113
11.3
8200
NVIDIA RTX A4000
_CudaDeviceProperties(name='NVIDIA RTX A4000', major=8, minor=6, total_memory=16117MB, multi_processor_count=48)
torch.Size([1, 3, 222, 222])

I get the exact same results except for print(torch.__version__)
where I get only 1.11.0 as an output.
Is it a problem ?

Is the code snippet working as well or are you seeing the same error message when executing it?

The code is working properly yes… Do you know if pytorch is available with CUDA 11.6 ?
I’ve tried to change CUDA version to 11.6 and I’ve installed pytorch with the command

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

(the latest version), but it doesn’t seem to work either. So I’m wondering if it needs CUDA 11.3.

In that case I don’t fully understand what is causing the issue with the missing GPU architecture.
Are you using a custom extension and didn’t build it for Ampere GPUs?

Yes, we have internal CUDA 11.6 wheels, which can be downloaded e.g. via:

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

but this would not fix the issue you are seeing as you are either using the CUDA 10.2 wheels (installed via a wrongly specified requirement) or are building an extension for older architectures.

Sorry, I don’t really know what a custom extension is.
My script is written in python and runs a flask application. It is a python’s framework used to create web applications easily.
What I’m trying to do is to display some images on this web app. Those images are passed through a neural network.
This code work well on a colleague’s computer who have a different GPU and a different version of CUDA.

I’ve installed CUDA 11.6 following these installation instructions :

wget https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run
sudo sh cuda_11.6.0_510.39.01_linux.run

Hello again !

UserWarning:
NVIDIA RTX A4000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

This was in my console too. Is it what you meant with GPU architectures ?

So I just found the solution to my issue. I had installed the wrong pytorch version for my gpu architecture (I think).
I installed pytorch 1.8.1 which is compatible with the architecture sm_86 and now it’s working fine !
Thank you @ptrblck for helping me !!

I don’t think https://download.pytorch.org/whl/cu116/ available. If you will do get request it access denied while https://download.pytorch.org/whl/cu115 show a listing of all packages.