Cuda runtime error (48) : no kernel image is available for execution

1

I am new to PyTorch, and attempting to run some Mask R-CNN code using PyTorch on AWS Ubuntu 16.04 with a Tesla K80 GPU. I am using python 3.6 and latest PyTorch code compiled from source. I am able to run PyTorch examples and some basic test code on GPU. However, when running a custom extension, i get this error:

RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu:386

If it helps, I am running a non-maximal suppression code from here: https://github.com/ruotianluo/pytorch-faster-rcnn/tree/master/lib/nms/ . I compiled the code using nvcc -c -o nms_kernel.cu.o nms_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_37 and then built using the build.py which all went fine as far as I can tell.

Full trace below:

THCudaCheck FAIL file=/home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu line=386 error=48 : no kernel image is available for execution on the device
Traceback (most recent call last):
File “model/mask_rcnn_lib/box.py”, line 335, in
run_check_nms()
File “model/mask_rcnn_lib/box.py”, line 324, in run_check_nms
keep = torch_nms(rois, 0.5)
File “/home/ubuntu/DSB_2018/HCK_MRCNN/model/mask_rcnn_lib/nms/torch_nms/init.py”, line 34, in torch_nms
areas = (x2 - x1 + 1) * (y2 - y1 + 1)
RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /home/ubuntu/pytorch/aten/src/THC/generated/…/generic/THCTensorMathPointwise.cu:386

Thanks and apologies if this is the wrong place to post this.

2

Could you try compiling with the following:

TORCH_CUDA_ARCH_LIST="3.7" python setup.py install

or

TORCH_CUDA_ARCH_LIST="3.7" python setup.py build develop

and see if the problem still persists?

1 Like
3

Hi, thanks for the response. I tried the first option, and it gave me the same error. For now, I am able to get around this problem using pre-compiled binaries for the custom code that I got from someone else. Thanks again!

4

Hi tavildar,

I was wondering if it possible to share the pre-compiled binaries you used to get around this problem. I met the same error message on AWS, and I didn’t find any solution yet. Thanks in advance!

5

Here is the link: you may have to rename the file. https://drive.google.com/open?id=1yjwrmSFzxdgBEZnFTAzSv9oz3Q-GHIG8

1 Like
6

Thank you so much!

– Runmin

7

Hi @tavildar,

I am getting similar error with my windows 10 X64 machine, with QuadraM1200 GPU, which is of compute compatibility 5.0 I am new to torch. May I know where we need to replace these binaries?

Thanks in advance.

1 Like
8

I am also facing this issue on my windows 10. Have you found a solution?

9

I am facing the same problem in centos,
RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at

10

Hello I am also facing same issue i have a Nvidia MQ1000M ubuntu 16.04, cuda 10 and nvidia-docker2