THCudaCheck FAIL error 48

#1

Hi

I have the following setup:
RTX 2080Ti, CUDA 10, compiled pytorch from source libcudnn7 Ubuntu 18.04

Now everything is absolutely fine with this. But, as soon as I swap my card with a GTX 1070 Ti
it fails. Same machine only different GPU:

THCudaCheck FAIL file=/home/geir/src/pytorch/aten/src/THC/generic/THCTensorMath.cu line=240 error=48 : no kernel image is available for execution on the device
terminate called after throwing an instance of 'std::runtime_error'
  what():  cuda runtime error (48) : no kernel image is available for execution on the device at /home/geir/src/pytorch/aten/src/THC/generic/THCTensorMath.cu:240
Aborted (core dumped)

Does this mean 1070 Ti is not supported on latest pytorch source? Should I have used some other settings while compiling the pytorch source?
Sorry if this is a stupid question, but I am not a CUDA/PyTorch guy, I only try to integrate some software modules.

#2

OK, so I fixed the issue. I had to add some more options before building. I used this:

export USE_OPENCV=1 TORCH_CUDA_ARCH_LIST="5.1;6.1;7.5"

I also added NCCL and MPI support
It was probably the CUDA_ARCH_LIST that was the key, i guess.

Anyway, it now works with both 1070 Ti and 2080 Ti

#3

Would you mind marking your own answer as a solution? You are right, CUDA_ARCH_LIST is the key, you only need 6.1 and 7.5 for your cards.

#4

I added the 5.1 just to see if it would run on a 970, and indeed it did!