THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp

Hey Guys,

I recently changed my gpu to an RTX2080ti and in the process also upgraded from CUDA8 to CUDA9 and the newest pytorch version. Ever since I get the following message when I use pytorch (the program keeps running though):

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument

I could not find much on this and like to understand in what context this error can appear and if it can cause performance problems.

Thanks your help!

Hi,

Do the CUDA 9.0 samples work on an RTX2080 card? I though only CUDA 10 worked on these?

Thanks! You may be right, I have the exact same problem with the CUDA samples as described here , that indicates that there might be a fundamental problem with RTX2080ti and cuda9 (on linux?).

Would be interested if anyone has experienced otherwise.

I thinks it’s just that most recent cards like RTX2080Ti only work with CUDA 10 (not OS dependant).

I have the same problem with cuda 10 and pytorch 1.0

Same problem for me. RTX2080Ti, Ubuntu 18.04, CUDA 10

I experience the same problem, I’m using CUDA 9 though.
Anyway, it seems that the GPU is used anyway, even though I’m not sure if it’s used “fully”.

By the way, for anyone following this topic, a solution to at least get things running is to set `torch.backends.cudnn.benchmark = False’. I found this solution in this thread: A error when using GPU

That may solve the problem of the error appearing, though it would be interesting to know why the rtx 2080 ti fails the cuda benchmark.

For me it does also fail the cudnn installation examples. Can anyone else confirm that?

Same with cuda 10, ubuntu 18.04, and Titan RTX, on python 3.6

I put ‘torch.backends.cudnn.benchmark = False’
at the beginning of my source code, but the error message still appears. Not sure why that happened.

are you sure that later in the code that attribute is not set to true? try to print to screen torch.backends.cudnn.benchmark at various points of execution, to make sure it is indeed False

Same issue here. Pytorch 1.0.0, CUDA 10.0, RTX 2080, on Fedora 28.
Set " torch.backends.cudnn.benchmark=False" doesn’t work. It shows:
cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Any information would be appreciated!

Okay I have solve the problem. You cannot directly install pytorch, instead “pip3 install -U https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl” work for me.

Thanks your reply! But I want to know how to solve this problem on Pytorch 1.0.0, CUDA 9.0, RTX 2080. Must change to CUDA 10.0?

While some users seem to got the RTX 2080 working with CUDA9.X, it seems CUDA10 is the way to go for these new GPUs.

Hi, I signed up today to share a solution :slight_smile:

My setup is 2080Ti, CUDA 10.1, python 3.6, installation method is pip3.

I originally installed the latest stable version suggested by pytorch home page https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
, which produces that error message.

Then I tried the version 1.0.0 suggested by Yiru_Shen, which still produces that error message.

Finally I tried the nightly build, which does NOT produce any error message, luckily i don’t have to build from source :grin:

1 Like

Welcome and thanks for the sign up and the information! :slight_smile:

1 Like

Thank you! This forum has really helped me a lot, and pytorch is the best framework I’ve seen so far in terms of code & documentation :smiley:.

1 Like

RTX only works with cuda 10 toolkit. I use this pytorch conda enviroment and it works (both python3.5 and python 3.7):

conda=/usr/local/anaconda3/bin/conda
activate=/usr/local/anaconda3/bin/activate
deactivate=/usr/local/anaconda3/bin/deactivate

#python3.7
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0 python=3.7
source $activate  /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate

#python 3.5
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0 python=3.5
source $activate  /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate