THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp

chrissikek · December 10, 2018, 9:03pm

Hey Guys,

I recently changed my gpu to an RTX2080ti and in the process also upgraded from CUDA8 to CUDA9 and the newest pytorch version. Ever since I get the following message when I use pytorch (the program keeps running though):

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument

I could not find much on this and like to understand in what context this error can appear and if it can cause performance problems.

Thanks your help!

albanD · December 11, 2018, 11:04am

Hi,

Do the CUDA 9.0 samples work on an RTX2080 card? I though only CUDA 10 worked on these?

chrissikek · December 11, 2018, 12:22pm

Thanks! You may be right, I have the exact same problem with the CUDA samples as described here , that indicates that there might be a fundamental problem with RTX2080ti and cuda9 (on linux?).

Would be interested if anyone has experienced otherwise.

albanD · December 11, 2018, 12:25pm

I thinks it’s just that most recent cards like RTX2080Ti only work with CUDA 10 (not OS dependant).

cristinasegalin · January 7, 2019, 4:54pm

I have the same problem with cuda 10 and pytorch 1.0

ggeor · January 16, 2019, 11:51pm

Same problem for me. RTX2080Ti, Ubuntu 18.04, CUDA 10

filippobistaffa · February 20, 2019, 9:36am

I experience the same problem, I’m using CUDA 9 though.
Anyway, it seems that the GPU is used anyway, even though I’m not sure if it’s used “fully”.

ggeor · February 20, 2019, 3:46pm

By the way, for anyone following this topic, a solution to at least get things running is to set `torch.backends.cudnn.benchmark = False’. I found this solution in this thread: A error when using GPU

chrissikek · February 20, 2019, 4:03pm

That may solve the problem of the error appearing, though it would be interesting to know why the rtx 2080 ti fails the cuda benchmark.

For me it does also fail the cudnn installation examples. Can anyone else confirm that?

Felix_Lessange · February 25, 2019, 10:49am

Same with cuda 10, ubuntu 18.04, and Titan RTX, on python 3.6

Rifle_Zhang · March 4, 2019, 5:12am

I put ‘torch.backends.cudnn.benchmark = False’
at the beginning of my source code, but the error message still appears. Not sure why that happened.

ggeor · March 4, 2019, 5:39pm

are you sure that later in the code that attribute is not set to true? try to print to screen torch.backends.cudnn.benchmark at various points of execution, to make sure it is indeed False

Yiru_Shen · March 12, 2019, 8:20pm

Same issue here. Pytorch 1.0.0, CUDA 10.0, RTX 2080, on Fedora 28.
Set " torch.backends.cudnn.benchmark=False" doesn’t work. It shows:
cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Any information would be appreciated!

Yiru_Shen · March 12, 2019, 8:27pm

Okay I have solve the problem. You cannot directly install pytorch, instead “pip3 install -U https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl” work for me.

Evergrow · March 29, 2019, 7:37am

Thanks your reply! But I want to know how to solve this problem on Pytorch 1.0.0, CUDA 9.0, RTX 2080. Must change to CUDA 10.0?

ptrblck · March 29, 2019, 10:57pm

While some users seem to got the RTX 2080 working with CUDA9.X, it seems CUDA10 is the way to go for these new GPUs.

zfzhang · April 1, 2019, 9:21pm

Hi, I signed up today to share a solution

My setup is 2080Ti, CUDA 10.1, python 3.6, installation method is pip3.

I originally installed the latest stable version suggested by pytorch home page https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl
, which produces that error message.

Then I tried the version 1.0.0 suggested by Yiru_Shen, which still produces that error message.

Finally I tried the nightly build, which does NOT produce any error message, luckily i don’t have to build from source

ptrblck · April 1, 2019, 11:52pm

Welcome and thanks for the sign up and the information!

zfzhang · April 2, 2019, 6:41am

Thank you! This forum has really helped me a lot, and pytorch is the best framework I’ve seen so far in terms of code & documentation .

jmaronas · April 2, 2019, 8:05am

RTX only works with cuda 10 toolkit. I use this pytorch conda enviroment and it works (both python3.5 and python 3.7):

conda=/usr/local/anaconda3/bin/conda
activate=/usr/local/anaconda3/bin/activate
deactivate=/usr/local/anaconda3/bin/deactivate

#python3.7
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0 python=3.7
source $activate  /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate

#python 3.5
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0 python=3.5
source $activate  /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate