RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS

1

My environment is

  • ubuntu 16.04
  • python 3.5
  • RTX 2080 ti
  • cuda-9.0
  • cudnn 7

When I tried to train the model, there was RuntimeError (RuntimeError: CuDNN error: CUDNN_STATUS_SUCCESS).

First of all, I totally could not understand what above error means.
Furthermore the same code what I tried to train worked well on gtx 1080 ti (the rest environments are totally same as above)

What is the problem?

2 Likes
2

I have the same issue, same GPU as you. Did you find a workaround?

3

No… I just know about the issue is occurred when I use RNN base model.
For example,
from torchvision import models
vgg = models.vgg19(pretrained = True)
vgg.cuda()
These lines are not problem. They work clearly.

However,
from torch import nn
gru = nn.gru(3, 3, 2)
gru.cuda()
Above code make issue and the problem is in …/torch/nn/modules/rnn.py file

Maybe, this issue is not cause from our environment and/or settings.

4

On windows with RTX 2080 (not TI) I’m seeing the same thing with an RNN

1 Like
5

When you update your CUDA to cuda-9.2 and update pytorch for cuda-9.2, the issue will be disappeared :smiley:

1 Like
6

I have the same problem with you. However , I used Cuda-10.0 . How do you solve this problem. Can you tell me please?

7

I also have this issues. same GPU as you guys.

8

@postero20 - that didn’t seem to do the trick for me. What version of cudnn do you have?

9

Same problem here with 2080 and CUDA 9.1

10

this problem can be solved by install cuda92, conda install cuda92 pytorch torchvision

1 Like
11

do you install cuda by conda? conda install cuda by cuda9.0, so you need to update the cuda by conda install cuda92

1 Like
12

@RainyFish - couldn’t get it to work even with installing cuda92 with conda.

13

you should check the version of pytorch if it is suitable for cuda.

14

@Zohaib_Ahmed I’m using pytorch 0.4.1, CUDA 9.2, and CUDNN 7104

15

@RainyFish No, I did install cuda using run file (https://developer.nvidia.com/cuda-92-download-archive?target_os=Linux)

16

I got the same issue, and my gpu is RTX 2080.
I didn’t figure it out, but I found a way to run my code.
The method is that running my code in Atom with the Hydrogen add-on. This error occur when I run the code at the first time, but it disappeared when I run it at the second time.
And I found that if I run my code just after restart my PC, there will be no such error.

So I guess that the reason of this error is that there is some other program using the GPU when I try to run my code.

17

RTX 2080ti, cuda 10.0, CNN, same problem.

18

I had the same error message but realised my pytorch install didn’t have CUDA installed with it so you can try the conda instructions from above (believe for win10 it’s just conda install pytorch cuda92 -c pytorch)

But I had CUDNN already downloaded so just follow the instructions here (https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html). All I needed to do was dump the CUDNN files here C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\

19

Thanks!!!

I installed the pytorch with cuda 9.0 before. I just update pytorch with cuda92 ( for win10 it’s just conda install pytorch cuda92 -c pytorch) and then the errer disappear.

1 Like
20

I have the same problem with RTX 2080, CUDA 10 and pytorch 0.4.1.