CuDNN status internal error with Ubuntu 14.04

[hyojin]

Hi,

I have CUDNN_STATUS_INTERNAL_ERROR on my Ubuntu 14.04 machine with Cuda 8. (GPUs: Maxwell Titan X, Pascal Titan X, Driver version: 384.66)
The problem goes away when I reboot the machine, but it is inconvenient because it happens every so often. Does anyone know why this would happen?

This seems identical to the problem in this post: https://stackoverflow.com/questions/45810356/runtimeerror-cudnn-status-internal-error

Traceback (most recent call last):
File “train_univ_incremental_itersize.py”, line 607, in
main()
File “train_univ_incremental_itersize.py”, line 341, in main
train_loss, cur_learning_rate = train(train_loader, net, criterion, optimizer, epoch)
File “train_univ_incremental_itersize.py”, line 399, in train
myOut = model(input_var)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 259, in call
result = self.forward(*input, **kwargs)
File “/playpen1/alternet.py”, line 109, in forward
low_out = self.alexnet_lower(img)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 259, in call
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/container.py”, line 67, in forward
input = module(input)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py”, line 259, in call
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/modules/conv.py”, line 254, in forward
self.padding, self.dilation, self.groups)
File “/usr/local/lib/python2.7/dist-packages/torch/nn/functional.py”, line 52, in conv2d
return f(input, weight, bias)

[richard]

What is your pytorch version (check torch.__version__) and what is your CUDNN version?

[hyojin]

Pytorch version is 0.2.0_3
CuDNN version is v5

Thanks

[richard]

Pytorch doesn’t support CuDNN v5. I would recommend upgrading to v6 (https://developer.nvidia.com/cudnn)

[hyojin]

Oops, I checked cudnn version for pytorch via torch.backends.cudnn.version() and it returns 6021. Which means it uses cuDNN v6.
I don’t know what is the problem now…

According to following post, pytorch ships its own cuDNN version (6.0)

Does Pytorch 0.2.0 still support cudnn 5.1 and how to manage different versions of cudnn?

Hi there, I have a question as shown in the title. First, does Pytorch 0.2.0 still support Cudnn 5.1? I have some Torch code to run, but currently Torch only support cudnn5.1, as far as I know. SO I install cudnn 5.1 globally in my CUDA installation folder. I then install pytorch using, conda install pytorch torchvision cuda80 -c soumith But when I check within Pytorch cudnn version using, torch.backends.cudnn.version() It shows cudnn version of 6021. But I do not have cudnn 6.0 installed…

[richard]

Interesting. Could you try the following: (any of these will be helpful in identifying the problem!)

1. build pytorch from master and see if it happens?
2. provide a minimal example that causes the CuDNN error. From the trace it looks like it’s happening in conv2d

[hyojin]

1. I can try this. Currently installed version is the official distributed version which I downloaded in August 2017.
2. This I cannot do. I don’t think this is specific to code. Indeed it seems to be caused by conv2 (or other conv operations), but the same code usually runs without problem. I did test with different codes.

But I do stop programs using keyboard interrupt (Ctrl+C). Could this mess up with the cuDNN status? That sometimes does not clear up the GPU memory, too.

[hyojin]

As mentioned in the following post, sudo rm -r ~/.nv works for this problem. Not sure why it happens though.

Cublas runtime error : library not initialized at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCGeneral.c:383

@Rohith_AP, @ShawnGuo “sudo rm -r ~/.nv” works for my 4-GPU machine to remove error below RuntimeError: cublas runtime error : library not initialized at /py/conda-bld/pytorch_1493681908901/work/torch/lib/THC/THCGeneral.c:394 FYI, it also solves error below File “/opt/anaconda/lib/python3.6/site-packages/torch/nn/functional.py”, line 40, in conv2d return f(input, weight, bias) RuntimeError: CUDNN_STATUS_INTERNAL_ERROR Thanks!

[ho0712 (ChanTzeho)]

yes. it’s work. I also have the error