Rnn.py RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

1

Hi,
I’m getting an CUDNN_STATUS_INTERNAL_ERROR error like below.

python train_v2.py

Traceback (most recent call last):
  File "train_v2.py", line 113, in <module>
    main()
  File "train_v2.py", line 74, in main
    model.cuda()
  File "/home/ahkim/Desktop/squad_vteam/src/model.py", line 234, in cuda
    self.network.cuda()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 105, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

What should I try to resolve this issue?
I tried deleting .nv but no success.

The same code runs without error using Nvidia Driver Version: 396.26 (cuda V9.1.85. torch.backends.cudnn.version(): 7102). I’m getting an error using Driver Version: 390.67 (cuda V9.1.85. torch.backends.cudnn.version(): 7102)

2

solved by below steps.

  1. export LD_LIBRARY_PATH= “/usr/local/cuda-9.1/lib64”

  2. Due to nfs issue, have pytoch cache not in nfs. For example:
    $ rm ~/.nv -rf
    $ mkdir -p /tmp/$USER/.nv
    $ ln -s /tmp/$USER/.nv ~/.nv