When trying to run .cuda(), I encounter the following error:
THCudaCheck FAIL file=/home/hendrik/pytorch/aten/src/THC/THCGeneral.c line=70 error=30 : unknown error
Traceback (most recent call last):
File “train_fader_network.py”, line 154, in
train_fader_network()
File “train_fader_network.py”, line 29, in train_fader_network
encoder_decoder.cuda(gpu_id)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 216, in cuda
return self._apply(lambda t: t.cuda(device))
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 146, in _apply
module._apply(fn)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 152, in _apply
param.data = fn(param.data)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 216, in
return self._apply(lambda t: t.cuda(device))
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/_utils.py”, line 61, in _cuda
with torch.cuda.device(device):
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/cuda/init.py”, line 218, in enter
_lazy_init()
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/cuda/init.py”, line 153, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /home/hendrik/pytorch/aten/src/THC/THCGeneral.c:70
I’m running Ubuntu 14.04, with CUDA 9.1 and cuDNN 7.
Output of nvidia-smi:
NVIDIA-SMI 387.34 Driver Version: 387.34
(And other information, showing three GPUs, GTX 660 and 2x TITAN X).
Output of nvcc --version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85
I’ve seen similar threads where this happens after a driver update, but even after reinstalling the drivers and rebooting, I’ve had no luck. Any suggestions on what to investigate? The error message doesn’t provide much information.