RuntimeError: cuda runtime error (30)

When trying to run .cuda(), I encounter the following error:

THCudaCheck FAIL file=/home/hendrik/pytorch/aten/src/THC/THCGeneral.c line=70 error=30 : unknown error
Traceback (most recent call last):
File “train_fader_network.py”, line 154, in
train_fader_network()
File “train_fader_network.py”, line 29, in train_fader_network
encoder_decoder.cuda(gpu_id)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 216, in cuda
return self._apply(lambda t: t.cuda(device))
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 146, in _apply
module._apply(fn)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 152, in _apply
param.data = fn(param.data)
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 216, in
return self._apply(lambda t: t.cuda(device))
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/_utils.py”, line 61, in _cuda
with torch.cuda.device(device):
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/cuda/init.py”, line 218, in enter
_lazy_init()
File “/home/hendrik/deep-learning/local/lib/python2.7/site-packages/torch/cuda/init.py”, line 153, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /home/hendrik/pytorch/aten/src/THC/THCGeneral.c:70

I’m running Ubuntu 14.04, with CUDA 9.1 and cuDNN 7.

Output of nvidia-smi:

NVIDIA-SMI 387.34 Driver Version: 387.34
(And other information, showing three GPUs, GTX 660 and 2x TITAN X).

Output of nvcc --version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

I’ve seen similar threads where this happens after a driver update, but even after reinstalling the drivers and rebooting, I’ve had no luck. Any suggestions on what to investigate? The error message doesn’t provide much information.

For anyone else coming across this post, it turns out CUDA 9.1 doesn’t support Ubuntu 14.04:
http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#system-requirements

Everything works with CUDA 8.0 and NVIDIA 375.26.