Cuda error when sending tensor to device

1

Hi folks,
I was wondering if anyone has seen this type of error before?
Any ideas on what’s going wrong here?
Thanks!

>> import torch
>>> import numpy as np
>>> ten = torch.from_numpy(np.random.randn(23,4))
>>> ten.cuda(2)
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549636813070/work/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kirk/miniconda3/envs/torch/lib/python3.7/site-packages/torch/cuda/__init__.py", line 162, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1549636813070/work/aten/src/THC/THCGeneral.cpp:51
2

The error may be because of your driver.
Try rebooting or update/reinstall the driver.

3

Easier said than done, that’s a server with multiple users so rebooting is not really an option since there are a lot of thing running.
But here’s what I don’t understand we can load and unlod modules so that we don’t have to reboot.
On a different conda environment I have tensorflow freshly installed version 1.13 and it works just fine which leads me to believe that it should also work with pytorch.