Need help on running code!

1

(problem is fixed now!!)
(problem is fixed now!!, I just need to change my label from 1-5 to 0-4, and then use n_classes = 5)
Hi, there. I am working on reproducing a paper ‘Very Deep Convolutional Networks for Text Classification’ as a final project. I have been runing with no problem on my own GPU.

Full Code

But, when I run it on a server, the following error pops up.

Traceback (most recent call last):
  File "ypf_1_9.py", line 313, in <module>
    loss.backward()
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cublas runtime error : an internal operation failed at /dev/shm/cmaclean/python-single/portage/sci-libs/caffe2-0.4.1/work/pytorch-0.4.1/aten/src/THC/THCBlas.cu:249

I am new to deep learning, my knowledge is limited. It could be my code is wrong somewhere but I can not tell right now. I have to get it working on the server to run hundreds of hours. Any help is appreciated!

2

Is this the complete stack trace ?

Post the stack trace you get after running your code in the following manner.

CUDA_LAUNCH_BLOCKING=1 python yourscript.py
3 
/dev/shm/cmaclean/python-single/portage/sci-libs/caffe2-0.4.1/work/pytorch-0.4.1/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [3,0,0] Assertion `t >= 0 && t < n_classes` failed.
THCudaCheck FAIL file=/dev/shm/cmaclean/python-single/portage/sci-libs/caffe2-0.4.1/work/pytorch-0.4.1/aten/src/THCUNN/generic/ClassNLLCriterion.cu line=111 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "ypf_1_9_test.py", line 308, in <module>
    loss = criterion(outputs, labels)
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 862, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/mnt/bwpy/single/usr/lib/python3.5/site-packages/torch/nn/functional.py", line 1407, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /dev/shm/cmaclean/python-single/portage/sci-libs/caffe2-0.4.1/work/pytorch-0.4.1/aten/src/THCUNN/generic/ClassNLLCriterion.cu:111
4 
Assertion `t >= 0 && t < n_classes` failed

It looks like my class labels are wrong. My input have labels from 1-5, and I used nclasses = 5 in the last linear layer.

5

Change labels from 1-5 to 0-4. I think this is causing the error. Because python, C++ uses 0 index.