RuntimeError: cuda runtime error (59) in loss.backward()

1

I have checked other posts on the Cuda runtime error 59 and most of them are related to accessing out-of-bound indices but my error seems to be occurring during backprop and not in the forward prop (if it was an out-of-bound index, it would have happened in forward prop). The detailed error message is below:

THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THCUNN/generic/Threshold.cu line=66 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "srnn/train.py", line 191, in <module>
    main()
  File "srnn/train.py", line 80, in main
    train(args)
  File "srnn/train.py", line 159, in train
    loss.backward()
  File "/home/anirudh/.virtualenvs/gp/local/lib/python2.7/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/anirudh/.virtualenvs/gp/local/lib/python2.7/site-packages/torch/nn/_functions/thnn/auto.py", line 174, in backward
    update_grad_input_fn(self._backend.library_state, input, grad_output, grad_input, *gi_args)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THCUNN/generic/Threshold.cu:66
2

Using CUDA_LAUNCH_BLOCKING=1, I get the following trace:

/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [0,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [1,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [2,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [3,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [4,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [5,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [6,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [7,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [8,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [9,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [10,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [11,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [12,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [13,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [14,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [15,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [16,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [17,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [18,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [19,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [20,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [21,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [22,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [23,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [24,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [25,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [26,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [27,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [28,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [29,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [30,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [31,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [32,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [33,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [34,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [35,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [36,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [37,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [38,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [39,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [40,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [41,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [42,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [43,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [44,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [45,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [46,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [47,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [48,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [49,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [50,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [51,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [52,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [53,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [54,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [55,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [56,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [57,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [58,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [59,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [60,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [61,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [62,0,0] Assertion `dstIndex < dstFillDimSize` failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:202: void indexFillSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, IndexType, long, T) [with T = float, IndexType = unsigned int, DstDim = 2, IdxDim = -2]: block: [0,0,0], thread: [63,0,0] Assertion `dstIndex < dstFillDimSize` failed.
THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorCopy.cu line=100 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "srnn/train.py", line 190, in <module>
    main()
  File "srnn/train.py", line 80, in main
    train(args)
  File "srnn/train.py", line 158, in train
    loss.backward()
  File "/home/anirudh/.virtualenvs/gp/local/lib/python2.7/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
  File "/home/anirudh/.virtualenvs/gp/local/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 44, in backward
    grad_value = grad_output.index(self.index).clone()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorCopy.cu:100
3

Error 59 is notoriously non-informative. Are you, by chance, using a pre-trained model when you get this error? I had a similar issue until I adjusted the number of outputs in the last layer to match the number of classes in my dataset.

2 Likes
4

Hi,

From the error message it looks like you are using indexing with wrong indices.
A good way to debug these errors is to simply run the exact same code on CPU, that way you will get a proper stack trace with the exact line that crashed.

5 Likes
5

@albanD 's answer helped me a lot, when I moved my model and variables to cpu I got an exact line number that caused it:

        model.cpu()
        input_features.cpu()
        result = model(input_features)

Thanks!