What the error means? RuntimeError,device-side assert triggered

xType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [97,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [97,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [97,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
THCudaCheck FAIL file=/data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu line=226 error=59 : device-side assert triggered
Traceback (most recent call last):
File “dialog_train.py”, line 537, in
main()
File “dialog_train.py”, line 533, in main
trainModel(model, trainData, validData, testData, dataset, optim, criterion)
File “dialog_train.py”, line 376, in trainModel
train_loss, train_acc, train_loss_ppl = trainEpoch(epoch)
File “dialog_train.py”, line 333, in trainEpoch
outputs, topic_dist = model(batch[0], targets)
File “/home/zeng/envs/pytorch_0.1.10_py27/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 202, in call
result = self.forward(*input, **kwargs)
File “/home/zeng/parlAI/conversation/OpenNMT-Dialog/Models.py”, line 243, in forward
out, dec_hidden, _attn = self.decoder(target_embedding, hidden_n, context, init_output)
File “/home/zeng/envs/pytorch_0.1.10_py27/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 202, in call
result = self.forward(*input, **kwargs)
File “/home/zeng/parlAI/conversation/OpenNMT-Dialog/Models.py”, line 189, in forward
emb_t = torch.cat([emb_t, output], 2)
File “/home/zeng/envs/pytorch_0.1.10_py27/lib/python2.7/site-packages/torch/autograd/variable.py”, line 840, in cat
return Concat(dim)(*iterable)
File “/home/zeng/envs/pytorch_0.1.10_py27/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py”, line 305, in forward
return torch.cat(inputs, self.dim)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /data/users/soumith/builder/wheel/pytorch-src/torch/lib/THC/generic/THCTensorMath.cu:226

this means that you are indexing out-of-bounds indices in your code. For example x[10] when x size is only 4

2 Likes

The weired thing is that, i output all the intermediate result, especially each input of embedding layer did not find any index big than the tensor size

Do i have the error like this?

it can be smaller index too, for example -1

thank you sir… it is