I’m getting this error in evaluation!!
Here is error snippet
/pytorch/aten/src/THC/THCTensorIndex.cu:362: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [7,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize
failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMathPointwise.cu line=59 error=59 : device-side assert triggered
Traceback (most recent call last):
File “evaluate.py”, line 102, in
main()
File “evaluate.py”, line 93, in main
results = evaluate(model, val_loader, maps[“aid_to_ans”], device)
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/autograd/grad_mode.py”, line 43, in decorate_no_grad
return func(*args, **kwargs)
File “evaluate.py”, line 30, in evaluate
output = model(img, q, lengths)
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/raid/cs15resch01005/vqa/VisualQA/models/mcb.py”, line 118, in forward
q = self.embedding(ques) # BxTxD
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/nn/modules/container.py”, line 92, in forward
input = module(input)
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 489, in call
result = self.forward(*input, **kwargs)
File “/home/cs15resch01005/.local/lib/python2.7/site-packages/torch/nn/modules/activation.py”, line 292, in forward
return torch.tanh(input)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorMathPointwise.cu:59
Can any one tell me how to go about it…? Thanks in advance
p.s.: used “CUDA_LAUNCH_BLOCKING=1”