Okay. Just to clarify, even though the traceback (below) shows the error at torch.nn.Linear and I’ve checked the inputs to this layer, the issue could be caused by advanced indexing happening somewhere else? Doesn’t exactly make sense to me, but I can certainly go through and check the indexing in the rest of my code. Thanks
Traceback (most recent call last):
File “example_server.py”, line 74, in
validate_every=100, validater_examples=validater_examples, validate_examples_every=1000, use_cuda=True)
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/train.py”, line 61, in train
_, logprobs = decoder.forward(V, y_0, state_attention, state_language, y_true=caption_true_packed)
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 78, in forward
(state_language[0][0:batch_size_t], state_language[1][0:batch_size_t]))
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 158, in forward
v_hat = self.attention.forward(V, new_state_attention[0]) # N x D
File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 191, in forward
V_hidden = [self.image_linear(v) for v in V] # List of length N, containing 1 x k x H
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 325, in call
result = self.forward(*input, **kwargs)
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/linear.py”, line 55, in forward
return F.linear(input, self.weight, self.bias)
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/functional.py”, line 837, in linear
output = input.matmul(weight.t())
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 386, in matmul
return torch.matmul(self, other)
File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/functional.py”, line 191, in matmul
output = torch.mm(tensor1, tensor2)
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/torch/lib/THC/THCBlas.cu:246
/pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.