# Cublas runtime error **URL:** \ **Category:** Uncategorized\ **Created:** [December 17, 2017, 2:37am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243 "2017-12-17T02:37:41Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 17, 2017, 2:37am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/1 "2017-12-17T02:37:41Z")
I get this cublas error after several epochs of training. The data is fed sequentially and the error always occurs at a different iteration, so it is difficult to reproduce in a simple example. Something is causing an assertion on line 97 of [https://github.com/torch/cutorch/blob/master/lib/THC/THCTensorScatterGather.cu](https://github.com/torch/cutorch/blob/master/lib/THC/THCTensorScatterGather.cu). Any ideas of what could be happening, or what I should be checking for? I’ve tried just catching it and continuing onto the next iteration, but then there are just more (different) cublas errors on the next iteration, too. The error always occurs during a forward call of a torch.nn.Linear object RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/torch/lib/THC/THCBlas.cu:246 /pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor\_gatherKernel(TensorInfo\, TensorInfo\, TensorInfo\, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion `indexValue >= 0 && indexValue < src.sizes[dim]` failed. Thanks ---
**Author:** ![SimonW](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/32/1702_2.png) [@SimonW](https://discuss.pytorch.org/u/SimonW)\ **Post date:** [December 17, 2017, 4:26am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/2 "2017-12-17T04:26:12Z")
What does you architecture look like? ---
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 17, 2017, 4:50am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/3 "2017-12-17T04:50:26Z")
It’s a custom RNN decoder for image captioning, described in this paper: [https://arxiv.org/abs/1707.07998](https://arxiv.org/abs/1707.07998). Suppose I could give the full source code, though really wish I could provide a simple minimal working example. Also if I do catch this exception and move onto the next iteration, I get this CUDA runtime error just trying to even move a tensor to the GPU for initialization. File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 106, in init\_state state\_attention = (hidden\_attention\_lstm.cuda(), cell\_attention\_lstm.cuda()) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 298, in cuda return CudaTransfer.apply(self, device, async) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/\_functions/tensor.py”, line 201, in forward return i.cuda(async=async) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/\_utils.py”, line 69, in _cuda return new\_type(self.size()).copy_(self, async) RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/generic/THCTensorCopy.c:20 /pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor\_gatherKernel(TensorInfo\, TensorInfo\, TensorInfo\, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [30,0,0] Assertion `indexValue >= 0 && indexValue < src.sizes[dim]` failed. ---
**Author:** ![ngimel](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/32/14766_2.png) [@ngimel](https://discuss.pytorch.org/u/ngimel)\ **Post date:** [December 17, 2017, 7:30am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/4 "2017-12-17T07:30:30Z")
Cublas error is a red herring, your true problem is an assert thrown by gather kernel - is it possible that something is wrong with the indices that are sent to it? After assert us thrown, cuda context is corrupted, so don’t try catching it and continuing, the errors that you see are expected. [http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#assertion](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#assertion) ---
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 18, 2017, 4:39pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/5 "2017-12-18T16:39:28Z")
@ngimel How can I check for that? I’ve asserted that all the tensors being fed to this torch.nn.Linear layer are the correct dimension and don’t contain nan’s or inf values. Not sure what else could be corrupted. ---
**Author:** ![ngimel](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/32/14766_2.png) [@ngimel](https://discuss.pytorch.org/u/ngimel)\ **Post date:** [December 18, 2017, 4:53pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/6 "2017-12-18T16:53:25Z")
The problematic layer is not torch.nn.Linear, it’s something like index\_select, or some advanced indexing that you are possibly using, hard to say without looking at your model. ---
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 18, 2017, 5:02pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/7 "2017-12-18T17:02:20Z")
Okay. Just to clarify, even though the traceback (below) shows the error at torch.nn.Linear and I’ve checked the inputs to this layer, the issue could be caused by advanced indexing happening somewhere else? Doesn’t exactly make sense to me, but I can certainly go through and check the indexing in the rest of my code. Thanks Traceback (most recent call last): File “example\_server.py”, line 74, in validate\_every=100, validater\_examples=validater\_examples, validate\_examples\_every=1000, use\_cuda=True) File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/train.py”, line 61, in train \_, logprobs = decoder.forward(V, y\_0, state\_attention, state\_language, y\_true=caption\_true\_packed) File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 78, in forward (state\_language[0][0:batch\_size\_t], state\_language[1][0:batch\_size\_t])) File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 158, in forward v\_hat = self.attention.forward(V, new\_state\_attention[0]) # N x D File “/zfsauton/home/mbarnes1/aggrevated/aggrevated-coco/models/networks.py”, line 191, in forward V\_hidden = [self.image\_linear(v) for v in V] # List of length N, containing 1 x k x H File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/module.py”, line 325, in **call** result = self.forward(\*input, \*\*kwargs) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/modules/linear.py”, line 55, in forward return F.linear(input, self.weight, self.bias) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/nn/functional.py”, line 837, in linear output = input.matmul(weight.t()) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/autograd/variable.py”, line 386, in matmul return torch.matmul(self, other) File “/zfsauton/home/mbarnes1/miniconda2/envs/torch/lib/python2.7/site-packages/torch/functional.py”, line 191, in matmul output = [torch.mm](http://torch.mm)(tensor1, tensor2) RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/torch/lib/THC/THCBlas.cu:246 /pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor\_gatherKernel(TensorInfo\, TensorInfo\, TensorInfo\, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion `indexValue >= 0 && indexValue < src.sizes[dim]` failed. ---
**Author:** ![ngimel](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngimel/32/14766_2.png) [@ngimel](https://discuss.pytorch.org/u/ngimel)\ **Post date:** [December 18, 2017, 5:06pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/8 "2017-12-18T17:06:15Z")
> [@mbarnes1](#): > > /pytorch/torch/lib/THC/THCTensorScatterGather.cu:97: void THCudaTensor\_gatherKernel(TensorInfo\, TensorInfo\, TensorInfo\, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 2]: block: [0,0,0], thread: [29,0,0] Assertion indexValue \>= 0 && indexValue \< src.sizes[dim] failed. Here’s the part of the stack trace that corrupted your cuda context, all subsequent errors are stemming from this. ---
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 18, 2017, 5:21pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/9 "2017-12-18T17:21:19Z")
Got it. Trying to understand where in my code there are corrupted indices, but it seems like all we know based on this traceback is its due to something that invokes the gather kernel. I’ll start with checking all the advanced indexing. Thanks for your help ---
**Author:** ![SimonW](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/32/1702_2.png) [@SimonW](https://discuss.pytorch.org/u/SimonW)\ **Post date:** [December 18, 2017, 5:49pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/10 "2017-12-18T17:49:25Z")
Try adding `torch.cuda.synchronize` after each cuda operation. That way the error won’t be deferred to later operations. [http://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize](http://pytorch.org/docs/master/cuda.html#torch.cuda.synchronize) If you can share your script, I can take a look. ---
**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\ **Post date:** [December 18, 2017, 5:51pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/11 "2017-12-18T17:51:21Z")
an easy way to get the correct stack trace with CUDA is to run: ```auto CUDA_LAUNCH_BLOCKING=1 python yourscript.py ``` This puts cuda in synchronous mode (rather than it’s default of asynchronous mode) ---
**Author:** ![mbarnes1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mbarnes1/32/2737_2.png) [@mbarnes1](https://discuss.pytorch.org/u/mbarnes1)\ **Post date:** [December 19, 2017, 3:08pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/12 "2017-12-19T15:08:27Z")
Ran a couple instances of training yesterday with CUDA\_LAUNCH\_BLOCKING=1, and was able to debug the issue based on the better stack trace. Long story short, it ended up being an issue with torch.gather very rarely receiving an invalid index. Thanks everyone for your help. ---
**Author:** ![Rafael\_Valle](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f9b8f/32.png) [@Rafael\_Valle](https://discuss.pytorch.org/u/Rafael_Valle)\ **Post date:** [January 17, 2018, 5:42am UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/13 "2018-01-17T05:42:25Z")
> [@SimonW](#): > > torch.cuda.synchronize Did the trick for me! ---
**Author:** ![ezyang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ezyang/32/486_2.png) [@ezyang](https://discuss.pytorch.org/u/ezyang)\ **Post date:** [April 26, 2018, 10:52pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/14 "2018-04-26T22:52:33Z")
@Rafael_Valle Erm, what exactly did you do? ---
**Author:** ![Rafael\_Valle](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f9b8f/32.png) [@Rafael\_Valle](https://discuss.pytorch.org/u/Rafael_Valle)\ **Post date:** [April 26, 2018, 11:27pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/15 "2018-04-26T23:27:49Z")
called torch.cuda.synchronize() ---
**Author:** ![isalirezag](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/fbc32d/32.png) [@isalirezag](https://discuss.pytorch.org/u/isalirezag)\ **Post date:** [February 25, 2019, 3:19pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/16 "2019-02-25T15:19:58Z")
> [@Rafael\_Valle](#): > > torch.cuda.synchronize() where did you put it? ---
**Author:** ![Rafael\_Valle](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/5f9b8f/32.png) [@Rafael\_Valle](https://discuss.pytorch.org/u/Rafael_Valle)\ **Post date:** [February 25, 2019, 6:38pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/17 "2019-02-25T18:38:58Z")
> [@SimonW](#): > > Try adding `torch.cuda.synchronize` after each cuda operation. Take a look at SimonW’s answer: “Try adding `torch.cuda.synchronize` after each cuda operation.” ---
**Author:** ![abdullahkhilji](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdullahkhilji/32/13636_2.png) [@abdullahkhilji](https://discuss.pytorch.org/u/abdullahkhilji)\ **Post date:** [June 30, 2019, 5:22pm UTC](https://discuss.pytorch.org/t/cublas-runtime-error/11243/18 "2019-06-30T17:22:20Z")
Yes, torch.gather is very efficient, but we need to be careful while using it. Thanks !