Illegal memory access when upsampling with cuda

1

code:

import torch
import torch.nn as nn
x=torch.randn(16, 32, 320).cuda()
nn.Upsample(scale_factor=4)(x)

$ CU_LAUNCH_BLOCKING=1 python
>>> import torch
>>> import torch.nn as nn
>>> torch.__version__
'0.2.0+c74f7d8'
>>> x=torch.randn(16, 32, 320).cuda()
>>> nn.Upsample(scale_factor=4)(x)
THCudaCheck FAIL file=/Users/qbx2/pytorch/torch/lib/THCUNN/generic/TemporalUpSamplingNearest.cu line=97 error=77 : an illegal memory access was encountered
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 259, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/upsampling.py", line 80, in forward
    return F.upsample(input, self.size, self.scale_factor, self.mode)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/functional.py", line 1005, in upsample
    return _functions.thnn.UpsamplingNearest1d.apply(input, _single(size), scale_factor)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/thnn/upsampling.py", line 66, in forward
    ctx.scale_factor
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /Users/qbx2/pytorch/torch/lib/THCUNN/generic/TemporalUpSamplingNearest.cu:97

Tested on Linux with pytorch 0.2.0+72f6b5a & GTX 750 and Mac OS X Sierra with pytorch 0.2.0+c74f7d8 & GT 750M
It seems to be work well with mode=‘linear’ and memory of gpu is larger than 1GB.
I think it’s something wrong.

1 Like
2

Same problem. When I try to forward a large image in CNN, same error occurs. I guess it’s because that a single variable takes too large GPU memory. Referring to https://discuss.pytorch.org/t/weird-cuda-illegal-memory-access-error/8848/2?u=liang, its solution might help.

3

Thanks for the report! I’ve submitted an issue for this at https://github.com/pytorch/pytorch/issues/3309.

4

@Liang @qbx2 The fix is now in master!

1 Like
5

@Liang Your problem is likely different from @qbx2 's. The issue behind this issue is a bug specific to the Upsample1d cuda function.

6

Too glad to hear that!!