CUDA illegal memory error when using nn.Conv1d and torch.backends.cudnn.benchmark = True

1

Hi,

I encountered a CUDA runtime error: illegal memory: an illegal memory access was encountered. This happens when calling model.to(‘cuda’), and the stack trace points to this line in the convert(t) method (line 903/905):

return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)

This happens when I build the model with nn.Conv1d, but is fine when I build the same model using nn.Conv2d. Does anybody have a clue where the problem might lie?

Thanks a lot and happy holidays!

Edit: So in my code I also set torch.backends.cudnn.benchmark to True. I found out that removing this resolves my illegal memory error (in the version that uses Conv2d, having the value as True causes no problem). I haven’t managed to understand yet why setting the benchmark to True causes a problem in the version that uses Conv1d

2

Could you post the output of python -m torch.utils.collect_env here, please?