Backward calcalution fails with batch size >1 while using cudnn with error CUDNN_STATUS_INTERNAL_ERROR

3

Thanks for replying.

I cleaned up my environment, reinstalled and verified cudnn and reinstalled torch using pip. The code snippet above still fails for me, as is the following code:

import torch
from torch.backends import cudnn
torch.backends.cudnn.benchmark = True
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)
net = torch.nn.Conv3d(4, 1, kernel_size=[3, 3, 3], padding=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]).cuda()
out = net(out)
out.sum().backward()

Setting torch.backends.cudnn.benchmark = False succeeds. Additionally the following code succeeds:

import torch
from torch.backends import cudnn
torch.backends.cudnn.benchmark = True
out = torch.randn([2, 4, 3, 360, 640], dtype=torch.float, device=‘cuda’, requires_grad=True)
out.sum().backward()

I’ve seen similar issues reported in this forum and in github:

Frankly I’m not sure how to proceed with this and setting benchmark=False is not the solution I’d want.