I have the same error, but I have a different issue than everyone else. Here’s a simple code that reproduces my issue:
import torch.nn as nn
import numpy as np
self.base_n_filter = 8
self.conv1 = nn.Conv3d(1, self.base_n_filter, 3, stride=1, padding=1)
self.conv2 = nn.Conv3d(self.base_n_filter, self.base_n_filter*2, 3, stride=1, padding=1)
self.conv3 = nn.Conv3d(self.base_n_filter*2, 1, 3, stride=1, padding=1)
def forward(self, img):
output1 = self.conv1(img)
output2 = self.conv2(output1)
output3 = self.conv3(output2)
if __name__ == '__main__':
img = torch.zeros([1,1,248,248,140]).cuda()
label = torch.rand_like(img, device=img.device)
model = Net().cuda()
output = model(img)
loss_fn = nn.MSELoss()
loss = loss_fn(output, label)
The error only occurs when I have an input image of a specific size. For example, if I change the input to shape [1,1,250,250,150], the error no longer occurs. So it doesn’t seem like an overclocking issue. Additionally, this error also only occurs when self.base_n_filter is 8 or greater. I’m not using any lists to store my convolutions, and have set my model to cuda.
I am also using up-to-date GPU and cuda. The error occurs on both Tesla K80 and GTX1080Ti, with pytorch 1.2 cudatoolkit=10.0, CUDA/10.0.130, and cudnn/18.104.22.168-CUDA-10.0.130. Also, this error only occurs on Linux machines.
Any help would be truly appreciated!