RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED when calling backward()

The input data changed from

Input image size: (304,304,96)
Input label (target): (304,304,96) binary image
pixel spacing = 0.25

to

Input size: (256,256,128)
Input (target): (256,256,128) distance map 
pixel spacing = 0.25
pytorch version: 1.0.1.post2
CUDA version: 10.0.130
cuDNN version: 7.4.2
GPUs: 2x NVIDIA Titan RTX 

The original workshop is from NVIDIA and its name is 3-D Segmentation for Medical Imaging with V-Net

You can find it here: https://courses.nvidia.com/courses

I only used it as a template though. I basically only used their VNET model and incorporated it into https://github.com/victoresque/pytorch-template

I hope that helps.

Let me know if you want me to try something. I am basically stuck right now because I can’t train the model without using cuDNN since it doesn’t fit into memory.

Thanks a lot,
Christian