Max_unpooling3d_forward_kernel failed with error code 0

After receiving a memory error I ran my training loop using CUDA_LAUNCH_BLOCKING=1 and now receive this rather esoteric error: max_unpooling3d_forward_kernel failed with error code 0. What does error code 0 stand for? Also, this happens after 3 successful forward and backward passes through the network.

As I changed a number of things about my training loop from the last working version, without touching the actual network, having an hint where to look for a bug would really help.

This is surprising indeed.
Are you running out of memory? Does reducing the batch size help?

When I had low memory issues in the past I would get a much clearer “out of memory” error.

@ptrblck any idea of what could be causing this?

What do you mean by “memory error”?
Was it an out of memory error or some memory access violation?
Could you post the complete stack trace and, if possible, a code snippet to reproduce this issue, so that we can debug it?