Random CUDA error: device-side assert triggered (once every week)

[ptrblck]

@Emil_Wallner

/opt/pytorch/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:144: operator(): block: [28,0,0], thread: [53,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.

You are running into an indexing error and should isolate which line of code calls into this scatter/gather kernel.

@Emil_Wallner

I only made a few updates a few months back, and all of a sudden it started crashing frequently

Try to correlate these changes to any indexing, scatter, gather operation and verify these ops are using valid indices.

@Emil_Wallner

I suspected a memory leak, shape mismatches, or NaN values, but nothing stood out.

I don’t think the OOM is real and just reported after running into the sticky assert, so I would ignore it for now and focus in the indexing error, which is real.