Training Job Stalls with no Logs & GPU Usage Spike

[ultramarine]

@ptrblck

for every increase in the reserved memory allocation.
Did you find any suspicious calls, which might append non-detached tensors t

Yes. Found a non-detached tensor getting accumulated.