Does slicing/trimming during training cause memory leak?

autograd
1

During training, at some point I need my tensors to be padded to the same size.
Later on (still during training), I want to remove the padding by slicing and keeping only a certain part of the tensors.

I appears to me that this causes a cuda memory leak, since the subgraph of the removed part is not involved in the loss computation/backprop then and thus it is never cleaned up. Is my understanding correct about this?

And if so, how can I “trim” a tensor during training without running into this memory leak?

2

No, PyTorch will not leak memory, which would mean you cannot deallocate the data and have lost it. The intermediate tensors would of course still use the additional memory which will be removed later but all intermediates will be freed after the backward pass.