Memory leak when using RPC for pipeline parallelism

I only provided one example, and it’s expected that my suggestions is not exhaustive.

Memory leak is a really tricky issue. You can try reverting some code and checking when the memory leak appears step by step.

On the other hand, check out this question: How to debug causes of GPU memory leaks? - #3 by smth