I think, the gradient calculations of next iterations won’t be affected because every forward call would be working on their own set of data. But if the computational graphs are not cleaned up, you would eventually face the Out of memory error such as: