If you add del loss, output at the end of the loop the memory usage will likely remain the same after the first iteration (what you see it probably a side-effect of Python’s scoping rules). It’s possible that cuDNN uses much less memory than the default backend.
18 Likes