Understanding multiple forward passes and their effect on back propagation

If I perform multiple forward passes through a module (e.g. with the discriminator of a GAN, once for real images, and once for generated images), what happens to the intermediate values in PyTorch? Are the intermediate values immediately overwritten on the second pass, meaning that if back propagation for the first pass is no longer available? Or is the intermediate values stored in some form of list? Or combined with the second pass somehow?

If the values are no longer available, is there a good way to save them so you can perform both passes and wait to do the back propagation?

Thank you for your time!

1 Like

Those are stored separately in different contexts. So you can safely fwd through multiple times. :slight_smile: