Does the generated grad-graph RETAIN inside a for-loop?

autograd
1

Does the generated graph retain inside a for-loop, even after the model output is overwritten?

Example:

for epoch in range(100):
    for img in dataloader:
         optimizer.zero_grad(); loss=0
         for i in range(5):   
             out_img = M[i](img)   
             loss += criterion_L1(out_img, target[i])
         loss.backward()
         optimizer.step()

Above, I am using 5 models, each applied to the input image img. So, my simple understanding is although out_img is overwritten each time, backpropagation will apply to all the M models; that is the graph is retained.

Please correct me if I am wrong.

2

There is not a single grad graph in pytorch. Whatever you do as computation will be the graph.
So here gradients will flow back to each model in M indeed.

1 Like