Should we perform optimizer.zero_grad() before we do forward pass or after that?

1

as titled. should I call zero grad before I do forward pass? Or I just need to make sure it is performed before I call loss.backward()?

1 Like
2

The forward pass doesn’t care about the grads, nor does it modify them.
You only need to zero the grads before calling .backward().

5 Likes
3

Thanks! It helps a lot

4

Does calling zero_grad before or after forward pass have any memory implications, however?