Why do we need to set the gradients manually to zero in pytorch?

[tom (Thomas V)]

A more explicit example in a similar direction as @ruotianluo is the ability to add gradients from several forward passes, for example in GANs:

How to use the backward functions for multiple losses?

Hello @Nabarun_Goswami, to try to clear this up, in the DCGAN example, you have (think about mathematical functions here, I left out everything not relevant). loss = criterion(netD(real, params))+criterion(netD(fake, params)) Spelling out the chain rule for the gradient of the loss w.r.t. the params: ∇params loss = ∇params netD(real, params)* ∇netD loss(netD (real,params)) + ∇params netD(fake, params)* ∇netD loss(netD (fake,params)), note how ∇params netD is evaluated at two different poin…

If you wanted, you could also achieve minibatches that are larger than fit in your memory by combining several sub-minibatches into one gradient step, but I have not really seen that done.

Best regards

Thomas