How to implement accumulated gradient？

[MORI]

Hi smth,

So, eventually, there is no necessity to divide loss with iter_size?
I’m still bit confusing since apaszke mentioned about dividing here.

PyTorch Gradients

Normally when we’re doing backprop we would do the following: loss.backward() # This calculates the gradients optimizer.step() # This updates the net However, what if I wish to accumulate the gradients? Meaning I want to run various loss.backward() multiple times first and accumulate the gradients before applying them in optimizer.step(). What do you recommend for storing the gradients? grads = loss.backward()? How do I feed the accumulated gradients to optimizer.step?

Thank you for your help.