# How to implement accumulated gradient？

**URL:** <https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822>\
**Category:** vision\
**Created:** [June 7, 2017, 6:33pm UTC](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822 "2017-06-07T18:33:24Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![Gopal\_Sharma](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gopal_sharma/32/512_2.png) [@Gopal\_Sharma](https://discuss.pytorch.org/u/Gopal_Sharma)\
**Post date:** [September 7, 2017, 4:55pm UTC](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822/8 "2017-09-07T16:55:51Z")

</div>

Note that, keeping the learning rate constant, it is important to feed the optimizer same gradients before and after using this trick. If we don’t use the trick of accumulating, we would be computing the gradient like this:

```
"blah-blah"
optimizer.zero_grad()
loss = 0
minibatch_size = old_batch_size / iter_size
for i in range(iter_size):
    # output here as the size of minibatch
    loss += criterion(output, target_var)
loss = loss / iter_size
loss.backward()
optimizer.step()

```

But when we are using this trick, we need to make sure that the accumulated gradient’s mean should be same as before.  
So, we divide the loss everytime with the iter\_size such that after summing up, gradients come out to be the same.

```
optimizer.zero_grad()
loss_sum = 0
for i in range(iter_size):
    loss = criterion(output, target_var) / iter_size
    loss.backward()
    loss_sum += loss
optimizer.step()

```

If you divide by the iter\_size, you don’t need to change the learning rate. If you don’t, then you should divide the learning rate by iter\_size to have the same performance. I am considering that you are using SGD as the optimizer.

---

_[View the full topic](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822)._