# How to implement accumulated gradient？

**URL:** <https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822>\
**Category:** vision\
**Created:** [June 7, 2017, 6:33pm UTC](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822 "2017-06-07T18:33:24Z")\
**Posts on this page:** 1\
**Showing post:** 19

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 26, 2018, 12:15pm UTC](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822/19 "2018-11-26T12:15:47Z")

</div>

Your general approach is right and I also assumed than BatchNorm layers might be a problem in this case.  
If you just have very few samples in each forward pass, you could use `InstanceNorm` or `GroupNorm` instead, which should work better for small batch sizes.  
Alternatively, you could also try to change the `momentum` of `BatchNorm`, but I’m not sure, if that will really help a lot.

---

_[View the full topic](https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822)._