# Is average the correct way for the gradient in DistributedDataParallel with multi nodes?

**URL:** <https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260>\
**Category:** distributed\
**Created:** [January 9, 2019, 1:37pm UTC](https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260 "2019-01-09T13:37:58Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![Lausanne](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9a140/32.png) [@Lausanne](https://discuss.pytorch.org/u/Lausanne)\
**Post date:** [February 10, 2019, 2:56am UTC](https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260/16 "2019-02-10T02:56:22Z")

</div>

@coincheung Your lr in torch.distributed mode should be 0.005

---

_[View the full topic](https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260)._