# DistributedDataParallel training not efficient

**URL:** <https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624>\
**Category:** distributed\
**Created:** [May 18, 2021, 11:22am UTC](https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624 "2021-05-18T11:22:57Z")\
**Posts on this page:** 1\
**Showing post:** 11

<div class="post-metadata">

**Author:** ![Pavel](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pavel/32/61230_2.png) [@Pavel](https://discuss.pytorch.org/u/Pavel)\
**Post date:** [June 12, 2023, 10:37pm UTC](https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624/11 "2023-06-12T22:37:41Z")

</div>

Hi! I’ve faced the same issue. The solution was to increase learning rate proportional to the increase in the total batch size - as explained in details [in this great article](https://towardsdatascience.com/why-parallelized-training-might-not-be-working-for-you-4c01f606ef2c).  
In particular when I’ve used same learning rate for training on 1 GPU and 4 GPUs - there was no speed up at all. But when I’ve multiplied the learning rate by 4 for the 4 GPUs case - it converged much faster. The speed up was almost 4x - linear to the number of GPUs actually.

---

_[View the full topic](https://discuss.pytorch.org/t/distributeddataparallel-training-not-efficient/121624)._