# DataParallel imbalanced memory usage

**URL:** <https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551>\
**Category:** Uncategorized\
**Created:** [August 6, 2018, 10:29pm UTC](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551 "2018-08-06T22:29:49Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![JuanFMontesinos](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/32/76115_2.png) [@JuanFMontesinos](https://discuss.pytorch.org/u/JuanFMontesinos)\
**Post date:** [August 8, 2018, 3:31pm UTC](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/3 "2018-08-08T15:31:32Z")

</div>

Well the op you cited is right, in the end pytorch collect the whole batch output in one gpu to calculate the loss. That’s what generates the imbalanced usage. At the same time optimizers’ parameters are stored in by-default gpu, what makes the problem worse. There is no apparent solution since dataparallel allows you to choose output gpu but the problem remains. There are several open threads asking about the same issue. I tried to play with dataparallel arguments: device\_ids and output\_device but no way.

---

_[View the full topic](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551)._