# DataParallel imbalanced memory usage

**URL:** <https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551>\
**Category:** Uncategorized\
**Created:** [August 6, 2018, 10:29pm UTC](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551 "2018-08-06T22:29:49Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [August 8, 2018, 4:30pm UTC](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/4 "2018-08-08T16:30:49Z")

</div>

If you want the loss to be splitted among gpus, just make your loss layer part of the DataParallel and add a `sum` or `mean` operation on what you get out of it. That way if you use DataParallel on 4 devices, only 4 extra numbers will be allocated on the output\_device. Is that a good solution for you?

---

_[View the full topic](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551)._