# Training on Multi-GPUs out of Memory

**URL:** <https://discuss.pytorch.org/t/training-on-multi-gpus-out-of-memory/55592>\
**Category:** Uncategorized\
**Created:** [September 10, 2019, 6:28am UTC](https://discuss.pytorch.org/t/training-on-multi-gpus-out-of-memory/55592 "2019-09-10T06:28:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![freelv](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/ebca7d/32.png) [@freelv](https://discuss.pytorch.org/u/freelv)\
**Post date:** [September 10, 2019, 6:28am UTC](https://discuss.pytorch.org/t/training-on-multi-gpus-out-of-memory/55592/1 "2019-09-10T06:28:06Z")

</div>

When I was training my model using multi-gpus, I set the batch\_size=8, and used two gpus. This setting cannot make the best of my machine(4 GPUs and each with memory of 12G), however, if I set the batch\_size bigger or use more GPUs, it would raise exception(GPU out of memory). It seems like that, the first GPU takes more assignment, so when I set the batch\_size bigger or used more GPUs, the first GPU was out of memory. How can I fix it?

 ![%E5%9B%BE%E5%83%8F%201](https://discuss.pytorch.org/uploads/default/original/2X/c/cfecd5d6e4edf6189c7fd10ae17b8dd6aa578647.png)  
The image show my GPU’s state. I used GPU 2 and 3.

---

<div class="post-metadata">

**Author:** ![alex.veuthey](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex.veuthey/32/12176_2.png) [@alex.veuthey](https://discuss.pytorch.org/u/alex.veuthey)\
**Post date:** [September 10, 2019, 6:45am UTC](https://discuss.pytorch.org/t/training-on-multi-gpus-out-of-memory/55592/2 "2019-09-10T06:45:31Z")

</div>

This is commonly observed in multi-gpu setups, because some kind of aggregation has to be performed on one selected GPU (in the default case, `cuda:0`). See [this answer](https://discuss.pytorch.org/t/dataparallel-imbalanced-memory-usage/22551/3), which explains the problem in a bit more details, and check the answer just after it for a possible solution (i.e. split the loss on multiple GPUs as well, not just the network).