# Why torch.nn.DataParallel causes unexpected CUDA OOM?

**URL:** <https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277>\
**Category:** distributed\
**Created:** [October 27, 2021, 11:36pm UTC](https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277 "2021-10-27T23:36:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Olivier-CR](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/c5a1d2/32.png) [@Olivier-CR](https://discuss.pytorch.org/u/Olivier-CR)\
**Post date:** [October 27, 2021, 11:36pm UTC](https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277/1 "2021-10-27T23:36:32Z")

</div>

Hi,

I’m using `torch.nn.DataParallel` to do single-node data parallelism , and I’m wondering the following: how should the DataLoader batch be scaled?

I’m asking since I have a code running fine with batch 16 on a T4 GPU, but doing CUDA OOM with batch 4_16 = 64 (and even with 48!) with `torch.nn.DataParallel` over 4x T4 GPUs. Is torch.nn.DataParallel doing anything weird with the memory, so that it has less memory available than N_1 GPU memory? or is `torch.nn.DataParallel` already applying a scaling rule so that the dataloader batch is the per-GPU batch and not the SGD-level batch? (I don’t think that’s the case as the doc says it “splits the input across the specified devices by chunking in the batch dimension”)

_ **note** :I know [PyTorch recommends DDP even for single-node data parallel](https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html), but honestly I’m not smart enough to figure out how to use all those torchrun/torch.distributed/launch.py tool, MPI, local\_rank things and [couldn’t make DDP work after a week and 7 issues opened](https://discuss.pytorch.org/u/olivier-cr/activity/topics) 🙂_

---

<div class="post-metadata">

**Author:** ![techkang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/techkang/32/43474_2.png) [@techkang](https://discuss.pytorch.org/u/techkang)\
**Post date:** [October 31, 2021, 7:29am UTC](https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277/2 "2021-10-31T07:29:51Z")

</div>

`nn.DataParallel` will transfer the data to GPU0 and scatter latter. So it is easy to cause OOM problems. You can refer to [pytorch ddp](https://kangsheng.xyz/2020/08/13/pytorch%E4%B8%AD%E5%8D%95%E6%9C%BA%E5%A4%9A%E5%8D%A1%E8%AE%AD%E7%BB%83/) to understand how to leverage `DDP` if you can read Chinese.

---

<div class="post-metadata">

**Author:** ![Olivier-CR](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/c5a1d2/32.png) [@Olivier-CR](https://discuss.pytorch.org/u/Olivier-CR)\
**Post date:** [November 2, 2021, 10:19pm UTC](https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277/3 "2021-11-02T22:19:34Z")

</div>

ok so if I understand correctly, in nn.DataParallel, all the cluster data must be able to fit in one GPU? That does not make any sense right? Because if all the data fits in one GPU, people would not use data parallelism in the first place 🙃 🤨

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 3, 2021, 6:48am UTC](https://discuss.pytorch.org/t/why-torch-nn-dataparallel-causes-unexpected-cuda-oom/135277/4 "2021-11-03T06:48:42Z")

</div>

The data would still be split in its batch dimension and the forward activations would thus be shared between all devices, which are usually much larger than the input data.  
However, @techkang is right and the scatter/gather ops from the default device create a memory imbalance which is why we recommend the usage of DDP (besides DDP also being faster).