# What is the difference between DataParallel and DistributedDataParallel?

**URL:** <https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108>\
**Category:** Uncategorized\
**Created:** [August 11, 2017, 12:40pm UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108 "2017-08-11T12:40:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dawny33](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dawny33/32/1249_2.png) [@Dawny33](https://discuss.pytorch.org/u/Dawny33)\
**Post date:** [August 11, 2017, 12:40pm UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/1 "2017-08-11T12:40:54Z")

</div>

I am going through this imagenet example: [https://github.com/pytorch/examples/blob/master/imagenet/main.py](https://github.com/pytorch/examples/blob/master/imagenet/main.py)

And, in line 88, the module `DistributedDataParallel` is used. When I searched for the same in the docs, I haven’t found anything. Possible to redirect me to it if any such doc exist for the module.

Else, would like to know what is the difference between the DataParallel and DistributedDataParallel modules.

---

<div class="post-metadata">

**Author:** ![fmassa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fmassa/32/58_2.png) [@fmassa](https://discuss.pytorch.org/u/fmassa)\
**Post date:** [August 11, 2017, 7:57pm UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/2 "2017-08-11T19:57:11Z")

</div>

`DataParallel` is for performing [training on multiple GPUs, single machine](http://pytorch.org/docs/master/nn.html?highlight=dataparallel#torch.nn.DataParallel).  
`DistributedDataParallel` is [useful when you want to use multiple machines](https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L24-L78).

---

<div class="post-metadata">

**Author:** ![csarofeen](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/csarofeen/32/4613_2.png) [@csarofeen](https://discuss.pytorch.org/u/csarofeen)\
**Post date:** [May 4, 2018, 4:54pm UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/3 "2018-05-04T16:54:10Z")

</div>

Sorry for resurrecting this old thread. The answer above made some confusion with some folks I’ve talked to.

Distributed Data Parallel can very much be advantageous perf wise for single node multi-gpu runs. When run in a 1 gpu / process configuration Distributed Data Parallel can be beneficial as CPU based overheads are now spread across multiple processes.

Perf gains will especially be prominent in networks that have many small layers/operations. I primarily recommend to folks that they use single gpu / process Distributed Data Parallel over Data Parallel even for single node cases if they want to scale past 2 GPUs.

---

<div class="post-metadata">

**Author:** ![Toru](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toru/32/3435_2.png) [@Toru](https://discuss.pytorch.org/u/Toru)\
**Post date:** [May 18, 2018, 2:37pm UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/4 "2018-05-18T14:37:55Z")

</div>

Can you please elaborate on “When run in a 1 gpu / process configuration Distributed Data Parallel can be beneficial as CPU based overheads are now spread across multiple processes”? Thanks!

---

<div class="post-metadata">

**Author:** ![Hjjiang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hjjiang/32/14712_2.png) [@Hjjiang](https://discuss.pytorch.org/u/Hjjiang)\
**Post date:** [April 1, 2021, 6:39am UTC](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108/5 "2021-04-01T06:39:00Z")

</div>

Totally agree with you！  
“I primarily recommend to folks that they use single gpu / process Distributed Data Parallel over Data Parallel even for single node cases if they want to scale past 2 GPUs.”