# DataParallel not effective

**URL:** <https://discuss.pytorch.org/t/dataparallel-not-effective/18221>\
**Category:** Uncategorized\
**Created:** [May 16, 2018, 4:59pm UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221 "2018-05-16T16:59:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andy-jqa](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e274bd/32.png) [@Andy-jqa](https://discuss.pytorch.org/u/Andy-jqa)\
**Post date:** [May 16, 2018, 4:59pm UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/1 "2018-05-16T16:59:45Z")

</div>

Hi all,  
I have a quite large model and need to do data parallel among multiple GPUs.  
I used:

```python
model = nn.DataParallel(model)

```

And there are three visible GPUs. The GPU usage is:

```bash
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 12444 C python 10741MiB |
| 1 12444 C python 4663MiB |
| 2 12444 C python 4633MiB |
+-----------------------------------------------------------------------------+

```

GPU1 and GPU2 are not fully utilized, but I cannot increase the batch size because then there will be memory error on GPU0.  
Does anyone know how to solve this problem?

---

<div class="post-metadata">

**Author:** ![soulless](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulless/32/40008_2.png) [@soulless](https://discuss.pytorch.org/u/soulless)\
**Post date:** [August 8, 2018, 7:54am UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/2 "2018-08-08T07:54:27Z")

</div>

Hello, I am having the same issue. Have you solved it? Thanks

---

<div class="post-metadata">

**Author:** ![liujie\_zhang](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b19c9b/32.png) [@liujie\_zhang](https://discuss.pytorch.org/u/liujie_zhang)\
**Post date:** [August 8, 2018, 8:22am UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/3 "2018-08-08T08:22:45Z")

</div>

> [@Andy-jqa](#):
>
> nn.DataParallel

Hi, Andy-jpa  
If the the first gpu (id:0) is occupied all the time and no more space to use, you could consider to use only 1 and 2 gpu explictly assigned in you code by device\_ids.

refer : [Multi-GPU Examples — PyTorch Tutorials 2.1.1+cu121 documentation](https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html)

code like this:

```auto
def data_parallel(module, input, device_ids, output_device=None):
    if not device_ids:
        return module(input)

    if output_device is None:
        output_device = device_ids[0]

    replicas = nn.parallel.replicate(module, device_ids)
    inputs = nn.parallel.scatter(input, device_ids)
    replicas = replicas[:len(inputs)]
    outputs = nn.parallel.parallel_apply(replicas, inputs)
    return nn.parallel.gather(outputs, output_device)

```

---

<div class="post-metadata">

**Author:** ![hashio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hashio/32/22971_2.png) [@hashio](https://discuss.pytorch.org/u/hashio)\
**Post date:** [April 19, 2020, 12:52am UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/4 "2020-04-19T00:52:29Z")

</div>

Hi I have same issue.  
I tried 16GPU.

I think this problem is batch\_size need increase to use efficiently multiple GPUs but outputs increase either.  
replicas … batch\_size / GPUs \* input\_size  
output\_device … batch\_size \* GPUs \* output\_size  
but output\_device must have replica in the current DataParallel imprementation.  
Therefore I try rewrite DataParallel to output\_device avoid from replicas.  
In my case, I could increase batch size to output\_device GPU use 30GB and each replica GPUs uses 20GB.

But, I’m not sure it’s correct.  
I know, it need resolve hide latency to get more efficiency process.  
best solution is change to concurrent processing.

I have used PyTorch since last week replace from TensorFlow2.  
I want to know best practice of PyTorch.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [April 19, 2020, 1:29am UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/5 "2020-04-19T01:29:43Z")

</div>

The fastest and recommended approach is `DistributedDataparallel` using a single process per GPU as described in [these docs](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel).

---

<div class="post-metadata">

**Author:** ![hashio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hashio/32/22971_2.png) [@hashio](https://discuss.pytorch.org/u/hashio)\
**Post date:** [April 19, 2020, 1:45am UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/6 "2020-04-19T01:45:53Z")

</div>

thanks quick reply  
i’ll try it

---

<div class="post-metadata">

**Author:** ![hashio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hashio/32/22971_2.png) [@hashio](https://discuss.pytorch.org/u/hashio)\
**Post date:** [April 21, 2020, 10:05pm UTC](https://discuss.pytorch.org/t/dataparallel-not-effective/18221/7 "2020-04-21T22:05:47Z")

</div>

DDP is a very good performance and nothing worried for memory management.  
thanks for ptrblck and pytorch team