# Batch chunking in DataParallel and DistributedDataParallel

**URL:** <https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193>\
**Category:** Uncategorized\
**Created:** [June 4, 2018, 11:58pm UTC](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193 "2018-06-04T23:58:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dashesy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dashesy/32/4869_2.png) [@dashesy](https://discuss.pytorch.org/u/dashesy)\
**Post date:** [June 4, 2018, 11:58pm UTC](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193/1 "2018-06-04T23:58:57Z")

</div>

I have seen [this](https://discuss.pytorch.org/t/what-is-the-difference-between-dataparallel-and-distributeddataparallel/6108) and tried to understand the documentation:

```nohighlight
    This container parallelizes the application of the given module by
    splitting the input across the specified devices by chunking in the batch
    dimension. The module is replicated on each machine and each device, and
    each such replica handles a portion of the input. During the backwards
    pass, gradients from each node are averaged.
    The batch size should be larger than the number of GPUs used locally. It
    should also be an integer multiple of the number of GPUs so that each chunk
    is the same size (so that each GPU processes the same number of samples).

```

Does that mean the entire batch of data is first loaded in rank 0, then chunks are sent to other ranks? or each rank will read only the part of the batch that is in its chunk portion?  
I do not know how that would be possible (for each worker to read only part of the batch) unless there is a requirement for data loader to subclass `torch.utils.data.Dataset` that has ` __getitem__ `. If for example data comes from a custom class, how would `DataParallel` know how to read the proper chunk? and if the entire batch is read each time, it could be inefficient to send huge amount of say images among nodes, where each node could just read the images from a shared directory.

---

<div class="post-metadata">

**Author:** ![dashesy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dashesy/32/4869_2.png) [@dashesy](https://discuss.pytorch.org/u/dashesy)\
**Post date:** [June 5, 2018, 6:24pm UTC](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193/2 "2018-06-05T18:24:44Z")

</div>

Looking at the [code](https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L215) it does look like `distributed` scatters the input! this is very inefficient when inputs are images for example. It would be much more efficient for each process to read its portion from a shared directory/file.  
I think I would subclass `DistributedDataParallel` and do just that.

How is the performance of distributed pytorch with large inputs on multi-gpu (say 32+ nodes)?

---

<div class="post-metadata">

**Author:** ![andfoy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andfoy/32/4435_2.png) [@andfoy](https://discuss.pytorch.org/u/andfoy)\
**Post date:** [June 5, 2018, 11:30pm UTC](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193/3 "2018-06-05T23:30:00Z")

</div>

@dashesy You can use `torch.utils.data.distributed.DistributedSampler` ([docs](https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler)) to prevent input data broadcasting among nodes. You can see it in action in the [distributed ImageNet example](https://github.com/pytorch/examples/blob/master/imagenet/main.py)

---

<div class="post-metadata">

**Author:** ![dashesy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dashesy/32/4869_2.png) [@dashesy](https://discuss.pytorch.org/u/dashesy)\
**Post date:** [June 7, 2018, 10:01pm UTC](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193/4 "2018-06-07T22:01:54Z")

</div>

A few more observations:

I am trying to understand how `DistributedSampler` prevents scattering inputs among nodes when using `DistributedDataParallel`. According to [this](https://discuss.pytorch.org/t/batch-chunking-in-dataparallel-and-distributeddataparallel/19193) answer, imagenet example uses `torch.utils.data.distributed.DistributedSampler` to do that.

[This](https://github.com/pytorch/examples/blob/master/imagenet/main.py#L192) line:

```python
       # compute output
        output = model(input)

```

however, calls forward on `DistributedDataParallel` which does [call scatter](https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L213):

```python
    def forward(self, *inputs, **kwargs):
        self.need_reduction = True
        inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

```

So even though `DistributedSampler` has the data already chunked up for each replica, the batches will be still scattered, but they will be scattered among local devices, not among nodes! It took me some time to realize that `device_ids` are all local devices for each node and `DistributedDataParallel` does not scatter among nodes.