Can DDP divide dataset unevenly to workers?

distributed
1

Hi,

Is there any native method in DDP to divide a given dataset unevenly? For example, 60% of Cifar10 data are distributed to the first worker in each epoch while the other 40% are run through by another worker.

Thanks for any comments in advance.

2

Hi, there is no native support for this at the moment as DistributedSampler does not take in a custom sampler so it will always do the default which is divide evenly.

One option is to implement your own data sampling across workers and use DDP’s uneven inputs join context manager: Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 1.11.0+cu102 documentation to make sure DDP does not hang due to uneven dataset size.

Curious about your use case though, what is the aim of having 60/40 skew of data across workers?

3

Hi, thanks for your response.

My motivation is that, I want the faster GPU process more data and let the slower GPU process fewer data. The reason behind is the different capabilities across GPUs. For example, batch size = 60 costs GPU1 the same time as batch size = 40 for GPU2. For this reason, I want to divide the data 6:4 to alleviate the speed gap.

4

Design a custom batch_sampler could make that.

# this is a super simple example, you should impl your own one.
class MyDistributedSampler(DistributedSampler):
    # customize the __iter__ method
    def __iter__(self):
        if self.rank == 0: # data 0-59 on node 0
            return [list(range(60))]
        if self.rank == 1: # data 60-99 on node 1
            return [list(range(60, 100))]

# use batch_sampler to init the dataloader
train_sampler = MyDistributedSampler(...)
trainloader = DataLoader(tr_set, batch_sampler=train_sampler, ...)

And note that you should do addtional normalization to the loss since batch sizes on nodes are not the same. See “Alternatives” at Support different batch size across GPUs with DDP · Issue #67253 · pytorch/pytorch (github.com)

5

Thank you very much. This is exactly what I am finding.

6

Thanks you and I have a question. When we instantiate a DataLoader object. The batch size is fixed among workers when we use torch DDP. If we define a uneven sampler and 60 data is on node 0 while 40 data on node 1, the two nodes will end up having different iterations. For example, if batch size is 5, node 0 will have 12 iterations while node 1 has 8 iterations. There will be an error. How can we solve this issue?

7

To clarify we’re discussing the same thing, there are two levels of “uneven data”:

  1. uneven samples in mini-batch, e.g. at step=0, GPU0 is fed with a mini-batch of 10 samples and for GPU1 it is 5 samples.
  2. uneven mini-batches across devices, e.g. for GPU0, 1 epoch runs for 20 steps (mini-batches); and for GPU1 it runs for 10 steps (mini-batches)

For situation 1, DDP is still synchronous and wouldn’t be blocked.

For 2, which I believe is what you are interested, there are multiple ways to address that, for example, one could repeat samples for the GPU worker with less mini-batches; or drop the rest of mini-batches once a GPU worker runs more steps than others.
And pytorch has natively supported a api to address such issue, please check this guide Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 2.3.0+cu121 documentation