Should we split batch_size according to ngpu_per_node when DistributedDataparallel

[mrshenli (Shen Li)]

@hhxx

(2) If dividing the batchsize by ngpu_per_node is a correct way, I wonder what will happen if we do not that.

• Does it means in each node, 4*batch_size images are processed per forward-process?
• Will 4*len(dataset) images are processed in one epoch, or the forward frequency are four times less than usual(i.e. the total number images proceeded per epoch keep same)?

You are correct. Each DataLoader instance pairs with a DDP instances. If you do not divide the batch-size=256 by 4, then each DDP instance will process 256 images. As your env has 8-GPUs in total, there will be 8 DDP instances. So one iteration will process 256 * 8 images in total.

However, DDP does divide the gradients by the world_size by default code. So, when configuring learning rate, you only need to consider the batch_size for a single DDP instance.