The acc in multi gpus is lower than one gpus

[mrshenli (Shen Li)]

Hey @sangyx, when using DDP, you might need to tune the batch size and learning rate a bit. See the discussion below:

Should we split batch_size according to ngpu_per_node when DistributedDataparallel

Assume we have two nodes: node-A and node-B, each has 4gpus(i.e. ngpu_per_node=4). We set args.batch_size = 256 on each node, means that we want each node process 256 images in each forward. (1) If we use DistributedDataparallel with 1gpu-per-process mode, shall we manually divide the batchsize by ngpu_per_node in torch.utils.data.DataLoader : torch.utils.data.DataLoader(batch_size = args.batch_size / 4)(the way used in pytorch-imagenet-official-example). In my original opinion, I think Distrib…