DDP on single machine with 4 GPU hangs at model = DDP(model, device_id=[rank]) step

[summer5e (Summer)]

I figure out the reason why my training will hang. It is due to early stop.
Sometimes two out of four GPUs will early stop and exit while the other two GPUs are still training. At the graident synch step, the training will hang since the gradient from the two GPUs who left the party are not there.

Also found a solution for adding early stop in DDP training via the post below.

torch.distributed.reduce(op=torch.distributed.ReduceOp.AVG) behavior when few gpus have input batches while others do not

I am new to DDP. I am collecting the training losses from all the GPUs and reducing them at gpu_id=0 with op=torch.distributed.ReduceOp.AVG. What happens in the last iteration where only a few GPUs (not all of them) have batches. How will the torch.distributed.ReduceOp.AVG behave? Ex: Assume in the last iteration of the epoch, only 2 (out of 8 GPUs) receive data from the DataLoader. Will torch.distributed.ReduceOp.AVG average the losses tensors in the 2 GPUs only as it should? Or will it assume…