I find a parameter in DDP:
model = DistributedDataParallel( model, device_ids=[train_device], output_device=train_device)
What is the meaning of output_device? I read the source codes and doc but cannot figure it out. Does the different device has the same output_device?
Output_device is usually the rank, which tells DDP where to output.
For details, please check out the following documentations.
https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html