In the source code as well, if the model’s device_ids is None, then scattering is not performed in the forward() pass of the model.
Yes, this is correct.
Input data split only occurs in two situations:
- When using DataParallel (single-process multi-thread)
- Using DistributedDataParallel (DDP), and provide a
device_idslist of multiple CUDA devices. In this case, each DDP process will operate on multiple devices and multiple model replicas, and hence need to split the input data. (This is not recommended, as this could be slow)
For the recommended use case of DDP (one device/replica per DDP process), DDP will NOT split input or distributed them into multiple processes. Each DDP process needs to read its own input data independently. You could try manually splitting those data (say on rank0) and pass them across processes though, if they are on the same machine. Or, I also saw many people using the DistributedSampler to load input data