What is the relationship between num_workers and the distributed process in DistributedDataParallel?

1

What is the relationship between num_workers of the data loader in DistributedDataParallel mode? For example, if the num_workers=8 and the number of GPUs is 4, then whether each distributed process in DistributedDataParallel mode will get num_workers 2 or 8?

2

cc @vincentqb for data loader question :slight_smile:

3

Hi,

They are completely independent.
DataLoader processes will only be used to load data from your Dataset.

The processes for DistributedDataParallel have nothing to do with it.

2 Likes