Dataloader is more than 4x slower

[mkserge (Sergey Mkrtchyan)]

The main advantage to the DataLoader is loading files from a hard drive. It does so iteratively because each process can be assigned to cpu cores with the num_workers argument.

I see, thanks! That explains why num_workers does not help in my scenario since all the data is already in memory.

Regarding DDP, in the example given here, there is no need to split data for each GPU. You just pass in the train data into the model and Pytorch will handle distribution behind the scenes.

From this post it looks like it will not shard the data and iterate over the entire dataset if you don’t pass a DistributedSampler