Hi,
I’m working with a system (Amazon SageMaker Training) capable of spreading remote files homogenously across machines. Meaning that at the beginning of a distributed DDP training, I have the files of my dataset (eg images, text files) homogeneously spread on the disks of each DDP nodes, as represented in the diagram below
What Dataset
& DataLoader
settings shall I use to train DDP over this distributed dataset?
Most DDP example I saw work do a virtual sharding with the DistributedSampler
in the DataLoader
, but here my data is already phyiscally distributed…