I am exploring the use of DistributedDataParallel to train on two GPUs. In all the examples I have found the DataLoader and Model are instanciated separately at each rank. Can I create the model and dataloader outside of the multiprocessing.spaw function and pass them as input arguments to multiprocessing.spawn? I mean something like this:
import torch.multiprocessing as mp loader = DataLoader(dataset, batch_size=128, shuffle=True) model = MyNet() if __name__ == '__main__': mp.spawn(fit, args=(model, devices, loader), nprocs=len(devices), join=True)
In this case, will be a new model and dataloader created at each rank without shared memory?
I would like to use the same iterable for the dataloader at each rank so each GPU works on the same epoch. Is this possible?
I know that it would be better to use a DistributtedSampler, but I am working with graphs and I cannot do so due to the irregular structure of my data. If you know other similar option I would be nice to hear.