I have a very big file list, which is organized with:
And I create a dataset to read this file list to the memory.
Since my training code is run with DistributedDataParallel and I have 8 GPUs, the dataset will be created 8 times.
python -m torch.distributed.launch --nproc_per_node=8 --nnodes=1 --node_rank=0 --master_addr="127.0.0.1" train.py
And they will cost large memories, nearly 30*8=240G in total. Is there a way to let those processes share a single dataset?
Thanks for your help