PyTorch’s data loader uses multiprocessing in Python and each process gets a replica of the dataset. When the dataset is huge, this data replication leads to memory issues. Normally, multiple processes should use shared memory to share data (unlike threads). I wonder if there is an easy way to shar…

If you are lazily loading the data (which is the common use case, if you are dealing with large datasets), the memory overhead from the copies might be small in comparison to the overall memory usage in the script. That being said, you could try to use shared arrays as described here instead.

How to share data among DataLoader processes to save memory

Memory Format

Pietro_Cicalese (Pietro Cicalese) July 5, 2022, 11:02pm 3

See here, @ptrblck

How to make one GPU work more than the other in DDP if the GPUs performance not the same? (Please answer this its really a unique question)