Num_workers in DataLoader will increase memory usage?

I am using DataLoader to load my training data.
According to the document, we can set num_workers to set the number of subprocess to speed up the loading process.
However, when I use it like this:

dataloader = DataLoader(dataset, batch_size=args.batch_size,
shuffle=True, drop_last=False, num_workers=10,
collate_fn=dataset.collate_fn)

I found the memory usage keep growing, which is not happening when I set num_worker=0.
Is this supposed behavior ? I have limited memory resources, so I don’t want the memory usage keeps growing.

P.S. My code is running on the GPU, every time I move a batch of data from cpu to gpu.

1 Like

how does your dataset code look like?

I have the same problem as him. Here is my topic

Please help us

I think that when using multiple workers, each of them loads its data on the cpu before sending it to gpu. Which makes the amount of data in memory bigger. Also it creates a new process per worker.

2 Likes

I found other similar question like this. Here is one of example.


In this thread, they said it is fixed recent day at:

I am fixing my local torch’s code. Hopefully I help

I am facing this issue even with the updated PyTorch nightly version. The Dataloder memory usage continuously increases until it runs of memory. My Dataset size is 26GB when initialized, it contains an ndarray from which I return an element based on index value. After running on 10% of data it ends up using another 30+GB of ram and 40GB + swap space. I tried upgrading the PyTorch with the latest nightly version. Tried both on python3.6 and Python3.7, but the issue persists.

Thanks,
Vishnu

Now, for PyTorch 1.2.0, multiple workers don’t make multiple memory copy of dataloader object. But still such problems some times. I find that setting num_worker < physical cpu kernels works fine.

How do you implement your dataset?
I am using PyTorch 1.5.0, but it still make memory copy, based on my observation.

I just load the whole dataset in the init() of DataLoader, others are just as normal. Be sure to make num_worker < physical cpu . I don’t know why, but it works well when your memory is limited. E.g., my dataset is about 30 GB, I can run on a two-GPU machine with 80 GB memory. Num_worker and physical cpu are 10 and 12 respectively.