Multiprocessing dataloader with num_workers = 1

data
1

Hello, I have custom map-style dataset created and I iterate with the native PyTorch Dataloader. I use num_workers = 1 for debugging purposes and understanding the bottlenecks with pytorch profiler. Here is what I see in PyTorch Profiler’s trace tab:

image
image897×199 36.8 KB

I’m confused why there are 7 different threads still created that wait for other thread to finish in each iteration of dataloader. Can someone help me understand and guide me how to definitely get a single process dataloader?

2

To get a single-process dataloader, you should specify num_wokers=0.

3

I wonder why. It doesn’t sound intuitive!

4

num_worker represents the number of subprocesses used by DataLoader. Using num_worker=0 means make everything running in the main process.

1 Like