Is the dataloader parallelizing __getitem__ or batch sampling?

#1

Hi,

I tried to google the answer, but couldn’t find it.
i) Is each worker in the the dataloader calling __ getitem __ within the same batch, and then collating it,
ii) or is each worker loading and collating each batch independently?

Imagine this scenario:
16 batches per epoch, each with 100 examples.
24 workers available.
Will 8 workers be iddle if I set num_workers = 24?

Thanks!

#2

The second case is currently used, i.e. each worker creates a full batch and pushes it to the queue.
Once it’s ready it’ll start creating the new batch until prefetch_factor*num_workers batches are available.

1 Like