About multiprocessing in dataloader when num_workers > 0

Hello, I use Pytorch for my research to train the machine-learning model.

As I know, “Dataloader” in PyTorch makes subprocesses and parallelizes the job that loads the data from drives to memory when the attribute “num_worker” is larger than zero.
However, I am a little bit confused that I understand exactly the process.
Are all subprocesses involved in loading data every time, or does each subprocess load and prepare data sequentially?


The main process generates the indices using Sampler and sends them to the workers, which then prepare the specific sample and send the result back to the main process.