MultiProcessing Batch by Batch

I want to ask a pure Python question here, :sweat_smile:

Suppose I want to run 40 processes, however I only have 20 cpu cores. I want to run the processes batch by batch. For example, the first batch run the 0-19 processes, after the first batch is done the second batch run the 20-39 processes. How to do this by MultiProcessing?

The logic is:

first_batch=[process_0 – process_19]
second_batch=[process_20 – process_39]
for p in first_batch:
wait for the first_batch finish
for p in second_batch:
wait for the second_batch finish

how to implemement wait for batch finish?
I have tried p.join() after the first batch, however join() puts all the processes to idle and the second batch never starts. Can anyone help me to debug?

In pytorch, you do not need to write all dataloader by yourself. You can write a class inherited from the dataset, then the dataloader in pytorch will help you prefetch by multiprocess. You can check this

for p in first_batch: