MultiProcessing Batch by Batch

1

I want to ask a pure Python question here, :sweat_smile:

Suppose I want to run 40 processes, however I only have 20 cpu cores. I want to run the processes batch by batch. For example, the first batch run the 0-19 processes, after the first batch is done the second batch run the 20-39 processes. How to do this by MultiProcessing?

The logic is:

first_batch=[process_0 – process_19]
second_batch=[process_20 – process_39]
for p in first_batch:
p.start()
wait for the first_batch finish
for p in second_batch:
p.start()
wait for the second_batch finish

how to implemement wait for batch finish?
I have tried p.join() after the first batch, however join() puts all the processes to idle and the second batch never starts. Can anyone help me to debug?

2

In pytorch, you do not need to write all dataloader by yourself. You can write a class inherited from the dataset, then the dataloader in pytorch will help you prefetch by multiprocess. You can check this

3 
for p in first_batch:
  p.join()