Hello, i was wondering how the dataloder with num_workers > 0 queu works. I imagine N wokers are created.
I see 2 options:
the program goes through all workers in sequence? This would mean that if one worker is delayed for some reason, the other workers have to wait until this specific worker can deliver the goods. This would also mean that if a worker gets stuck into an infinite loop while fetching its data, the training process is going to be waiting forever.
The program takes a batch from whichever worker is done first. This means that if a worker gets into an infinite loop, another worker will always be able to get the training process new data. In this case i am wondering what would happen to the worker in the infinite loop. Does it get killed, does it get sidelined in some way ?
To give some context to my reason for asking this question:
I am doing object detection. And in my data loader i have a transform that does a random crop. However, i don’t want to do random crops with no bounding boxes contained within it. So i wrote a loop inside the transform that does random crops on the same image until it finds a candidate with a bounding box contained in it.
My fear is that there might be cases in which no such crop is ever found ( maybe there is no bounding box in the image). In this case i want to understand how the dataloader/training process would deal with this.
Thanks a lot.