Hello,
I have a data.Dataset and a corresponding DataLoader with multiple workers. I would like the workers __getitem__ to have a specific behavior depending on some variable in my main training loop (e.g. training phases). How can I send a message from the main process to the workers ?
More details :
I work with video data, and train with short video-clips extracted from long sequences of frames. The dataloader only returns the small clip. This sampling strategy needs to evolve throughout training (e.g. retrieve the clip at a specific position in the video) so I need the workers to receive a message from the main process (a general message for all workers).
Currently I am using files written in the main loop and read by the workers to send the message, but I dont think this is a good practice. In the opposite way, I send information from workers to main process within the __getitem__ return data.
Thanks for your help