I’m trying to use PyTorch inside a Celery container and running into some issues related to threading. Initializing a DataLoader with num_workers greater than 0 successfully starts a new thread - but instead of acting as a dataloader, the new thread runs as a new celery worker and waits for a new task to be submitted! Is there a way to control how a new thread is spawned?
The Dataloader will spawn new processes if num_workers>0 is used and won’t use threads.
I’m not familiar with Celery and don’t know about its specific requirements etc.