Hi, I’m confused. I’ve got 96 CPUs with 2 threads each.
I chose num_workers=12 for my DataLoader but I’m getting a warning:
UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
Why is the suggested number only 2?
I can see from the
DataLoader source code that it sets
max_num_worker_suggest = len(os.sched_getaffinity(0))
and when I run
len(os.sched_getaffinity(0)) manually I get 96.