In two projects I have the same problem.
Evnironemt: Linux, Ubunitu 16.04
Python: 3.9.1
Torch: 1.7.1
The problem lies in the case of using DataLoaders instantiated in the thread (main) for completely different datasets. After that data-loaders are used in different threads.
DataLoaderes are configured with default “num_workers = 4”
An attempt to have the ability from different threads to use different data loaders is crashing.
One workaround is to use “num_workers = 0” and it solves the problem…
It very likely has a connection with bug in Python: Issue 41567: multiprocessing.Pool from concurrent threads failure on 3.9.0rc1 - Python tracker
Stack traces:
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/threading.py”, line 954, in _bootstrap_inner
self.run()
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/workers_threads.py”, line 50, in run
retValues = f(self, *args)
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/model_funcs.py”, line 145, in local_training
metrics_meter = train_model(
File “/home/burlack/pl_new/pl_pytorch/pl_pytorch/utils/model_funcs.py”, line 223, in train_model
for i, (data, label) in enumerate(train_loader):
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 352, in iter
return self._get_iterator()
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 294, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 776, in init
self._worker_result_queue = multiprocessing_context.Queue() # type: ignore
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/context.py”, line 103, in Queue
return Queue(maxsize, ctx=self.get_context())
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/queues.py”, line 40, in init
from .synchronize import SEM_VALUE_MAX as maxsize
ImportError: cannot import name ‘SEM_VALUE_MAX’ from partially initialized module ‘multiprocessing.synchronize’ (most likely due to a circular import) (/home/burlack/miniconda3/envs/pl/lib/python3.9/multiprocessing/synchronize.py)
Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x14c8389d61f0>
Traceback (most recent call last):
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1203, in del
self._shutdown_workers()
File “/home/burlack/miniconda3/envs/pl/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1150, in _shutdown_workers
if not self._shutdown: