Error message Ignite Engine Thread Pickle

ignite
1

Hi I am using pytorch ignite wrapped in the MONAI package. My code worked until 3h ago when suddenly I started receiving this code. I reverted all my changes back and the error still persists. Does anyone know more?

ERROR:ignite.engine.engine.SupervisedTrainer:Exception: cannot pickle '_thread.lock' object
Traceback (most recent call last):
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/ignite/engine/engine.py", line 728, in _internal_run
    self._setup_engine()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/ignite/engine/engine.py", line 710, in _setup_engine
    self._dataloader_iter = iter(self.state.dataloader)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 801, in __init__
    w.start()
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/cornelius/opt/anaconda3/envs/LungSegChallenge/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
2

@quornmd could you please provide versions of MONAI and Ignite ?

3

thank you for the quick response.

I just solved the problem by deleting my Conda environment and reinstalling it.
In both, old and new, environments I was using: pytorch-ignite 0.4.2 and MONAI 0.3.0+95.g535561e.
I did install simpleITK and ITK packages about an hour before pytorch started breaking, but I was able to start my script in-between without it crashing.

4

Sounds good. Please feel free to ask here if you experience this issue again and if we could reproduce it :slight_smile:

5

Add a link for visibility from MONAI side : https://github.com/Project-MONAI/MONAI/pull/1286