# Issue with multiprocessing semaphore tracking **URL:** \ **Category:** Uncategorized\ **Created:** [August 12, 2018, 2:10am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943 "2018-08-12T02:10:23Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![gamma](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/ce7236/32.png) [@gamma](https://discuss.pytorch.org/u/gamma)\ **Post date:** [August 12, 2018, 2:10am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/1 "2018-08-12T02:10:23Z")
Hi I am facing an issue with semaphore tracking while using pytorch multiprocessing over multiple GPUs. I get the following warning message multiple number of times every time and it is slowing down the code execution substantially (8 times slower). ```auto /anaconda3/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) ``` I didn’t face this issue a couple of days back and started facing it when I updated my PyTorch to 0.4.1 hence I downgraded it to 0.3.1 (hoping that it was my previous Pytorch version). But, I am still facing the warning and slowdown. I really need to understand the cause of this leakage and how can I fix it; my code is wrapped inside if **name** ==‘ **main** ’ which is one of the solution commonly suggested for this warning. I would really appreciate any suggestions on this. Please let me know what details should I provide. Thanks ---
**Author:** ![Michael\_Petrochuk](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_petrochuk/32/1117_2.png) [@Michael\_Petrochuk](https://discuss.pytorch.org/u/Michael_Petrochuk)\ **Post date:** [September 14, 2018, 2:43pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/2 "2018-09-14T14:43:15Z")
Similarly, I have the same issue and i do not know where it is originating from! I’m using DataLoader with DistributedDataParallel. Ditto with my name wrapped inside `name==main` ---
**Author:** ![kittoku](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kittoku/32/7985_2.png) [@kittoku](https://discuss.pytorch.org/u/kittoku)\ **Post date:** [September 26, 2018, 4:15am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/3 "2018-09-26T04:15:24Z")
I encountered a similar problem with `multiprocessing` and `for` loops in CPU. It’s not a fundamental solution, but `-W` option works for me, e.g. `python -W ignore main.py`. This makes training stable in my environment. I tried using `warnings` module to ignore the warning, but it’s not worked. I think this is because semaphore trackers’ processes are generated internally. my environment.: linux, python=3.7, pytorch-cpu=0.4.1, start\_method=spawn ---
**Author:** ![sbelharbi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/32.png) [@sbelharbi](https://discuss.pytorch.org/u/sbelharbi)\ **Post date:** [January 9, 2019, 4:57pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/4 "2019-01-09T16:57:26Z")
I run to the same issue with the warning (when running some preprocessing on GPU), except for now, I didn’t see a slow down in the running time of the code, yet. ```bash python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown len(cache)) ``` The message seems harmful, but pretty annoying since it is displayed whenever the multiprocessing takes place. One way to hide this warning is to ignore it using the Python module `warnings`. Also, redirecting the `stderr` may be temporary solution. However, [`warnings.filterwarnings()`](https://docs.python.org/3/library/warnings.html#warnings.filterwarnings) DOES not work for this particular warning (and probably for all the warnings of the Python module `multiprocessing.semaphor_tracker()`. I don’t really understand why. I tried different ways using regular expressions, but nothing seem to work. **Things that work** : 1. `python -W ignore your_script.py` --\> ignore ALL the warnings within your script. (not recommended). 2. `python -W ignore:semaphore_tracker:UserWarning your_script.py` or `python -W 'ignore:semaphore_tracker:UserWarning' your_script.py` seem to ignore exactly the above warning. It can also done by using the environment variable [PYTHONWARNINGS](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONWARNINGS): ```bash export PYTHONWARNINGS='ignore:semaphore_tracker:UserWarning' python your_script.py ``` Python 3.7.1, Pytorch 1.0.0. ---
**Author:** ![aliutkus](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aliutkus/32/5600_2.png) [@aliutkus](https://discuss.pytorch.org/u/aliutkus)\ **Post date:** [March 21, 2019, 12:17pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/5 "2019-03-21T12:17:03Z")
Is there some understanding concerning why this warning happens ? > UserWarning: semaphore\_tracker: There appear to be 1 leaked semaphores to clean up at shutdown It seems pretty difficult to find the cause for this to me… ---
**Author:** ![aliutkus](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aliutkus/32/5600_2.png) [@aliutkus](https://discuss.pytorch.org/u/aliutkus)\ **Post date:** [April 17, 2019, 8:23am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/6 "2019-04-17T08:23:25Z")
still no news on that ? Maybe it’s related to some conflict with tqdm ? It’s looking ugly to have this displayed all the time… ---
**Author:** ![sbelharbi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/32.png) [@sbelharbi](https://discuss.pytorch.org/u/sbelharbi)\ **Post date:** [April 17, 2019, 7:56pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/7 "2019-04-17T19:56:38Z")
If the issue was raised because you tried to use multiprocessing within cuda … It is better to ignore it as mentioned [above](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/4?u=sbelharbi), and move on (see [here](https://github.com/pytorch/pytorch/issues/13883#issuecomment-440004307), and [here](https://github.com/pytorch/pytorch/issues/2517#issuecomment-452153555)). ---
**Author:** ![zaccharieramzi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zaccharieramzi/32/26294_2.png) [@zaccharieramzi](https://discuss.pytorch.org/u/zaccharieramzi)\ **Post date:** [July 3, 2020, 3:30pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/8 "2020-07-03T15:30:32Z")
I am actually also interested in understanding this error, since I have a similar one using tensorflow, and it uses a lot of memory in my case (too much for my workers which die). ---
**Author:** ![Enamex](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/enamex/32/3329_2.png) [@Enamex](https://discuss.pytorch.org/u/Enamex)\ **Post date:** [October 13, 2021, 8:51am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/9 "2021-10-13T08:51:56Z")
I’m getting the same error. Only my latest run had **5779** leaked semaphores 🙃. I’m _not_ using PyTorch MP manually; fairseq is using it. Fairseq master with PyTorch 1.9. ---
**Author:** ![lvrcek](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lvrcek/32/34792_2.png) [@lvrcek](https://discuss.pytorch.org/u/lvrcek)\ **Post date:** [January 14, 2022, 10:37am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/10 "2022-01-14T10:37:38Z")
Same issue, I am getting 6 leaked semaphores, and am not using multiprocessing explicitly. The mind-boggling thing about this is that sometimes it happens, sometimes not, and I cannot find any pattern what leads to this. The memory doesn’t seem I use PyTorch 1.9.0 and DGL 0.7.1 and I’m not sure which one is causing the warning to arise. Would greatly appreciate if anyone had any insight. ---
**Author:** ![jingpengw](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jingpengw/32/15007_2.png) [@jingpengw](https://discuss.pytorch.org/u/jingpengw)\ **Post date:** [August 29, 2022, 1:34am UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/11 "2022-08-29T01:34:28Z")
I got the same issue. It seems that the dataloader has to be constructed under the “if **name** ==‘ **main** ’:” or in a function. Whenever I move it to a function in a class, it will have this issue… ---
**Author:** ![log2win](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/log2win/32/66084_2.png) [@log2win](https://discuss.pytorch.org/u/log2win)\ **Post date:** [December 7, 2023, 2:04pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/12 "2023-12-07T14:04:46Z")
I found simple example which leads to “UserWarning: resource\_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown”. I think it may be helpful to someone, as using barriers with multiprocessing should be pretty common ```auto import torch.multiprocessing as mp def foo(barrier = mp.Barrier(1)): print("default barrier leads to leak") # def foo(barrier = None): # print("default None not leads to leak") if __name__ == ' __main__': processes = [] bar = mp.Barrier(1) mp.set_start_method('spawn', force=True) for rank in range(1): p = mp.Process(target=foo, args=(bar, )) p.start() processes.append(p) for p in processes: p.join() ``` ---
**Author:** ![AI\_my\_dream](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ai_my_dream/32/79458_2.png) [@AI\_my\_dream](https://discuss.pytorch.org/u/AI_my_dream)\ **Post date:** [March 28, 2026, 2:00pm UTC](https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/13 "2026-03-28T14:00:15Z")
I have the issue: /home/ari/miniconda3/envs/GiuliaEnv/lib/python3.10/multiprocessing/resource\_tracker.py:224: UserWarning: resource\_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown warnings.warn('resource\_tracker: There appear to be %d ’ Segmentation fault (core dumped) can someone help or give idea how to fix it??