# RuntimeError: unable to open shared memory object (depending on the model)

**URL:** <https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090>\
**Category:** distributed\
**Created:** [March 26, 2021, 2:53pm UTC](https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090 "2021-03-26T14:53:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ClementPla](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@ClementPla](https://discuss.pytorch.org/u/ClementPla)\
**Post date:** [March 26, 2021, 2:53pm UTC](https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090/1 "2021-03-26T14:53:06Z")

</div>

Hi,

I’m facing an issue I’ve seen covered in others topics but I haven’t found any working solutions.  
I’m trying to use my library to train different models on a distributed way on a server containing 4 GPUs.  
For most models, it works as expected (DeepLabV3+, ResNet101, UNeT, Transformers…). There is one model though that does not work which is the HRN for Semantic Segmentation,  
[following the original implementation](https://github.com/HRNet/HRNet-Semantic-Segmentation/blob/pytorch-v1.1/lib/models/seg_hrnet.py)  
I did minor modification to this model to remove anything that wasn’t Pytorch/pure python objects. I also removed any class attributes (not sure if this helps). My [re-implementation is here.](https://pastebin.com/K8wgGJcK)

On a single GPU, the model works fine, but on multi-gpu DDP, I get the following message:

```auto
File "/usagers/clpla/.conda/envs/torch18/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 321, in reduce_storage
RuntimeError: unable to open shared memory object </torch_403534_2756454997> in read-write mode

```

This error raises when calling the torch.multiprocessing.spawn function [in my code](https://github.com/ClementPla/NNTools/blob/347629c326202af839b002b2bf0b40b7675b7e31/nntools/experiment.py#L311), so before the dataloaders are even created. I’ve still tried varying the num\_workers (even set to 0), with no effects.  
Following other recommendations, I’ve tried increasing the shared memory size:

```auto
# cat /proc/sys/kernel/shmmni
8192

```

Which did not helped neither. Finally, I tried changing the sharing strategy to file\_system, as described in the [pytorch multiprocessing documentation](https://pytorch.org/docs/stable/multiprocessing.html)  
In this case, the previous error does not appeared, but it leads to memory leaks (in regular RAM) after a few epochs and I suspect it comes from the dataloader. It also does not explain why the file\_descriptor strategy would work on certain models and not others.

I’m running out of idea, I’m open to any suggestions. Let me know if more details are needed. I could try to provide a minimal (non)-working example.

Specifications:

- OS : CentOS 7
- Python 3.8.8
- Pytorch: 1.8.0
- 4 GPUs RTX 2080
- CUDA: 11.0
- Driver: 450.66
- RAM: 125G

Thanks!

---

<div class="post-metadata">

**Author:** ![wayi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/32/32374_2.png) [@wayi](https://discuss.pytorch.org/u/wayi)\
**Post date:** [March 27, 2021, 6:21am UTC](https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090/2 "2021-03-27T06:21:44Z")

</div>

To summarize, you have tried 3 approaches (as also suggested in this [thread](https://github.com/facebookresearch/maskrcnn-benchmark/issues/103)):

1. Set `num_workers=0` (i.e., `self.config['Manager']['num_workers']=0`) when calling `DataLoader` constructor;
2. Increase shared memory size;
3. Change the [sharing strategy](https://pytorch.org/docs/master/multiprocessing.html?highlight=sharing%20strategy#sharing-strategies):

```auto
import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

```

Can you try a few more suggestions in [possible deadlock in dataloader · Issue #1355 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/1355)?

It seems that someone also tried `ulimit -n` or `setNumThreads(0)`. If all the suggestions are exhausted, feel free to open a bug on Github.

---

<div class="post-metadata">

**Author:** ![ClementPla](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/5e9695/32.png) [@ClementPla](https://discuss.pytorch.org/u/ClementPla)\
**Post date:** [March 31, 2021, 8:17pm UTC](https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090/3 "2021-03-31T20:17:28Z")

</div>

Hi!  
Thanks for your reply, you perfectly summarized my approaches.  
And thanks to your suggestion, my model finally works!  
As you suggested, I had to call:

```auto
ulimit -n 64000

```

The 64000 is kinda arbitrary, it corresponds to the open files limit. I’m not really sure how this is is related to distributed training and what a good value should be. The default value of 1024 was enough for some models but not for other.

Anyway, some relevant infos if someones ends up with the same issue:

You need admin rights to call ulimit. Note that calling ulimit will only affect the current shell. Values will be resetted at reboot.  
Thanks again.

---

<div class="post-metadata">

**Author:** ![tianhan4](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tianhan4/32/45568_2.png) [@tianhan4](https://discuss.pytorch.org/u/tianhan4)\
**Post date:** [January 8, 2022, 5:35am UTC](https://discuss.pytorch.org/t/runtimeerror-unable-to-open-shared-memory-object-depending-on-the-model/116090/4 "2022-01-08T05:35:20Z")

</div>

Great, it also helps me. Thank you!