# Segmentation Fault when importing PyTorch

**URL:** <https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486>\
**Category:** Uncategorized\
**Created:** [October 18, 2021, 4:52am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486 "2021-10-18T04:52:41Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [October 18, 2021, 4:52am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/1 "2021-10-18T04:52:41Z")

</div>

When I tried to import PyTorch in python, it crashed with a segfault error:

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b1c75b435bf0e36c28c335327820a2770e0239.png)

“Segmentation fault (core dumped)” is all I have about the issue. Since the sys admin is very disagreeable, I have to figure out what the problem is myself. But I really don’t know what the cause of the crash could be. [This thread](https://github.com/pytorch/pytorch/issues/4101) talked about this issue, but I couldn’t find a solution therein that works for me. I’m at my wits end so I ask here for help. Following is the configuration of the machine:

- Remote Linux with core version 5.8.0. I am not a super user.
- Python 3.8.6
- CUDA Version: 11.1
- GPU is RTX 3090 with driver version 455.23.05
- CPU: Intel Core i9-10900K
- PyTorch version: 1.8.0+cu111 (I found it at /usr/local/lib/python3.8/dist-packages/torch/version.py)
- System imposed RAM quota: 4GB
- System imposed number of threads: 512198
- System imposed RLIMIT\_NPROC value: 300

If you need other information related to this error, please let me know. Thank you for helping me troubleshoot this problem.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [October 18, 2021, 6:17am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/2 "2021-10-18T06:17:43Z")

</div>

Could you update to the latest PyTorch release (`1.9.1`) or the nightly and check, if this would solve the issue?

---

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [October 18, 2021, 6:52am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/3 "2021-10-18T06:52:11Z")

</div>

It’s a remote Linux. I am not the admin, so I don’t have privilege to upgrade PyTorch.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [October 18, 2021, 6:56am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/4 "2021-10-18T06:56:39Z")

</div>

You could try to grab the backtrace via:

```python
gdb --args python -c "import torch"
...
run
...
bt

```

but given that you are not able to update PyTorch I don’t know which access rights you have on this node to debug the issue.

---

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [November 1, 2021, 9:31am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/6 "2021-11-01T09:31:18Z")

</div>

@ptrblck  
I installed gdb 11.1 on the remote Linux node. If I run gdb commands below

```auto
gdb python3
r -c "import torch"
bt

```

I got the following output:

> Starting program: /usr/bin/python3 -c “import torch”  
> [Thread debugging using libthread\_db enabled]  
> Using host libthread\_db library “/lib/x86\_64-linux-gnu/libthread\_db.so.1”.  
> [Detaching after fork from child process 2451545]  
> [New Thread 0x7fff09280640 (LWP 2451546)]  
> [New Thread 0x7fff08a7f640 (LWP 2451547)]  
> [New Thread 0x7fff0027e640 (LWP 2451548)]  
> [New Thread 0x7ffeffa7d640 (LWP 2451549)]
> 
> Thread 3 “python3” received signal SIGSEGV, Segmentation fault.  
> [Switching to Thread 0x7fff08a7f640 (LWP 2451547)]  
> 0x0000000000000000 in ?? ()  
> (gdb) bt  
> #0 0x0000000000000000 in ?? ()  
> #1 0x00007fff09e23760 in blas\_memory\_alloc ()  
> from /usr/local/lib/python3.8/dist-packages/numpy/core/…/…/numpy.libs/libopenblasp-r0-09e95953.3.13.so  
> #2 0x00007fff09e23f24 in blas\_thread\_server ()  
> from /usr/local/lib/python3.8/dist-packages/numpy/core/…/…/numpy.libs/libopenblasp-r0-09e95953.3.13.so  
> #3 0x00007ffff7f90590 in start\_thread () from /lib/x86\_64-linux-gnu/libpthread.so.0  
> #4 0x00007ffff7d0f223 in clone () from /lib/x86\_64-linux-gnu/libc.so.6  
> (gdb)

Can we find out any clue of the segfault? Please let me know what I need to do next to probe the problem. Thanks.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 1, 2021, 10:15am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/7 "2021-11-01T10:15:19Z")

</div>

Based on the backtrace it seems that numpy’s `libopenblas` creates the seg fault. Did you install numpy with the PyTorch wheels? If not, install it or update to the latest PyTorch release, as recently we’ve found [this issue](https://github.com/pytorch/pytorch/issues/66353), which might be related.

---

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [November 1, 2021, 10:27am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/8 "2021-11-01T10:27:15Z")

</div>

> [@ptrblck](#):
>
> `libopenblas`

numpy is installed in the system folder by the system admin, so I’m not sure if it is installed together with PyTorch. But I can retrieve the version installed as follows:

> Name: numpy  
> Version: 1.19.5  
> Summary: NumPy is the fundamental package for array computing with Python.  
> Home-page: [https://www.numpy.org](https://www.numpy.org)  
> Author: Travis E. Oliphant et al.  
> Author-email:  
> License: BSD  
> Location: /usr/local/lib/python3.8/dist-packages  
> Requires:  
> Required-by: opencv-python, imageio, torchvision, torch, tensorflow, tensorboard, opt-einsum, Keras-Preprocessing, h5py

By “install it” do you mean install numpy? If so, I’ll try to install numpy in my user space. Would the latest numpy that pip will install do? How do I make sure that it is the numpy installed in my user space that is used when running import PyTorch (i.e., will the numpy installed in the system be masked)? Thank you.

---

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [November 1, 2021, 11:12am UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/9 "2021-11-01T11:12:24Z")

</div>

I installed numpy in my user space. The version is 1.21.3. Now when I run import torch, there are some new error messages:

> import torch  
> OpenBLAS blas\_thread\_init: pthread\_create failed for thread 10 of 20: Resource temporarily unavailable  
> OpenBLAS blas\_thread\_init: RLIMIT\_NPROC 300 current, 300 max  
> Segmentation fault (core dumped)

Is this because the RLIMIT\_NPROC number is too small? I think 300 concurrent processes sounds large enough. If this is the cause of segfault, how large this number should be? Thank you.

PS, according to the error message, OpenBLAS failed at creating new threads. The system imposed number of threads is 512198, I think that is also large enough. In addition, why does OpenBLAS have to create 20 threads (is OpenBLAS trying to build a thread pool)? Is it possible to create only 9 so that the failure would not occur (if it does not affect the performance)?

---

<div class="post-metadata">

**Author:** ![zzzhhh](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/65b543/32.png) [@zzzhhh](https://discuss.pytorch.org/u/zzzhhh)\
**Post date:** [November 1, 2021, 12:25pm UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/10 "2021-11-01T12:25:38Z")

</div>

@ptrblck

I’m so glad to let you know that the problem is solved by adding the following lines before `import torch`:

```auto
import os
os.environ['OPENBLAS_NUM_THREADS'] = '5'

```

Maybe different machine needs different number. For the numpy version 1.21.3 just installed, 5 is the max I can reach. I would get error “ImportError: numpy.core.multiarray failed to import” if the number is set to 6. For numpy version 1.19.5 originally installed system-wide, the max is only 2. Above 2, the import statement just crashes with segfault. So installing the latest numpy is not needed but can improve the performance because max allowed thread# is increased. Anyway, it’s solved! Thank you so much for your help, ptrblck.

🙂

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 1, 2021, 5:49pm UTC](https://discuss.pytorch.org/t/segmentation-fault-when-importing-pytorch/134486/11 "2021-11-01T17:49:44Z")

</div>

Good to hear you were able to come up with a workaround! In case you are seeing the same seg fault in openblas by purely importing numpy, it might be a good idea to create a GitHub issue in the numpy repository so that the devs are aware of it.
