[solved] Segmentation fault (core dump)

Hi,
I’m running into an segmentation fault error while running my code. Here are the details.

Version:
Cuda: 8.0
PyTorch 0.4.0
gcc 5.2.0
g++ 5.2.0

Running my script with gdb gives following outputs.

(GE) [15084015@compute-01 BraTS]$ gdb python
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/eee/ug/15084015/miniconda3/envs/GE/bin/python3.6...done.
(gdb) 
(gdb) r predict.py 
Starting program: /home/eee/ug/15084015/miniconda3/envs/GE/bin/python predict.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /home/eee/ug/15084015/miniconda3/envs/GE/lib/python3.6/site-packages/numpy/../../../libiomp5.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/50/126f8244a53ed88a1531546fcaa8dedc4bc85c.debug
Missing separate debuginfo for /home/eee/ug/15084015/miniconda3/envs/GE/lib/python3.6/site-packages/scipy/linalg/../.libs/libgfortran-ed201abd.so.3.0.0
[New Thread 0x7fffdfcdc700 (LWP 32504)]
[New Thread 0x7fffdf4db700 (LWP 32505)]
[New Thread 0x7fffdccda700 (LWP 32506)]
[New Thread 0x7fffda4d9700 (LWP 32507)]
[New Thread 0x7fffd7cd8700 (LWP 32508)]
[New Thread 0x7fffd54d7700 (LWP 32509)]
[New Thread 0x7fffd2cd6700 (LWP 32510)]
[New Thread 0x7fffd04d5700 (LWP 32511)]
[New Thread 0x7fffcdcd4700 (LWP 32512)]
[New Thread 0x7fffcb4d3700 (LWP 32513)]
[New Thread 0x7fffc8cd2700 (LWP 32514)]
[New Thread 0x7fffc64d1700 (LWP 32515)]
[New Thread 0x7fffc3cd0700 (LWP 32516)]
[New Thread 0x7fffc14cf700 (LWP 32517)]
[New Thread 0x7fffbecce700 (LWP 32518)]
[New Thread 0x7fffbc4cd700 (LWP 32519)]
[New Thread 0x7fffb9ccc700 (LWP 32520)]
[New Thread 0x7fffb74cb700 (LWP 32521)]
[New Thread 0x7fffb4cca700 (LWP 32522)]
[New Thread 0x7fffb24c9700 (LWP 32523)]
[New Thread 0x7fffafcc8700 (LWP 32524)]
[New Thread 0x7fffad4c7700 (LWP 32525)]
[New Thread 0x7fffaacc6700 (LWP 32526)]
[New Thread 0x7fffa84c5700 (LWP 32527)]
[New Thread 0x7fffa5cc4700 (LWP 32528)]
[New Thread 0x7fffa34c3700 (LWP 32529)]
[New Thread 0x7fffa0cc2700 (LWP 32530)]
[New Thread 0x7fff9e4c1700 (LWP 32531)]
[New Thread 0x7fff9bcc0700 (LWP 32532)]
[New Thread 0x7fff994bf700 (LWP 32533)]
[New Thread 0x7fff96cbe700 (LWP 32534)]
[New Thread 0x7fff944bd700 (LWP 32535)]
[New Thread 0x7fff91cbc700 (LWP 32536)]
[New Thread 0x7fff8f4bb700 (LWP 32537)]
[New Thread 0x7fff8ccba700 (LWP 32538)]
[New Thread 0x7fff8a4b9700 (LWP 32539)]
[New Thread 0x7fff87cb8700 (LWP 32540)]
[New Thread 0x7fff854b7700 (LWP 32541)]
[New Thread 0x7fff82cb6700 (LWP 32542)]
[New Thread 0x7fff804b5700 (LWP 32543)]
[New Thread 0x7fff7dcb4700 (LWP 32544)]
[New Thread 0x7fff794b3700 (LWP 32545)]
[New Thread 0x7fff78cb2700 (LWP 32546)]
[New Thread 0x7fff764b1700 (LWP 32547)]
[New Thread 0x7fff73cb0700 (LWP 32548)]
[New Thread 0x7fff714af700 (LWP 32549)]
[New Thread 0x7fff6ecae700 (LWP 32550)]
[New Thread 0x7fff6c4ad700 (LWP 32551)]
[New Thread 0x7fff69cac700 (LWP 32552)]
[New Thread 0x7fff674ab700 (LWP 32553)]
[New Thread 0x7fff64caa700 (LWP 32554)]
[New Thread 0x7fff624a9700 (LWP 32555)]
[New Thread 0x7fff5fca8700 (LWP 32556)]
[New Thread 0x7fff5d4a7700 (LWP 32557)]
[New Thread 0x7fff5aca6700 (LWP 32558)]
[New Thread 0x7fff584a5700 (LWP 32559)]
[New Thread 0x7fff55ca4700 (LWP 32560)]
[New Thread 0x7fff534a3700 (LWP 32561)]
[New Thread 0x7fff50ca2700 (LWP 32562)]
[New Thread 0x7fff4e4a1700 (LWP 32563)]
[New Thread 0x7fff4bca0700 (LWP 32564)]
[New Thread 0x7fff4949f700 (LWP 32565)]
[New Thread 0x7fff46c9e700 (LWP 32566)]
warning: File "/home/eee/ug/15084015/miniconda3/envs/GE/lib/libstdc++.so.6.0.21-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /home/eee/ug/15084015/miniconda3/envs/GE/lib/libstdc++.so.6.0.21-gdb.py
line to your configuration file "/home/eee/ug/15084015/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/eee/ug/15084015/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread 0x7fff764b1700 (LWP 32547) exited]
[Thread 0x7fff4bca0700 (LWP 32564) exited]
[Thread 0x7fff8ccba700 (LWP 32538) exited]
[Thread 0x7fff4949f700 (LWP 32565) exited]
[Thread 0x7fffcdcd4700 (LWP 32512) exited]
[Thread 0x7fff46c9e700 (LWP 32566) exited]
[Thread 0x7fffa34c3700 (LWP 32529) exited]
[Thread 0x7fff4e4a1700 (LWP 32563) exited]
[Thread 0x7fff96cbe700 (LWP 32534) exited]
[Thread 0x7fffd7cd8700 (LWP 32508) exited]
[Thread 0x7fff6ecae700 (LWP 32550) exited]
[Thread 0x7fff50ca2700 (LWP 32562) exited]
[Thread 0x7fffbc4cd700 (LWP 32519) exited]
[Thread 0x7fff534a3700 (LWP 32561) exited]
[Thread 0x7fff78cb2700 (LWP 32546) exited]
[Thread 0x7fff55ca4700 (LWP 32560) exited]
[Thread 0x7fff5d4a7700 (LWP 32557) exited]
[Thread 0x7fff584a5700 (LWP 32559) exited]
[Thread 0x7fffa84c5700 (LWP 32527) exited]
[Thread 0x7fff5aca6700 (LWP 32558) exited]
[Thread 0x7fff9bcc0700 (LWP 32532) exited]
[Thread 0x7fff5fca8700 (LWP 32556) exited]
[Thread 0x7fffbecce700 (LWP 32518) exited]
[Thread 0x7fff624a9700 (LWP 32555) exited]
[Thread 0x7fff854b7700 (LWP 32541) exited]
[Thread 0x7fff64caa700 (LWP 32554) exited]
[Thread 0x7fff804b5700 (LWP 32543) exited]
[Thread 0x7fff674ab700 (LWP 32553) exited]
[Thread 0x7fffd04d5700 (LWP 32511) exited]
[Thread 0x7fff69cac700 (LWP 32552) exited]
[Thread 0x7fffa0cc2700 (LWP 32530) exited]
[Thread 0x7fff6c4ad700 (LWP 32551) exited]
[Thread 0x7fffb24c9700 (LWP 32523) exited]
[Thread 0x7fff714af700 (LWP 32549) exited]
[Thread 0x7fffc3cd0700 (LWP 32516) exited]
[Thread 0x7fff73cb0700 (LWP 32548) exited]
[Thread 0x7fffdfcdc700 (LWP 32504) exited]
[Thread 0x7fff794b3700 (LWP 32545) exited]
[Thread 0x7fffaacc6700 (LWP 32526) exited]
[Thread 0x7fff7dcb4700 (LWP 32544) exited]
[Thread 0x7fffda4d9700 (LWP 32507) exited]
[Thread 0x7fff82cb6700 (LWP 32542) exited]
[Thread 0x7fff8f4bb700 (LWP 32537) exited]
[Thread 0x7fff87cb8700 (LWP 32540) exited]
[Thread 0x7fffdccda700 (LWP 32506) exited]
[Thread 0x7fff8a4b9700 (LWP 32539) exited]
[Thread 0x7fffcb4d3700 (LWP 32513) exited]
[Thread 0x7fff91cbc700 (LWP 32536) exited]
[Thread 0x7fffb9ccc700 (LWP 32520) exited]
[Thread 0x7fff944bd700 (LWP 32535) exited]
[Thread 0x7fffb74cb700 (LWP 32521) exited]
[Thread 0x7fff994bf700 (LWP 32533) exited]
[Thread 0x7fffdf4db700 (LWP 32505) exited]
[Thread 0x7fff9e4c1700 (LWP 32531) exited]
[Thread 0x7fffd54d7700 (LWP 32509) exited]
[Thread 0x7fffa5cc4700 (LWP 32528) exited]
[Thread 0x7fffafcc8700 (LWP 32524) exited]
[Thread 0x7fffad4c7700 (LWP 32525) exited]
[Thread 0x7fffc8cd2700 (LWP 32514) exited]
[Thread 0x7fffb4cca700 (LWP 32522) exited]
[Thread 0x7fffc14cf700 (LWP 32517) exited]
[Thread 0x7fffc64d1700 (LWP 32515) exited]
[Thread 0x7fffd2cd6700 (LWP 32510) exited]
Detaching after fork from child process 32567.
Detaching after fork from child process 32569.
[New Thread 0x7fff46c9e700 (LWP 32615)]
[New Thread 0x7fff4949f700 (LWP 32616)]
valid_list: /home/eee/ug/15084015/GE/data/MICCAI_BraTS_2018_Data_Validation/test.txt
[New Thread 0x7fff4bca0700 (LWP 32646)]
Detaching after fork from child process 32659.
Detaching after fork from child process 32664.
Detaching after fork from child process 32665.
Detaching after fork from child process 32666.
[New Thread 0x7fff4e4a1700 (LWP 32671)]
[New Thread 0x7fffd54d6800 (LWP 32672)]
[New Thread 0x7fffd2cd5880 (LWP 32673)]
[New Thread 0x7fffd04d4900 (LWP 32674)]
[New Thread 0x7fffcdcd3980 (LWP 32675)]
[New Thread 0x7fffcb4d2a00 (LWP 32676)]
[New Thread 0x7fffc8cd1a80 (LWP 32677)]
[New Thread 0x7fffc64d0b00 (LWP 32678)]
[New Thread 0x7fffc3ccfb80 (LWP 32679)]
[New Thread 0x7fffc14cec00 (LWP 32680)]
[New Thread 0x7fffbeccdc80 (LWP 32681)]
[New Thread 0x7fffbc4ccd00 (LWP 32682)]
[New Thread 0x7fffb9ccbd80 (LWP 32683)]
[New Thread 0x7fffb74cae00 (LWP 32684)]
[New Thread 0x7fffb4cc9e80 (LWP 32685)]
[New Thread 0x7fffb24c8f00 (LWP 32686)]
[New Thread 0x7fffafcc7f80 (LWP 32687)]
[New Thread 0x7fffad4c7000 (LWP 32688)]
[New Thread 0x7fffaacc6080 (LWP 32690)]
[New Thread 0x7fffa84c5100 (LWP 32691)]
[New Thread 0x7fffa5cc4180 (LWP 32692)]
[New Thread 0x7fffa34c3200 (LWP 32693)]
[New Thread 0x7fffa0cc2280 (LWP 32694)]
[New Thread 0x7fff9e4c1300 (LWP 32695)]
[New Thread 0x7fff9bcc0380 (LWP 32696)]
[New Thread 0x7fff994bf400 (LWP 32697)]
[New Thread 0x7fff96cbe480 (LWP 32698)]
[New Thread 0x7fff944bd500 (LWP 32699)]
[New Thread 0x7fff91cbc580 (LWP 32700)]
[New Thread 0x7fff8f4bb600 (LWP 32701)]
[New Thread 0x7fff8ccba680 (LWP 32702)]
[New Thread 0x7fff8a4b9700 (LWP 32703)]
[New Thread 0x7fff87cb7780 (LWP 32704)]
[New Thread 0x7fff854b6800 (LWP 32705)]
[New Thread 0x7fff82cb5880 (LWP 32706)]
[New Thread 0x7fff804b4900 (LWP 32707)]
[New Thread 0x7fff7dcb3980 (LWP 32708)]
[New Thread 0x7fff78abca00 (LWP 32709)]
[New Thread 0x7fff764b0a80 (LWP 32710)]
[New Thread 0x7fff73cafb00 (LWP 32711)]
[New Thread 0x7fff714aeb80 (LWP 32712)]
[New Thread 0x7fff6ecadc00 (LWP 32713)]
[New Thread 0x7fff6c4acc80 (LWP 32714)]
[New Thread 0x7fff69cabd00 (LWP 32715)]
[New Thread 0x7fff674aad80 (LWP 32716)]
[New Thread 0x7fff64ca9e00 (LWP 32717)]
[New Thread 0x7fff624a8e80 (LWP 32718)]
[New Thread 0x7fff5fca7f00 (LWP 32719)]
[New Thread 0x7fff5d4a6f80 (LWP 32720)]
[New Thread 0x7fff5aca6000 (LWP 32721)]
[New Thread 0x7fff584a5080 (LWP 32722)]
[New Thread 0x7fff55ca4100 (LWP 32723)]
[New Thread 0x7fff534a3180 (LWP 32724)]
[New Thread 0x7fff50ca2200 (LWP 32725)]
[New Thread 0x7fff07673280 (LWP 32726)]
[New Thread 0x7fff07271300 (LWP 32727)]
[New Thread 0x7fff06e6f380 (LWP 32728)]
[New Thread 0x7fff057ff400 (LWP 32729)]
[New Thread 0x7fff053fd480 (LWP 32730)]
[New Thread 0x7fff04ffb500 (LWP 32731)]
[New Thread 0x7fff04bf9580 (LWP 32732)]
[New Thread 0x7fff047f7600 (LWP 32733)]
[New Thread 0x7ffefed75680 (LWP 32734)]
[New Thread 0x7ffefe973700 (LWP 32735)]
[New Thread 0x7ffefe570780 (LWP 32736)]
[New Thread 0x7ffefe16d800 (LWP 32737)]
[New Thread 0x7ffefdd6a880 (LWP 32738)]
[New Thread 0x7ffefd967900 (LWP 32739)]
[New Thread 0x7ffefa3c2980 (LWP 32740)]
[New Thread 0x7ffef9fbfa00 (LWP 32741)]
[New Thread 0x7ffef9bbca80 (LWP 32742)]
[New Thread 0x7ffef97ba700 (LWP 32743)]
0it [00:00, ?it/s]
Program received signal SIGSEGV, Segmentation fault.
0x00007fffd7a667b4 in construct<_object*, _object*> (__p=0xb, this=0x555556bd9ac8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120
120		{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7_3.5.x86_64 libuuid-2.23.2-33.el7_3.2.x86_64
(gdb) where
#0  0x00007fffd7a667b4 in construct<_object*, _object*> (__p=0xb, this=0x555556bd9ac8) at /usr/include/c++/4.8.2/ext/new_allocator.h:120
#1  _S_construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:254
#2  construct<_object*, _object*> (__p=0xb, __a=...) at /usr/include/c++/4.8.2/bits/alloc_traits.h:393
#3  emplace_back<_object*> (this=0x555556bd9ac8) at /usr/include/c++/4.8.2/bits/vector.tcc:96
#4  push_back (__x=<unknown type in /home/eee/ug/15084015/miniconda3/envs/GE/lib/python3.6/site-packages/multicrop-0.0.0-py3.6-linux-x86_64.egg/multicrop.cpython-36m-x86_64-linux-gnu.so, CU 0x0, DIE 0x5d935>, 
    this=0x555556bd9ac8) at /usr/include/c++/4.8.2/bits/stl_vector.h:920
#5  loader_life_support (this=0x7fffffffd0e0) at /home/eee/ug/15084015/miniconda3/envs/GE/lib/python3.6/site-packages/torch/lib/include/pybind11/cast.h:43
#6  pybind11::cpp_function::dispatcher (self=<optimized out>, args_in=0x7fffd793a5f8, kwargs_in=0x0)
    at /home/eee/ug/15084015/miniconda3/envs/GE/lib/python3.6/site-packages/torch/lib/include/pybind11/pybind11.h:618
#7  0x00005555556673d4 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1540319457073/work/Objects/methodobject.c:231
#8  0x00005555556f0c1c in call_function () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4837
#9  0x000055555571371a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
#10 0x00005555556ea124 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
#11 0x00005555556eafc1 in fast_function () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4978
#12 0x00005555556f0cf5 in call_function () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
#13 0x00005555557144d8 in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3351
#14 0x00005555556ea42e in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
#15 0x00005555556eafc1 in fast_function () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4978
#16 0x00005555556f0cf5 in call_function () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
#17 0x000055555571371a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
#18 0x00005555556ebad9 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=0x0, argcount=0, 
    args=0x0, locals=0x7ffff7f431b0, globals=0x7ffff7f431b0, _co=0x7ffff0905780) at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
#19 PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4187
#20 0x00005555556ec87c in PyEval_EvalCode (co=co@entry=0x7ffff0905780, globals=globals@entry=0x7ffff7f431b0, locals=locals@entry=0x7ffff7f431b0)
    at /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:731
#21 0x000055555576d074 in run_mod () at /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:1025
#22 0x000055555576d471 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:978
#23 0x000055555576d673 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:419
#24 0x000055555576d77d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:81
#25 0x00005555557711b0 in run_file (p_cf=0x7fffffffdbdc, filename=0x5555558ac310 L"predict.py", fp=0x55555593a360) at /tmp/build/80754af9/python_1540319457073/work/Modules/main.c:340
#26 Py_Main () at /tmp/build/80754af9/python_1540319457073/work/Modules/main.c:811
#27 0x0000555555638b4e in main () at /tmp/build/80754af9/python_1540319457073/work/Programs/python.c:69
#28 0x00007ffff7820b35 in __libc_start_main () from /lib64/libc.so.6
#29 0x000055555571a1a8 in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb) 

I’ve spent a lot of time googling and figuring out what’s wrong with my code/system. Please help me.

So, I’ve traced down the issue. It is being caused by mutlicrop module which I’m using as an dependency for my project. I recloned the multicrop repo, reinstalled it and now it works.