Pytorch from source install fails

gakadam · September 15, 2020, 2:44am

Set up:
CUDA: 10.2
GCC: 8.4.0

Error:

Building wheel torch-1.7.0a0+8daaa3b
-- Building version 1.7.0a0+8daaa3b
cmake --build . --target install --config Release -- -j 56
[1/5] Building CXX object c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o
FAILED: c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o
/home/gakadam/tools/gcc-8.4/bin/g++  -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMAGMA_V2 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTH_BLAS_MKL -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Iaten/src -I../aten/src -I. -I../ -I../cmake/../third_party/benchmark/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -I../c10/.. -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../cmake/../third_party/googletest/googlemock/include -isystem ../cmake/../third_party/googletest/googletest/include -isystem ../third_party/protobuf/src -isystem /home/gakadam/tools/anaconda3/include -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isystem ../third_party/XNNPACK/include -isystem ../third_party -isystem ../cmake/../third_party/eigen -isystem /home/gakadam/tools/anaconda3/include/python3.8 -isystem /home/gakadam/tools/anaconda3/lib/python3.8/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -isystem ../cmake/../third_party/cub -isystem /home/gakadam/tools/cuda-10.2/include -isystem ../third_party/ideep/mkl-dnn/include -isystem ../third_party/ideep/include -isystem ../third_party/googletest/googlemock/include -isystem ../third_party/googletest/googlemock -isystem ../third_party/googletest/googletest/include -isystem ../third_party/googletest/googletest -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -fPIE   -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -pthread -std=gnu++14 -MD -MT c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o -MF c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o.d -o c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o -c ../c10/test/util/complex_math_test.cpp
init2.c:52: MPFR assertion failed: p >= 2 && p <= ((mpfr_prec_t)((mpfr_uprec_t)(~(mpfr_uprec_t)0)>>1))
during GIMPLE pass: fre
In file included from ../c10/test/util/complex_math_test.cpp:4:
../c10/test/util/complex_math_test_common.h: In member function ‘virtual void TestRevTrigonometric_Rev_Test::TestBody()’:
../c10/test/util/complex_math_test_common.h:503:1: internal compiler error: Aborted
 }
 ^
0xb3805f crash_signal
        ../../gcc/toplev.c:325
0x7fe84604624f ???
        /usr/src/debug/glibc-2.17-c758a686/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7fe8460461d7 __GI_raise
        ../nptl/sysdeps/unix/sysv/linux/raise.c:56
0x7fe8460478c7 __GI_abort
        /usr/src/debug/glibc-2.17-c758a686/stdlib/abort.c:90
0x8df310 do_mpc_arg1
        ../../gcc/fold-const-call.c:346
0x8df310 do_mpc_arg1
        ../../gcc/fold-const-call.c:326
0x8e0cf2 fold_const_call_cc
        ../../gcc/fold-const-call.c:1032
0x8e0cf2 fold_const_call_1
        ../../gcc/fold-const-call.c:1125
0xe42331 gimple_resimplify1(gimple**, code_helper*, tree_node*, tree_node**, tree_node* (*)(tree_node*))
        ../../gcc/gimple-match-head.c:89
0xf06e05 gimple_simplify(gimple*, code_helper*, tree_node**, gimple**, tree_node* (*)(tree_node*), tree_node* (*)(tree_node*))
        ../../gcc/gimple-match-head.c:782
0x904bca gimple_fold_stmt_to_constant_1(gimple*, tree_node* (*)(tree_node*), tree_node* (*)(tree_node*))
        ../../gcc/gimple-fold.c:6107
0xcadc89 visit_use
        ../../gcc/tree-ssa-sccvn.c:4132
0xcafa46 process_scc
        ../../gcc/tree-ssa-sccvn.c:4346
0xcafa46 extract_and_process_scc_for_name
        ../../gcc/tree-ssa-sccvn.c:4448
0xcafa46 DFS
        ../../gcc/tree-ssa-sccvn.c:4498
0xcafdc7 sccvn_dom_walker::before_dom_children(basic_block_def*)
        ../../gcc/tree-ssa-sccvn.c:4940
0x11f40f7 dom_walker::walk(basic_block_def*)
        ../../gcc/domwalk.c:353
0xcb0a98 run_scc_vn(vn_lookup_kind)
        ../../gcc/tree-ssa-sccvn.c:5047
0xcb126a execute
        ../../gcc/tree-ssa-sccvn.c:6080
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.
[2/5] Building CXX object caffe2/torch/CMakeFiles/torch_python.dir/csrc/DataLoader.cpp.o
FAILED: caffe2/torch/CMakeFiles/torch_python.dir/csrc/DataLoader.cpp.o
/home/gakadam/tools/gcc-8.4/bin/g++  -DBUILDING_TESTS -DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMAGMA_V2 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTHP_BUILD_MAIN_LIB -DTH_BLAS_MKL -DUSE_C10D -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_CUDA -DUSE_CUDNN -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_NCCL -DUSE_NUMPY -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_python_EXPORTS -Iaten/src -I../aten/src -I. -I../ -I../cmake/../third_party/benchmark/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -I../torch/.. -I../torch/../aten/src -I../torch/../aten/src/TH -Icaffe2/aten/src -Ithird_party -I../torch/../third_party/gloo -I../torch/../third_party/onnx -I../torch/csrc -I../torch/csrc/api/include -I../torch/lib -I../torch/lib/libshm -I../torch/csrc/api -I../c10/.. -Ithird_party/ideep/mkl-dnn/include -I../third_party/ideep/mkl-dnn/src/../include -I../c10/cuda/../.. -I../torch/lib/libshm/../../../torch/lib -I../third_party/fmt/include -I../torch/lib/c10d/.. -I../torch/lib/c10d/../../.. -Inccl/include -I../third_party/tensorpipe -Ithird_party/tensorpipe -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../cmake/../third_party/googletest/googlemock/include -isystem ../cmake/../third_party/googletest/googletest/include -isystem ../third_party/protobuf/src -isystem /home/gakadam/tools/anaconda3/include -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isystem ../third_party/XNNPACK/include -isystem ../third_party -isystem ../cmake/../third_party/eigen -isystem /home/gakadam/tools/anaconda3/include/python3.8 -isystem /home/gakadam/tools/anaconda3/lib/python3.8/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -isystem ../cmake/../third_party/cub -isystem /home/gakadam/tools/cuda-10.2/include -isystem ../third_party/ideep/mkl-dnn/include -isystem ../third_party/ideep/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -fPIC   -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -fno-strict-aliasing -Wno-write-strings -Wno-strict-aliasing -pthread -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unknown-pragmas -std=gnu++14 -MD -MT caffe2/torch/CMakeFiles/torch_python.dir/csrc/DataLoader.cpp.o -MF caffe2/torch/CMakeFiles/torch_python.dir/csrc/DataLoader.cpp.o.d -o caffe2/torch/CMakeFiles/torch_python.dir/csrc/DataLoader.cpp.o -c ../torch/csrc/DataLoader.cpp
../torch/csrc/DataLoader.cpp: In function ‘PyObject* THPModule_removeWorkerPIDs(PyObject*, PyObject*)’:
../torch/csrc/DataLoader.cpp:185:21: error: expected primary-expression before ‘(’ token
     throw ValueError("Cannot find worker information for _BaseDataLoaderIter with id %" PRId64, key);
                     ^
../torch/csrc/DataLoader.cpp:185:88: error: expected ‘)’ before ‘PRId64’
     throw ValueError("Cannot find worker information for _BaseDataLoaderIter with id %" PRId64, key);
                     ~                                                                  ^~~~~~~
                                                                                        )
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 747, in <module>
    build_deps()
  File "setup.py", line 326, in build_deps
    build_caffe2(version=version,
  File "/home/gakadam/tools/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/gakadam/tools/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/gakadam/tools/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/gakadam/tools/anaconda3/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '56']' returned non-zero exit status 1.

How to rectify this? Please let me know if more details are needed.

thank you.

albanD · September 15, 2020, 2:09pm

Hi,

This is surprising indeed.
Can you try a different gcc version to make sure that the “internal compiler error” is just due to gcc 8.4?
If it is, can you please open an issue on github so that we can fix it

gakadam · September 15, 2020, 6:14pm

I get the same result with gcc-7.5.0 / CUDA-10.2.

albanD · September 15, 2020, 6:27pm

Our ci only tests 4.8, 5.4 and 7.2 IIRC.
But this file wasn’t changed for several month so I’m a bit surprise that it would trigger an internal compiler error… Is there anything specific about the way you install gcc? Also which distro are you using?

prashantserai · September 22, 2020, 3:32pm

I’m facing the same error with gcc 6.3.1 /cuda-10.2 on RHEL 6

malfet · September 22, 2020, 4:39pm

And will it pass if you are to build PyTorch without CUDA?
No luck reproducing it with CUDA-10.2 + gcc-8.3.0…

malfet · September 22, 2020, 5:32pm

Can you share a bit more info on how /home/gakadam/tools/gcc-8.4/bin/g++ was built?
I can not reproduce the same crash using gcc-8.4 from Ubuntu-18.04:

/pytorch/build# g++-8  -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMAGMA_V2 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTH_BLAS_MKL -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Iaten/src -I../aten/src -I. -I../ -I../cmake/../third_party/benchmark/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -I../c10/.. -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../cmake/../third_party/googletest/googlemock/include -isystem ../cmake/../third_party/googletest/googletest/include -isystem ../third_party/protobuf/src -isystem /home/gakadam/tools/anaconda3/include -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isystem ../third_party/XNNPACK/include -isystem ../third_party -isystem ../cmake/../third_party/eigen -isystem /home/gakadam/tools/anaconda3/include/python3.8 -isystem /home/gakadam/tools/anaconda3/lib/python3.8/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -isystem ../cmake/../third_party/cub -isystem /home/gakadam/tools/cuda-10.2/include -isystem ../third_party/ideep/mkl-dnn/include -isystem ../third_party/ideep/include -isystem ../third_party/googletest/googlemock/include -isystem ../third_party/googletest/googlemock -isystem ../third_party/googletest/googletest/include -isystem ../third_party/googletest/googletest -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -fPIE   -DCAFFE2_USE_GLOO -DCUDA_HAS_FP16=1 -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -pthread -std=gnu++14 -MD -MT c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o -MF c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o.d -o c10/test/CMakeFiles/c10_complex_math_test.dir/util/complex_math_test.cpp.o -c ../c10/test/util/complex_math_test.cpp
root@f02210591fe6:/pytorch/build# g++-8
g++-8: fatal error: no input files
compilation terminated.
root@f02210591fe6:/pytorch/build# g++-8 -v
Using built-in specs.
COLLECT_GCC=g++-8
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 8.4.0-1ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 8.4.0 (Ubuntu 8.4.0-1ubuntu1~18.04)

prashantserai · September 22, 2020, 11:01pm

I just realized that this was just part of the logs and not the terminal error in my case, the terminal error was just below this so I got fooled. When I cleaned my install and tried again, the terminal error was further different and this message was nowhere to be seen near it in the logs.

@gakadam might still have this issue but I don’t seem to