Compiling pytorch from source failed

Hi,

Compiling PyTorch from source failed;
Thanks for any help!

PyTorch Version (e.g., 1.0): master
Box: OpenSuse 15.2
How you installed PyTorch (conda, pip, source): source
Build command you used (if compiling from source): python3 setup.py install
following instructions here
that is

git clone --recursive https://github.com/pytorch/pytorch cd pytorch git submodule sync git submodule update --init --recursive python3 setup.py install

Python version: 3.6.12
CUDA/cuDNN version: 10.1

`
– The CXX compiler identification is GNU 7.5.0
– The C compiler identification is GNU 7.5.0
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Check for working CXX compiler: /usr/bin/c++ - skipped
– Detecting CXX compile features
– Detecting CXX compile features - done
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working C compiler: /usr/bin/cc - skipped
– Detecting C compile features
– Detecting C compile features - done
– Not forcing any particular BLAS to be found
– Performing Test COMPILER_WORKS
– Performing Test COMPILER_WORKS - Success
– Performing Test SUPPORT_GLIBCXX_USE_C99
– Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
– Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
– Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
– std::exception_ptr is supported.
– Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
– Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
– Turning off deprecation warning due to glog.
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
– Current compiler supports avx2 extension. Will build perfkernels.
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
– Current compiler supports avx512f extension. Will build fbgemm.
– Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
– Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
– Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
– Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
– Performing Test COMPILER_SUPPORTS_RDYNAMIC
– Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
– Building using own protobuf under third_party per request.
– Use custom protobuf build.

– 3.11.4.0
– Looking for pthread.h
– Looking for pthread.h - found
– Performing Test CMAKE_HAVE_LIBC_PTHREAD
– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
– Check if compiler accepts -pthread
– Check if compiler accepts -pthread - yes
– Found Threads: TRUE
– Performing Test protobuf_HAVE_BUILTIN_ATOMICS
– Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success
– Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/laurent/Download/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
– Trying to find preferred BLAS backend of choice: MKL
– MKL_THREADING = OMP
– Looking for sys/types.h
– Looking for sys/types.h - found
– Looking for stdint.h
– Looking for stdint.h - found
– Looking for stddef.h
– Looking for stddef.h - found
– Check size of void*
– Check size of void* - done
– MKL_THREADING = OMP
CMake Warning at cmake/Dependencies.cmake:150 (message):
MKL could not be found. Defaulting to Eigen
Call Stack (most recent call first):
CMakeLists.txt:533 (include)

CMake Warning at cmake/Dependencies.cmake:173 (message):
Preferred BLAS (MKL) cannot be found, now searching for a general BLAS
library
Call Stack (most recent call first):
CMakeLists.txt:533 (include)

– MKL_THREADING = OMP
– Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_intel - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_gf - mkl_intel_thread - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_intel - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_gnu_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_gf - mkl_intel_thread - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_gnu_thread - mkl_core - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_intel - mkl_intel_thread - mkl_core - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_gnu_thread - mkl_core - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_gf - mkl_intel_thread - mkl_core - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_sequential - mkl_core - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_sequential - mkl_core - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_sequential - mkl_core - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_sequential - mkl_core - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_core - gomp - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_core - gomp - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_core - iomp5 - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl_intel_lp64 - mkl_core - pthread - m - dl]
– Library mkl_intel_lp64: not found
– Checking for [mkl_intel - mkl_core - pthread - m - dl]
– Library mkl_intel: not found
– Checking for [mkl_gf_lp64 - mkl_core - pthread - m - dl]
– Library mkl_gf_lp64: not found
– Checking for [mkl_gf - mkl_core - pthread - m - dl]
– Library mkl_gf: not found
– Checking for [mkl - guide - pthread - m]
– Library mkl: not found
– MKL library not found
– Checking for [Accelerate]
– Library Accelerate: BLAS_Accelerate_LIBRARY-NOTFOUND
– Checking for [vecLib]
– Library vecLib: BLAS_vecLib_LIBRARY-NOTFOUND
– Found OpenBLAS libraries: /usr/lib64/libopenblas.so
– Found OpenBLAS include: /usr/include
– Performing Test BLAS_F2C_DOUBLE_WORKS
– Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
– Performing Test BLAS_F2C_FLOAT_WORKS
– Performing Test BLAS_F2C_FLOAT_WORKS - Success
– Performing Test BLAS_USE_CBLAS_DOT
– Performing Test BLAS_USE_CBLAS_DOT - Success
– Found a library with BLAS API (open). Full path: (/usr/lib64/libopenblas.so)
– The ASM compiler identification is GNU
– Found assembler: /usr/bin/cc
– Brace yourself, we are building NNPACK
– Performing Test NNPACK_ARCH_IS_X86_32
– Performing Test NNPACK_ARCH_IS_X86_32 - Failed
– Found PythonInterp: /usr/bin/python3 (found version β€œ3.6.12”)
– NNPACK backend is x86-64
CMake Deprecation Warning at third_party/googletest/CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a … suffix to tell
CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googlemock/CMakeLists.txt:42 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a … suffix to tell
CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googletest/CMakeLists.txt:49 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a … suffix to tell
CMake that the project does not need compatibility with older versions.

– Failed to find LLVM FileCheck
– Found Git: /usr/bin/git (found version β€œ2.26.2”)
– git Version: v1.4.0-505be96a
– Version: 1.4.0
– Performing Test HAVE_CXX_FLAG_STD_CXX11
– Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
– Performing Test HAVE_CXX_FLAG_WALL
– Performing Test HAVE_CXX_FLAG_WALL - Success
– Performing Test HAVE_CXX_FLAG_WEXTRA
– Performing Test HAVE_CXX_FLAG_WEXTRA - Success
– Performing Test HAVE_CXX_FLAG_WSHADOW
– Performing Test HAVE_CXX_FLAG_WSHADOW - Success
– Performing Test HAVE_CXX_FLAG_WERROR
– Performing Test HAVE_CXX_FLAG_WERROR - Success
– Performing Test HAVE_CXX_FLAG_PEDANTIC
– Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
– Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
– Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
– Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
– Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
– Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
– Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
– Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
– Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
– Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
– Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
– Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
– Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
– Performing Test HAVE_CXX_FLAG_WD654
– Performing Test HAVE_CXX_FLAG_WD654 - Failed
– Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
– Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
– Performing Test HAVE_CXX_FLAG_COVERAGE
– Performing Test HAVE_CXX_FLAG_COVERAGE - Success
– Performing Test HAVE_STD_REGEX
– Performing Test HAVE_STD_REGEX
– Performing Test HAVE_STD_REGEX – success
– Performing Test HAVE_GNU_POSIX_REGEX
– Performing Test HAVE_GNU_POSIX_REGEX
– Performing Test HAVE_GNU_POSIX_REGEX – failed to compile
– Performing Test HAVE_POSIX_REGEX
– Performing Test HAVE_POSIX_REGEX
– Performing Test HAVE_POSIX_REGEX – success
– Performing Test HAVE_STEADY_CLOCK
– Performing Test HAVE_STEADY_CLOCK
– Performing Test HAVE_STEADY_CLOCK – success
– Performing Test COMPILER_SUPPORTS_AVX512
– Performing Test COMPILER_SUPPORTS_AVX512 - Success
CMake Warning (dev) at /usr/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:424 (message):
The package name passed to find_package_handle_standard_args (OpenMP_C)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
third_party/fbgemm/CMakeLists.txt:59 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.

– Found OpenMP_C: -fopenmp (found version β€œ4.5”)
CMake Warning (dev) at /usr/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:424 (message):
The package name passed to find_package_handle_standard_args (OpenMP_CXX)
does not match the name of the calling package (OpenMP). This can lead to
problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Call Stack (most recent call first):
cmake/Modules/FindOpenMP.cmake:565 (find_package_handle_standard_args)
third_party/fbgemm/CMakeLists.txt:59 (find_package)
This warning is for project developers. Use -Wno-dev to suppress it.

– Found OpenMP_CXX: -fopenmp (found version β€œ4.5”)
– Found OpenMP: TRUE (found version β€œ4.5”)
CMake Warning at third_party/fbgemm/CMakeLists.txt:61 (message):
OpenMP found! OpenMP_C_INCLUDE_DIRS =

CMake Warning at third_party/fbgemm/CMakeLists.txt:136 (message):

CMake Warning at third_party/fbgemm/CMakeLists.txt:137 (message):
CMAKE_BUILD_TYPE = Release

CMake Warning at third_party/fbgemm/CMakeLists.txt:138 (message):
CMAKE_CXX_FLAGS_DEBUG is -g

CMake Warning at third_party/fbgemm/CMakeLists.txt:139 (message):
CMAKE_CXX_FLAGS_RELEASE is -O3 -DNDEBUG

CMake Warning at third_party/fbgemm/CMakeLists.txt:140 (message):

– Performing Test __CxxFlag__fno_threadsafe_statics
– Performing Test __CxxFlag__fno_threadsafe_statics - Success
– Performing Test __CxxFlag__fno_semantic_interposition
– Performing Test __CxxFlag__fno_semantic_interposition - Success
– Performing Test __CxxFlag__fmerge_all_constants
– Performing Test __CxxFlag__fmerge_all_constants - Success
** AsmJit Summary **
ASMJIT_DIR=/home/laurent/Download/pytorch/third_party/fbgemm/third_party/asmjit
ASMJIT_TEST=FALSE
ASMJIT_TARGET_TYPE=STATIC
ASMJIT_DEPS=pthread;rt
ASMJIT_LIBS=asmjit;pthread;rt
ASMJIT_CFLAGS=-DASMJIT_STATIC
ASMJIT_PRIVATE_CFLAGS=-Wall;-Wextra;-fno-math-errno;-fno-threadsafe-statics;-fno-semantic-interposition;-DASMJIT_STATIC
ASMJIT_PRIVATE_CFLAGS_DBG=
ASMJIT_PRIVATE_CFLAGS_REL=-O2;-fmerge-all-constants
– Found Numa: /usr/include
– Found Numa (include: /usr/include, library: /usr/lib64/libnuma.so)
– Using third party subdirectory Eigen.
– Found PythonInterp: /usr/bin/python3 (found suitable version β€œ3.6.12”, minimum required is β€œ3.0”)
– Found PythonLibs: /usr/lib64/libpython3.6m.so.1.0 (Required is at least version β€œ3.0”)
CMake Error at cmake/Dependencies.cmake:942 (if):
if given arguments:

"VERSION_LESS" "3"

Unknown arguments specified
Call Stack (most recent call first):
CMakeLists.txt:533 (include)

– Configuring incomplete, errors occurred!
See also β€œ/home/laurent/Download/pytorch/build/CMakeFiles/CMakeOutput.log”.
See also β€œ/home/laurent/Download/pytorch/build/CMakeFiles/CMakeError.log”.
Building wheel torch-1.8.0a0
– Building version 1.8.0a0
cmake -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/laurent/Download/pytorch/torch -DCMAKE_PREFIX_PATH=/usr -DJAVA_HOME=/usr/lib64/jvm/jre-11-openjdk -DNUMPY_INCLUDE_DIR=/usr/lib64/python3.6/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6m -DPYTHON_LIBRARY=/usr/lib64/libpython3.6m.so.1.0 -DTORCH_BUILD_VERSION=1.8.0a0 -DUSE_NUMPY=True /home/laurent/Download/pytorch
Traceback (most recent call last):
File β€œsetup.py”, line 773, in
build_deps()
File β€œsetup.py”, line 320, in build_deps
cmake=cmake)
File β€œ/home/laurent/Download/pytorch/tools/build_pytorch_libs.py”, line 55, in build_caffe2
rerun_cmake)
File β€œ/home/laurent/Download/pytorch/tools/setup_helpers/cmake.py”, line 330, in generate
self.run(args, env=my_env)
File β€œ/home/laurent/Download/pytorch/tools/setup_helpers/cmake.py”, line 141, in run
check_call(command, cwd=self.build_dir, env=env)
File β€œ/usr/lib64/python3.6/subprocess.py”, line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command β€˜[β€˜cmake’, β€˜-DBUILD_PYTHON=True’, β€˜-DBUILD_TEST=True’, β€˜-DCMAKE_BUILD_TYPE=Release’, β€˜-DCMAKE_INSTALL_PREFIX=/home/laurent/Download/pytorch/torch’, β€˜-DCMAKE_PREFIX_PATH=/usr’, β€˜-DJAVA_HOME=/usr/lib64/jvm/jre-11-openjdk’, β€˜-DNUMPY_INCLUDE_DIR=/usr/lib64/python3.6/site-packages/numpy/core/include’, β€˜-DPYTHON_EXECUTABLE=/usr/bin/python3’, β€˜-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m’, β€˜-DPYTHON_LIBRARY=/usr/lib64/libpython3.6m.so.1.0’, β€˜-DTORCH_BUILD_VERSION=1.8.0a0’, β€˜-DUSE_NUMPY=True’, β€˜/home/laurent/Download/pytorch’]’ returned non-zero exit status 1.
`

Ran into the same issue, you’re probably missing headers for libpython on your system, you can install them with something like sudo yum install python3-devel.x86_64.

Our CMake needs to be tweaked a bit, it seems that here the version string is empty despite PYTHONLIBS_FOUND being true: https://github.com/pytorch/pytorch/blob/db5e5b439c454d657cfa8f08a096cf68e203f2a8/cmake/Dependencies.cmake#L959

cc @malfet FYI

2 Likes

Thanks for your help; I installed the required dev packages and it worked like a charm.
The error message from CMake is just misleading here.

2 Likes