Help Pytorch build constantly failing, with "Configuring incomplete, errors occurred!"

So I used to be able to build PyTorch pretty easily but this time I’m really stuck. In a real pickle right now help will be greatly appreciated.

OS: Ubuntu 20.04
GPU: Nvidia Geforce GT 750m
gcc/g++: 8.4.0

Following are the commands I am using to build:

# pytorch-build is by build environment
conda create --name pytorch-build python numpy

conda activate pytorch-build

conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi

conda install -c pytorch magma-cuda102

# now cd into the pytorch repo
cd pytorch/

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
export USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1 TORCH_CUDA_ARCH_LIST="3.0

# finally install
python setup.py install

This fails with the following error log:

Building wheel torch-1.6.0a0+9111ae7
-- Building version 1.6.0a0+9111ae7
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/rafay/pytorch/torch -DCMAKE_PREFIX_PATH=/home/rafay/anaconda3/envs/pytorch-build -DNUMPY_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/home/rafay/anaconda3/envs/pytorch-build/bin/python -DPYTHON_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build/include/python3.8 -DPYTHON_LIBRARY=/home/rafay/anaconda3/envs/pytorch-build/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.6.0a0+9111ae7 -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_MKLDNN=1 -DUSE_NUMPY=True /home/rafay/pytorch
-- std::exception_ptr is supported.
-- Turning off deprecation warning due to glog.
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- 
-- 3.11.4.0
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/rafay/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- MKL_THREADING = OMP
-- MKL libraries: /home/rafay/anaconda3/envs/pytorch-build/lib/libmkl_intel_lp64.so;/home/rafay/anaconda3/envs/pytorch-build/lib/libmkl_gnu_thread.so;/home/rafay/anaconda3/envs/pytorch-build/lib/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so
-- MKL include directory: /home/rafay/anaconda3/envs/pytorch-build/include
-- MKL OpenMP type: GNU
-- MKL OpenMP library: -fopenmp
-- Brace yourself, we are building NNPACK
-- Found PythonInterp: /home/rafay/anaconda3/envs/pytorch-build/bin/python (found version "3.8.3") 
-- NNPACK backend is x86-64
-- Failed to find LLVM FileCheck
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK -- success
CMake Warning at third_party/fbgemm/CMakeLists.txt:61 (message):
  OpenMP found! OpenMP_C_INCLUDE_DIRS =


CMake Warning at third_party/fbgemm/CMakeLists.txt:130 (message):
  ==========


CMake Warning at third_party/fbgemm/CMakeLists.txt:131 (message):
  CMAKE_BUILD_TYPE = Release


CMake Warning at third_party/fbgemm/CMakeLists.txt:132 (message):
  CMAKE_CXX_FLAGS_DEBUG is -g


CMake Warning at third_party/fbgemm/CMakeLists.txt:133 (message):
  CMAKE_CXX_FLAGS_RELEASE is -O3 -DNDEBUG


CMake Warning at third_party/fbgemm/CMakeLists.txt:134 (message):
  ==========


** AsmJit Summary **
   ASMJIT_DIR=/home/rafay/pytorch/third_party/fbgemm/third_party/asmjit
   ASMJIT_TEST=FALSE
   ASMJIT_TARGET_TYPE=STATIC
   ASMJIT_DEPS=pthread;rt
   ASMJIT_LIBS=asmjit;pthread;rt
   ASMJIT_CFLAGS=-DASMJIT_STATIC
   ASMJIT_PRIVATE_CFLAGS=-Wall;-Wextra;-fno-math-errno;-fno-threadsafe-statics;-fno-semantic-interposition;-DASMJIT_STATIC
   ASMJIT_PRIVATE_CFLAGS_DBG=
   ASMJIT_PRIVATE_CFLAGS_REL=-O2;-fmerge-all-constants
-- Using third party subdirectory Eigen.
-- Found PythonInterp: /home/rafay/anaconda3/envs/pytorch-build/bin/python (found suitable version "3.8.3", minimum required is "3.0") 
-- Using third_party/pybind11.
-- pybind11 include dirs: /home/rafay/pytorch/cmake/../third_party/pybind11/include
-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr (found version "10.1") 
-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr
-- Caffe2: Header version is: 10.1
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) 
CMake Warning at cmake/public/cuda.cmake:109 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1060 (include)
  CMakeLists.txt:469 (include)


CMake Warning at cmake/public/utils.cmake:196 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  cmake/public/cuda.cmake:452 (torch_cuda_get_nvcc_gencode_flag)
  cmake/Dependencies.cmake:1060 (include)
  CMakeLists.txt:469 (include)


-- Added CUDA NVCC flags for: -gencode;arch=compute_30,code=sm_30
CMake Warning at cmake/public/utils.cmake:196 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  cmake/External/nccl.cmake:13 (torch_cuda_get_nvcc_gencode_flag)
  cmake/Dependencies.cmake:1182 (include)
  CMakeLists.txt:469 (include)


CMake Warning at cmake/External/nccl.cmake:65 (message):
  Objcopy version is too old to support NCCL library slimming
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1182 (include)
  CMakeLists.txt:469 (include)


-- Could NOT find CUB (missing: CUB_INCLUDE_DIR) 
-- Found CUDA: /usr (found suitable version "10.1", minimum required is "7.0") 
-- CUDA detected: 10.1
-- Could NOT find NCCL (missing: NCCL_INCLUDE_DIR NCCL_LIBRARY) 
CMake Warning at third_party/gloo/cmake/Dependencies.cmake:96 (message):
  Not compiling with NCCL support.  Suppress this warning with
  -DUSE_NCCL=OFF.
Call Stack (most recent call first):
  third_party/gloo/CMakeLists.txt:75 (include)


Generated: /home/rafay/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.proto
Generated: /home/rafay/pytorch/build/third_party/onnx/onnx/onnx-operators_onnx_torch-ml.proto
-- 
-- ******** Summary ********
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 8.4.0
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNXIFI_ENABLE_EXT=1
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/rafay/pytorch/cmake/Modules;/home/rafay/pytorch/cmake/public/../Modules_CUDA_fix
-- 
--   ONNX version          : 1.7.0
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
--   ONNXIFI_ENABLE_EXT    : OFF
-- 
--   Protobuf compiler     : 
--   Protobuf includes     : 
--   Protobuf libraries    : 
--   BUILD_ONNX_PYTHON     : OFF
-- 
-- ******** Summary ********
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 8.4.0
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNXIFI_ENABLE_EXT=1
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/rafay/pytorch/cmake/Modules;/home/rafay/pytorch/cmake/public/../Modules_CUDA_fix
-- 
--   ONNX version          : 1.4.1
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
-- 
--   Protobuf compiler     : 
--   Protobuf includes     : 
--   Protobuf libraries    : 
--   BUILD_ONNX_PYTHON     : OFF
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
-- Adding -DNDEBUG to compile flags
-- Compiling with MAGMA support
-- MAGMA INCLUDE DIRECTORIES: /home/rafay/anaconda3/envs/pytorch-build/include
-- MAGMA LIBRARIES: /home/rafay/anaconda3/envs/pytorch-build/lib/libmagma.a
-- MAGMA V2 check: 1
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- AVX compiler support found
-- AVX2 compiler support found
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API (mkl).
-- USE_CUDNN is set to 0. Compiling without cuDNN support
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
-- MKLDNN_CPU_RUNTIME = OMP
-- Using the obsolete way to specify the CPU runtime. Use DNNL_CPU_RUNTIME= instead.
-- GPU support is disabled
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
-- Primitive cache is enabled
-- Found MKL-DNN: TRUE
-- Version: 6.2.0
-- Build type: Release
-- CXX_STANDARD: 14
-- Required features: cxx_variadic_templates
-- GCC 8.4.0: Adding gcc and gcc_s libs to link line
-- don't use NUMA
-- Configuring build for SLEEF-v3.4.0
   Target system: Linux-5.4.0-37-generic
   Target processor: x86_64
   Host system: Linux-5.4.0-37-generic
   Host processor: x86_64
   Detected C compiler: GNU @ /usr/bin/cc
-- Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
-- Building shared libs : OFF
-- MPFR : LIB_MPFR-NOTFOUND
-- GMP : LIBGMP-NOTFOUND
-- RT : /usr/lib/x86_64-linux-gnu/librt.so
-- FFTW3 : LIBFFTW3-NOTFOUND
-- OPENSSL : 1.1.1g
-- SDE : SDE_COMMAND-NOTFOUND
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
AT_INSTALL_INCLUDE_DIR include/ATen/core
core header install: /home/rafay/pytorch/build/aten/src/ATen/core/TensorBody.h
-- Include NCCL operators
-- Excluding FakeLowP operators
-- Including IDEEP operators
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- MPI operators skipped due to no MPI support
-- Include Observer library
CMake Error at test/cpp/api/CMakeLists.txt:39 (add_executable):
  add_executable cannot create target "test_api" because another target with
  the same name already exists.  The existing target is an executable created
  in source directory
  "/home/rafay/pytorch/third_party/ideep/mkl-dnn/tests/gtests/api".  See
  documentation for policy CMP0002 for more details.


CMake Error at test/cpp/api/CMakeLists.txt:41 (target_link_libraries):
  The plain signature for target_link_libraries has already been used with
  the target "test_api".  All uses of target_link_libraries with a target
  must be either all-keyword or all-plain.

  The uses of the plain signature are here:

   * third_party/ideep/mkl-dnn/cmake/utils.cmake:45 (target_link_libraries)



CMake Error at test/cpp/api/CMakeLists.txt:44 (target_link_libraries):
  Attempt to add link library "/usr/lib/x86_64-linux-gnu/libcudart.so" to
  target "test_api" which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.


-- /usr/bin/c++ /home/rafay/pytorch/caffe2/../torch/abi-check.cpp -o /home/rafay/pytorch/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
-- pytorch is compiling with OpenMP. 
OpenMP CXX_FLAGS: -fopenmp. 
OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so.
-- Caffe2 is compiling with OpenMP. 
OpenMP CXX_FLAGS: -fopenmp. 
OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so.
-- Using ATen parallel backend: OMP
-- Using lib/python3.7/site-packages as python relative installation path
CMake Warning at CMakeLists.txt:690 (message):
  Generated cmake files are only fully tested if one builds with system glog,
  gflags, and protobuf.  Other settings may generate files that are not well
  tested.


-- 
-- ******** Summary ********
-- General:
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler id       : GNU
--   C++ compiler version  : 8.4.0
--   BLAS                  : MKL
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;MAGMA_V2;IDEEP_USE_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
-- 
--   TORCH_VERSION         : 1.6.0
--   CAFFE2_VERSION        : 1.6.0
--   BUILD_CAFFE2_MOBILE   : OFF
--   USE_STATIC_DISPATCH   : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.8.3
--     Python executable   : /home/rafay/anaconda3/envs/pytorch-build/bin/python
--     Pythonlibs version  : 3.8.3
--     Python library      : /home/rafay/anaconda3/envs/pytorch-build/lib/libpython3.8.so.1.0
--     Python includes     : /home/rafay/anaconda3/envs/pytorch-build/include/python3.8
--     Python site-packages: lib/python3.8/site-packages
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   INTERN_BUILD_MOBILE   : 
--   USE_ASAN              : OFF
--   USE_CUDA              : 1
--     CUDA static link    : OFF
--     USE_CUDNN           : OFF
--     CUDA version        : 10.1
--     CUDA root directory : /usr
--     CUDA library        : /usr/lib/x86_64-linux-gnu/libcuda.so
--     cudart library      : /usr/lib/x86_64-linux-gnu/libcudart.so
--     cublas library      : /usr/lib/x86_64-linux-gnu/libcublas.so
--     cufft library       : /usr/lib/x86_64-linux-gnu/libcufft.so
--     curand library      : /usr/lib/x86_64-linux-gnu/libcurand.so
--     nvrtc               : /usr/lib/x86_64-linux-gnu/libnvrtc.so
--     CUDA include path   : /usr/include
--     NVCC executable     : /usr/bin/nvcc
--     NVCC flags          : -DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_30,code=sm_30;-Xcudafe;--diag_suppress=cc_clobber_ignored;-Xcudafe;--diag_suppress=integer_sign_change;-Xcudafe;--diag_suppress=useless_using_declaration;-Xcudafe;--diag_suppress=set_but_not_used;-Xcudafe;--diag_suppress=field_without_dll_interface;-Xcudafe;--diag_suppress=base_class_has_different_dll_interface;-Xcudafe;--diag_suppress=dll_interface_conflict_none_assumed;-Xcudafe;--diag_suppress=dll_interface_conflict_dllexport_assumed;-Xcudafe;--diag_suppress=implicit_return_from_non_void_function;-Xcudafe;--diag_suppress=unsigned_compare_with_zero;-Xcudafe;--diag_suppress=declared_but_not_referenced;-Xcudafe;--diag_suppress=bad_friend_decl;-std=c++14;-Xcompiler;-fPIC;--expt-relaxed-constexpr;--expt-extended-lambda;-Wno-deprecated-gpu-targets;--expt-extended-lambda;-gencode;arch=compute_30,code=sm_30;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__
--     CUDA host compiler  : /usr/bin/cc
--     NVCC --device-c     : OFF
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : 
--   USE_FBGEMM            : ON
--     USE_FAKELOWP          : OFF
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : 1
--   USE_MKLDNN_CBLAS      : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : OFF
--   USE_NNPACK            : ON
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_VULKAN            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_PYTORCH_QNNPACK   : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : OFF
--     USE_GLOO            : ON
--     USE_TENSORPIPE      : ON
--   Public Dependencies  : Threads::Threads;caffe2::mkl;caffe2::mkldnn
--   Private Dependencies : cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;fp16;gloo;tensorpipe;aten_op_header_gen;foxi_loader;rt;fmt::fmt-header-only;gcc_s;gcc;dl
-- Configuring incomplete, errors occurred!
See also "/home/rafay/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/home/rafay/pytorch/build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 732, in <module>
    build_deps()
  File "setup.py", line 311, in build_deps
    build_caffe2(version=version,
  File "/home/rafay/pytorch/tools/build_pytorch_libs.py", line 54, in build_caffe2
    cmake.generate(version,
  File "/home/rafay/pytorch/tools/setup_helpers/cmake.py", line 329, in generate
    self.run(args, env=my_env)
  File "/home/rafay/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/rafay/anaconda3/envs/pytorch-build/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/rafay/pytorch/torch', '-DCMAKE_PREFIX_PATH=/home/rafay/anaconda3/envs/pytorch-build', '-DNUMPY_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build/lib/python3.8/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/home/rafay/anaconda3/envs/pytorch-build/bin/python', '-DPYTHON_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build/include/python3.8', '-DPYTHON_LIBRARY=/home/rafay/anaconda3/envs/pytorch-build/lib/libpython3.8.so.1.0', '-DTORCH_BUILD_VERSION=1.6.0a0+9111ae7', '-DUSE_CUDA=1', '-DUSE_CUDNN=1', '-DUSE_MKLDNN=1', '-DUSE_NUMPY=True', '/home/rafay/pytorch']' returned non-zero exit status 1.

You might want to uninstall the pytorch you already have installed? Also clean the repo with python setup.py clean after that.
Also if the issue is still with test_api, you can try disabling building the tests with BUILD_TEST=0.

1 Like

@albanD on one of the PCs I finally got it working using the master branch of PyTorch (don’t know if python setup.py clean helped or not), while on the other machine checking out the stable version git checkout v1.5.0 I can’t seem to get past the above error. Any idea why?

Ho,

the log you send is building from master.
Can you share the log for the 1.5.0 build with no pytorch pre-installed and after cleaning?

@albanD Here is is the build log when I check out v1.5.0. It fails configuration steps when I use python setup.py bdist_wheel:

Building wheel torch-1.5.0a0+4ff3872
-- Building version 1.5.0a0+4ff3872
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/rafay/pytorch/torch -DCMAKE_PREFIX_PATH=/home/rafay/anaconda3/envs/pytorch-build-env -DNUMPY_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build-env/lib/python3.7/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/home/rafay/anaconda3/envs/pytorch-build-env/bin/python -DPYTHON_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build-env/include/python3.7m -DPYTHON_LIBRARY=/home/rafay/anaconda3/envs/pytorch-build-env/lib/libpython3.7m.so.1.0 -DTORCH_BUILD_VERSION=1.5.0a0+4ff3872 -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_MKLDNN=1 -DUSE_NUMPY=True /home/rafay/pytorch
-- The CXX compiler identification is GNU 8.4.0
-- The C compiler identification is GNU 8.4.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Not forcing any particular BLAS to be found
-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
-- Turning off deprecation warning due to glog.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- 
-- 3.11.4.0
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS
-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/rafay/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- MKL_THREADING = OMP
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Looking for cblas_sgemm
-- Looking for cblas_sgemm - found
-- MKL libraries: /home/rafay/anaconda3/envs/pytorch-build-env/lib/libmkl_intel_lp64.so;/home/rafay/anaconda3/envs/pytorch-build-env/lib/libmkl_gnu_thread.so;/home/rafay/anaconda3/envs/pytorch-build-env/lib/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so
-- MKL include directory: /home/rafay/anaconda3/envs/pytorch-build-env/include
-- MKL OpenMP type: GNU
-- MKL OpenMP library: -fopenmp
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Brace yourself, we are building NNPACK
-- Performing Test NNPACK_ARCH_IS_X86_32
-- Performing Test NNPACK_ARCH_IS_X86_32 - Failed
-- Found PythonInterp: /home/rafay/anaconda3/envs/pytorch-build-env/bin/python (found version "3.7.7") 
-- NNPACK backend is x86-64
-- Failed to find LLVM FileCheck
-- Found Git: /usr/bin/git (found version "2.25.1") 
-- git Version: v1.4.0-505be96a
-- Version: 1.4.0
-- Performing Test HAVE_CXX_FLAG_STD_CXX11
-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WSHADOW
-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success
-- Performing Test HAVE_CXX_FLAG_WERROR
-- Performing Test HAVE_CXX_FLAG_WERROR - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS
-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32
-- Performing Test HAVE_CXX_FLAG_WSHORTEN_64_TO_32 - Failed
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL
-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS
-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING
-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success
-- Performing Test HAVE_CXX_FLAG_WD654
-- Performing Test HAVE_CXX_FLAG_WD654 - Failed
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY
-- Performing Test HAVE_CXX_FLAG_WTHREAD_SAFETY - Failed
-- Performing Test HAVE_CXX_FLAG_COVERAGE
-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- success
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Performing Test COMPILER_SUPPORTS_AVX512
-- Performing Test COMPILER_SUPPORTS_AVX512 - Success
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
CMake Warning at third_party/fbgemm/CMakeLists.txt:61 (message):
  OpenMP found! OpenMP_C_INCLUDE_DIRS =


CMake Warning at third_party/fbgemm/CMakeLists.txt:130 (message):
  ==========


CMake Warning at third_party/fbgemm/CMakeLists.txt:131 (message):
  CMAKE_BUILD_TYPE = Release


CMake Warning at third_party/fbgemm/CMakeLists.txt:132 (message):
  CMAKE_CXX_FLAGS_DEBUG is -g


CMake Warning at third_party/fbgemm/CMakeLists.txt:133 (message):
  CMAKE_CXX_FLAGS_RELEASE is -O3 -DNDEBUG


CMake Warning at third_party/fbgemm/CMakeLists.txt:134 (message):
  ==========


-- Performing Test __CxxFlag__fno_threadsafe_statics
-- Performing Test __CxxFlag__fno_threadsafe_statics - Success
-- Performing Test __CxxFlag__fno_semantic_interposition
-- Performing Test __CxxFlag__fno_semantic_interposition - Success
-- Performing Test __CxxFlag__fmerge_all_constants
-- Performing Test __CxxFlag__fmerge_all_constants - Success
** AsmJit Summary **
   ASMJIT_DIR=/home/rafay/pytorch/third_party/fbgemm/third_party/asmjit
   ASMJIT_TEST=FALSE
   ASMJIT_TARGET_TYPE=STATIC
   ASMJIT_DEPS=pthread;rt
   ASMJIT_LIBS=asmjit;pthread;rt
   ASMJIT_CFLAGS=-DASMJIT_STATIC
   ASMJIT_PRIVATE_CFLAGS=-Wall;-Wextra;-fno-math-errno;-fno-threadsafe-statics;-fno-semantic-interposition;-DASMJIT_STATIC
   ASMJIT_PRIVATE_CFLAGS_DBG=
   ASMJIT_PRIVATE_CFLAGS_REL=-O2;-fmerge-all-constants
-- Could NOT find Numa (missing: Numa_INCLUDE_DIR Numa_LIBRARIES) 
CMake Warning at cmake/Dependencies.cmake:641 (message):
  Not compiling with NUMA.  Suppress this warning with -DUSE_NUMA=OFF
Call Stack (most recent call first):
  CMakeLists.txt:411 (include)


-- Using third party subdirectory Eigen.
Python 3.7.7
-- Found PythonInterp: /home/rafay/anaconda3/envs/pytorch-build-env/bin/python (found suitable version "3.7.7", minimum required is "2.7") 
-- Found PythonLibs: /home/rafay/anaconda3/envs/pytorch-build-env/lib/libpython3.7m.so.1.0 (found suitable version "3.7.7", minimum required is "2.7") 
-- Could NOT find pybind11 (missing: pybind11_DIR)
-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR) 
-- Using third_party/pybind11.
-- pybind11 include dirs: /home/rafay/pytorch/cmake/../third_party/pybind11/include
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS) 
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS) 
-- Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND) 
CMake Warning at cmake/Dependencies.cmake:877 (message):
  Not compiling with MPI.  Suppress this warning with -DUSE_MPI=OFF
Call Stack (most recent call first):
  CMakeLists.txt:411 (include)


-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr (found version "10.1") 
-- Caffe2: CUDA detected: 10.1
-- Caffe2: CUDA nvcc is: /usr/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr
-- Caffe2: Header version is: 10.1
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH) 
CMake Warning at cmake/public/cuda.cmake:109 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  cmake/Dependencies.cmake:956 (include)
  CMakeLists.txt:411 (include)


CMake Warning at cmake/public/utils.cmake:196 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  cmake/public/cuda.cmake:420 (torch_cuda_get_nvcc_gencode_flag)
  cmake/Dependencies.cmake:956 (include)
  CMakeLists.txt:411 (include)


-- Added CUDA NVCC flags for: -gencode;arch=compute_30,code=sm_30
CMake Warning at cmake/public/utils.cmake:196 (message):
  In the future we will require one to explicitly pass TORCH_CUDA_ARCH_LIST
  to cmake instead of implicitly setting it as an env variable.  This will
  become a FATAL_ERROR in future version of pytorch.
Call Stack (most recent call first):
  cmake/External/nccl.cmake:13 (torch_cuda_get_nvcc_gencode_flag)
  cmake/Dependencies.cmake:1074 (include)
  CMakeLists.txt:411 (include)


CMake Warning at cmake/External/nccl.cmake:65 (message):
  Objcopy version is too old to support NCCL library slimming
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1074 (include)
  CMakeLists.txt:411 (include)


-- Could NOT find CUB (missing: CUB_INCLUDE_DIR) 
CMake Warning (dev) at third_party/gloo/CMakeLists.txt:21 (option):
  Policy CMP0077 is not set: option() honors normal variables.  Run "cmake
  --help-policy CMP0077" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  For compatibility with older versions of CMake, option is clearing the
  normal variable 'BUILD_BENCHMARK'.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found CUDA: /usr (found suitable version "10.1", minimum required is "7.0") 
-- CUDA detected: 10.1
-- Could NOT find NCCL (missing: NCCL_INCLUDE_DIR NCCL_LIBRARY) 
CMake Warning at third_party/gloo/cmake/Dependencies.cmake:96 (message):
  Not compiling with NCCL support.  Suppress this warning with
  -DUSE_NCCL=OFF.
Call Stack (most recent call first):
  third_party/gloo/CMakeLists.txt:75 (include)


CMake Warning at cmake/Dependencies.cmake:1159 (message):
  Metal is only used in ios builds.
Call Stack (most recent call first):
  CMakeLists.txt:411 (include)


Generated: /home/rafay/pytorch/build/third_party/onnx/onnx/onnx_onnx_torch-ml.proto
Generated: /home/rafay/pytorch/build/third_party/onnx/onnx/onnx-operators_onnx_torch-ml.proto
-- 
-- ******** Summary ********
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build-env/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 8.4.0
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build-env;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/rafay/pytorch/cmake/Modules;/home/rafay/pytorch/cmake/public/../Modules_CUDA_fix
-- 
--   ONNX version          : 1.7.0
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
--   ONNXIFI_ENABLE_EXT    : OFF
-- 
--   Protobuf compiler     : 
--   Protobuf includes     : 
--   Protobuf libraries    : 
--   BUILD_ONNX_PYTHON     : OFF
-- 
-- ******** Summary ********
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build-env/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 8.4.0
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build-env;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/rafay/pytorch/cmake/Modules;/home/rafay/pytorch/cmake/public/../Modules_CUDA_fix
-- 
--   ONNX version          : 1.4.1
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
-- 
--   Protobuf compiler     : 
--   Protobuf includes     : 
--   Protobuf libraries    : 
--   BUILD_ONNX_PYTHON     : OFF
-- Found CUDA with FP16 support, compiling with torch.cuda.HalfTensor
-- Adding -DNDEBUG to compile flags
-- Checking prototype magma_get_sgeqrf_nb for MAGMA_V2 - True
-- Compiling with MAGMA support
-- MAGMA INCLUDE DIRECTORIES: /home/rafay/anaconda3/envs/pytorch-build-env/include
-- MAGMA LIBRARIES: /home/rafay/anaconda3/envs/pytorch-build-env/lib/libmagma.a
-- MAGMA V2 check: 1
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Performing Test HAVE_GCC_GET_CPUID
-- Performing Test HAVE_GCC_GET_CPUID - Success
-- Performing Test NO_GCC_EBX_FPIC_BUG
-- Performing Test NO_GCC_EBX_FPIC_BUG - Success
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- AVX compiler support found
-- AVX2 compiler support found
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API (mkl).
-- USE_CUDNN is set to 0. Compiling without cuDNN support
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
-- MKLDNN_THREADING = OMP:COMP
-- MKL-DNN compat: set DNNL_ENABLE_CONCURRENT_EXEC to MKLDNN_ENABLE_CONCURRENT_EXEC with value `1`
-- MKL-DNN compat: set DNNL_LIBRARY_TYPE to MKLDNN_LIBRARY_TYPE with value `STATIC`
-- Using the obsolete way to specify the CPU runtime. Use DNNL_CPU_RUNTIME= instead.
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- GPU support is disabled
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
Failed to include MKL-DNN target
CMake Warning at cmake/Dependencies.cmake:1459 (MESSAGE):
  MKLDNN could not be found.
Call Stack (most recent call first):
  CMakeLists.txt:411 (include)


-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for shm_open
-- Looking for shm_open - found
-- Looking for shm_unlink
-- Looking for shm_unlink - found
-- Looking for malloc_usable_size
-- Looking for malloc_usable_size - found
-- Performing Test C_HAS_THREAD
-- Performing Test C_HAS_THREAD - Success
-- GCC 8.4.0: Adding gcc and gcc_s libs to link line
-- Performing Test HAS_WERROR_FORMAT
-- Performing Test HAS_WERROR_FORMAT - Success
-- don't use NUMA
-- Performing Test COMPILER_SUPPORTS_NO_AVX256_SPLIT
-- Performing Test COMPILER_SUPPORTS_NO_AVX256_SPLIT - Success
-- Found OpenSSL: /home/rafay/anaconda3/envs/pytorch-build-env/lib/libcrypto.so (found version "1.1.1g")  
-- Check size of long double
-- Check size of long double - done
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE - Success
-- Performing Test COMPILER_SUPPORTS_FLOAT128
-- Performing Test COMPILER_SUPPORTS_FLOAT128 - Success
-- Performing Test COMPILER_SUPPORTS_SSE2
-- Performing Test COMPILER_SUPPORTS_SSE2 - Success
-- Performing Test COMPILER_SUPPORTS_SSE4
-- Performing Test COMPILER_SUPPORTS_SSE4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX
-- Performing Test COMPILER_SUPPORTS_AVX - Success
-- Performing Test COMPILER_SUPPORTS_FMA4
-- Performing Test COMPILER_SUPPORTS_FMA4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX2
-- Performing Test COMPILER_SUPPORTS_AVX2 - Success
-- Performing Test COMPILER_SUPPORTS_AVX512F
-- Performing Test COMPILER_SUPPORTS_AVX512F - Success
-- Performing Test COMPILER_SUPPORTS_OPENMP
-- Performing Test COMPILER_SUPPORTS_OPENMP - Success
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES - Success
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH - Success
-- Performing Test COMPILER_SUPPORTS_SYS_GETRANDOM
-- Performing Test COMPILER_SUPPORTS_SYS_GETRANDOM - Success
-- Configuring build for SLEEF-v3.4.0
   Target system: Linux-5.4.0-37-generic
   Target processor: x86_64
   Host system: Linux-5.4.0-37-generic
   Host processor: x86_64
   Detected C compiler: GNU @ /usr/bin/cc
-- Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
-- Building shared libs : OFF
-- MPFR : LIB_MPFR-NOTFOUND
-- GMP : LIBGMP-NOTFOUND
-- RT : /usr/lib/x86_64-linux-gnu/librt.so
-- FFTW3 : LIBFFTW3-NOTFOUND
-- OPENSSL : 1.1.1g
-- SDE : SDE_COMMAND-NOTFOUND
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
AT_INSTALL_INCLUDE_DIR include/ATen/core
core header install: /home/rafay/pytorch/build/aten/src/ATen/core/TensorBody.h
core header install: /home/rafay/pytorch/build/aten/src/ATen/core/TensorMethods.h
-- Include NCCL operators
-- Excluding ideep operators as we are not using ideep
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- MPI operators skipped due to no MPI support
-- Include Observer library
CMake Error at test/cpp/api/CMakeLists.txt:39 (add_executable):
  add_executable cannot create target "test_api" because another target with
  the same name already exists.  The existing target is an executable created
  in source directory
  "/home/rafay/pytorch/third_party/ideep/mkl-dnn/tests/gtests/api".  See
  documentation for policy CMP0002 for more details.


CMake Error at test/cpp/api/CMakeLists.txt:41 (target_link_libraries):
  The plain signature for target_link_libraries has already been used with
  the target "test_api".  All uses of target_link_libraries with a target
  must be either all-keyword or all-plain.

  The uses of the plain signature are here:

   * third_party/ideep/mkl-dnn/cmake/utils.cmake:45 (target_link_libraries)



CMake Error at test/cpp/api/CMakeLists.txt:44 (target_link_libraries):
  Attempt to add link library "/usr/lib/x86_64-linux-gnu/libcudart.so" to
  target "test_api" which is not built in this directory.

  This is allowed only when policy CMP0079 is set to NEW.


-- /usr/bin/c++ /home/rafay/pytorch/caffe2/../torch/abi-check.cpp -o /home/rafay/pytorch/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
-- pytorch is compiling with OpenMP. 
OpenMP CXX_FLAGS: -fopenmp. 
OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so.
-- Caffe2 is compiling with OpenMP. 
OpenMP CXX_FLAGS: -fopenmp. 
OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/8/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so.
-- Using ATen parallel backend: OMP
-- Using lib/python3.7/site-packages as python relative installation path
CMake Warning at CMakeLists.txt:619 (message):
  Generated cmake files are only fully tested if one builds with system glog,
  gflags, and protobuf.  Other settings may generate files that are not well
  tested.


-- 
-- ******** Summary ********
-- General:
--   CMake version         : 3.14.0
--   CMake command         : /home/rafay/anaconda3/envs/pytorch-build-env/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler id       : GNU
--   C++ compiler version  : 8.4.0
--   BLAS                  : MKL
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_INTERNAL_THREADPOOL_IMPL -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNX_NAMESPACE=onnx_torch;MAGMA_V2;IDEEP_USE_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : /home/rafay/anaconda3/envs/pytorch-build-env;/usr
--   CMAKE_INSTALL_PREFIX  : /home/rafay/pytorch/torch
-- 
--   TORCH_VERSION         : 1.5.0
--   CAFFE2_VERSION        : 1.5.0
--   BUILD_CAFFE2_MOBILE   : OFF
--   USE_STATIC_DISPATCH   : OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.7.7
--     Python executable   : /home/rafay/anaconda3/envs/pytorch-build-env/bin/python
--     Pythonlibs version  : 3.7.7
--     Python library      : /home/rafay/anaconda3/envs/pytorch-build-env/lib/libpython3.7m.so.1.0
--     Python includes     : /home/rafay/anaconda3/envs/pytorch-build-env/include/python3.7m
--     Python site-packages: lib/python3.7/site-packages
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   INTERN_BUILD_MOBILE   : 
--   USE_ASAN              : OFF
--   USE_CUDA              : 1
--     CUDA static link    : OFF
--     USE_CUDNN           : OFF
--     CUDA version        : 10.1
--     CUDA root directory : /usr
--     CUDA library        : /usr/lib/x86_64-linux-gnu/libcuda.so
--     cudart library      : /usr/lib/x86_64-linux-gnu/libcudart.so
--     cublas library      : /usr/lib/x86_64-linux-gnu/libcublas.so
--     cufft library       : /usr/lib/x86_64-linux-gnu/libcufft.so
--     curand library      : /usr/lib/x86_64-linux-gnu/libcurand.so
--     nvrtc               : /usr/lib/x86_64-linux-gnu/libnvrtc.so
--     CUDA include path   : /usr/include
--     NVCC executable     : /usr/bin/nvcc
--     NVCC flags          : -DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_30,code=sm_30;-Xcudafe;--diag_suppress=cc_clobber_ignored;-Xcudafe;--diag_suppress=integer_sign_change;-Xcudafe;--diag_suppress=useless_using_declaration;-Xcudafe;--diag_suppress=set_but_not_used;-std=c++14;-Xcompiler;-fPIC;--expt-relaxed-constexpr;--expt-extended-lambda;-Wno-deprecated-gpu-targets;--expt-extended-lambda;-gencode;arch=compute_30,code=sm_30;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__
--     CUDA host compiler  : /usr/bin/cc
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : 
--   USE_FBGEMM            : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : 1
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : OFF
--   USE_NNPACK            : ON
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_PYTORCH_QNNPACK   : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : OFF
--     USE_GLOO            : ON
--   Public Dependencies  : Threads::Threads;caffe2::mkl
--   Private Dependencies : cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;fp16;gloo;aten_op_header_gen;foxi_loader;rt;gcc_s;gcc;dl
-- Configuring incomplete, errors occurred!
See also "/home/rafay/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/home/rafay/pytorch/build/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 745, in <module>
    build_deps()
  File "setup.py", line 316, in build_deps
    cmake=cmake)
  File "/home/rafay/pytorch/tools/build_pytorch_libs.py", line 59, in build_caffe2
    rerun_cmake)
  File "/home/rafay/pytorch/tools/setup_helpers/cmake.py", line 323, in generate
    self.run(args, env=my_env)
  File "/home/rafay/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/rafay/anaconda3/envs/pytorch-build-env/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/rafay/pytorch/torch', '-DCMAKE_PREFIX_PATH=/home/rafay/anaconda3/envs/pytorch-build-env', '-DNUMPY_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build-env/lib/python3.7/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/home/rafay/anaconda3/envs/pytorch-build-env/bin/python', '-DPYTHON_INCLUDE_DIR=/home/rafay/anaconda3/envs/pytorch-build-env/include/python3.7m', '-DPYTHON_LIBRARY=/home/rafay/anaconda3/envs/pytorch-build-env/lib/libpython3.7m.so.1.0', '-DTORCH_BUILD_VERSION=1.5.0a0+4ff3872', '-DUSE_CUDA=1', '-DUSE_CUDNN=1', '-DUSE_MKLDNN=1', '-DUSE_NUMPY=True', '/home/rafay/pytorch']' returned non-zero exit status 1.

Okay so I have been racking my brain trying to figure out why sometimes the build successfully passes and why sometimes it doesn’t. This apparently happens when “mkldnn is out of sync”, the following few git commands solves this problem:

git submodule sync
git submodule update --init --recursive

source: https://github.com/pytorch/pytorch/issues/35602

3 Likes