Build from source failed in linux

Nazmul_Alom · July 28, 2020, 7:09am

I am using Ubuntu 20.04
My GPU: GTX 710
conda 4.8.3
Python 3.7.7
I am following the instruction from Github to install it from source. But when I run this command:

python setup.py install

The following error occured:

Traceback (most recent call last):
  File "setup.py", line 737, in <module>
    build_deps()
  File "setup.py", line 321, in build_deps
    cmake=cmake)
  File "/home/nazmul/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/nazmul/miniconda3/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '6']' returned non-zero exit status 1.

ptrblck · July 28, 2020, 10:03am

Could you post the build log directly by wrapping it into three backticks ```?
I think the error might have been raised before the actual failure and your screenshot doesn’t seem to show it.

Nazmul_Alom · July 29, 2020, 4:33pm

@ptrblck this is the log

Building wheel torch-1.7.0a0+d6f1346
-- Building version 1.7.0a0+d6f1346
cmake --build . --target install --config Release -- -j 6
[2/4962] Performing build step for 'nccl_external'
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a 
cd /home/nazmul/pytorch/third_party/nccl/nccl && env CCACHE_DISABLE=1 SCCACHE_DISABLE=1 make CXX=/usr/bin/c++ CUDA_HOME=/usr NVCC=/usr/bin/nvcc NVCC_GENCODE=-gencode=arch=compute_35,code=sm_35 BUILDDIR=/home/nazmul/pytorch/build/nccl VERBOSE=0 -j && /home/nazmul/miniconda3/bin/cmake -E touch /home/nazmul/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build
make -C src build BUILDDIR=/home/nazmul/pytorch/build/nccl
make[1]: Entering directory '/home/nazmul/pytorch/third_party/nccl/nccl/src'
make[2]: Entering directory '/home/nazmul/pytorch/third_party/nccl/nccl/src/collectives/device'
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/functions.dep] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/sendrecv.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/broadcast.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/reduce.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/all_gather.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/reduce_scatter.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/all_reduce.dep] Error 1
make[2]: Leaving directory '/home/nazmul/pytorch/third_party/nccl/nccl/src/collectives/device'
make[1]: *** [Makefile:50: /home/nazmul/pytorch/build/nccl/obj/collectives/device/colldevice.a] Error 2
make[1]: Leaving directory '/home/nazmul/pytorch/third_party/nccl/nccl/src'
make: *** [Makefile:25: src.build] Error 2
[7/4962] Building CXX object third_par.../src/google/protobuf/message_lite.cc.o
In file included from /usr/include/string.h:495,
                 from ../third_party/protobuf/src/google/protobuf/stubs/port.h:39,
                 from ../third_party/protobuf/src/google/protobuf/stubs/common.h:46,
                 from ../third_party/protobuf/src/google/protobuf/message_lite.h:45,
                 from ../third_party/protobuf/src/google/protobuf/message_lite.cc:36:
In function ‘void* memcpy(void*, const void*, size_t)’,
    inlined from ‘google::protobuf::uint8* google::protobuf::io::EpsCopyOutputStream::WriteRaw(const void*, int, google::protobuf::uint8*)’ at ../third_party/protobuf/src/google/protobuf/io/coded_stream.h:699:16,
    inlined from ‘bool google::protobuf::MessageLite::SerializePartialToZeroCopyStream(google::protobuf::io::ZeroCopyOutputStream*) const’ at ../third_party/protobuf/src/google/protobuf/implicit_weak_message.h:87:28:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34:33: warning: ‘void* __builtin___memcpy_chk(void*, const void*, long unsigned int, long unsigned int)’ specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
   34 |   return __builtin___memcpy_chk (__dest, __src, __len, __bos0 (__dest));
      |          ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 737, in <module>
    build_deps()
  File "setup.py", line 321, in build_deps
    cmake=cmake)
  File "/home/nazmul/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/nazmul/miniconda3/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '6']' returned non-zero exit status 1.

Usama_Hasan · July 29, 2020, 7:31pm

Hy @Nazmul_Alom.
It looks like your gcc version (C++ compiler) is incompatible with cuda.
You can check your gcc version through terminal like this.

gcc --version

and

sudo apt install gcc-7 g++-7

I guess this should help, if not you should probably have to set cmake to gcc specific version.

cmake  -D CMAKE_C_COMPILER=/path/to/ gcc/bin/gcc -D CMAKE_CXX_COMPILER=/path/to/gcc/bin/g++`

Nazmul_Alom · July 30, 2020, 6:45am

Hey @Usama_Hasan,
My gcc and g++ version is 9.3.0
though, I installed version 7 as you said. and tried to set compiler in pytorch directory. but the following errors occured:

CMake Warning:
  No source or binary directory provided.  Both will be assumed to be the
  same as the current working directory, but note that this warning will
  become a fatal error in future CMake releases.


-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:23 (project):
  The CMAKE_CXX_COMPILER:

    /urs/bin/g++-9

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.


-- Configuring incomplete, errors occurred!
See also "/home/nazmul/pytorch/CMakeFiles/CMakeOutput.log".
See also "/home/nazmul/pytorch/CMakeFiles/CMakeError.log".
You have changed variables that require your cache to be deleted.
Configure will be re-run and you may have to reset some variables.
The following variables have changed:
CMAKE_C_COMPILER= /usr/bin/gcc-9

-- Generating done
-- Build files have been written to: /home/nazmul/pytorch

Nazmul_Alom · July 30, 2020, 6:46am

and using gcc and g++ version 7 I was getting same error

Nazmul_Alom · July 30, 2020, 6:48am

and my python setup.py install returns this now:

Building wheel torch-1.7.0a0+d6f1346
-- Building version 1.7.0a0+d6f1346
cmake --build . --target install --config Release -- -j 6
[1/4942] Performing build step for 'nccl_external'
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a 
cd /home/nazmul/pytorch/third_party/nccl/nccl && env CCACHE_DISABLE=1 SCCACHE_DISABLE=1 make CXX=/usr/bin/c++ CUDA_HOME=/usr NVCC=/usr/bin/nvcc NVCC_GENCODE=-gencode=arch=compute_35,code=sm_35 BUILDDIR=/home/nazmul/pytorch/build/nccl VERBOSE=0 -j && /home/nazmul/miniconda3/bin/cmake -E touch /home/nazmul/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build
make -C src build BUILDDIR=/home/nazmul/pytorch/build/nccl
make[1]: Entering directory '/home/nazmul/pytorch/third_party/nccl/nccl/src'
make[2]: Entering directory '/home/nazmul/pytorch/third_party/nccl/nccl/src/collectives/device'
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
In file included from /usr/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
  138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
      |  ^~~~~
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/all_gather.dep] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/broadcast.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/sendrecv.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/reduce.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/reduce_scatter.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/all_reduce.dep] Error 1
make[2]: *** [Makefile:53: /home/nazmul/pytorch/build/nccl/obj/collectives/device/functions.dep] Error 1
make[2]: Leaving directory '/home/nazmul/pytorch/third_party/nccl/nccl/src/collectives/device'
make[1]: *** [Makefile:50: /home/nazmul/pytorch/build/nccl/obj/collectives/device/colldevice.a] Error 2
make[1]: Leaving directory '/home/nazmul/pytorch/third_party/nccl/nccl/src'
make: *** [Makefile:25: src.build] Error 2
[6/4942] Building CXX object third_par...f/compiler/command_line_interface.cc.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 737, in <module>
    build_deps()
  File "setup.py", line 321, in build_deps
    cmake=cmake)
  File "/home/nazmul/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/nazmul/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/nazmul/miniconda3/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '6']' returned non-zero exit status 1.