Ninja: build stopped: subcommand failed. when installing pytorch from source

thomasB · March 31, 2021, 4:27pm

Hi all,
I’m trying to install pytorch from source, but I’m getting the “ninja: build stopped: subcomman failed. error”. Here are the versions I’m using:
Ubuntu: 16.04.5 LTS
GCC: 5.4.0
CUDA: 11.0
CUDNN: 8.1.0
Python: 3.8.8
pytorch: latest, just cloned from git
The error message I get is
[2708/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/cpu_reorder.cpp.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File “setup.py”, line 876, in
build_deps()
File “setup.py”, line 354, in build_deps
build_caffe2(version=version,
File “/local/scratch/public/pytorch/tools/build_pytorch_libs.py”, line 58, in build_caffe2
cmake.build(my_env)
File “/local/scratch/public/pytorch/tools/setup_helpers/cmake.py”, line 345, in build
self.run(build_args, my_env)
File “/local/scratch/public/pytorch/tools/setup_helpers/cmake.py”, line 140, in run
check_call(command, cwd=self.build_dir, env=env)
File “/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/python3.8/subprocess.py”, line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘48’]’ returned non-zero exit status 1.
I tried

git clean -xdf
python setup.py clean
git submodule sync
git submodule deinit -f .
git submodule update --init --recursive
python setup.py install

but that didn’t help. Does anyone know how to fix this?
Thanks!

ptrblck · April 1, 2021, 6:42am

The error message should have been printed in the install log before ninja is failing, so you would have to recheck the log.
In case you cannot fix the error, feel free to post it here and we can have a look.

thomasB · April 1, 2021, 8:51am

Thanks! Is there an install log that is automatically created when installing pytorch from source?
When I tried the installation I used

python setup.py install > install_log.txt

The last few lines in this document are

[2703/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/rnn/ref_rnn.cpp.o
[2704/6039] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o
[2705/6039] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8Depthwise3DAvx2.cc.o
[2706/6039] Building CXX object third_party/ideep/mkl-dnn/src/cpu/CMakeFiles/dnnl_cpu.dir/cpu_reorder.cpp.o
ninja: build stopped: subcommand failed.
Building wheel torch-1.9.0a0+git09756e7
– Building version 1.9.0a0+git09756e7
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/local/scratch/public/pytorch/torch -DCMAKE_PREFIX_PATH=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env -DCUDNN_INCLUDE_DIR=/local/scratch/public/cudnn/cuda/include -DCUDNN_LIBRARY=/local/scratch/public/cudnn/cuda/lib64 -DNUMPY_INCLUDE_DIR=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/bin/python -DPYTHON_INCLUDE_DIR=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/include/python3.8 -DPYTHON_LIBRARY=/alt/applic/user-maint/tb588/Conda/envs/pytorch_env/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.9.0a0+git09756e7 -DUSE_NUMPY=True /local/scratch/public/pytorch
cmake --build . --target install --config Release – -j 48

Thanks for the help!

CLaireeeee · April 1, 2021, 9:53am

I also had this problem for a long time while installing on linux and have no idea how to deal with it. Any help will be appreciated!

Here is my console output:

Building wheel torch-1.9.0a0+git55dfb4a

-- Building version 1.9.0a0+git55dfb4a

cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/caowanlu/fnet/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/lib/python3/dist-packages -DNUMPY_INCLUDE_DIR=/home/caowanlu/.local/lib/python3.6/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/usr/bin/python3 -DPYTHON_INCLUDE_DIR=/usr/include/python3.6m -DPYTHON_LIBRARY=/usr/lib/libpython3.6m.so.1.0 -DTORCH_BUILD_VERSION=1.9.0a0+git55dfb4a -DUSE_NUMPY=True /home/caowanlu/fnet/pytorch

-- The CXX compiler identification is GNU 7.5.0

-- The C compiler identification is GNU 7.5.0

-- Detecting CXX compiler ABI info

-- Detecting CXX compiler ABI info - done

-- Check for working CXX compiler: /usr/bin/c++ - skipped

-- Detecting CXX compile features

-- Detecting CXX compile features - done

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Check for working C compiler: /usr/bin/cc - skipped

-- Detecting C compile features

-- Detecting C compile features - done

-- Not forcing any particular BLAS to be found

-- Performing Test COMPILER_WORKS

-- Performing Test COMPILER_WORKS - Success

-- Performing Test SUPPORT_GLIBCXX_USE_C99

-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success

-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED

-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success

-- std::exception_ptr is supported.

-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING

-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed

-- Turning off deprecation warning due to glog.

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success

-- Current compiler supports avx2 extension. Will build perfkernels.

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS

-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success

-- Current compiler supports avx512f extension. Will build fbgemm.

-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY

-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success

-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY

-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success

-- Performing Test COMPILER_SUPPORTS_RDYNAMIC

-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success

-- Building using own protobuf under third_party per request.

-- Use custom protobuf build.

--

-- 3.11.4.0

-- Looking for pthread.h

-- Looking for pthread.h - found

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed

-- Check if compiler accepts -pthread

-- Check if compiler accepts -pthread - yes

-- Found Threads: TRUE

-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS

-- Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success

-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/caowanlu/fnet/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>

-- Trying to find preferred BLAS backend of choice: MKL

-- MKL_THREADING = OMP

-- Looking for sys/types.h

-- Looking for sys/types.h - found

-- Looking for stdint.h

-- Looking for stdint.h - found

-- Looking for stddef.h

-- Looking for stddef.h - found

-- Check size of void*

-- Check size of void* - done

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)

cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)

cmake/Dependencies.cmake:140 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Modules/FindMKL.cmake:213 (FIND_PACKAGE)

cmake/Modules/FindMKL.cmake:307 (CHECK_ALL_LIBRARIES)

cmake/Dependencies.cmake:140 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Looking for cblas_sgemm

-- Looking for cblas_sgemm - found

-- MKL libraries: /home/caowanlu/anaconda3/lib/libmkl_intel_lp64.so;/home/caowanlu/anaconda3/lib/libmkl_gnu_thread.so;/home/caowanlu/anaconda3/lib/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so

-- MKL include directory: /home/caowanlu/anaconda3/include

-- MKL OpenMP type: GNU

-- MKL OpenMP library: -fopenmp

-- The ASM compiler identification is GNU

-- Found assembler: /usr/bin/cc

-- Brace yourself, we are building NNPACK

-- Performing Test NNPACK_ARCH_IS_X86_32

-- Performing Test NNPACK_ARCH_IS_X86_32 - Failed

-- Found PythonInterp: /usr/bin/python3 (found version "3.6.9")

-- NNPACK backend is x86-64

CMake Deprecation Warning at third_party/googletest/CMakeLists.txt:1 (cmake_minimum_required):

Compatibility with CMake < 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument <min> value or use a ...<max> suffix to tell

CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googlemock/CMakeLists.txt:42 (cmake_minimum_required):

Compatibility with CMake < 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument <min> value or use a ...<max> suffix to tell

CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at third_party/googletest/googletest/CMakeLists.txt:49 (cmake_minimum_required):

Compatibility with CMake < 2.8.12 will be removed from a future version of

CMake.

Update the VERSION argument <min> value or use a ...<max> suffix to tell

CMake that the project does not need compatibility with older versions.

-- Failed to find LLVM FileCheck

-- Found Git: /usr/bin/git (found version "2.17.1")

-- git Version: v1.4.0-505be96a-dirty

-- Version: 1.4.0

-- Performing Test HAVE_CXX_FLAG_STD_CXX11

-- Performing Test HAVE_CXX_FLAG_STD_CXX11 - Success

-- Performing Test HAVE_CXX_FLAG_WALL

-- Performing Test HAVE_CXX_FLAG_WALL - Success

-- Performing Test HAVE_CXX_FLAG_LPTHREAD

-- Performing Test HAVE_CXX_FLAG_LPTHREAD - Success

-- Performing Test HAVE_CXX_FLAG_WEXTRA

-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success

-- Performing Test HAVE_CXX_FLAG_WSHADOW

-- Performing Test HAVE_CXX_FLAG_WSHADOW - Success

-- Performing Test HAVE_CXX_FLAG_WERROR

-- Performing Test HAVE_CXX_FLAG_WERROR - Success

-- Performing Test HAVE_CXX_FLAG_PEDANTIC

-- Performing Test HAVE_CXX_FLAG_PEDANTIC - Success

-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS

-- Performing Test HAVE_CXX_FLAG_PEDANTIC_ERRORS - Success

-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL

-- Performing Test HAVE_CXX_FLAG_WFLOAT_EQUAL - Success

-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING

-- Performing Test HAVE_CXX_FLAG_FSTRICT_ALIASING - Success

-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS

-- Performing Test HAVE_CXX_FLAG_WNO_DEPRECATED_DECLARATIONS - Success

-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING

-- Performing Test HAVE_CXX_FLAG_WSTRICT_ALIASING - Success

-- Performing Test HAVE_CXX_FLAG_COVERAGE

-- Performing Test HAVE_CXX_FLAG_COVERAGE - Success

-- Performing Test HAVE_STD_REGEX

-- Performing Test HAVE_STD_REGEX

-- Performing Test HAVE_STD_REGEX -- success

-- Performing Test HAVE_GNU_POSIX_REGEX

-- Performing Test HAVE_GNU_POSIX_REGEX

-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile

-- Performing Test HAVE_POSIX_REGEX

-- Performing Test HAVE_POSIX_REGEX

-- Performing Test HAVE_POSIX_REGEX -- success

-- Performing Test HAVE_STEADY_CLOCK

-- Performing Test HAVE_STEADY_CLOCK

-- Performing Test HAVE_STEADY_CLOCK -- success

-- Performing Test COMPILER_SUPPORTS_AVX512

-- Performing Test COMPILER_SUPPORTS_AVX512 - Success

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

third_party/fbgemm/CMakeLists.txt:60 (find_package)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Found OpenMP_C: -fopenmp (found version "4.5")

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

third_party/fbgemm/CMakeLists.txt:60 (find_package)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Found OpenMP_CXX: -fopenmp (found version "4.5")

-- Found OpenMP: TRUE (found version "4.5")

CMake Warning at third_party/fbgemm/CMakeLists.txt:62 (message):

OpenMP found! OpenMP_C_INCLUDE_DIRS =

CMake Warning at third_party/fbgemm/CMakeLists.txt:142 (message):

==========

CMake Warning at third_party/fbgemm/CMakeLists.txt:143 (message):

CMAKE_BUILD_TYPE = Release

CMake Warning at third_party/fbgemm/CMakeLists.txt:144 (message):

CMAKE_CXX_FLAGS_DEBUG is -g

CMake Warning at third_party/fbgemm/CMakeLists.txt:145 (message):

CMAKE_CXX_FLAGS_RELEASE is -O3 -DNDEBUG

CMake Warning at third_party/fbgemm/CMakeLists.txt:146 (message):

==========

-- Performing Test __CxxFlag__fno_threadsafe_statics

-- Performing Test __CxxFlag__fno_threadsafe_statics - Success

-- Performing Test __CxxFlag__fno_semantic_interposition

-- Performing Test __CxxFlag__fno_semantic_interposition - Success

-- Performing Test __CxxFlag__fmerge_all_constants

-- Performing Test __CxxFlag__fmerge_all_constants - Success

-- Performing Test __CxxFlag__fno_enforce_eh_specs

-- Performing Test __CxxFlag__fno_enforce_eh_specs - Success

** AsmJit Summary **

ASMJIT_DIR=/home/caowanlu/fnet/pytorch/third_party/fbgemm/third_party/asmjit

ASMJIT_TEST=FALSE

ASMJIT_TARGET_TYPE=STATIC

ASMJIT_DEPS=pthread;rt

ASMJIT_LIBS=asmjit;pthread;rt

ASMJIT_CFLAGS=-DASMJIT_STATIC

ASMJIT_PRIVATE_CFLAGS=-Wall;-Wextra;-Wconversion;-fno-math-errno;-fno-threadsafe-statics;-fno-semantic-interposition;-DASMJIT_STATIC

ASMJIT_PRIVATE_CFLAGS_DBG=

ASMJIT_PRIVATE_CFLAGS_REL=-O2;-fmerge-all-constants;-fno-enforce-eh-specs

-- Found Numa: /usr/include

-- Found Numa (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libnuma.so)

-- Using third party subdirectory Eigen.

-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.6.9", minimum required is "3.0")

-- Found PythonLibs: /usr/lib/libpython3.6m.so.1.0 (found suitable version "3.6.9", minimum required is "3.0")

-- Could NOT find pybind11 (missing: pybind11_DIR)

-- Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR)

-- Using third_party/pybind11.

-- pybind11 include dirs: /home/caowanlu/fnet/pytorch/cmake/../third_party/pybind11/include

-- Found MPI_C: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so (found version "3.1")

-- Found MPI_CXX: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so (found version "3.1")

-- Found MPI: TRUE (found version "3.1")

-- MPI support found

-- MPI compile flags: -pthread

-- MPI include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include

-- MPI LINK flags path: -L/usr/lib -pthread

-- MPI libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so

CMake Warning at cmake/Dependencies.cmake:1040 (message):

OpenMPI found, but it is not built with CUDA support.

Call Stack (most recent call first):

CMakeLists.txt:615 (include)

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_C)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Dependencies.cmake:1100 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at /home/caowanlu/anaconda3/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:426 (message):

The package name passed to `find_package_handle_standard_args` (OpenMP_CXX)

does not match the name of the calling package (OpenMP). This can lead to

problems in calling code that expects `find_package` result variables

(e.g., `_FOUND`) to follow a certain pattern.

Call Stack (most recent call first):

cmake/Modules/FindOpenMP.cmake:576 (find_package_handle_standard_args)

cmake/Dependencies.cmake:1100 (find_package)

CMakeLists.txt:615 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

-- Adding OpenMP CXX_FLAGS: -fopenmp

-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so

-- Found CUDA: /usr/local/cuda (found version "11.0")

-- Caffe2: CUDA detected: 11.0

-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc

-- Caffe2: CUDA toolkit directory: /usr/local/cuda

-- Caffe2: Header version is: 11.0

-- Found CUDNN: /usr/local/cuda/lib64/libcudnn.so

-- Found cuDNN: v8.1.0 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)

-- /usr/local/cuda/lib64/libnvrtc.so shorthash is f4909b87

-- Autodetected CUDA architecture(s): 7.5

-- Added CUDA NVCC flags for: -gencode;arch=compute_75,code=sm_75

-- Found HIP: /opt/rocm-4.0.0/hip (found suitable version "4.0.20496-4f163c68", minimum required is "1.0")

HIP VERSION: 4.0.20496-4f163c68

***** ROCm version from /opt/rocm/.info/version-dev ****

ROCM_VERSION_DEV: 4.0.0

ROCM_VERSION_DEV_MAJOR: 4

ROCM_VERSION_DEV_MINOR: 0

ROCM_VERSION_DEV_PATCH: 0

***** Library versions from dpkg *****

rocm-dev VERSION: 4.0.0.40000-23

rocm-device-libs VERSION: 1.0.0.637-rocm-rel-4.0-23-db8c0c3

hsakmt-roct VERSION: 20201016.1.0269-mainline-20201016-1-g0269ce3

hsakmt-roct-dev VERSION: 20201016.1.0269-mainline-20201016-1-g0269ce3

hsa-rocr-dev VERSION: 1.2.40000.0-rocm-rel-4.0-23-a5173c90

***** Library versions from cmake find_package *****

hip VERSION: 4.0.20496

hsa-runtime64 VERSION: 1.2.40000

amd_comgr VERSION: 1.9.0.194-rocm-rel-4.0-23-0fa438b

CMake Error at cmake/public/LoadHIP.cmake:138 (find_package):

By not providing "Findrocrand.cmake" in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by "rocrand", but

CMake did not find one.

Could not find a package configuration file provided by "rocrand" with any

of the following names:

rocrandConfig.cmake

rocrand-config.cmake

Add the installation prefix of "rocrand" to CMAKE_PREFIX_PATH or set

"rocrand_DIR" to a directory containing one of the above files. If

"rocrand" provides a separate development package or SDK, be sure it has

been installed.

Call Stack (most recent call first):

cmake/public/LoadHIP.cmake:197 (find_package_and_print_version)

cmake/Dependencies.cmake:1189 (include)

CMakeLists.txt:615 (include)

-- Configuring incomplete, errors occurred!

See also "/home/caowanlu/fnet/pytorch/build/CMakeFiles/CMakeOutput.log".

See also "/home/caowanlu/fnet/pytorch/build/CMakeFiles/CMakeError.log".

Traceback (most recent call last):

File "setup.py", line 876, in <module>

build_deps()

File "setup.py", line 359, in build_deps

cmake=cmake)

File "/home/caowanlu/fnet/pytorch/tools/build_pytorch_libs.py", line 55, in build_caffe2

rerun_cmake)

File "/home/caowanlu/fnet/pytorch/tools/setup_helpers/cmake.py", line 329, in generate

self.run(args, env=my_env)

File "/home/caowanlu/fnet/pytorch/tools/setup_helpers/cmake.py", line 140, in run

check_call(command, cwd=self.build_dir, env=env)

File "/usr/lib/python3.6/subprocess.py", line 311, in check_call

raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/caowanlu/fnet/pytorch/torch', '-DCMAKE_PREFIX_PATH=/usr/lib/python3/dist-packages', '-DNUMPY_INCLUDE_DIR=/home/caowanlu/.local/lib/python3.6/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m', '-DPYTHON_LIBRARY=/usr/lib/libpython3.6m.so.1.0', '-DTORCH_BUILD_VERSION=1.9.0a0+git55dfb4a', '-DUSE_NUMPY=True', '/home/caowanlu/fnet/pytorch']' returned non-zero exit status 1.

ptrblck · April 2, 2021, 12:41am

The error might be earlier than the last couple of lines and you could grep for it or alternatively post the complete log (or the last part at least) as a code snippet here.

@CLaireeeee
The build seems to fail with:

CMake Error at cmake/public/LoadHIP.cmake:138 (find_package):

By not providing “Findrocrand.cmake” in CMAKE_MODULE_PATH this project has

asked CMake to find a package configuration file provided by “rocrand”, but

CMake did not find one.

Could not find a package configuration file provided by “rocrand” with any

of the following names:

rocrandConfig.cmake

rocrand-config.cmake

Add the installation prefix of “rocrand” to CMAKE_PREFIX_PATH or set

“rocrand_DIR” to a directory containing one of the above files. If

“rocrand” provides a separate development package or SDK, be sure it has

been installed.

Call Stack (most recent call first):

cmake/public/LoadHIP.cmake:197 (find_package_and_print_version)

cmake/Dependencies.cmake:1189 (include)

CMakeLists.txt:615 (include)

– Configuring incomplete, errors occurred!

PS: you can post code snippets (or these types of logs) by wrapping them into three backticks ```, which would make debugging easier.

CLaireeeee · April 2, 2021, 7:15am

@ptrblck Thank you for your reply! I edited it and used ``` now.
I saw the “rocrand” problem you mentioned and exported environment variable USE_ROCM=0 (as I really do not need it)

export USE_ROCM=0

Then the building continued!
Thank you again!

craigyang · August 9, 2021, 3:38am

Hi @ptrblck, when building from source, I also get the Ninja error in the title, but the message seems different from the above. Referring to ninja: build stopped: subcommand failed. · Issue #32694 · pytorch/pytorch · GitHub, I tried both the git clean code snippet (also posted in the question body) and conda (or pip) install ninja, but neither helped. Could you help take a look? Thanks!

Last several lines of output when I run python setup.py develop (with some environment variables set beforehand):

...
-- Generating done
-- Build files have been written to: /home/craig/pytorch/build
[17/381] Linking CXX executable bin/utility_ops_gpu_test
FAILED: bin/utility_ops_gpu_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/utility_ops_gpu_test.dir/operators/utility_ops_gpu_test.cc.o -o bin/utility_ops_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[19/381] Linking CXX executable bin/generate_proposals_op_gpu_test
FAILED: bin/generate_proposals_op_gpu_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/generate_proposals_op_gpu_test.dir/operators/generate_proposals_op_gpu_test.cc.o -o bin/generate_proposals_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[20/381] Linking CXX executable bin/roi_align_op_gpu_test
FAILED: bin/roi_align_op_gpu_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/roi_align_op_gpu_test.dir/operators/roi_align_op_gpu_test.cc.o -o bin/roi_align_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[21/381] Linking CXX executable bin/batch_matmul_op_gpu_test
FAILED: bin/batch_matmul_op_gpu_test 
: && /usr/bin/c++ -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/local/openmpi/openmpi-4.1.0/lib -Wl,--enable-new-dtags -pthread caffe2/CMakeFiles/batch_matmul_op_gpu_test.dir/operators/batch_matmul_op_gpu_test.cc.o -o bin/batch_matmul_op_gpu_test  -Wl,-rpath,/home/craig/anaconda3/envs/py/lib:/usr/local/cuda/lib64:/home/craig/pytorch/build/lib:  /usr/local/cuda/lib64/libcudart.so  lib/libgtest_main.a  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch.so" -Wl,--as-needed  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cpu.so" -Wl,--as-needed  lib/libprotobuf.a  /home/craig/anaconda3/envs/py/lib/libmkl_intel_lp64.so  /home/craig/anaconda3/envs/py/lib/libmkl_gnu_thread.so  /home/craig/anaconda3/envs/py/lib/libmkl_core.so  -fopenmp  /usr/lib/x86_64-linux-gnu/libpthread.so  -lm  /usr/lib/x86_64-linux-gnu/libdl.so  lib/libdnnl.a  -ldl  -Wl,--no-as-needed,"/home/craig/pytorch/build/lib/libtorch_cuda.so" -Wl,--as-needed  lib/libc10_cuda.so  lib/libc10.so  /usr/local/cuda/lib64/libcudart.so  /home/craig/anaconda3/envs/py/lib/libnvToolsExt.so  /usr/local/cuda/lib64/libcufft.so  /usr/local/cuda/lib64/libcurand.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcudnn.so  lib/libgtest.a  -pthread && :
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtSSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmDDD@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmACC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtZZZMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtACCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationZZZ@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtDDDMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTST@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHHH@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmTSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmSSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmul@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHHHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmHSS@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSHMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtCCCMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtBSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmBII@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtLegacyGemmUtilizationCCC@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtTSTMatmulAlgoGetHeuristic@libcublasLt.so.11'
/usr/bin/ld: /usr/local/cuda/lib64/libcublas.so: undefined reference to `cublasLtHSSMatmulAlgoGetHeuristic@libcublasLt.so.11'
collect2: error: ld returned 1 exit status
[40/381] Building CXX object test_api/CMakeFiles/test_api.dir/modules.cpp.o
ninja: build stopped: subcommand failed.

ptrblck · August 9, 2021, 4:22am

Based on the error message it seems that linking cublas fails.
Did you install different CUDA toolkit versions or tried to manipulate the cublas version manually somehow?

craigyang · August 9, 2021, 4:53am

Thanks for the instant reply! I am building on a Ubuntu cluster with CUDA_HOME set to the directory of a preinstalled CUDA toolkit. There are indeed other CUDA versions installed as well, but not in that directory (I think, but will double check). Just curious, how will this impact the linking?

For reference, below is what’s in the version.json file in that CUDA_HOME directory:

{
   "cuda" : {
      "name" : "CUDA SDK",
      "version" : "11.2.20210204"
   },
   "cuda_cudart" : {
      "name" : "CUDA Runtime (cudart)",
      "version" : "11.2.146"
   },
   "cuda_cuobjdump" : {
      "name" : "cuobjdump",
      "version" : "11.2.135"
   },
   "cuda_cupti" : {
      "name" : "CUPTI",
      "version" : "11.2.135"
   },
   "cuda_cuxxfilt" : {
      "name" : "CUDA cu++ filt",
      "version" : "11.2.135"
   },
   "cuda_demo_suite" : {
      "name" : "CUDA Demo Suite",
      "version" : "11.2.67"
   },
   "cuda_gdb" : {
      "name" : "CUDA GDB",
      "version" : "11.2.135"
   },
   "cuda_memcheck" : {
      "name" : "CUDA Memcheck",
      "version" : "11.2.135"
   },
   "cuda_nsight" : {
      "name" : "Nsight Eclipse Plugins",
      "version" : "11.2.135"
   },
   "cuda_nvcc" : {
      "name" : "CUDA NVCC",
      "version" : "11.2.142"
},
   "cuda_nvdisasm" : {
      "name" : "CUDA nvdisasm",
      "version" : "11.2.135"
   },
   "cuda_nvml_dev" : {
      "name" : "CUDA NVML Headers",
      "version" : "11.2.67"
   },
   "cuda_nvprof" : {
      "name" : "CUDA nvprof",
      "version" : "11.2.135"
   },
   "cuda_nvprune" : {
      "name" : "CUDA nvprune",
      "version" : "11.2.135"
   },
   "cuda_nvrtc" : {
      "name" : "CUDA NVRTC",
      "version" : "11.2.142"
   },
   "cuda_nvtx" : {
      "name" : "CUDA NVTX",
      "version" : "11.2.67"
   },
   "cuda_nvvp" : {
      "name" : "CUDA NVVP",
      "version" : "11.2.135"
   },
   "cuda_samples" : {
      "name" : "CUDA Samples",
      "version" : "11.2.135"
   },
   "cuda_sanitizer_api" : {
      "name" : "CUDA Compute Sanitizer API",
      "version" : "11.2.135"
  },
   "libcublas" : {
      "name" : "CUDA cuBLAS",
      "version" : "11.4.1.1026"
   },
   "libcufft" : {
      "name" : "CUDA cuFFT",
      "version" : "10.4.0.135"
   },
   "libcurand" : {
      "name" : "CUDA cuRAND",
      "version" : "10.2.3.135"
   },
   "libcusolver" : {
      "name" : "CUDA cuSOLVER",
      "version" : "11.1.0.135"
   },
   "libcusparse" : {
      "name" : "CUDA cuSPARSE",
      "version" : "11.4.0.135"
   },
   "libnpp" : {
      "name" : "CUDA NPP",
      "version" : "11.3.2.139"
   },
   "libnvjpeg" : {
      "name" : "CUDA nvJPEG",
      "version" : "11.4.0.135"
   },
   "nsight_compute" : {
      "name" : "Nsight Compute",
      "version" : "2020.3.1.3"
   },
   "nsight_systems" : {
      "name" : "Nsight Systems",
      "version" : "2020.4.3.7"
   },
   "nvidia_driver" : {
      "name" : "NVIDIA Linux Driver",
      "version" : "460.32.03"
   }
}

lcxywfe · November 23, 2021, 11:45am

I met the same problem, and why cuda toolkit in other directory which not in LIBRARY_PATH could disturb linking?

lamb_ele · July 21, 2022, 12:36am

I’m getting the same error when I try to install pytorch from the source with cuda 11.4

[1/871] Building CXX object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o
FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o 
/usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_C10D_NCCL -DUSE_CUDA -DUSE_DISTRIBUTED -DUSE_EXPERIMENTAL_CUDNN_V8_API -DUSE_EXTERNAL_MZCRC -DUSE_NCCL -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -I/home/jupyter/pytorch/build/aten/src -I/home/jupyter/pytorch/aten/src -I/home/jupyter/pytorch/build -I/home/jupyter/pytorch -I/home/jupyter/pytorch/cmake/../third_party/benchmark/include -I/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include -I/home/jupyter/pytorch/third_party/onnx -I/home/jupyter/pytorch/build/third_party/onnx -I/home/jupyter/pytorch/third_party/foxi -I/home/jupyter/pytorch/build/third_party/foxi -I/home/jupyter/pytorch/build/include -I/home/jupyter/pytorch/torch/csrc/distributed -I/home/jupyter/pytorch/aten/src/THC -I/home/jupyter/pytorch/aten/src/ATen/cuda -I/home/jupyter/pytorch/build/caffe2/aten/src -I/home/jupyter/pytorch/aten/../third_party/catch/single_include -I/home/jupyter/pytorch/aten/src/ATen/.. -I/home/jupyter/pytorch/build/nccl/include -I/home/jupyter/pytorch/c10/cuda/../.. -I/home/jupyter/pytorch/c10/.. -I/home/jupyter/pytorch/third_party/tensorpipe -I/home/jupyter/pytorch/build/third_party/tensorpipe -I/home/jupyter/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/jupyter/pytorch/torch/csrc/api -I/home/jupyter/pytorch/torch/csrc/api/include -isystem /home/jupyter/pytorch/build/third_party/gloo -isystem /home/jupyter/pytorch/cmake/../third_party/gloo -isystem /home/jupyter/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /home/jupyter/pytorch/cmake/../third_party/googletest/googletest/include -isystem /home/jupyter/pytorch/third_party/protobuf/src -isystem /opt/conda/include -isystem /home/jupyter/pytorch/third_party/gemmlowp -isystem /home/jupyter/pytorch/third_party/neon2sse -isystem /home/jupyter/pytorch/third_party/XNNPACK/include -isystem /home/jupyter/pytorch/third_party/ittapi/include -isystem /home/jupyter/pytorch/cmake/../third_party/eigen -isystem /usr/local/cuda/include -isystem /home/jupyter/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN/include -isystem /home/jupyter/pytorch/third_party/ideep/include -isystem /home/jupyter/pytorch/third_party/ideep/mkl-dnn/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -fPIC -DCAFFE2_USE_GLOO -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-sign-compare -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-missing-braces -Wno-maybe-uninitialized -fvisibility=hidden -O2 -DTORCH_CUDA_BUILD_MAIN_LIB -pthread -std=gnu++14 -MD -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o -MF caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o.d -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp.o -c /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp
In file included from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_ConvDesc.h:35,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend.h:100,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/utils.h:20,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp:11:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h: In function ‘std::__cxx11::string cudnn_frontend::to_string(cudnnDataType_t)’:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: error: ‘CUDNN_DATA_INT64’ was not declared in this scope
         case CUDNN_DATA_INT64:
              ^~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: note: suggested alternative: ‘CUDNN_DATA_INT8x4’
         case CUDNN_DATA_INT64:
              ^~~~~~~~~~~~~~~~
              CUDNN_DATA_INT8x4
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:95:14: error: ‘CUDNN_DATA_BFLOAT16’ was not declared in this scope
         case CUDNN_DATA_BFLOAT16:
              ^~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:95:14: note: suggested alternative: ‘CUDNN_DATA_FLOAT’
         case CUDNN_DATA_BFLOAT16:
              ^~~~~~~~~~~~~~~~~~~
              CUDNN_DATA_FLOAT
In file included from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Operation.h:36,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_OperationGraph.h:36,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_Heuristics.h:31,
                 from /home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend.h:101,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/utils.h:20,
                 from /home/jupyter/pytorch/aten/src/ATen/native/quantized/cudnn/LinearUnpackImpl.cpp:11:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h: In member function ‘int64_t cudnn_frontend::PointWiseDesc_v8::getPortCount() const’:
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:87:18: error: ‘CUDNN_POINTWISE_RELU_BWD’ was not declared in this scope
             case CUDNN_POINTWISE_RELU_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:87:18: note: suggested alternative: ‘CUDNN_POINTWISE_RELU_FWD’
             case CUDNN_POINTWISE_RELU_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
                  CUDNN_POINTWISE_RELU_FWD
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:88:18: error: ‘CUDNN_POINTWISE_TANH_BWD’ was not declared in this scope
             case CUDNN_POINTWISE_TANH_BWD:
                  ^~~~~~~~~~~~~~~~~~~~~~~~
/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_PointWiseDesc.h:88:18: note: suggested alternative: ‘CUDNN_POINTWISE_TANH_FWD’
             case CUDNN_POINTWISE_TANH_BWD:
....
ninja: build stopped: subcommand failed.
Building wheel torch-1.13.0a0+gitc949788
-- Building version 1.13.0a0+gitc949788
cmake --build . --target install --config Release -- -j 8

ptrblck · July 21, 2022, 2:26am

/home/jupyter/pytorch/cmake/../third_party/cudnn_frontend/include/cudnn_frontend_utils.h:93:14: error: ‘CUDNN_DATA_INT64’ was not declared in this scope

Did you initialize all submodules before starting the build?

lamb_ele · July 21, 2022, 4:09am

I was following the instructions from pytorch github readme: GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration Specifically I ran: git submodule update --init --recursive --jobs 0 before running:

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install

which gave the error I posted above. Now that I saw this other link and ran git submodule update --init --recursive instead and got different errors.

Anyways, I gave up. It shouldn’t be this hard to get pytorch to work with gpu. I’m selecting a different VM instance that has everything set up already. Thanks for the help

ptrblck · July 21, 2022, 4:26am

Yes, you are right and the common approach is to install the pip wheels or conda binaries which already ship with the CUDA runtime. If you need to latest nightly release, you can also install the nightly binaries with the corresponding CUDA version.
A source build would be needed in case you want a specific CUDA/cuDNN/NCCL etc. version or want to add changes to the framework and while I haven’t seen your issue before (if seems you indeed did not synchronize the submodules) I would also consider it a more complicated approach as you would need a locally installed and working build toolchain (C++ compiler, CUDA compiler etc.).

Thomas_Buddenkotte · January 28, 2023, 12:03pm

Hi all,
a few months later I hit the same problem again. I used the same commands as before (git submodule …) and exported USE_ROCM=0 and CUDNN_{LIB_DIR/INCLUDE_DIR/LIBRARY} to the library and pathes of the cudnn files. I now get the following error:

[5410/6989] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCPU.cpp.o
[5411/6989] Performing build step for ‘nccl_external’
make -C src build BUILDDIR=/local/scratch/public/tb588/github/pytorch/build/nccl
make[1]: Entering directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src’
Grabbing include/nccl_net.h > /local/scratch/public/tb588/github/pytorch/build/nccl/include/nccl_net.h
Generating nccl.h.in > /local/scratch/public/tb588/github/pytorch/build/nccl/include/nccl.h
Generating nccl.pc.in > /local/scratch/public/tb588/github/pytorch/build/nccl/lib/pkgconfig/nccl.pc
Compiling init.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/init.o
Compiling channel.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/channel.o
Compiling bootstrap.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/bootstrap.o
Compiling transport.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/transport.o
Compiling enqueue.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/enqueue.o
enqueue.cc: In function ‘ncclResult_t scheduleCollTasksToPlan(ncclComm*, ncclKernelPlan*, int*)’:
enqueue.cc:578:60: warning: ‘workFuncIndex’ may be used uninitialized in this function [-Wmaybe-uninitialized]
plan->kernelSpecialized = ncclKerns[workFuncIndex].specialized;
~~~~~~~~~~~~~~~^
Compiling group.cc > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/group.o
…
Compiling functions.cu > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/functions.o
Compiling onerank_reduce.cu > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/onerank_reduce.o
Archiving objects > /local/scratch/public/tb588/github/pytorch/build/nccl/obj/collectives/device/colldevice.a
make[2]: Leaving directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src/collectives/device’
Linking libnccl.so.2.14.3 > /local/scratch/public/tb588/github/pytorch/build/nccl/lib/libnccl.so.2.14.3
Archiving libnccl_static.a > /local/scratch/public/tb588/github/pytorch/build/nccl/lib/libnccl_static.a
make[1]: Leaving directory ‘/local/scratch/public/tb588/github/pytorch/third_party/nccl/nccl/src’
ninja: build stopped: subcommand failed.
Building wheel torch-2.0.0a0+gitf40183d
– Building version 2.0.0a0+gitf40183d
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/local/scratch/public/tb588/github/pytorch/torch -DCMAKE_PREFIX_PATH=/local/scratch/public/tb588/miniconda3/e
nvs/pytorch_env/lib/python3.8/site-packages;/local/scratch/public/tb588/miniconda3/envs/pytorch_env -DCMAKE_PREFIX_PATH_modshare=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cuda-11.1.1-joqpcn7aufehhcop2
vxsczwivpgsgi4o/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecr
ba66pm44a5yd/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpc-1.0.3-tumbpshu5hjxwextoudk5hmic6nspb3z/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cuda-11.2.0-4cnusrbsscayn2fudsrlkkwmxuatqnn
q/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/isl-0.18-vttqoutnsmjpm3ogb52rninksc7hq5ax/:1:/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpfr-3.1.5-mdi6irzvxcbemt7yredzr36dvo6ty4sl/:1:/alt/app
lic/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-gmnpoqci26qz55qnegn2kvt7hq67hnsy/:1 -DCUDNN_INCLUDE_DIR=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecrba66pm4
4a5yd/include -DCUDNN_LIBRARY=/alt/applic/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/cudnn-8.0.5.39-11.1-aogmnpiibuxgcu47oecrba66pm44a5yd/lib64 -DNUMPY_INCLUDE_DIR=/local/scratch/public/tb588/miniconda3/envs/pytor
ch_env/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/local/scratch/public/tb588/miniconda3/envs/pytorch_env/bin/python -DPYTHON_INCLUDE_DIR=/local/scratch/public/tb588/miniconda3/envs/pytor
ch_env/include/python3.8 -DPYTHON_LIBRARY=/local/scratch/public/tb588/miniconda3/envs/pytorch_env/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=2.0.0a0+gitf40183d -DUSE_NUMPY=True -DUSE_ROCM=0 /local/scratch/pub
lic/tb588/github/pytorch
cmake --build . --target install --config Release

Does anyone know where this error might come from?

Many thanks!