Custom C++ and CUDA Extensions: compiling exception

o.o · July 14, 2022, 8:43am

When I compile the customized pytorch c++ extension and CUDA kernel with Nijia, the following exception occurs. I don’t know how to solve it. The environment I use is:
ubuntu18.04
python3.8
cuda 11.1
pytorch 1.10.1

Blockquote
Traceback (most recent call last):
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1717, in _run_ninja_build
subprocess.run(
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/subprocess.py”, line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

Blockquote
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “natten/gradcheck.py”, line 11, in
from nattencuda import NATTENAVFunction, NATTENQKRPBFunction
File “/home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/nattencuda.py”, line 20, in
nattenav_cuda = load(
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1124, in load
return jit_compile(
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1337, in jit_compile
write_ninja_file_and_build_library(
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1449, in write_ninja_file_and_build_library
run_ninja_build(
File “/home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/utils/cpp_extension.py”, line 1733, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension ‘nattenav_cuda’: [1/3] :/usr/local/cuda-11.1/bin/nvcc -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda_kernel.cu -o nattenav_cuda_kernel.cuda.o
FAILED: nattenav_cuda_kernel.cuda.o
:/usr/local/cuda-11.1/bin/nvcc -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda_kernel.cu -o nattenav_cuda_kernel.cuda.o
/bin/sh: 1: :/usr/local/cuda-11.1/bin/nvcc: not found
[2/3] c++ -MMD -MF nattenav_cuda.o.d -DTORCH_EXTENSION_NAME=nattenav_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/TH -isystem /home/vipuser/anaconda3/envs/venv3.8/lib/python3.8/site-packages/torch/include/THC -isystem :/usr/local/cuda-11.1/include -isystem /home/vipuser/anaconda3/envs/venv3.8/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/vipuser/Desktop/NAT/Neighborhood-Attention-Transformer-main/natten/src/nattenav_cuda.cpp -o nattenav_cuda.o
ninja: build stopped: subcommand failed.

ptrblck · July 14, 2022, 8:55am

It seems the CUDA compiler (nvcc) cannot be found:

/bin/sh: 1: :/usr/local/cuda-11.1/bin/nvcc: not found

What does which nvcc return and are you able to find the binary in /usr/local/cuda-11.1/bin?

o.o · July 14, 2022, 9:34am

thank you for you replay, and the

which nvcc

return

/usr/local/cuda-11.1/bin/nvcc

And I can find the binary file nvcc in

/usr/local/cuda-11.1/bin

image983×601 14.6 KB

I’m not sure whether the result is expected, please give me more advice. Thanks a lot!

ptrblck · July 14, 2022, 5:46pm

I don’t know what’s causing the issue, but maybe the Python process doesn’t have access to execute nvcc in this folder? Could you check for permission issues and make sure Python is able to compile the code?