Ninja: build stopped: subcommand failed

vision
1

my gcc version is 9.4.0

Traceback (most recent call last):
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1808, in _run_ninja_build
subprocess.run(
File “/opt/conda/envs/difgan/lib/python3.9/subprocess.py”, line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.py”, line 41, in init
plugin = custom_ops.get_plugin(‘bias_act_plugin’, sources=sources, extra_cuda_cflags=[‘–use_fast_math’])
File “/home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/custom_ops.py”, line 103, in get_plugin
torch.utils.cpp_extension.load(name=module_name, verbose=verbose_build, sources=sources, **build_kwargs)
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1202, in load
return jit_compile(
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1425, in jit_compile
write_ninja_file_and_build_library(
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1537, in write_ninja_file_and_build_library
run_ninja_build(
File “/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/utils/cpp_extension.py”, line 1824, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension ‘bias_act_plugin’: [1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options ‘-fPIC’ --use_fast_math -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
FAILED: bias_act.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options ‘-fPIC’ --use_fast_math -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cu -o bias_act.cuda.o
/bin/sh: 1: /usr/local/cuda/bin/nvcc: not found
[2/3] c++ -MMD -MF bias_act.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=”_gcc” -DPYBIND11_STDLIB=“_libstdcpp” -DPYBIND11_BUILD_ABI=“_cxxabi1011” -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp -o bias_act.o
FAILED: bias_act.o
c++ -MMD -MF bias_act.o.d -DTORCH_EXTENSION_NAME=bias_act_plugin -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=“_gcc” -DPYBIND11_STDLIB=“_libstdcpp” -DPYBIND11_BUILD_ABI=“_cxxabi1011” -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/TH -isystem /opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/envs/difgan/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp -o bias_act.o
In file included from /home/octadion/diffusion-gan/Diffusion-GAN/diffusion-stylegan2/torch_utils/ops/bias_act.cpp:10:
/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory
5 | #include <cuda_runtime_api.h>
| ^~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

2

You didn’t explain your use case, but just posted a stacktrace, so I don’t know if you are trying to build PyTorch from source or a custom extension.
In any case, the error:

/opt/conda/envs/difgan/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:5:10: fatal error: cuda_runtime_api.h: No such file or directory

points to a missing file and it seems your locally installed CUDA toolkit might be missing these files, so make sure this file is available in your CUDA toolkit e.g. at:

/usr/local/cuda-11.7/targets/x86_64-linux/include/cuda_runtime_api.h

PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.

3

Apparently, your installation are missing cuda_runtime_api.h, typically found in /usr/local/cuda/include or so. Maybe the CUDA installation is not quite complete?

4

i tried to run diffusion-style gan sir i installed pytorch with this pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl /torch_stable.html and yes it turns out that after I checked the runtime_api was missing, how could this happen even though the download and install process had finished to the end? and there’s 2 cuda version in /usr/local, i think because previously i tried to install 2 different version of cuda. both cuda have no runtime_api, should i uninstall one of them?

5

yes sir it turns out that after I checked the runtime_api was missing, how could this happen even though the download and install process had finished to the end?

6

PyTorch itself won’t need this file if you execute a workload using an installed binary unless you are trying to build a custom extension. Is this the case and if so, what are you trying to build? You can use a verbose=True in the extension builds for more verbose outputs during the build.

7

actually i tried to run stylegan2-ada-pytorch / diffusion-stylegan2 sir. on my different machines, everything works normally.