Nvcc fatal : Unsupported gpu architecture 'compute_86'

Traceback (most recent call last):
File “venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1814, in _run_ninja_build
env=env)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/subprocess.py”, line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “scripts/train.py”, line 16, in
from training.coach import Coach
File “./training/coach.py”, line 19, in
from models.psp import pSp
File “./models/psp.py”, line 6, in
from models.encoders import psp_encoders
File “./models/encoders/psp_encoders.py”, line 9, in
from models.stylegan2.model import EqualLinear
File “./models/stylegan2/model.py”, line 7, in
from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File “./models/stylegan2/op/init.py”, line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File “./models/stylegan2/op/fused_act.py”, line 13, in
os.path.join(module_path, ‘fused_bias_act_kernel.cu’),
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1214, in load
keep_intermediates=keep_intermediates)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1435, in jit_compile
is_standalone=is_standalone)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1540, in write_ninja_file_and_build_library
error_prefix=f"Error building extension ‘{name}’")
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1824, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension ‘fused’: [1/3] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/TH -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS
-D__CUDA_NO_BFLOAT16_CONVERSIONS
-D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/TH -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options ‘-fPIC’ -std=c++14 -c /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
nvcc fatal : Unsupported gpu architecture ‘compute_86’
[2/3] c++ -MMD -MF fused_bias_act.o.d -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/TH -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp -o fused_bias_act.o
In file included from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/Device.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/Allocator.h:6,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/ATen.h:7,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:1:
/vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp: In function ‘at::Tensor fused_bias_act(const at::Tensor&, const at::Tensor&, const at::Tensor&, int, int, float, float)’:
/vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:7:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:13:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(input);
^
In file included from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/core/Tensor.h:3:0,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/DeviceGuard.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/ATen.h:11,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:1:
/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:213:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
In file included from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/DeviceType.h:8:0,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/Device.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/c10/core/Allocator.h:6,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/ATen.h:7,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:1:
/vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:7:42: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
#define CHECK_CUDA(x) TORCH_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
^
/vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:14:5: note: in expansion of macro ‘CHECK_CUDA’
CHECK_CUDA(bias);
^
In file included from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/core/Tensor.h:3:0,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/DeviceGuard.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/ATen.h:11,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
from /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
from /vol/research/AttriGAN/encoder4editing/models/stylegan2/op/fused_bias_act.cpp:1:
/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:213:30: note: declared here
DeprecatedTypeProperties & type() const {
^~~~
ninja: build stopped: subcommand failed.

Conda Env:
packages in environment at /vol/research/AttriGAN/venv/anaconda3/envs/myenv2:
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 0.9.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
blas 1.0 mkl
brotlipy 0.7.0 py37h540881e_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2022.07.19 h06a4308_0
cachetools 4.1.0 pypi_0 pypi
certifi 2022.6.15 py37h06a4308_0
cffi 1.14.6 py37hc58025e_0 conda-forge
chardet 3.0.4 pypi_0 pypi
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
cryptography 37.0.2 py37h38fbfac_0 conda-forge
cudatoolkit 11.6.0 hecad31d_10 conda-forge
cycler 0.10.0 pypi_0 pypi
debugpy 1.6.3 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
fonttools 4.37.1 pypi_0 pypi
freetype 2.10.4 h0708190_1 conda-forge
future 0.18.2 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 1.15.0 pypi_0 pypi
google-auth-oauthlib 0.4.1 pypi_0 pypi
grpcio 1.29.0 pypi_0 pypi
idna 2.9 pypi_0 pypi
imageio 2.9.0 pypi_0 pypi
importlib-metadata 1.6.0 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561
ipykernel 6.15.2 pypi_0 pypi
ipython 7.34.0 pypi_0 pypi
jedi 0.18.1 pypi_0 pypi
jpeg 9e h166bdaf_1 conda-forge
jupyter-client 7.2.0 pypi_0 pypi
jupyter-core 4.11.1 pypi_0 pypi
kiwisolver 1.2.0 pypi_0 pypi
lame 3.100 h7f98852_1001 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1
libedit 3.1.20210910 h7f8727e_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libiconv 1.17 h166bdaf_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.2.0 hf544144_3 conda-forge
libwebp-base 1.2.2 h7f98852_1 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markdown 3.2.2 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.3 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py37h402132d_0 conda-forge
mkl_fft 1.3.1 py37h3e078e5_1 conda-forge
mkl_random 1.2.2 py37h219a48f_0 conda-forge
mxnet 1.9.1 pypi_0 pypi
ncurses 6.3 h5eee18b_3
nest-asyncio 1.5.5 pypi_0 pypi
nettle 3.6 he412f7d_0 conda-forge
networkx 2.4 pypi_0 pypi
ninja 1.10.2 h06a4308_5
ninja-base 1.10.2 hd09550d_5
numpy 1.21.5 py37h6c91a56_3
numpy-base 1.21.5 py37ha15fc14_3
oauthlib 3.1.0 pypi_0 pypi
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.6.0.66 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1q h7f8727e_0
packaging 21.3 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 7.1.2 pypi_0 pypi
pip 22.1.2 py37h06a4308_0
prompt-toolkit 3.0.31 pypi_0 pypi
protobuf 3.12.2 pypi_0 pypi
psutil 5.9.2 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pyaml 21.10.1 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.13.0 pypi_0 pypi
pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pypi_0 pypi
pysocks 1.7.1 py37h89c1867_5 conda-forge
python 3.7.13 h12debd9_0
python-dateutil 2.8.1 pypi_0 pypi
python-graphviz 0.8.4 pypi_0 pypi
python_abi 3.7 2_cp37m conda-forge
pytorch 1.12.1 py3.7_cuda11.6_cudnn8.3.2_0 pytorch
pytorch-lightning 0.7.1 pypi_0 pypi
pytorch-mutex 1.0 cuda pytorch
pywavelets 1.1.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.1 pypi_0 pypi
readline 8.1.2 h7f8727e_1
requests 2.23.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rsa 4.0 pypi_0 pypi
scikit-image 0.17.2 pypi_0 pypi
scipy 1.7.3 pypi_0 pypi
setuptools 63.4.1 py37h06a4308_0
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.39.2 h5082296_0
tensorboard 2.10.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorboardx 2.5.1 pypi_0 pypi
tifffile 2021.11.2 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
torchaudio 0.12.1 py37_cu116 pytorch
torchvision 0.13.1 py37_cu116 pytorch
tornado 6.2 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
traitlets 5.3.0 pypi_0 pypi
typing_extensions 4.3.0 pyha770c72_0 conda-forge
urllib3 1.25.11 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7f8727e_1
zipp 3.8.1 pypi_0 pypi
zlib 1.2.12 h7f8727e_2
zstd 1.5.0 ha95c52a_0 conda-forge

Question:

How do I reslove this issue, I am using python version 3.7, cuda-11.6 all in a conda virtual env. Furthermore, this issue only arises in the RTX30 series GPUS(using RTX 3090).

Your local CUDA compiler (nvcc) might be too old and you should check a) which one is used in case you have installed multiple compilers, and b) if it supports sm_86.

I am running the task on a condor cluster and it has its own installation of cuda compiler. However this problem is preseistant to only my case as others on the cluster dont seem to have this problem. All my dependencies are in the conda venv that i have created as stated above.

Your conda environment does not ship with a CUDA compiler and the locally installed nvcc is used as seen in the error and the corresponding path:

RuntimeError: Error building extension ‘fused’: [1/3] /usr/local/cuda/bin/nvcc 

Again, check your local CUDA toolkit as it will be used to build custom CUDA extensions, not the CUDA runtime installed via conda.

Traceback (most recent call last):
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1814, in _run_ninja_build
env=env)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/subprocess.py”, line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “scripts/train.py”, line 16, in
from training.coach import Coach
File “./training/coach.py”, line 19, in
from models.psp import pSp
File “./models/psp.py”, line 6, in
from models.encoders import psp_encoders
File “./models/encoders/psp_encoders.py”, line 9, in
from models.stylegan2.model import EqualLinear
File “./models/stylegan2/model.py”, line 7, in
from models.stylegan2.op import FusedLeakyReLU, fused_leaky_relu, upfirdn2d
File “./models/stylegan2/op/init.py”, line 1, in
from .fused_act import FusedLeakyReLU, fused_leaky_relu
File “./models/stylegan2/op/fused_act.py”, line 13, in
os.path.join(module_path, ‘fused_bias_act_kernel.cu’),
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1214, in load
keep_intermediates=keep_intermediates)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1435, in _jit_compile
is_standalone=is_standalone)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1540, in _write_ninja_file_and_build_library
error_prefix=f"Error building extension ‘{name}’“)
File “/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/lib/python3.7/site-packages/torch/utils/cpp_extension.py”, line 1824, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension ‘fused’: [1/2] /vol/research/AttriGAN/venv/anaconda3/envs/myenv2/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=”_gcc" -DPYBIND11_STDLIB=“_li$
FAILED: fused_bias_act_kernel.cuda.o
/vol/research/AttriGAN/venv/anaconda3/envs/myenv2/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=”_gcc" -DPYBIND11_STDLIB=“_libstdcpp” -DPYBIND11_BUILD_ABI=“_cxxabi1011” -isyste$
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
ninja: build stopped: subcommand failed.

Even after installing nvcc in the conda enviroment it is still giving the same error