Hello, I’m trying to compile pytorch from source and encountering the following build error.
$ CC=gcc-10 CXX=g++-10 python setup.py develop
...
[5995/6841] Linking CXX executable bin/HashStoreTest
Warning: Unused direct dependencies:
/home/netfpga/research/collective/pytorch/build/lib/libc10.so
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_intel_lp64.so.1
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_gnu_thread.so.1
/home/netfpga/anaconda3/envs/pytorch_base/lib/libmkl_core.so.1
/lib/x86_64-linux-gnu/libdl.so.2
/home/netfpga/anaconda3/envs/pytorch_base/lib/libgomp.so.1
[5996/6841] Performing build step for 'nccl_external'
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a
cd /home/netfpga/research/collective/pytorch/third_party/nccl/nccl && make -j24 -l24 CXX=/usr/bin/g++-10 CUDA_HOME=/home/netfpga/anaconda3/envs/pytorch_base NVCC=/home/netfpga/anaconda3/envs/pytorch_base/bin/nvcc "NVCC_GENCODE=-gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86"
BUILDDIR=/home/netfpga/research/collective/pytorch/build/nccl VERBOSE=0 DEBUG=0 && /home/netfpga/.local/lib/python3.8/site-packages/cmake/data/bin/cmake -E touch /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-
stamp/nccl_external-build
make -C src build BUILDDIR=/home/netfpga/research/collective/pytorch/build/nccl
make[1]: Entering directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src'
NVCC_GENCODE is -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_86,code=compute_86
...
...
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_f32.cu
[264/1934]nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_f64.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u32.cu
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u64.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
Compiling /home/netfpga/research/collective/pytorch/build/nccl/obj/device/gensrc/reduce_prod_u8.cu nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
make[2]: Leaving directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src/device'
Linking libnccl.so.2.19.3 > /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl.so.2.19.3
Archiving libnccl_static.a >
/home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a
/usr/bin/ld: cannot find -lcudart_static
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:79: /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl.so.2.19.3] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory '/home/netfpga/research/collective/pytorch/third_party/nccl/nccl/src'
make: *** [Makefile:25: src.build] Error 2
ninja: build stopped: subcommand failed.
Access full log here. In case it’s helpful.
The main error is:
[5996/6841] Performing build step for 'nccl_external'
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a /home/netfpga/research/collective/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build /home/netfpga/research/collective/pytorch/build/nccl/lib/libnccl_static.a
I’m on a tesla K80, so:
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__9_20:32:38_PDT_2021
Cuda compilation tools, release 11.3, V11.3.122
Build cuda_11.3.r11.3/compiler.30059648_0
$ nvidia-smi
Fri Nov 17 00:48:09 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02 Driver Version: 470.223.02 CUDA Version: 12.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla K80 Off | 00000000:04:00.0 Off | 0 |
| N/A 38C P8 26W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K80 Off | 00000000:05:00.0 Off | 0 |
| N/A 32C P8 30W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla K80 Off | 00000000:08:00.0 Off | 0 |
| N/A 32C P8 26W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla K80 Off | 00000000:09:00.0 Off | 0 |
| N/A 25C P8 30W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 Tesla K80 Off | 00000000:84:00.0 Off | 0 |
| N/A 31C P8 26W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 Tesla K80 Off | 00000000:85:00.0 Off | 0 |
| N/A 24C P8 31W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 Tesla K80 Off | 00000000:88:00.0 Off | 0 |
| N/A 31C P8 26W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 Tesla K80 Off | 00000000:89:00.0 Off | 0 |
| N/A 25C P8 29W / 149W | 4MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 1 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 2 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 3 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 4 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 5 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 6 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
| 7 N/A N/A 2079 G /usr/lib/xorg/Xorg 3MiB |
+-----------------------------------------------------------------------------+
Though nvidia-smi shows CUDA as 12.3, I’m on 11.3.1, which is the supported version for K80. The conda environment is:
$ conda list
# packages in environment at /home/netfpga/anaconda3/envs/pytorch_base:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
anaconda-client 1.11.2 py310h06a4308_0
anaconda-navigator 2.4.1 py310h06a4308_0
anaconda-project 0.11.1 py310h06a4308_0
asttokens 2.2.1 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
attrs 22.1.0 py310h06a4308_0
backcall 0.2.0 pypi_0 pypi
backports 1.1 pyhd3eb1b0_0
backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0
backports.tempfile 1.0 pyhd3eb1b0_1
backports.weakref 1.0.post1 py_1
beautifulsoup4 4.12.2 py310h06a4308_0
boltons 23.0.0 py310h06a4308_0
brotli-python 1.0.9 py310h6a678d5_7
brotlipy 0.7.0 py310h7f8727e_1002
bzip2 1.0.8 h7b6447c_0
c-ares 1.19.0 h5eee18b_0
ca-certificates 2023.08.22 h06a4308_0
certifi 2022.12.7 pypi_0 pypi
cffi 1.15.1 py310h5eee18b_3
chardet 4.0.0 py310h06a4308_1003
charset-normalizer 3.1.0 pypi_0 pypi
click 8.0.4 py310h06a4308_0
clyent 1.2.2 py310h06a4308_1
cmake 3.26.3 pypi_0 pypi
conda 23.5.0 py310h06a4308_0
conda-build 3.24.0 py310h06a4308_0
conda-content-trust 0.1.3 py310h06a4308_0
conda-pack 0.6.0 pyhd3eb1b0_0
conda-package-handling 2.0.2 py310h06a4308_0
conda-package-streaming 0.7.0 py310h06a4308_0
conda-repo-cli 1.0.41 py310h06a4308_0
conda-token 0.4.0 pyhd3eb1b0_0
conda-verify 3.4.2 py_1
cryptography 39.0.1 py310h9ce1e76_0
cuda-command-line-tools 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-compiler 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-cudart 11.3.109 hfb95d0c_0 nvidia/label/cuda-11.3.1
cuda-cuobjdump 11.3.122 hbf6ec6b_0 nvidia/label/cuda-11.3.1
cuda-cupti 11.3.111 h12ad217_0 nvidia/label/cuda-11.3.1
cuda-cuxxfilt 11.3.122 h4dc11a3_0 nvidia/label/cuda-11.3.1
cuda-gdb 11.3.109 h33b7820_0 nvidia/label/cuda-11.3.1
cuda-libraries 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-libraries-dev 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-memcheck 11.3.109 hf5cb439_0 nvidia/label/cuda-11.3.1
cuda-nvcc 11.3.122 h4814707_0 nvidia/label/cuda-11.3.1
cuda-nvdisasm 11.3.122 ha26faa6_0 nvidia/label/cuda-11.3.1
cuda-nvml-dev 11.3.58 hc25e488_0 nvidia/label/cuda-11.3.1
cuda-nvprof 11.3.111 h95a27d4_0 nvidia/label/cuda-11.3.1
cuda-nvprune 11.3.122 hb3346b8_0 nvidia/label/cuda-11.3.1
cuda-nvrtc 11.3.122 h1aa17d8_0 nvidia/label/cuda-11.3.1
cuda-nvtx 11.3.109 h4ec7630_0 nvidia/label/cuda-11.3.1
cuda-nvvp 11.3.111 h4c4416a_0 nvidia/label/cuda-11.3.1
cuda-samples 11.3.58 h6d5b628_0 nvidia/label/cuda-11.3.1
cuda-sanitizer-api 11.3.111 h2446cfc_0 nvidia/label/cuda-11.3.1
cuda-thrust 11.3.109 he8b717c_0 nvidia/label/cuda-11.3.1
cuda-toolkit 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-tools 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cuda-visual-tools 11.3.1 h712c49d_0 nvidia/label/cuda-11.3.1
cudatoolkit 11.3.1 h2bc3f7f_2
cudnn 8.2.1 cuda11.3_0
dbus 1.13.18 hb2f20db_0
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pyhd3eb1b0_0
exceptiongroup 1.1.1 pypi_0 pypi
executing 1.2.0 pypi_0 pypi
expat 2.4.9 h6a678d5_0
expecttest 0.1.4 pypi_0 pypi
filelock 3.12.0 pypi_0 pypi
fontconfig 2.14.1 h4c34cd2_2
freetype 2.12.1 h4a9f257_0
fsspec 2023.6.0 pypi_0 pypi
future 0.18.3 py310h06a4308_0
gdb 11.2 py310he3c4487_2
giflib 5.2.1 h5eee18b_3
glib 2.69.1 he621ea3_2
glob2 0.7 pyhd3eb1b0_0
gst-plugins-base 1.14.1 h6a678d5_1
gstreamer 1.14.1 h5eee18b_1
hypothesis 6.75.1 pypi_0 pypi
icu 58.2 he6710b0_3
idna 3.4 py310h06a4308_0
intel-openmp 2021.4.0 h06a4308_3561
ipython 8.14.0 pypi_0 pypi
jedi 0.19.0 pypi_0 pypi
jinja2 3.1.2 py310h06a4308_0
jpeg 9e h5eee18b_1
jsonpatch 1.32 pyhd3eb1b0_0
jsonpointer 2.1 pyhd3eb1b0_0
jsonschema 4.17.3 py310h06a4308_0
jupyter_core 5.3.0 py310h06a4308_0
krb5 1.19.4 h568e23c_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.38 h1181459_1
lerc 3.0 h295c915_0
libarchive 3.6.2 hb4bd9a0_1
libclang 14.0.6 default_hc6dbbc7_1
libclang13 14.0.6 default_he11475f_1
libcublas 11.5.1.109 h0fd73e7_0 nvidia/label/cuda-11.3.1
libcufft 10.4.2.109 h2344711_0 nvidia/label/cuda-11.3.1
libcurand 10.2.4.109 h0189693_0 nvidia/label/cuda-11.3.1
libcurl 7.88.1 h91b91d3_0
libcusolver 11.1.2.109 h1e009e5_0 nvidia/label/cuda-11.3.1
libcusparse 11.6.0.109 hf5bfba9_0 nvidia/label/cuda-11.3.1
libdeflate 1.17 h5eee18b_0
libedit 3.1.20221030 h5eee18b_0
libev 4.33 h7f8727e_1
libevent 2.1.12 h8f2d780_0
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
liblief 0.12.3 h6a678d5_0
libllvm14 14.0.6 hdb19cb5_3
libnghttp2 1.52.0 ha637b67_1
libnpp 11.3.3.95 h122bb27_0 nvidia/label/cuda-11.3.1
libnvjpeg 11.5.0.109 h159916b_0 nvidia/label/cuda-11.3.1
libpng 1.6.39 h5eee18b_0
libpq 12.9 h16c4e8d_3
libssh2 1.10.0 h37d81fd_2
libstdcxx-ng 11.2.0 h1234567_1
libtiff 4.5.0 h6a678d5_2
libuuid 1.41.5 h5eee18b_0
libuv 1.44.2 h5eee18b_0
libwebp 1.2.4 h11a3e52_1
libwebp-base 1.2.4 h5eee18b_1
libxcb 1.15 h7f8727e_0
libxkbcommon 1.0.1 h5eee18b_1
libxml2 2.10.3 hcbfbd50_0
libxslt 1.1.37 h2085143_0
lit 16.0.1 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_0
magma-cuda113 2.5.2 1 pytorch
markupsafe 2.1.2 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
mkl 2021.4.0 h06a4308_640
mkl-include 2023.1.0 h06a4308_46344
mpmath 1.3.0 pypi_0 pypi
navigator-updater 0.4.0 py310h06a4308_0
nbformat 5.7.0 py310h06a4308_0
ncurses 6.4 h6a678d5_0
networkx 3.1 pypi_0 pypi
ninja 1.10.2 h06a4308_5
ninja-base 1.10.2 hd09550d_5
nspr 4.35 h6a678d5_0
nss 3.89.1 h6a678d5_0
numpy 1.24.2 pypi_0 pypi
nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi
nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi
nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi
nvidia-cuda-runtime-cu113 11.3.109 pypi_0 pypi
nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi
nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
nvidia-curand-cu11 10.2.10.91 pypi_0 pypi
nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi
nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi
nvidia-ml-py3 7.352.0 pypi_0 pypi
nvidia-nccl-cu11 2.14.3 pypi_0 pypi
nvidia-nvtx-cu11 11.7.91 pypi_0 pypi
nvidia-pyindex 1.0.9 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
optree 0.10.0 pypi_0 pypi
packaging 23.0 py310h06a4308_0
parso 0.8.3 pypi_0 pypi
patch 2.7.6 h7b6447c_1001
patchelf 0.17.2 h6a678d5_0
pathlib 1.0.1 pyhd3eb1b0_1
pcre 8.45 h295c915_0
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.5.0 pypi_0 pypi
pip 22.3.1 py310h06a4308_0
pkginfo 1.9.6 py310h06a4308_0
platformdirs 2.5.2 py310h06a4308_0
pluggy 1.0.0 py310h06a4308_1
ply 3.11 py310h06a4308_0
prompt-toolkit 3.0.39 pypi_0 pypi
psutil 5.9.0 py310h5eee18b_0
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
py-lief 0.12.3 py310h6a678d5_0
pycosat 0.6.4 py310h5eee18b_0
pycparser 2.21 pyhd3eb1b0_0
pygments 2.15.1 py310h06a4308_1
pyjwt 2.4.0 py310h06a4308_0
pyopenssl 23.0.0 py310h06a4308_0
pyqt 5.15.7 py310h6a678d5_1
pyqt5-sip 12.11.0 pypi_0 pypi
pyrsistent 0.18.0 py310h7f8727e_0
pysocks 1.7.1 py310h06a4308_0
python 3.10.9 h7a1cb2a_1
python-dateutil 2.8.2 pyhd3eb1b0_0
python-fastjsonschema 2.16.2 py310h06a4308_0
python-libarchive-c 2.9 pyhd3eb1b0_1
pytorch-triton 2.1.0+6e4932cda8 pypi_0 pypi
pytz 2022.7 py310h06a4308_0
pyyaml 6.0 py310h5eee18b_1
qt-main 5.15.2 h8373d8f_8
qt-webengine 5.15.9 hbbf29b9_6
qtpy 2.2.0 py310h06a4308_0
qtwebkit 5.212 h3fafdc1_5
readline 8.2 h5eee18b_0
requests 2.28.2 pypi_0 pypi
requests-toolbelt 0.9.1 pyhd3eb1b0_0
rhash 1.4.1 h3c74f83_1
ripgrep 13.0.0 hbdeaff8_0
ruamel.yaml 0.17.21 py310h5eee18b_0
ruamel.yaml.clib 0.2.6 py310h5eee18b_1
ruamel_yaml 0.17.21 py310h5eee18b_0
scapy 2.5.0 pypi_0 pypi
setuptools 65.6.3 py310h06a4308_0
sip 6.6.2 py310h6a678d5_0
six 1.16.0 pyhd3eb1b0_1
sortedcontainers 2.4.0 pypi_0 pypi
soupsieve 2.4 py310h06a4308_0
sqlite 3.41.2 h5eee18b_0
stack-data 0.6.2 pypi_0 pypi
sympy 1.11.1 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
tomli 2.0.1 py310h06a4308_0
toolz 0.12.0 py310h06a4308_0
torch 1.13.1 pypi_0 pypi
torchaudio 0.13.1 pypi_0 pypi
torchtext 0.14.1 pypi_0 pypi
torchvision 0.14.1 pypi_0 pypi
tornado 6.2 py310h5eee18b_0
tqdm 4.65.0 py310h2f386ee_0
traitlets 5.7.1 py310h06a4308_0
types-dataclasses 0.6.6 pypi_0 pypi
typing-extensions 4.5.0 pypi_0 pypi
tzdata 2023c h04d1e81_0
ujson 5.4.0 py310h6a678d5_0
urllib3 1.26.15 pypi_0 pypi
wcwidth 0.2.6 pypi_0 pypi
wheel 0.38.4 py310h06a4308_0
xz 5.4.2 h5eee18b_0
yaml 0.2.5 h7b6447c_0
zlib 1.2.13 h5eee18b_0
zstandard 0.19.0 py310h5eee18b_0
zstd 1.5.5 hc292b87_0
I tried so many solutions on the forum including purging and reinstalling drivers to using gcc 10 and specifying CUDA home to be the Conda enviornment and many other countless solutions. But I still get this error. Any help in this regard is most welcome. I assume /usr/bin/ld: cannot find -lcudart_static has something to do with the error but can’t seems to solve it.
Also, please provide me full details for the solutions as i’m too new to this.
Thank you in advance.