I installed Pytorch using conda with CUDA on my local machine.
On running python -m torch.utils.collect_env and this is what I get -
Collecting environment information...
/home/adwaykanhere/anaconda3/lib/python3.9/site-packages/torch/cuda/__init__.py:88: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /opt/conda/conda-bld/pytorch_1670525539683/work/c10/cuda/CUDAFunctions.cpp:109.)
return torch._C._cuda_getDeviceCount() > 0
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
OS: Linux Mint 21.1 (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.9.13 (main, Aug 25 2022, 23:26:10) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 525.78.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] numpydoc==1.4.0
[pip3] pytorch-ignite==0.4.10
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas 1.0 mkl
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] magma-cuda111 2.5.2 1 pytorch
[conda] mkl 2021.4.0 h06a4308_640
[conda] mkl-include 2022.1.0 h06a4308_224
[conda] mkl-service 2.4.0 py39h7f8727e_0
[conda] mkl_fft 1.3.1 py39hd3c417c_0
[conda] mkl_random 1.2.2 py39h51133e4_0
[conda] numpy 1.21.5 py39h6c91a56_3
[conda] numpy-base 1.21.5 py39ha15fc14_3
[conda] numpydoc 1.4.0 py39h06a4308_0
[conda] pytorch 1.13.1 py3.9_cuda11.7_cudnn8.5.0_0 pytorch
[conda] pytorch-cuda 11.7 h67b0de4_1 pytorch
[conda] pytorch-ignite 0.4.10 pypi_0 pypi
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchaudio 0.13.1 py39_cu117 pytorch
[conda] torchvision 0.14.1 py39_cu117 pytorch
I have CUDA version 12.0 and the nvidia driver 525 installed. When I run torch.cuda.is_available() , I get FALSE. Nvidia-smi does detect the GPU but apparently, PyTorch cannot. Please help!
ptrblck
February 17, 2023, 5:33pm
2
This sounds like a driver or setup issue. E.g. just recently I helped a user who was unknowingly masking GPUs via CUDA_VISIBLE_DEVICES="" which then also disallowed them to use any available GPU.
Thanks @ptrblck . I’m not masking at all in code. I uninstalled cuda and nvidia-drivers again on my machine. Now I’m trying to build Pytorch from source and I get the build fails for the following -
pytorch) adwaykanhere@CaptainA:~$ sudo apt-get -y install cuda
Unpacking libcufft-12-0 (11.0.0.21-1) ...
Selecting previously unselected package libcufile-12-0.
Preparing to unpack .../08-libcufile-12-0_1.5.0.59-1_amd64.deb ...
Unpacking libcufile-12-0 (1.5.0.59-1) ...
Selecting previously unselected package libcurand-12-0.
Preparing to unpack .../09-libcurand-12-0_10.3.1.50-1_amd64.deb ...
Unpacking libcurand-12-0 (10.3.1.50-1) ...
Selecting previously unselected package libcusolver-12-0.
Preparing to unpack .../10-libcusolver-12-0_11.4.2.57-1_amd64.deb ...
Unpacking libcusolver-12-0 (11.4.2.57-1) ...
Selecting previously unselected package libcusparse-12-0.
Preparing to unpack .../11-libcusparse-12-0_12.0.0.76-1_amd64.deb ...
Unpacking libcusparse-12-0 (12.0.0.76-1) ...
Selecting previously unselected package libnpp-12-0.
Preparing to unpack .../12-libnpp-12-0_12.0.0.30-1_amd64.deb ...
Unpacking libnpp-12-0 (12.0.0.30-1) ...
Selecting previously unselected package libnvjitlink-12-0.
Preparing to unpack .../13-libnvjitlink-12-0_12.0.76-1_amd64.deb ...
Unpacking libnvjitlink-12-0 (12.0.76-1) ...
Selecting previously unselected package libnvjpeg-12-0.
Preparing to unpack .../14-libnvjpeg-12-0_12.0.0.28-1_amd64.deb ...
Unpacking libnvjpeg-12-0 (12.0.0.28-1) ...
Selecting previously unselected package cuda-libraries-12-0.
Preparing to unpack .../15-cuda-libraries-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-libraries-12-0 (12.0.0-1) ...
Selecting previously unselected package nvidia-modprobe.
Preparing to unpack .../16-nvidia-modprobe_525.60.13-0ubuntu1_amd64.deb ...
Unpacking nvidia-modprobe (525.60.13-0ubuntu1) ...
Preparing to unpack .../17-nvidia-settings_525.60.13-0ubuntu1_amd64.deb ...
Unpacking nvidia-settings (525.60.13-0ubuntu1) over (510.47.03-0ubuntu1) ...
Selecting previously unselected package cuda-drivers-525.
Preparing to unpack .../18-cuda-drivers-525_525.60.13-1_amd64.deb ...
Unpacking cuda-drivers-525 (525.60.13-1) ...
Selecting previously unselected package cuda-drivers.
Preparing to unpack .../19-cuda-drivers_525.60.13-1_amd64.deb ...
Unpacking cuda-drivers (525.60.13-1) ...
Selecting previously unselected package cuda-runtime-12-0.
Preparing to unpack .../20-cuda-runtime-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-runtime-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-cuobjdump-12-0.
Preparing to unpack .../21-cuda-cuobjdump-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-cuobjdump-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-cuxxfilt-12-0.
Preparing to unpack .../22-cuda-cuxxfilt-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-cuxxfilt-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-cccl-12-0.
Preparing to unpack .../23-cuda-cccl-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-cccl-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-driver-dev-12-0.
Preparing to unpack .../24-cuda-driver-dev-12-0_12.0.107-1_amd64.deb ...
Unpacking cuda-driver-dev-12-0 (12.0.107-1) ...
Selecting previously unselected package cuda-cudart-dev-12-0.
Preparing to unpack .../25-cuda-cudart-dev-12-0_12.0.107-1_amd64.deb ...
Unpacking cuda-cudart-dev-12-0 (12.0.107-1) ...
Selecting previously unselected package cuda-nvcc-12-0.
Preparing to unpack .../26-cuda-nvcc-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvcc-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-nvprune-12-0.
Preparing to unpack .../27-cuda-nvprune-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvprune-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-compiler-12-0.
Preparing to unpack .../28-cuda-compiler-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-compiler-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-profiler-api-12-0.
Preparing to unpack .../29-cuda-profiler-api-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-profiler-api-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-nvrtc-dev-12-0.
Preparing to unpack .../30-cuda-nvrtc-dev-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvrtc-dev-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-opencl-dev-12-0.
Preparing to unpack .../31-cuda-opencl-dev-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-opencl-dev-12-0 (12.0.76-1) ...
Selecting previously unselected package libcublas-dev-12-0.
Preparing to unpack .../32-libcublas-dev-12-0_12.0.1.189-1_amd64.deb ...
Unpacking libcublas-dev-12-0 (12.0.1.189-1) ...
Selecting previously unselected package libcufft-dev-12-0.
Preparing to unpack .../33-libcufft-dev-12-0_11.0.0.21-1_amd64.deb ...
Unpacking libcufft-dev-12-0 (11.0.0.21-1) ...
Selecting previously unselected package libcufile-dev-12-0.
Preparing to unpack .../34-libcufile-dev-12-0_1.5.0.59-1_amd64.deb ...
Unpacking libcufile-dev-12-0 (1.5.0.59-1) ...
Selecting previously unselected package libcurand-dev-12-0.
Preparing to unpack .../35-libcurand-dev-12-0_10.3.1.50-1_amd64.deb ...
Unpacking libcurand-dev-12-0 (10.3.1.50-1) ...
Selecting previously unselected package libcusolver-dev-12-0.
Preparing to unpack .../36-libcusolver-dev-12-0_11.4.2.57-1_amd64.deb ...
Unpacking libcusolver-dev-12-0 (11.4.2.57-1) ...
Selecting previously unselected package libcusparse-dev-12-0.
Preparing to unpack .../37-libcusparse-dev-12-0_12.0.0.76-1_amd64.deb ...
Unpacking libcusparse-dev-12-0 (12.0.0.76-1) ...
Selecting previously unselected package libnpp-dev-12-0.
Preparing to unpack .../38-libnpp-dev-12-0_12.0.0.30-1_amd64.deb ...
Unpacking libnpp-dev-12-0 (12.0.0.30-1) ...
Selecting previously unselected package libnvjitlink-dev-12-0.
Preparing to unpack .../39-libnvjitlink-dev-12-0_12.0.76-1_amd64.deb ...
Unpacking libnvjitlink-dev-12-0 (12.0.76-1) ...
Selecting previously unselected package libnvjpeg-dev-12-0.
Preparing to unpack .../40-libnvjpeg-dev-12-0_12.0.0.28-1_amd64.deb ...
Unpacking libnvjpeg-dev-12-0 (12.0.0.28-1) ...
Selecting previously unselected package cuda-libraries-dev-12-0.
Preparing to unpack .../41-cuda-libraries-dev-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-libraries-dev-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-cupti-12-0.
Preparing to unpack .../42-cuda-cupti-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-cupti-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-cupti-dev-12-0.
Preparing to unpack .../43-cuda-cupti-dev-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-cupti-dev-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-nvdisasm-12-0.
Preparing to unpack .../44-cuda-nvdisasm-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvdisasm-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-gdb-12-0.
Preparing to unpack .../45-cuda-gdb-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-gdb-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-nvprof-12-0.
Preparing to unpack .../46-cuda-nvprof-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-nvprof-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-nvtx-12-0.
Preparing to unpack .../47-cuda-nvtx-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvtx-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-sanitizer-12-0.
Preparing to unpack .../48-cuda-sanitizer-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-sanitizer-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-command-line-tools-12-0.
Preparing to unpack .../49-cuda-command-line-tools-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-command-line-tools-12-0 (12.0.0-1) ...
Selecting previously unselected package nsight-compute-2022.4.0.
Preparing to unpack .../50-nsight-compute-2022.4.0_2022.4.0.15-1_amd64.deb ...
Unpacking nsight-compute-2022.4.0 (2022.4.0.15-1) ...
Selecting previously unselected package cuda-nsight-compute-12-0.
Preparing to unpack .../51-cuda-nsight-compute-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-nsight-compute-12-0 (12.0.0-1) ...
Selecting previously unselected package libtinfo5:amd64.
Preparing to unpack .../52-libtinfo5_6.3-2_amd64.deb ...
Unpacking libtinfo5:amd64 (6.3-2) ...
Selecting previously unselected package nsight-systems-2022.4.2.
Preparing to unpack .../53-nsight-systems-2022.4.2_2022.4.2.18-1_amd64.deb ...
Unpacking nsight-systems-2022.4.2 (2022.4.2.18-32044700v0) ...
Selecting previously unselected package cuda-nsight-systems-12-0.
Preparing to unpack .../54-cuda-nsight-systems-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-nsight-systems-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-nsight-12-0.
Preparing to unpack .../55-cuda-nsight-12-0_12.0.78-1_amd64.deb ...
Unpacking cuda-nsight-12-0 (12.0.78-1) ...
Selecting previously unselected package cuda-nvml-dev-12-0.
Preparing to unpack .../56-cuda-nvml-dev-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-nvml-dev-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-nvvp-12-0.
Preparing to unpack .../57-cuda-nvvp-12-0_12.0.90-1_amd64.deb ...
Unpacking cuda-nvvp-12-0 (12.0.90-1) ...
Selecting previously unselected package cuda-visual-tools-12-0.
Preparing to unpack .../58-cuda-visual-tools-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-visual-tools-12-0 (12.0.0-1) ...
Selecting previously unselected package gds-tools-12-0.
Preparing to unpack .../59-gds-tools-12-0_1.5.0.59-1_amd64.deb ...
Unpacking gds-tools-12-0 (1.5.0.59-1) ...
Selecting previously unselected package cuda-tools-12-0.
Preparing to unpack .../60-cuda-tools-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-tools-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-documentation-12-0.
Preparing to unpack .../61-cuda-documentation-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-documentation-12-0 (12.0.76-1) ...
Selecting previously unselected package libnvvm-samples-12-0.
Preparing to unpack .../62-libnvvm-samples-12-0_12.0.94-1_amd64.deb ...
Unpacking libnvvm-samples-12-0 (12.0.94-1) ...
Selecting previously unselected package cuda-toolkit-12-0.
Preparing to unpack .../63-cuda-toolkit-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-toolkit-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda-demo-suite-12-0.
Preparing to unpack .../64-cuda-demo-suite-12-0_12.0.76-1_amd64.deb ...
Unpacking cuda-demo-suite-12-0 (12.0.76-1) ...
Selecting previously unselected package cuda-12-0.
Preparing to unpack .../65-cuda-12-0_12.0.0-1_amd64.deb ...
Unpacking cuda-12-0 (12.0.0-1) ...
Selecting previously unselected package cuda.
Preparing to unpack .../66-cuda_12.0.0-1_amd64.deb ...
Unpacking cuda (12.0.0-1) ...
Setting up cuda-driver-dev-12-0 (12.0.107-1) ...
Setting up cuda-toolkit-config-common (12.0.107-1) ...
Setting up cuda-nvvp-12-0 (12.0.90-1) ...
Setting up nsight-compute-2022.4.0 (2022.4.0.15-1) ...
Setting up cuda-profiler-api-12-0 (12.0.76-1) ...
Setting up cuda-nvdisasm-12-0 (12.0.76-1) ...
Setting up cuda-toolkit-12-0-config-common (12.0.107-1) ...
Setting alternatives
update-alternatives: using /usr/local/cuda-12.0 to provide /usr/local/cuda (cuda) in auto mode
update-alternatives: using /usr/local/cuda-12.0 to provide /usr/local/cuda-12 (cuda-12) in auto mode
Setting up cuda-nsight-compute-12-0 (12.0.0-1) ...
Setting up cuda-nvtx-12-0 (12.0.76-1) ...
Setting up cuda-cuxxfilt-12-0 (12.0.76-1) ...
Setting up libnvvm-samples-12-0 (12.0.94-1) ...
Setting up cuda-cccl-12-0 (12.0.90-1) ...
Setting up nvidia-modprobe (525.60.13-0ubuntu1) ...
Setting up cuda-nsight-12-0 (12.0.78-1) ...
Setting up gds-tools-12-0 (1.5.0.59-1) ...
Setting up cuda-nvprof-12-0 (12.0.90-1) ...
Setting up cuda-cuobjdump-12-0 (12.0.76-1) ...
Setting up cuda-nvrtc-12-0 (12.0.76-1) ...
Setting up cuda-sanitizer-12-0 (12.0.90-1) ...
Setting up cuda-nvprune-12-0 (12.0.76-1) ...
Setting up cuda-cupti-12-0 (12.0.90-1) ...
Setting up nvidia-settings (525.60.13-0ubuntu1) ...
Setting up cuda-nvml-dev-12-0 (12.0.76-1) ...
Setting up libtinfo5:amd64 (6.3-2) ...
Setting up cuda-nvrtc-dev-12-0 (12.0.76-1) ...
Setting up cuda-toolkit-12-config-common (12.0.107-1) ...
Setting up cuda-gdb-12-0 (12.0.90-1) ...
Setting up cuda-documentation-12-0 (12.0.76-1) ...
Setting up nsight-systems-2022.4.2 (2022.4.2.18-32044700v0) ...
update-alternatives: using /opt/nvidia/nsight-systems/2022.4.2/target-linux-x64/nsys to provide /usr/local/bin/nsys (nsys) in auto mode
update-alternatives: using /opt/nvidia/nsight-systems/2022.4.2/host-linux-x64/nsys-ui to provide /usr/local/bin/nsys-ui (nsys-ui) in auto mode
Setting up cuda-drivers-525 (525.60.13-1) ...
Setting up libcusolver-12-0 (11.4.2.57-1) ...
Setting up cuda-nsight-systems-12-0 (12.0.0-1) ...
Setting up libnvjpeg-12-0 (12.0.0.28-1) ...
Setting up libcusolver-dev-12-0 (11.4.2.57-1) ...
Setting up libcufile-12-0 (1.5.0.59-1) ...
Setting alternatives
update-alternatives: using /usr/local/cuda-12.0/gds/cufile.json to provide /etc/cufile.json (cufile.json) in auto mode
Setting up cuda-cudart-12-0 (12.0.107-1) ...
Setting up cuda-cupti-dev-12-0 (12.0.90-1) ...
Setting up cuda-cudart-dev-12-0 (12.0.107-1) ...
Setting up libcufft-12-0 (11.0.0.21-1) ...
Setting up libcublas-12-0 (12.0.1.189-1) ...
Setting up libnvjpeg-dev-12-0 (12.0.0.28-1) ...
Setting up libcusparse-12-0 (12.0.0.76-1) ...
Setting up cuda-drivers (525.60.13-1) ...
Setting up libnvjitlink-12-0 (12.0.76-1) ...
Setting up libcufile-dev-12-0 (1.5.0.59-1) ...
Setting up libcurand-12-0 (10.3.1.50-1) ...
Setting up libcufft-dev-12-0 (11.0.0.21-1) ...
Setting up libnpp-12-0 (12.0.0.30-1) ...
Setting up cuda-command-line-tools-12-0 (12.0.0-1) ...
Setting up libnpp-dev-12-0 (12.0.0.30-1) ...
Setting up cuda-opencl-12-0 (12.0.76-1) ...
Setting up libnvjitlink-dev-12-0 (12.0.76-1) ...
Setting up cuda-libraries-12-0 (12.0.0-1) ...
Setting up cuda-runtime-12-0 (12.0.0-1) ...
Setting up libcurand-dev-12-0 (10.3.1.50-1) ...
Setting up libcusparse-dev-12-0 (12.0.0.76-1) ...
Setting up libcublas-dev-12-0 (12.0.1.189-1) ...
Setting up cuda-nvcc-12-0 (12.0.76-1) ...
Setting up cuda-opencl-dev-12-0 (12.0.76-1) ...
Setting up cuda-compiler-12-0 (12.0.0-1) ...
Setting up cuda-libraries-dev-12-0 (12.0.0-1) ...
Setting up cuda-demo-suite-12-0 (12.0.76-1) ...
Setting up cuda-visual-tools-12-0 (12.0.0-1) ...
Setting up cuda-tools-12-0 (12.0.0-1) ...
Setting up cuda-toolkit-12-0 (12.0.0-1) ...
Setting up cuda-12-0 (12.0.0-1) ...
Setting up cuda (12.0.0-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Processing triggers for desktop-file-utils (0.26+mint2+vera) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
(pytorch) adwaykanhere@CaptainA:~$ cd pytorch/
(pytorch) adwaykanhere@CaptainA:~/pytorch$ export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py develop
Building wheel torch-2.0.0a0+git69bcefc
-- Building version 2.0.0a0+git69bcefc
cmake --build . --target install --config Release
[4/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks
FAILED: bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks.dir/impl/CUDAAssertionsTest_multiple_writes_from_multiple_blocks.cu.o -o bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[5/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_1_var_test
FAILED: bin/c10_cuda_CUDAAssertionsTest_1_var_test
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_1_var_test.dir/impl/CUDAAssertionsTest_1_var_test.cu.o -o bin/c10_cuda_CUDAAssertionsTest_1_var_test -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[6/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_from_2_processes
FAILED: bin/c10_cuda_CUDAAssertionsTest_from_2_processes
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_from_2_processes.dir/impl/CUDAAssertionsTest_from_2_processes.cu.o -o bin/c10_cuda_CUDAAssertionsTest_from_2_processes -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[7/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device
FAILED: bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device.dir/impl/CUDAAssertionsTest_catches_thread_and_block_and_device.cu.o -o bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[8/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_catches_stream
FAILED: bin/c10_cuda_CUDAAssertionsTest_catches_stream
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_catches_stream.dir/impl/CUDAAssertionsTest_catches_stream.cu.o -o bin/c10_cuda_CUDAAssertionsTest_catches_stream -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[9/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block
FAILED: bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block.dir/impl/CUDAAssertionsTest_multiple_writes_from_same_block.cu.o -o bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[10/2056] Linking CXX executable bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads
FAILED: bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads
: && /usr/bin/c++ -D_GLIBCXX_USE_CXX11_ABI=1 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -O3 -DNDEBUG -DNDEBUG -rdynamic c10/cuda/test/CMakeFiles/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads.dir/impl/CUDAAssertionsTest_multiple_writes_from_blocks_and_threads.cu.o -o bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads -L/home/adwaykanhere/anaconda3/lib/stubs -L/home/adwaykanhere/anaconda3/lib -Wl,-rpath,/home/adwaykanhere/pytorch/build/lib:/home/adwaykanhere/anaconda3/lib: lib/libc10_cuda.so lib/libgtest_main.a lib/libc10.so /home/adwaykanhere/anaconda3/lib/libcudart.so lib/libgtest.a -lcudadevrt -lcudart_static -lrt -lpthread -ldl && :
/usr/bin/ld: lib/libc10.so: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
[11/2056] Performing build step for 'nccl_external'
make -C src build BUILDDIR=/home/adwaykanhere/pytorch/build/nccl
make[1]: Entering directory '/home/adwaykanhere/pytorch/third_party/nccl/nccl/src'
make[2]: Entering directory '/home/adwaykanhere/pytorch/third_party/nccl/nccl/src/collectives/device'
make[2]: Leaving directory '/home/adwaykanhere/pytorch/third_party/nccl/nccl/src/collectives/device'
Linking libnccl.so.2.14.3 > /home/adwaykanhere/pytorch/build/nccl/lib/libnccl.so.2.14.3
make[1]: Leaving directory '/home/adwaykanhere/pytorch/third_party/nccl/nccl/src'
[21/2056] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/SparseTensorImpl.cpp.o
ninja: build stopped: subcommand failed.
ptrblck
February 17, 2023, 6:41pm
4
Building from source won’t solve the issue and I would still recommend checking your setup.
Thanks @ptrblck I’m new to using CUDA. What do you mean by writing cuda samples? Like writing cuda tensors operations?
ptrblck
February 17, 2023, 9:54pm
6
You could start with building and running these cuda-samples and make sure your actual setup is working fine.
But it seems you’ve already solved the issue, which is great to hear!