NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

1

I’m running ubuntu 18.04
I upgraded cuda 9 to 10, but I cannot get nvidia-smi to work properly

I tried every suggestions on this thread without success.

I installed cuda10 and the drivers with this method

Some more info about my setup:

$ uname -a
Linux carmen 4.15.0-041500-generic #201802011154 SMP Thu Feb 1 11:55:45 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

$ nvidia-settings
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help` for usage information.

$ dpkg -l | grep -i nvidia
ii  cuda-nsight-compute-10-1                        10.1.168-1                                          amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-10-1                        10.1.168-1                                          amd64        NVIDIA Nsight Systems
ii  cuda-nvtx-10-1                                  10.1.168-1                                          amd64        NVIDIA Tools Extension
ii  libcupti-dev:amd64                              9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Profiler Tools Interface development files
ii  libcupti-doc                                    9.1.85-3ubuntu1                                     all          NVIDIA CUDA Profiler Tools Interface documentation
ii  libcupti9.1:amd64                               9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Profiler Tools Interface runtime library
ii  libnvidia-cfg1-418:amd64                        418.67-0ubuntu1                                     amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-418                            418.67-0ubuntu1                                     all          Shared files used by the NVIDIA libraries
rc  libnvidia-compute-390:i386                      390.116-0ubuntu0.18.04.1                            i386         NVIDIA libcompute package
ii  libnvidia-compute-418:amd64                     418.67-0ubuntu1                                     amd64        NVIDIA libcompute package
ii  libnvidia-decode-418:amd64                      418.67-0ubuntu1                                     amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-418:amd64                      418.67-0ubuntu1                                     amd64        NVENC Video Encoding runtime library
ii  libnvidia-fbc1-418:amd64                        418.67-0ubuntu1                                     amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-418:amd64                          418.67-0ubuntu1                                     amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-ifr1-418:amd64                        418.67-0ubuntu1                                     amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  nvidia-compute-utils-418                        418.67-0ubuntu1                                     amd64        NVIDIA compute utilities
ii  nvidia-dkms-418                                 418.67-0ubuntu1                                     amd64        NVIDIA DKMS package
ii  nvidia-driver-418                               418.67-0ubuntu1                                     amd64        NVIDIA driver metapackage
ii  nvidia-kernel-common-418                        418.67-0ubuntu1                                     amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-418                        418.67-0ubuntu1                                     amd64        NVIDIA kernel source package
ii  nvidia-modprobe                                 418.67-0ubuntu1                                     amd64        Load the NVIDIA kernel driver and create device files
ii  nvidia-prime                                    0.8.8.2                                             all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                                 418.67-0ubuntu1                                     amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-418                                418.67-0ubuntu1                                     amd64        NVIDIA driver support binaries
ii  xserver-xorg-video-nvidia-418                   418.67-0ubuntu1                                     amd64        NVIDIA binary Xorg driver

$ dpkg -l | grep -i cuda
ii  cuda                                            10.1.168-1                                          amd64        CUDA meta-package
ii  cuda-10-1                                       10.1.168-1                                          amd64        CUDA 10.1 meta-package
ii  cuda-command-line-tools-10-1                    10.1.168-1                                          amd64        CUDA command-line tools
rc  cuda-command-line-tools-9-0                     9.0.176-1                                           amd64        CUDA command-line tools
ii  cuda-compiler-10-1                              10.1.168-1                                          amd64        CUDA compiler
rc  cuda-core-9-0                                   9.0.176-1                                           amd64        CUDA core tools
ii  cuda-cudart-10-1                                10.1.168-1                                          amd64        CUDA Runtime native Libraries
rc  cuda-cudart-9-0                                 9.0.176-1                                           amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-10-1                            10.1.168-1                                          amd64        CUDA Runtime native dev links, headers
rc  cuda-cudart-dev-9-0                             9.0.176-1                                           amd64        CUDA Runtime native dev links, headers
ii  cuda-cufft-10-1                                 10.1.168-1                                          amd64        CUFFT native runtime libraries
rc  cuda-cufft-9-0                                  9.0.176-1                                           amd64        CUFFT native runtime libraries
ii  cuda-cufft-dev-10-1                             10.1.168-1                                          amd64        CUFFT native dev links, headers
ii  cuda-cuobjdump-10-1                             10.1.168-1                                          amd64        CUDA cuobjdump
ii  cuda-cupti-10-1                                 10.1.168-1                                          amd64        CUDA profiling tools interface.
ii  cuda-curand-10-1                                10.1.168-1                                          amd64        CURAND native runtime libraries
rc  cuda-curand-9-0                                 9.0.176-1                                           amd64        CURAND native runtime libraries
ii  cuda-curand-dev-10-1                            10.1.168-1                                          amd64        CURAND native dev links, headers
ii  cuda-cusolver-10-1                              10.1.168-1                                          amd64        CUDA solver native runtime libraries
rc  cuda-cusolver-9-0                               9.0.176-1                                           amd64        CUDA solver native runtime libraries
ii  cuda-cusolver-dev-10-1                          10.1.168-1                                          amd64        CUDA solver native dev links, headers
ii  cuda-cusparse-10-1                              10.1.168-1                                          amd64        CUSPARSE native runtime libraries
rc  cuda-cusparse-9-0                               9.0.176-1                                           amd64        CUSPARSE native runtime libraries
ii  cuda-cusparse-dev-10-1                          10.1.168-1                                          amd64        CUSPARSE native dev links, headers
ii  cuda-demo-suite-10-1                            10.1.168-1                                          amd64        Demo suite for CUDA
ii  cuda-documentation-10-1                         10.1.168-1                                          amd64        CUDA documentation
ii  cuda-driver-dev-10-1                            10.1.168-1                                          amd64        CUDA Driver native dev stub library
ii  cuda-drivers                                    418.67-1                                            amd64        CUDA Driver meta-package
ii  cuda-gdb-10-1                                   10.1.168-1                                          amd64        CUDA-GDB
ii  cuda-gpu-library-advisor-10-1                   10.1.168-1                                          amd64        CUDA GPU Library Advisor.
ii  cuda-libraries-10-1                             10.1.168-1                                          amd64        CUDA Libraries 10.1 meta-package
ii  cuda-libraries-dev-10-1                         10.1.168-1                                          amd64        CUDA Libraries 10.1 development meta-package
ii  cuda-license-10-1                               10.1.168-1                                          amd64        CUDA licenses
ii  cuda-memcheck-10-1                              10.1.168-1                                          amd64        CUDA-MEMCHECK
ii  cuda-misc-headers-10-1                          10.1.168-1                                          amd64        CUDA miscellaneous headers
ii  cuda-npp-10-1                                   10.1.168-1                                          amd64        NPP native runtime libraries
rc  cuda-npp-9-0                                    9.0.176-1                                           amd64        NPP native runtime libraries
ii  cuda-npp-dev-10-1                               10.1.168-1                                          amd64        NPP native dev links, headers
ii  cuda-nsight-10-1                                10.1.168-1                                          amd64        CUDA nsight
ii  cuda-nsight-compute-10-1                        10.1.168-1                                          amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-10-1                        10.1.168-1                                          amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-10-1                                  10.1.168-1                                          amd64        CUDA nvcc
ii  cuda-nvdisasm-10-1                              10.1.168-1                                          amd64        CUDA disassembler
ii  cuda-nvgraph-10-1                               10.1.168-1                                          amd64        NVGRAPH native runtime libraries
rc  cuda-nvgraph-9-0                                9.0.176-1                                           amd64        NVGRAPH native runtime libraries
ii  cuda-nvgraph-dev-10-1                           10.1.168-1                                          amd64        NVGRAPH native dev links, headers
ii  cuda-nvjpeg-10-1                                10.1.168-1                                          amd64        NVJPEG native runtime libraries
ii  cuda-nvjpeg-dev-10-1                            10.1.168-1                                          amd64        NVJPEG native dev links, headers
ii  cuda-nvml-dev-10-1                              10.1.168-1                                          amd64        NVML native dev links, headers
ii  cuda-nvprof-10-1                                10.1.168-1                                          amd64        CUDA Profiler tools
ii  cuda-nvprune-10-1                               10.1.168-1                                          amd64        CUDA nvprune
ii  cuda-nvrtc-10-1                                 10.1.168-1                                          amd64        NVRTC native runtime libraries
rc  cuda-nvrtc-9-0                                  9.0.176-1                                           amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-10-1                             10.1.168-1                                          amd64        NVRTC native dev links, headers
ii  cuda-nvtx-10-1                                  10.1.168-1                                          amd64        NVIDIA Tools Extension
ii  cuda-nvvp-10-1                                  10.1.168-1                                          amd64        CUDA nvvp
ii  cuda-repo-ubuntu1604-8-0-local-ga2              8.0.61-1                                            amd64        cuda repository configuration files
ii  cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67 1.0-1                                               amd64        cuda repository configuration files
ii  cuda-runtime-10-1                               10.1.168-1                                          amd64        CUDA Runtime 10.1 meta-package
ii  cuda-samples-10-1                               10.1.168-1                                          amd64        CUDA example applications
ii  cuda-sanitizer-api-10-1                         10.1.168-1                                          amd64        CUDA Sanitizer API
ii  cuda-toolkit-10-1                               10.1.168-1                                          amd64        CUDA Toolkit 10.1 meta-package
ii  cuda-tools-10-1                                 10.1.168-1                                          amd64        CUDA Tools meta-package
ii  cuda-visual-tools-10-1                          10.1.168-1                                          amd64        CUDA visual tools
ii  libcudnn7                                       7.0.5.15-1+cuda9.0                                  amd64        cuDNN runtime libraries
ii  libcupti-dev:amd64                              9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Profiler Tools Interface development files
ii  libcupti-doc                                    9.1.85-3ubuntu1                                     all          NVIDIA CUDA Profiler Tools Interface documentation
ii  libcupti9.1:amd64                               9.1.85-3ubuntu1                                     amd64        NVIDIA CUDA Profiler Tools Interface runtime library
1 Like
2

I tried reinstalling cuda-10.0 (down from 10.1) and nvidia-driver-410 without success. I was worried that cuda-cublas was not listed when listing cuda packages with the cuda-10.1 install. But it looks like this is not the issue. cublas is now listed with cuda 10.0 and I have the same output with nvidia-smi

I tried some of the suggestions listed here. Same issue

$ dmesg | grep NVIDIA
[    6.416468] nouveau 0000:24:00.0: NVIDIA GP102 (132000a1)

$ dpkg -l | grep nvidia
ii  libnvidia-cfg1-410:amd64                        410.48-0ubuntu1                                     amd64        NVIDIA binary OpenGL/GLX configuration library
ii  libnvidia-common-410                            410.48-0ubuntu1                                     all          Shared files used by the NVIDIA libraries
rc  libnvidia-compute-390:i386                      390.116-0ubuntu0.18.04.1                            i386         NVIDIA libcompute package
ii  libnvidia-compute-410:amd64                     410.48-0ubuntu1                                     amd64        NVIDIA libcompute package
ii  libnvidia-decode-410:amd64                      410.48-0ubuntu1                                     amd64        NVIDIA Video Decoding runtime libraries
ii  libnvidia-encode-410:amd64                      410.48-0ubuntu1                                     amd64        NVENC Video Encoding runtime library
ii  libnvidia-fbc1-410:amd64                        410.48-0ubuntu1                                     amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
ii  libnvidia-gl-410:amd64                          410.48-0ubuntu1                                     amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii  libnvidia-ifr1-410:amd64                        410.48-0ubuntu1                                     amd64        NVIDIA OpenGL-based Inband Frame Readback runtime library
ii  nvidia-compute-utils-410                        410.48-0ubuntu1                                     amd64        NVIDIA compute utilities
ii  nvidia-dkms-410                                 410.48-0ubuntu1                                     amd64        NVIDIA DKMS package
ii  nvidia-driver-410                               410.48-0ubuntu1                                     amd64        NVIDIA driver metapackage
ii  nvidia-kernel-common-410                        410.48-0ubuntu1                                     amd64        Shared files used with the kernel module
ii  nvidia-kernel-source-410                        410.48-0ubuntu1                                     amd64        NVIDIA kernel source package
ii  nvidia-prime                                    0.8.8.2                                             all          Tools to enable NVIDIA's Prime
ii  nvidia-settings                                 410.48-0ubuntu1                                     amd64        Tool for configuring the NVIDIA graphics driver
ii  nvidia-utils-410                                410.48-0ubuntu1                                     amd64        NVIDIA driver support binaries
ii  xserver-xorg-video-nvidia-410                   410.48-0ubuntu1                                     amd64        NVIDIA binary Xorg driver

$ dpkg -l | grep cuda
ii  cuda-command-line-tools-10-0                    10.0.130-1                                          amd64        CUDA command-line tools
ii  cuda-compiler-10-0                              10.0.130-1                                          amd64        CUDA compiler
ii  cuda-cublas-10-0                                10.0.130-1                                          amd64        CUBLAS native runtime libraries
ii  cuda-cublas-dev-10-0                            10.0.130-1                                          amd64        CUBLAS native dev links, headers
ii  cuda-cudart-10-0                                10.0.130-1                                          amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-10-0                            10.0.130-1                                          amd64        CUDA Runtime native dev links, headers
ii  cuda-cufft-10-0                                 10.0.130-1                                          amd64        CUFFT native runtime libraries
ii  cuda-cufft-dev-10-0                             10.0.130-1                                          amd64        CUFFT native dev links, headers
ii  cuda-cuobjdump-10-0                             10.0.130-1                                          amd64        CUDA cuobjdump
ii  cuda-cupti-10-0                                 10.0.130-1                                          amd64        CUDA profiling tools interface.
ii  cuda-curand-10-0                                10.0.130-1                                          amd64        CURAND native runtime libraries
ii  cuda-curand-dev-10-0                            10.0.130-1                                          amd64        CURAND native dev links, headers
ii  cuda-cusolver-10-0                              10.0.130-1                                          amd64        CUDA solver native runtime libraries
ii  cuda-cusolver-dev-10-0                          10.0.130-1                                          amd64        CUDA solver native dev links, headers
ii  cuda-cusparse-10-0                              10.0.130-1                                          amd64        CUSPARSE native runtime libraries
ii  cuda-cusparse-dev-10-0                          10.0.130-1                                          amd64        CUSPARSE native dev links, headers
ii  cuda-documentation-10-0                         10.0.130-1                                          amd64        CUDA documentation
ii  cuda-driver-dev-10-0                            10.0.130-1                                          amd64        CUDA Driver native dev stub library
ii  cuda-gdb-10-0                                   10.0.130-1                                          amd64        CUDA-GDB
ii  cuda-gpu-library-advisor-10-0                   10.0.130-1                                          amd64        CUDA GPU Library Advisor.
ii  cuda-libraries-10-0                             10.0.130-1                                          amd64        CUDA Libraries 10.0 meta-package
ii  cuda-libraries-dev-10-0                         10.0.130-1                                          amd64        CUDA Libraries 10.0 development meta-package
ii  cuda-license-10-0                               10.0.130-1                                          amd64        CUDA licenses
ii  cuda-memcheck-10-0                              10.0.130-1                                          amd64        CUDA-MEMCHECK
ii  cuda-misc-headers-10-0                          10.0.130-1                                          amd64        CUDA miscellaneous headers
ii  cuda-npp-10-0                                   10.0.130-1                                          amd64        NPP native runtime libraries
ii  cuda-npp-dev-10-0                               10.0.130-1                                          amd64        NPP native dev links, headers
ii  cuda-nsight-10-0                                10.0.130-1                                          amd64        CUDA nsight
ii  cuda-nsight-compute-10-0                        10.0.130-1                                          amd64        NVIDIA Nsight Compute
ii  cuda-nvcc-10-0                                  10.0.130-1                                          amd64        CUDA nvcc
ii  cuda-nvdisasm-10-0                              10.0.130-1                                          amd64        CUDA disassembler
ii  cuda-nvgraph-10-0                               10.0.130-1                                          amd64        NVGRAPH native runtime libraries
ii  cuda-nvgraph-dev-10-0                           10.0.130-1                                          amd64        NVGRAPH native dev links, headers
ii  cuda-nvjpeg-10-0                                10.0.130-1                                          amd64        NVJPEG native runtime libraries
ii  cuda-nvjpeg-dev-10-0                            10.0.130-1                                          amd64        NVJPEG native dev links, headers
ii  cuda-nvml-dev-10-0                              10.0.130-1                                          amd64        NVML native dev links, headers
ii  cuda-nvprof-10-0                                10.0.130-1                                          amd64        CUDA Profiler tools
ii  cuda-nvprune-10-0                               10.0.130-1                                          amd64        CUDA nvprune
ii  cuda-nvrtc-10-0                                 10.0.130-1                                          amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-10-0                             10.0.130-1                                          amd64        NVRTC native dev links, headers
ii  cuda-nvtx-10-0                                  10.0.130-1                                          amd64        NVIDIA Tools Extension
ii  cuda-nvvp-10-0                                  10.0.130-1                                          amd64        CUDA nvvp
ii  cuda-repo-ubuntu1804-10-0-local-10.0.130-410.48 1.0-1                                               amd64        cuda repository configuration files
ii  cuda-samples-10-0                               10.0.130-1                                          amd64        CUDA example applications
ii  cuda-toolkit-10-0                               10.0.130-1                                          amd64        CUDA Toolkit 10.0 meta-package
ii  cuda-tools-10-0                                 10.0.130-1                                          amd64        CUDA Tools meta-package
ii  cuda-visual-tools-10-0                          10.0.130-1                                          amd64        CUDA visual tools
3

After a day of trying to find the issue, I ended up doing a clean reinstall with ubuntu 16.40, cuda 10.0 and pytorch 1.1. And the issue didn’t come up during the cuda install.

1 Like