Cuda devices inaccessible suddenly

1

my environmen setup as

NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0
torch                     1.13.0a0+git7c98e70
cuda                      11.7
os                         Ubuntu 20.04.6 LTS              Debian 5.10.158-2 (2022-12-13) x86_64

run the instruction: python -c “import torch; print(torch.cuda.is_available())”, console output

/output/workspace/pytorch/torch/cuda/__init__.py:88: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /output/workspace/pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
...

a few days ago, that is normal but today suddenly unusable, very weird.

2

The issue doesn’t seem to e PyTorch-related as it seems your driver cannot initialize the GPU. Check what has changed between the last working setup, e.g. an unplanned driver update etc., and try reinstall the driver if that’s the case.