In a slurm environment: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero

1

I am in a slurm environment. Having this error everytime I run python3.11 -c “import torch; print(‘CUDA available:’, torch.cuda.is_available()); print(‘CUDA version:’, torch.version.cuda)” before using cuda and since the cuda is not found the device gets set to cpu.

But my program needs gpu. Please provide a solution that I can apply as a slurm regular user. Thank you.

2

Responded to your double post here.