Dear PyTorch and NVIDIA teams,
I am writing to report an unexpected behavior I’ve encountered when working with PyTorch and CUDA on a wsl2 on Windows 11 system equipped with multiple NVIDIA RTX 3090 GPUs.
- Operating System: Windows 11
- CUDA Version: 12.2
- WSL Version: 2
- GPUs: 4x NVIDIA RTX 3090
- PyTorch Version: 2.01 (CUDA 11.8)
Problem Statement: When I set the
CUDA_VISIBLE_DEVICES environment variable to enable all the GPUs (0,1,2,3) on the system and then run a PyTorch script that calls
torch.cuda.is_available(), I encounter an “Out of Memory” error. Notably, this error does not occur if I only enable GPU 1 or a combination of GPU 0,2,3. Furthermore, this error can be circumvented if I call
Steps to Reproduce:
- Set the environment variable:
- Run a Python script that imports PyTorch and calls
Expected Behavior: The
torch.cuda.is_available() function should return
True if GPUs are available and accessible.
Observed Behavior: An “Out of Memory” error is triggered internally at
torch.cuda.is_available() function returns
Workaround: I found that calling
torch.cuda.is_available() circumvents the error. However, this workaround requires modifying each script to include this extra call.
While the workaround is effective, it may be beneficial to investigate and address the root cause of this issue. I wanted to bring this to your attention and look forward to any insights or potential solutions you might provide.
Thank you for your time and assistance.
Best Regards, Peter Deng, Language and Technology Research Group at TC, Columbia