I am having a strange issue with pytorch on a machine that has 2 GPUs. When I run the following code with VRAM of both GPUs empty:
Python 3.11.6 | packaged by conda-forge | (main, Oct 3 2023, 10:40:35) [GCC 12.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch >>> dev = torch.device('cuda:0') >>> a = torch.rand(1000,1000,1000, device=dev)
The ‘a’ tensor gets copied over to both GPUs even though I explicitly piped it to GPU #0.
However, the same thing doesn’t happen on another system with similar hardware with the same code.
Specs of the faulty setup:
GPU: Quadro P6000 (2x) $ conda list torch* # packages in environment at ....: # # Name Version Build Channel pytorch 2.1.0 py3.11_cuda11.8_cudnn8.7.0_0 pytorch pytorch-cuda 11.8 h7e8668a_5 pytorch pytorch-mutex 1.0 cuda pytorch torchaudio 2.1.0 py311_cu118 pytorch torchtriton 2.1.0 py311 pytorch torchvision 0.16.0 py311_cu118 pytorch