Use two GPUs in C++

[GabrielB]

Hello, I am trying the C++ frontend with .to(torch::Device(torch::kCUDA, 1)) to load all my data and models onto the second GPU. The task manager shows the memory of the second GPU going up but the first GPU’s activity going up. How could it be processing the data on the wrong GPU? The same code works fine in Python. Can someone give an example of loading a model onto the second GPU so I can test it to see if it’s working properly?

[ptrblck]

@rtwolfe94022

it might be related to CUDA context initialization or other operations that PyTorch performs on the default device before your instructions take effect.

That’s not the case as PyTorch will not initialize a CUDA context on the default device. This would also be visible in an increase in its memory usage.

@rtwolfe94022

and that the second GPU is visible and accessible by your application. You can check this by running nvidia-smi in your terminal.

nvidia-smi will not tell you anything about the visibility of devices to processes.
@GabrielB you can mask a device via CUDA_VISIBLE_DEVICES to mask devices if needed.

@rtwolfe94022

by checking their .device() property.

NIT: .device is an attribute and not a function.

@rtwolfe94022

CUDA initializes its context on the first accessed GPU, which might explain the activity on the first GPU. This is a common behavior and doesn’t necessarily indicate that computation is happening on the wrong GPU.

This is not common at all and should be considered a bug.

@rtwolfe94022

PyTorch performs certain optimizations and background operations that might involve the GPUs. This activity might not directly relate to your data processing or model computation.

Could you describe what exactly you mean by that?

@rtwolfe94022 as previously asked, could you please review your ChatGPT posts before answering?
Otherwise the text sounds plausible, but is wrong as seen in this example.