Use two GPUs in C++

Hello, I am trying the C++ frontend with .to(torch::Device(torch::kCUDA, 1)) to load all my data and models onto the second GPU. The task manager shows the memory of the second GPU going up but the first GPU’s activity going up. How could it be processing the data on the wrong GPU? The same code works fine in Python. Can someone give an example of loading a model onto the second GPU so I can test it to see if it’s working properly?

That’s not the case as PyTorch will not initialize a CUDA context on the default device. This would also be visible in an increase in its memory usage.

nvidia-smi will not tell you anything about the visibility of devices to processes.
@GabrielB you can mask a device via CUDA_VISIBLE_DEVICES to mask devices if needed.

NIT: .device is an attribute and not a function.

This is not common at all and should be considered a bug.

Could you describe what exactly you mean by that?

@rtwolfe94022 as previously asked, could you please review your ChatGPT posts before answering?
Otherwise the text sounds plausible, but is wrong as seen in this example.

1 Like