Have a look at this solution, it’s the preferred approach!
The other solution is to specify which GPUs to use with PyTorch internal functions like torch.cuda.set_device(), but it’s not recommended, I think it’s because there might be some index mismatch between nvidia-smi and the internal values… Most likely other reasons too.