GPU Device Ordering

1

I have a system with a GTX 1070 and a Titan RTX installed in it.

Nvidia-smi yields:

| 0 GeForce GTX 1070 Off | 00000000:03:00.0 Off | N/A |
| 1 TITAN RTX Off | 00000000:04:00.0 Off | N/A |

However,

print(torch.cuda.get_device_name(“cuda:0”))
print(torch.cuda.get_device_name(“cuda:1”))

yields:

TITAN RTX
GeForce GTX 1070

Clearly, the results are out of order with each other. This presents no real difficulty, since I can always identify them at least up to the card type directly, so this is mostly curiosity:

Is this typical/intended behavior? Are there intended guarantees (or no guarantees) for card order of nvidia-smi vs the internal ordering seen through PyTorch?

2

nvidia-smi shows the devices by PCI order, while the CUDA driver and Runtime API should sort the devices based on their “performance”.
To get the same order in your PyTorch script, you could try to set

export CUDA_DEVICE_ORDER=PCI_BUS_ID

before running your script.

3

Yes, that looks like the “solution” to this not-quite-a-problem.
Previously, all my systems were either homogeneous, or sorted in the correct order by luck.

Thanks!