Can Torch support training on multiple GPUs which have different memory size?

distributed
1

Few years ago, I remembered that I had trained my model on 2 RTX 3060, and 1 RTX 2060. There was an error raised up.

best regards,
SW.

2

Yes, PyTorch supports multi-GPU training using different GPUs, but you would have to make sure you are not running out of memory etc. The training will also most likely be bottlenecked by the slowest GPU.

3

Great and thanks. Let me try.

4

A.

bottlenecked by the slowest GPU

Is it still the case?
Caz, slower GPU can be tasked with lower params and batch size
in relation to the faster GPU right?

The same holds for unequal vrams as well right?

B.

make sure you are not running out of memory etc

Well, it’d make sense for the PyTorch module (I don’t know, FSDP) to take care of all these right?
What is the current scenario? Does PyTorch handle these by itself?

C.
How about cpu offloading?
For large models requiring high vram, but in a setup with a single gpu and limited vram,
(given batchsize of 1, so that I get a better picture of the bare minimum weights, activation, opt. states, gradients vram requirements) is PyTorch currently able to divide the model and cpu offload automatically so that even if the training takes longer, it still works (leaving aside dynamic batchsize etc, and wanting to know this base case).

D. How does all the above questions apply for 3 scenarios:
a. 1 nvidia gpu only
b. 1 latest amd gpu only
c. a & b - ie, 2 in total (nvidia and amd)

Please give as much detail as possible.
Would also welcome pointers to learning resources and others implementations, real world stuff, etc.

PS: WAITING EAGERLY

5

Of course a proper balancing is possible if you implement it. Since these scenarios are uncommon (and can hardly be tested on available machines for leases) standard utils. might not support them.