Single-node Multi-GPU with different cards

Hi all!

As far as I understood, DataParallel splits batches into equal chunks. But if I have different GPUs, it will run at the speed of slowest card. Is it possible to set chunks proportions manually or somehow adjust chunks sizes?

You can modify the DataParallel function to send 40% / 60% or 30% / 70% splits across two cards, instead of 50%-50% default split.

However, if you work with multiple model architectures and parameter sizes it will be really hard to balance the workloads across different cards. Both GPUs may not finish a minibatch at the exact same time causing inefficiencies in training. I would recommend selling slower card on eBay and replacing it with a matching card, so you can do a 50/50 split on mini-batches.

How can I send different size of data to different GPUs?

Take a look at the code for DataParallel function in PyTorch repo @BrianHuang .