DistributedDataParallel with different 2 GPUs

distributed
1

Hi,

I have a huge image dataset and would like to train on my rtx 2060 and rtx 3060 simultaneously.
Since they do have quite a different memory an compute capability i wonder if this is viable at all or just train on the 3060.

Since they are so different would it be better to run each process with another batch size fitted to their vram but would this make it necessary to customize the DistributedSampler so that they receive an uneven split of the dataset?

Thanks for your help.

2

Yes, your overall training might run into a bottleneck caused by the slower GPU. Also, you are right that a custom sampler might be needed to create the imbalanced split for both devices.
This post describes potential issues a bit more and you could also profile the actual workload to see how severe the slowdown is.

1 Like