Parallel Training with INVIDIA MIG's

[ptrblck]

@gnnewton

Specifically, the model was working perfectly before the GPU(A1000) being used was carved up into a set of 7 5GB MIG’s. Now, if I keep the same training batch_size defined when the GPU was whole, I get an OOM error immediately. In order to avoid the OOM error I have been forced to reduce the batch size by more than half (from 512 - 200).

This would of course be expected, since each MIG slice can only use its assigned memory.

@gnnewton

However when I train with these 3 MIG’s, only 1 of them is visible to CUDA, so I can only use 5GB of gpu instead of the potential 15. I don’t know if its possible to simultaneously utilize 3 MIG’s during training in pytorch, I have looked through the web and couldn’t find anything that helped. If anyone has any thoughts, please share!

No, you cannot use multiple MIG slices in a “distributed” manner and could just use another MIG config which would assign more compute resources and memory to each MIG.