In a multi-node multi-GPU scenario, I use mp.spawn to spawn nproc_per_node processes on each node. But when I’m using multiple nodes with only 1 GPU on each node, should I still do the spawn? because spawning 1 process sounds redundant. Will spawning in this case be slower or faster or have little effect?
Thanks!
I’d say you wouldn’t need to spawn processes in general one-GPU-per-node cases.