How works dist.ProcessGroupGloo?

distributed
1

Hello together,

I have the following setup: I have three different computenodes each with two processes (rank 0 and rank 1).

So on each node two ranks are launched. I have some code creating some gloo Processgroups with:

global_pg = dist.ProcessGroupGloo(
prefix_store, {id of node}, {global_world_size}, some timeout
)

This means:


node 0:
rank 0: dist.ProcessGroupGloo(prefix_store, 0, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 0, 3}

node 1:
rank 0: dist.ProcessGroupGloo(prefix_store, 1, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 1, 3}

node 2:
rank 0: dist.ProcessGroupGloo(prefix_store, 2, 3}
rank 1: dist.ProcessGroupGloo(prefix_store, 2, 3}

I do not understand what is happening there. I get two processgroups on each node, with different ids (id(glogal_pg)), what makes sense. For both processgroups I use the same parameters for {id of node} and {global_world_size} on a node.

Which ranks are now connected to which process group? Can someone help me with this?
After having a look at the code I would expect, that I overall get two processgroups. The first is connecting all ranks 0 from all nodes, and the second one all ranks 1 from the nodes. (not sure)
Is this correct and how recognizes e.g. node1 rank1 that it belongs to node0 rank1 and not node0 rank0?