What is the difference between tag and group in distributed framwork?

distributed
1

Hi, I saw tag optional input at dist.send dist.recv api.
And I find that there is also group in pytorch distributed api.
So If I use multiple dist.isend in sam group, is it necessary for me to add tag at each send and receive?
What is the difference between tag and group?
https://pytorch.org/docs/stable/distributed.html#:~:text=isend(tensor%2C%20dst%2C%20group%3DNone%2C-,tag,-%3D0)

2

Hi, group is an id for the group of processes which wish to communicate with each other. It is similar to the “communicator” concept in the collective communication.

tag is an optional field that explicitly matches a send call with a recv call. Thus these two fields are orthogonal.

1 Like