Order of the list returned by torch.distributed.all_gather()?

distributed
1

Hi, I was wondering what is the order of the list returned by torch.distributed.all_gather.
Is the tensor in position i coming from the torch.distributed.get_rank() = i? Thanks!

1 Like
2

Yes, the ith tensor in the list comes from rank i.

2 Likes
3

is it stated anywhere in the doc? how can i be sure of this? I need to all gather 2 lists and the they have to be all gathered in the same order so that item i in the first list can be matched to item i in the second list.