All_reduce a list of tensors

distributed
1

I am getting the following error when I am trying to all reduce a list of tensor

RuntimeError: Tensors must be contiguous

Here is a snippet of code

t_ = [...] # list of tensors
for t in t_: 
   dist.all_reduce(t, , op=dist.ReduceOp.SUM, group=group)
2

Figure it out! I have to make the tensor contiguous before reducing it. This article provides a nice description of it.

1 Like