Set longer timeout for torch distributed training

distributed distributed-rpc
1

Hi, I wonder if there is a mechanism to synchronize all processes with unlimited waiting time.

The scenario is in distributed training where one of processes in each node needs to deal with some CPU-related tasks, while other processes keep waiting until finish. Currently I use torch.distributed.barrier() (with nccl backend) and find it will timeout in half an hour. Is there another way that can make synchronize on all processes without timeout limitations? It would be better if torch already provides something like this.

2 Likes
2

Could you try Distributed communication package - torch.distributed — PyTorch 1.13 documentation instead?

3 Likes
3

Worked for me! Thank you!

4

this is not a solution to the question. monitored_barrier is implemented only for GLOO, whereas the question specifically states they’re using NCCL.

1 Like
5

Hello,
I try this and it really worked, Thanks for sharing.

6

Hello, I found init_process_group works for my scenario. You can customize the timeout with timeout argument. Then all the NCCL operations under this group will be with this timeout.

1 Like