Timed out initializing process group in store based barrier

2

Update: reboot my instance and it works, wondering if there’s a better solution?

3

@EDENP, has this issue persisted for you? I’m also using PyTorch 1.8.1 and experiencing this issue when submitting a distributed training job with 2 nodes, each having 4 GPUs. I eventually get the message: Timed out initializing process group in store based barrier on rank: 4, for key: store_based_barrier_key:1 (world_size=8, worker_count=2, timeout=0:30:00). Thanks for any help.

1 Like
4

I’m also experiencing this exact same error when trying to run on a single node with 8 GPUs. I’ve tried rebooting my machine and setting export NCCL_IB_DISABLE=1 but neither of those have worked.

5

I am alse experiencing this when try to run on 2 nodes with 2 GPUs per node. pytorch 1.9+cudnn cuda 11.1 ,I have set export NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0 but those didn’t work. by the way ,they can work well in single node with 2 GPUs.

6

i am having exact the same issue when run distributed training with 2 nodes, have you solved this problem?

7

Same problem, any updates?

8

Hello, have you found a suitable solution to this problem.

9

Any updates? Same problem here.

10

Same problem here. Would appreciate any updates!

11

I’m getting this exact same problem. Are there any updates for this?

12

I’m receiving the same error message

13

same issue here using 2 nodes and 3 gpu per node

14

Is there any solution to this? I am facing same error with torch 1.11.0

15

I am facing the same problem. Did anyone manage to solve it?

16

Same problem! anyone from the developers? @ptrblck

17

I would recommend to check these docs and try to debug the application with the env variables etc.
Unfortunately, this topic does not contain enough information besides users claiming they are running into the same issue, which is not actionable.
Let me know once you have more logging information which you could post here.

18

Sucks that nobody has an answer for this. I am getting the same error out of the blue and just using a multi-gpu single machine. I suspect it has something to do with memory but really don’t know how to track it down.

1 Like
19

same error, but after set the NCCL_SOCKET_IFNAME environment, problem solved.

1 Like
20

Hey, what did you set this variable to?

21

I had the same issue trying to parallel process multiple models for model combination.

I fixed the problem for me by NEVER calling destroy_process_group(). Instead, once I call init_process_group(…), I only check if the process_group has been initialised with torch.distributed.is_initialized().

I observe that if you ever destroy a process group, I cannot initalise anymore, because I run into this timeout.