Upgrading from 1.8 to 1.10.2 causes DDP to fail, socket timeout

distributed
1

I’m doing multinode DDP, one process per node. Upgraded pytorch causes problems for some reason.

The specific error is a timeout after calling barrier()

  File "/remote/idiap.svm/temp.speech01/rbraun/code/speechbrain/speechbrain/utils/distributed.py", line 104, in ddp_barrier
    torch.distributed.barrier()
  File "/idiap/temp/rbraun/programs/anaconda3/envs/speech/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2709, in barrier
    work = default_pg.barrier(opts=opts)
RuntimeError: Socket Timeout

Why would an upgrade to pytorch cause this?

2

@divinho Any chance you could share a minimal repro for this passing on 1.8 and failing on 1.10.2? Also, I’m assuming you are using ProcessGroupGloo here?

3

Installing 1.10.1 with cuda 11.3 fixed the problem. think the problem might have been caused by the framework I was using (speechbrain)

1 Like