Migrating from torch.distributed.launch to torchrun

distributed
1

Hi guys I am working rewriting some previous training code using multiple nodes, on Azure ML.
The previous code uses the command
python -m torch.distributed.launch --nnodes $NUM_NODES --nproc_per_node $NUM_TRAINERS --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT $TRAINING_COMMAND
and works well now.
When I followed this to migrate the code to
torchrun --nnodes=$NUM_NODES --nproc_per_node $NUM_TRAINERS --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT $TRAINING_COMMAND
I got RendezvousTimeoutError on the master node and RendezvousConnectionError on the other node.
Could anyone help find solution or figure what I did wrong? Thanks!

2

@aivanou @Kiuk_Chung ?

3

Could you provide us with the actual command (with the real values for nnodes, nprocs_per_node, etc)? We’re you running across multiple hosts for both commands? torchrun and torch.distributed.launch both use the same underlying main entrypoint (torch.distributed.run), so I’m wondering if the two commands were invoked in exactly the same setup.

4

Sure. The command for torch.distributed.launch is
python3 -m torch.distributed.launch --nnodes 2 --nproc_per_node 4 --node_rank 0 --master_addr 10.0.0.8 --master_port 6000 $TRAINING_COMMAND, where the node rank is changed to 1, 2 or 3 for other processes.
For torchrun the command is
torchrun --nodes=2 --nproc_per_node=4 --rdzv_id=0 --rdzv_backend=c10d --rdzv_endpoint=10.0.0.8:6000 $TRAINING_COMMAND. I just use the node rank as rendezvous ID.
Actually I’ve tried passing other rendezvous backend like etcd but also failed.

5

Please check the reply above. Sorry for not replying your post.

6

Could you paste the full log output of both nodes? I believe what is happening is that the fully-qualified-domain-name (socket.getfqdn()) on node0 returns the hostname.domain that has no DNS route from node1 (hence, the timeout). There’s two ways around this:

  1. Make sure that the 2 nodes have a public ip
  2. Run with “static” rendezvous (just skip the --rdzv_backend option, it will default to “static”), which has the same options as torch.distributed.launch:
    $ torchrun --node_rank=0 # or 1 
                 --nnodes=2 
                 --nproc_per_node=4 
                 --master_addr=10.0.0.8 
                 --master_port=6000
                 $TRAINING_CMD