A small NCCL probe to distinguish initialization hangs from the first collective

Disclosure: I maintain TBR8, where the companion tutorial is published in Chinese. I am sharing the essential probe here so it can be reviewed without following a link.

The goal is to identify the last checkpoint each rank reaches: entering process-group initialization, initialization returned, or the first all_reduce completed. This helps narrow an investigation; it does not identify the root cause by itself. This version passes device_id to init_process_group to initialize the NCCL communicator eagerly; it needs a PyTorch version that supports that argument.

Save this as tbr8-nccl-init-probe.py:

"""Run one NCCL collective, with a flushed checkpoint for each phase."""
import os
from datetime import timedelta

import torch
import torch.distributed as dist


def main():
    rank = int(os.environ["RANK"])
    world = int(os.environ["WORLD_SIZE"])
    local_rank = int(os.environ["LOCAL_RANK"])
    device = torch.device("cuda", local_rank)
    torch.cuda.set_device(device)
    print(f"rank={rank}/{world} local_rank={local_rank} enter init", flush=True)
    dist.init_process_group(
        "nccl", device_id=device, timeout=timedelta(seconds=45)
    )
    try:
        print(f"rank={rank} init OK; enter all_reduce", flush=True)
        value = torch.tensor(rank + 1, device=device, dtype=torch.int64)
        dist.all_reduce(value)
        actual = value.item()
        expected = world * (world + 1) // 2
        if actual != expected:
            raise RuntimeError(f"rank={rank} got {actual}, expected {expected}")
        print(f"rank={rank} all_reduce OK; sum={actual}", flush=True)
    finally:
        dist.destroy_process_group()


if __name__ == "__main__":
    main()

For a two-node test with one GPU per node, run this on both nodes. Replace MASTER_ADDR with the reachable address of node 0; use NODE_RANK=0 on node 0 and NODE_RANK=1 on node 1. Select each node’s actual task-network interface (the interface names need not match).

export MASTER_ADDR=10.0.0.11 MASTER_PORT=29500
NODE_RANK=0
export NCCL_SOCKET_IFNAME='=ens5'
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,ENV
export NCCL_DEBUG_FILE='/tmp/nccl-%h-%p.log'
timeout --kill-after=10s 180s torchrun   --nnodes=2 --nproc-per-node=1 --node-rank="$NODE_RANK"   --master-addr="$MASTER_ADDR" --master-port="$MASTER_PORT"   tbr8-nccl-init-probe.py

For two ranks, the expected sum is 3. If a rank never prints the first checkpoint, inspect the launcher/rendezvous and process startup first. If initialization fails, keep the earliest error from every rank and compare the selected addresses and interfaces. If initialization returns but the collective stalls, compare participation, operation order, devices and tensor details as well as the data path. With lazy initialization in other scripts, communicator errors can first appear at the collective.

Validation scope: the published controlled test used WSL2, one RTX 2080 Ti, PyTorch 2.13.0+cu130 and NCCL 2.29.7. Selecting a nonexistent interface produced “Bootstrap : no socket interface found” after the first checkpoint; using loopback for the single-rank control completed with sum=1. This is not a validation of two-node connectivity or RDMA. Loopback is only for that local control, not the two-node command above. Reaching the rendezvous port alone also does not prove that all NCCL peer connections are allowed.

Companion explanation and downloadable script: NCCL 多机初始化卡住怎么查?网卡、端口与 rank 日志 | 汤不热吧

Feedback on the checkpoint placement and diagnostic limitations would be useful, especially from users testing a real two-node setup.