ValueError: Default process group has not been initialized, please make sure to call init_process_group.

nlp
1

Traceback (most recent call last):
File “/content/xl-sum/seq2seq/pipeline.py”, line 517, in
main()
File “/content/xl-sum/seq2seq/pipeline.py”, line 293, in main

  • (torch.distributed.get_world_size() if training_args.local_rank != -1 else 1)
    File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 1555, in get_world_size
    return _get_group_size(group)
    File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 836, in _get_group_size
    default_pg = _get_default_group()
    File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 977, in _get_default_group
    raise ValueError(
    ValueError: Default process group has not been initialized, please make sure to call init_process_group.
2

Did you initialize the default ProcessGroup in your DDP use case as described here?