NVLS support in pytorch

[rajagond]

Does PyTorch support NVLS? If not, how does it manage to call NCCL’s NVLS algorithm using `torch.distributed.all_reduce?

[HyperHyper]

I think it is not supported as I see from:

• Nvls channel setup inside the container · Issue #477 · microsoft/mscclpp · GitHub
• [RFC] Offload collectives to NVSwitch when possible · Issue #136567 · pytorch/pytorch · GitHub

[ptrblck]

It’s possible to use NVLS via the torch.cuda.MemPool API which landed in this PR. We are also working on enabling it in e.g. DDP related to this PR.