Nvlink and Horovod

1

Hi @ptrblck , we generally use Nvlink for Parallelism and the Horovod framework for distributed training, where a common task will be executed on multiple processors. Do we have any connection between these two? What I mean is, can we connect the Horovod framework with Nvllink? Please, do help me if you know anything about this.
Thank you

2

Not familiar with Horovod implementation. But if it internally uses PyTorch ProcessGroup or DistributedDataParallel, it would work with NVLink, if you specify the nccl backend when calling init_process_group.

3

Thank you so much @mrshenli for your reply. Could you please share some of the code related links with me, if you have any.
Thank you

4

Sure. Here two tutorials with code c10d, ddp, and a minimum ddp example.

5

Thank you so much @mrshenli for your reply, I will work on it.