Hi,
We are trying to understand the LLM (>70b; two variants: dense, MOE) pre-training communication. To better leverage the communication paradigms available in PyTorch, I explored relatively newer additions, i.e., Symmetric Memory with backends: cuda, nccl, & nvshmem.
If I understand correctly, dist backend nccl uses nccl for the control plane and data plane, whereas the symm mem backend nccl uses NCCL for the control plane (like bootstrapping) while giving some additional features than symm mem backend cuda as all symm mem backends use CUDA VMM features intra-node.
Also, is comparing model training using purely torch dist backend NCCL to purely torch symm mem backend NVSHMEM the same as saying we compared NCCL vs NVSHMEM? I do understand that more recent frontier MOE-based LLM trainings leverage a hybrid approach, i.e., both of them, based on the communication type (TP/DP/PP: NCCL; EP: NVSHMEM).
Kindly let me know if I misunderstood something. This is really important for us to use the resources efficiently for our LLM training.
Thank you. 
from India.