distributed

torchtitan Use this subcategory for all discussions related to end to end composability of distributed technologies using torchtitan. distributed-rpc

Topic	Replies	Views	Activity
About the distributed category distributed	2	2939	November 28, 2025
PyTorch Symmetric Memory backend NCCL/NVSHMEM vs PyTorch Distributed backend NCCL distributed	0	37	September 10, 2026
1.43× Faster MoE Training: LoongForge Redefines EP Expert Load Balancing with Topology-Aware Optimal Transport distributed	0	45	September 8, 2026
Is 32× A100 40 GB with 1 GPU per node a reasonable distributed-training testbed? distributed	2	142	August 16, 2026
Fix gradient accumulation for DDP with static graph and no sync distributed	0	76	July 11, 2026
Weights dont sychronise in multi GPU training distributed	7	232	July 9, 2026
How to train PyTorch model on multiple CPU nodes (SLURM)? distributed	2	278	May 22, 2026
[Distributed w/ TorchTitan] Introducing Async Tensor Parallelism in PyTorch torchtitan	13	19856	May 19, 2026
FSDP2 - inspecting parameter sharding distributed	0	76	May 14, 2026
Help with DDP in kaggle notebook distributed	3	475	May 14, 2026
PyTorch Distributed (Gloo) fails with system error: 10049 - The requested address is not valid in its context distributed	0	103	May 7, 2026
[c10d] The hostname of the client socket cannot be retrieved. err=-3 distributed	0	203	May 2, 2026
[Distributed w/ TorchTitan] Breaking Barriers: Training Long Context LLMs with 1M Sequence Length in PyTorch Using Context Parallel torchtitan	13	11689	April 22, 2026
`AveragedModel` and FSDP2 torchtitan	0	67	April 15, 2026
Transfer data GPU -> CPU and compute on GPU in parallel distributed	6	383	March 24, 2026
Qlora+fsdp2 training distributed	0	87	March 15, 2026
Parallel Training with INVIDIA MIG's distributed	8	5853	March 9, 2026
Balanced batch sampling with DistributedSampler/DDP distributed	1	109	March 4, 2026
PersistentTensorDict send data to GPU without blocking the computations distributed	0	59	March 4, 2026
Potential issue of "errno: 98- Address already in use" error in DDP (with torchrun) distributed	2	1106	February 25, 2026
[Solved] RTX 5090 (sm_120) Training Segfault - DDP Was the Cause distributed	4	785	February 25, 2026
Question About Backward–ReduceScatter Overlap in FSDP Figure 5 distributed	2	134	February 17, 2026
Is torch Muon optimizer compatible with FSDP/HSDP? distributed	1	208	February 12, 2026
Fully_shard with 2D mesh (4,1) still runs all-gather / reduce-scatter on the shard dimension distributed	0	64	February 5, 2026
FSDP2 post backward hook registration distributed	2	139	January 31, 2026
FSDP: Can users control which parameters are offloaded to CPU? distributed	0	110	January 30, 2026
Difference between torch.cuda.synchronize() and dist.barrier() distributed	3	5039	January 29, 2026
Runtime error raised in DDP when using .detach() to skip gradient computation in some DP ranks distributed	2	132	January 28, 2026
FSDP2 vs DDP gradient mismatch on Embeddings (Flex Attention + Compile) distributed	0	158	January 27, 2026
Multi GPU training on single node with DistributedDataParallel distributed	3	5579	January 27, 2026