# distributed **URL:** https://discuss.pytorch.org/c/distributed/12.md [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) --- ## [About the distributed category](https://discuss.pytorch.org/t/about-the-distributed-category/33465)
**Author:** [@smth](https://discuss.pytorch.org/u/smth)\ **Replies:** 2\ **Last updated:** [November 28, 2025, 10:58am UTC](https://discuss.pytorch.org/t/about-the-distributed-category/33465 "2025-11-28T10:58:50Z")
--- ## [Ease development by running computations on remote GPU](https://discuss.pytorch.org/t/ease-development-by-running-computations-on-remote-gpu/121002)
**Author:** [@meandmymind](https://discuss.pytorch.org/u/meandmymind)\ **Replies:** 11\ **Last updated:** [October 4, 2026, 9:08pm UTC](https://discuss.pytorch.org/t/ease-development-by-running-computations-on-remote-gpu/121002 "2026-10-04T21:08:54Z")
Hello! For development I use local machine with no GPU and have a remote machine with GPU. I like to debug my code via IDE tools but also want to have access to gpu. Using something a-la vscode over ssh is kinda slow … --- ## [PyTorch Symmetric Memory backend NCCL/NVSHMEM vs PyTorch Distributed backend NCCL](https://discuss.pytorch.org/t/pytorch-symmetric-memory-backend-nccl-nvshmem-vs-pytorch-distributed-backend-nccl/225396)
**Author:** [@VasudevaK](https://discuss.pytorch.org/u/VasudevaK)\ **Replies:** 0\ **Last updated:** [September 10, 2026, 2:41am UTC](https://discuss.pytorch.org/t/pytorch-symmetric-memory-backend-nccl-nvshmem-vs-pytorch-distributed-backend-nccl/225396 "2026-09-10T02:41:51Z")
Hi, We are trying to understand the LLM (\>70b; two variants: dense, MOE) pre-training communication. To better leverage the communication paradigms available in PyTorch, I explored relatively newer additions, i.e., Symm… --- ## [1.43× Faster MoE Training: LoongForge Redefines EP Expert Load Balancing with Topology-Aware Optimal Transport](https://discuss.pytorch.org/t/1-43x-faster-moe-training-loongforge-redefines-ep-expert-load-balancing-with-topology-aware-optimal-transport/225390)
**Author:** [@nullnonenilNULL](https://discuss.pytorch.org/u/nullnonenilNULL)\ **Replies:** 0\ **Last updated:** [September 8, 2026, 3:30pm UTC](https://discuss.pytorch.org/t/1-43x-faster-moe-training-loongforge-redefines-ep-expert-load-balancing-with-topology-aware-optimal-transport/225390 "2026-09-08T15:30:08Z")
MoE is now the default architecture for frontier models, and the direction of the next generation is clear: more experts, sparser activation. DeepSeek-V4-Pro raised the number of routed experts per layer from 256 in V3 t… --- ## [Is 32× A100 40 GB with 1 GPU per node a reasonable distributed-training testbed?](https://discuss.pytorch.org/t/is-32x-a100-40-gb-with-1-gpu-per-node-a-reasonable-distributed-training-testbed/225289)
**Author:** [@Atharva\_Rasane](https://discuss.pytorch.org/u/Atharva_Rasane)\ **Replies:** 2\ **Last updated:** [August 16, 2026, 3:44pm UTC](https://discuss.pytorch.org/t/is-32x-a100-40-gb-with-1-gpu-per-node-a-reasonable-distributed-training-testbed/225289 "2026-08-16T15:44:19Z")
We are designing a distributed-training experiment and currently have access to 32 NVIDIA A100 40 GB GPUs. For both Jetstream2 and Google Cloud, the 1× A100 40 GB per node configurations are the lowest-cost options avai… --- ## [Fix gradient accumulation for DDP with static graph and no sync](https://discuss.pytorch.org/t/fix-gradient-accumulation-for-ddp-with-static-graph-and-no-sync/225166)
**Author:** [@kapper](https://discuss.pytorch.org/u/kapper)\ **Replies:** 0\ **Last updated:** [July 11, 2026, 2:23pm UTC](https://discuss.pytorch.org/t/fix-gradient-accumulation-for-ddp-with-static-graph-and-no-sync/225166 "2026-07-11T14:23:06Z")
Hi all, I have a PR to fix a bug when DDP gradient accumulation is used with static\_graph=True and no\_sync. Can I get anyone to take a look at \[DDP\] Fix static\_graph=True + no\_sync() gradient accumulation regression by … --- ## [Weights dont sychronise in multi GPU training](https://discuss.pytorch.org/t/weights-dont-sychronise-in-multi-gpu-training/225141)
**Author:** [@pbellos](https://discuss.pytorch.org/u/pbellos)\ **Replies:** 7\ **Last updated:** [July 9, 2026, 2:13pm UTC](https://discuss.pytorch.org/t/weights-dont-sychronise-in-multi-gpu-training/225141 "2026-07-09T14:13:58Z")
Hi all, I have been trying to train a GNN in multiple gpus. I use the standard python libraries, PyG and DDP. I can see the data distributed in the GPUs, and each of them training a model. However the weights are not sy… --- ## [How to train PyTorch model on multiple CPU nodes (SLURM)?](https://discuss.pytorch.org/t/how-to-train-pytorch-model-on-multiple-cpu-nodes-slurm/224744)
**Author:** [@Moviaso1](https://discuss.pytorch.org/u/Moviaso1)\ **Replies:** 2\ **Last updated:** [May 22, 2026, 12:54am UTC](https://discuss.pytorch.org/t/how-to-train-pytorch-model-on-multiple-cpu-nodes-slurm/224744 "2026-05-22T00:54:55Z")
Hello, I have a problem/question regarding training a PyTorch model on a cluster with multiple CPU nodes using SLURM. I saw that the easiest way would be to utilize DDP and torchrun, but all examples that I found were o… --- ## [\[Distributed w/ TorchTitan\] Introducing Async Tensor Parallelism in PyTorch](https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487)
**Author:** [@yifuwang](https://discuss.pytorch.org/u/yifuwang)\ **Replies:** 13\ **Last updated:** [May 19, 2026, 1:54pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487 "2026-05-19T13:54:02Z")
with Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, Wanchao Liang TL;DR We implemented experimental async tensor parallelism support in PyTorch. We integrated it in TorchTitan and observed: Up to ~29% forward pas… --- ## [FSDP2 - inspecting parameter sharding](https://discuss.pytorch.org/t/fsdp2-inspecting-parameter-sharding/224940)
**Author:** [@farisd16](https://discuss.pytorch.org/u/farisd16)\ **Replies:** 0\ **Last updated:** [May 14, 2026, 1:26pm UTC](https://discuss.pytorch.org/t/fsdp2-inspecting-parameter-sharding/224940 "2026-05-14T13:26:28Z")
After applying fully\_shard to a model and its submodules, is there a way to see how parameters in each FSDP unit got sharded? Perhaps something like a mapping of submodules to a rank? Thanks in advance :slight\_smile: --- ## [Help with DDP in kaggle notebook](https://discuss.pytorch.org/t/help-with-ddp-in-kaggle-notebook/213369)
**Author:** [@Atia](https://discuss.pytorch.org/u/Atia)\ **Replies:** 3\ **Last updated:** [May 14, 2026, 2:59am UTC](https://discuss.pytorch.org/t/help-with-ddp-in-kaggle-notebook/213369 "2026-05-14T02:59:47Z")
I am trying to do DDP in kaggle notebook as I don’t have multiple gpus on my local machine. Here is my code: def main\_worker(rank, world\_size, arch: str = "resnet50",resume: str=None, start\_epoch: int=0, EPOCHS=100, … --- ## [PyTorch Distributed (Gloo) fails with system error: 10049 - The requested address is not valid in its context](https://discuss.pytorch.org/t/pytorch-distributed-gloo-fails-with-system-error-10049-the-requested-address-is-not-valid-in-its-context/224907)
**Author:** [@tanaya\_joshi](https://discuss.pytorch.org/u/tanaya_joshi)\ **Replies:** 0\ **Last updated:** [May 7, 2026, 1:16pm UTC](https://discuss.pytorch.org/t/pytorch-distributed-gloo-fails-with-system-error-10049-the-requested-address-is-not-valid-in-its-context/224907 "2026-05-07T13:16:16Z")
Hi everyone, I am trying to test basic distributed communication between two Windows PCs using torch.distributed with the gloo backend. Here is the test script: import sys import torch.distributed as dist MASTER\_IP … --- ## [\[c10d\] The hostname of the client socket cannot be retrieved. err=-3](https://discuss.pytorch.org/t/c10d-the-hostname-of-the-client-socket-cannot-be-retrieved-err-3/224888)
**Author:** [@wodaxia](https://discuss.pytorch.org/u/wodaxia)\ **Replies:** 0\ **Last updated:** [May 2, 2026, 4:19am UTC](https://discuss.pytorch.org/t/c10d-the-hostname-of-the-client-socket-cannot-be-retrieved-err-3/224888 "2026-05-02T04:19:31Z")
I encountered this issue when running a training using accelerate in kaggle notebook. W502 03:41:26.169817470 socket.cpp:200\] \[c10d\] The hostname of the client socket cannot be retrieved. err=-3 \[W502 03:41:26.25075240… --- ## [\[Distributed w/ TorchTitan\] Breaking Barriers: Training Long Context LLMs with 1M Sequence Length in PyTorch Using Context Parallel](https://discuss.pytorch.org/t/distributed-w-torchtitan-breaking-barriers-training-long-context-llms-with-1m-sequence-length-in-pytorch-using-context-parallel/215082)
**Author:** [@XWu](https://discuss.pytorch.org/u/XWu)\ **Replies:** 13\ **Last updated:** [April 22, 2026, 7:25am UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-breaking-barriers-training-long-context-llms-with-1m-sequence-length-in-pytorch-using-context-parallel/215082 "2026-04-22T07:25:13Z")
with Chien-Chin Huang (@fegin), Less Wright (@lessw2020), Tianyu Liu (@tianyu), Will Constable (@wconstab), Gokul Nadathur (@gnadathur) TL;DR We implemented pass-KV Ring Attention for Context Parallel in PyTorch We… --- ## [\`AveragedModel\` and FSDP2](https://discuss.pytorch.org/t/averagedmodel-and-fsdp2/224812)
**Author:** [@Chenchao\_Zhao](https://discuss.pytorch.org/u/Chenchao_Zhao)\ **Replies:** 0\ **Last updated:** [April 15, 2026, 4:16pm UTC](https://discuss.pytorch.org/t/averagedmodel-and-fsdp2/224812 "2026-04-15T16:16:06Z")
Is there a way to combine AveragedModel with modern ND parallel or at least FSDP2 with DTensor? --- ## [Transfer data GPU -\> CPU and compute on GPU in parallel](https://discuss.pytorch.org/t/transfer-data-gpu-cpu-and-compute-on-gpu-in-parallel/224695)
**Author:** [@mseeger](https://discuss.pytorch.org/u/mseeger)\ **Replies:** 6\ **Last updated:** [March 24, 2026, 1:36pm UTC](https://discuss.pytorch.org/t/transfer-data-gpu-cpu-and-compute-on-gpu-in-parallel/224695 "2026-03-24T13:36:09Z")
I am a beginner with PyTorch distributed, but never found this point explained anywhere. I am trying to implement gradient computation with activation checkpointing, where the model weights are kept on CPU, and only wei… --- ## [Qlora+fsdp2 training](https://discuss.pytorch.org/t/qlora-fsdp2-training/224696)
**Author:** [@sharmaarush](https://discuss.pytorch.org/u/sharmaarush)\ **Replies:** 0\ **Last updated:** [March 15, 2026, 6:17am UTC](https://discuss.pytorch.org/t/qlora-fsdp2-training/224696 "2026-03-15T06:17:19Z")
Hi guys I am trying to train a model using qlora+fsdp2 on kaggle 2x tpus (the unlsoth assignment, problem B) but I keep running into the error RuntimeError: only Tensors of floating point dtype can require gradients I … --- ## [Parallel Training with INVIDIA MIG's](https://discuss.pytorch.org/t/parallel-training-with-invidia-migs/159445)
**Author:** [@gnnewton](https://discuss.pytorch.org/u/gnnewton)\ **Replies:** 8\ **Last updated:** [March 9, 2026, 6:30pm UTC](https://discuss.pytorch.org/t/parallel-training-with-invidia-migs/159445 "2026-03-09T18:30:58Z")
Hello all! I have been recently running into troubles while attempting to train a pytorch nn.Module model, hopefully someone has knowledge on the subject. Specifically, the model was working perfectly before the GPU(A10… --- ## [Balanced batch sampling with DistributedSampler/DDP](https://discuss.pytorch.org/t/balanced-batch-sampling-with-distributedsampler-ddp/224652)
**Author:** [@lpereira](https://discuss.pytorch.org/u/lpereira)\ **Replies:** 1\ **Last updated:** [March 4, 2026, 9:57pm UTC](https://discuss.pytorch.org/t/balanced-batch-sampling-with-distributedsampler-ddp/224652 "2026-03-04T21:57:20Z")
Hello, I am trying to use a balanced/pk batch sampler for triplet loss learning, as in Hermans et al. 2017 (https://arxiv.org/pdf/1703.07737). However, I am using multiple GPUs with DistributedDataParallel, and have bee… --- ## [PersistentTensorDict send data to GPU without blocking the computations](https://discuss.pytorch.org/t/persistenttensordict-send-data-to-gpu-without-blocking-the-computations/224653)
**Author:** [@lsrosa](https://discuss.pytorch.org/u/lsrosa)\ **Replies:** 0\ **Last updated:** [March 4, 2026, 6:20pm UTC](https://discuss.pytorch.org/t/persistenttensordict-send-data-to-gpu-without-blocking-the-computations/224653 "2026-03-04T18:20:55Z")
I have a lot of data saved on PersistentTensorDicts (PTDs), which I send to the GPU for processing. However, there is a lot of idle time between sending data (which is read from disc) to the GPU and writing it back. I … --- ## [Potential issue of "errno: 98- Address already in use" error in DDP (with torchrun)](https://discuss.pytorch.org/t/potential-issue-of-errno-98-address-already-in-use-error-in-ddp-with-torchrun/202922)
**Author:** [@duan22677](https://discuss.pytorch.org/u/duan22677)\ **Replies:** 2\ **Last updated:** [February 25, 2026, 1:58am UTC](https://discuss.pytorch.org/t/potential-issue-of-errno-98-address-already-in-use-error-in-ddp-with-torchrun/202922 "2026-02-25T01:58:40Z")
During the use of torch run (with ddp), sometimes there may be random occurrences of ‘errno: 98- Address already in use’, for example: \[W socket.cpp:436\] \[c10d\] The server socket has failed to bind to \[::\]:29400 (errno:… --- ## [\[Solved\] RTX 5090 (sm\_120) Training Segfault - DDP Was the Cause](https://discuss.pytorch.org/t/solved-rtx-5090-sm-120-training-segfault-ddp-was-the-cause/224584)
**Author:** [@k\_k1](https://discuss.pytorch.org/u/k_k1)\ **Replies:** 4\ **Last updated:** [February 25, 2026, 1:19am UTC](https://discuss.pytorch.org/t/solved-rtx-5090-sm-120-training-segfault-ddp-was-the-cause/224584 "2026-02-25T01:19:08Z")
Hi everyone, I spent a few weeks debugging training crashes on my RTX 5090 and finally figured it out — sharing in case it helps anyone with the same GPU. Setup GPU: RTX 5090 Laptop (GB203, sm\_120, 24GB VRAM) OS: Windo… --- ## [Question About Backward–ReduceScatter Overlap in FSDP Figure 5](https://discuss.pytorch.org/t/question-about-backward-reducescatter-overlap-in-fsdp-figure-5/224536)
**Author:** [@BitCalSaul](https://discuss.pytorch.org/u/BitCalSaul)\ **Replies:** 2\ **Last updated:** [February 17, 2026, 10:45pm UTC](https://discuss.pytorch.org/t/question-about-backward-reducescatter-overlap-in-fsdp-figure-5/224536 "2026-02-17T22:45:11Z")
I have a question regarding Figure 5 in the PyTorch FSDP paper (Overlap Communication and Computation). In the timeline, BWD1 and RS1 appear to start concurrently. My understanding is that Reduce-Scatter should only be … --- ## [Is torch Muon optimizer compatible with FSDP/HSDP?](https://discuss.pytorch.org/t/is-torch-muon-optimizer-compatible-with-fsdp-hsdp/224467)
**Author:** [@DiMarzioBian](https://discuss.pytorch.org/u/DiMarzioBian)\ **Replies:** 1\ **Last updated:** [February 12, 2026, 2:44pm UTC](https://discuss.pytorch.org/t/is-torch-muon-optimizer-compatible-with-fsdp-hsdp/224467 "2026-02-12T14:44:42Z")
Hi all, I’m curious whether Muon’s official implementation supports advanced data parallelism strategies other than DDP. Thanks --- ## [Fully\_shard with 2D mesh (4,1) still runs all-gather / reduce-scatter on the shard dimension](https://discuss.pytorch.org/t/fully-shard-with-2d-mesh-4-1-still-runs-all-gather-reduce-scatter-on-the-shard-dimension/224477)
**Author:** [@harryjing](https://discuss.pytorch.org/u/harryjing)\ **Replies:** 0\ **Last updated:** [February 5, 2026, 6:54am UTC](https://discuss.pytorch.org/t/fully-shard-with-2d-mesh-4-1-still-runs-all-gather-reduce-scatter-on-the-shard-dimension/224477 "2026-02-05T06:54:27Z")
Description When passing a 2D mesh (4,1) to fully\_shard, the shard dimension has world size 1, which is semantically equivalent to pure replication . However, the current implementation still takes the HSDP path and perf… --- ## [FSDP2 post backward hook registration](https://discuss.pytorch.org/t/fsdp2-post-backward-hook-registration/224455)
**Author:** [@lovanto](https://discuss.pytorch.org/u/lovanto)\ **Replies:** 2\ **Last updated:** [January 31, 2026, 9:07pm UTC](https://discuss.pytorch.org/t/fsdp2-post-backward-hook-registration/224455 "2026-01-31T21:07:29Z")
Hi In FSDP2 module’s post backward hook is registered through an autograd function which runs the hook when the gradients of the module’s inputs have been computed. But it seems like the hook assumes that at the mome… --- ## [FSDP: Can users control which parameters are offloaded to CPU?](https://discuss.pytorch.org/t/fsdp-can-users-control-which-parameters-are-offloaded-to-cpu/224443)
**Author:** [@fopdoodle8](https://discuss.pytorch.org/u/fopdoodle8)\ **Replies:** 0\ **Last updated:** [January 30, 2026, 10:34am UTC](https://discuss.pytorch.org/t/fsdp-can-users-control-which-parameters-are-offloaded-to-cpu/224443 "2026-01-30T10:34:38Z")
Hi all, I have a scenario where I’m training an RL actor model using FSDP. Both rollout and training share the same backbone. The model weights, after sharding, fit on GPU. However, the gradients and optimizer states … --- ## [Difference between torch.cuda.synchronize() and dist.barrier()](https://discuss.pytorch.org/t/difference-between-torch-cuda-synchronize-and-dist-barrier/171250)
**Author:** [@Taejune](https://discuss.pytorch.org/u/Taejune)\ **Replies:** 3\ **Last updated:** [January 29, 2026, 6:21am UTC](https://discuss.pytorch.org/t/difference-between-torch-cuda-synchronize-and-dist-barrier/171250 "2026-01-29T06:21:27Z")
Hi, I’m wondering what is the main difference between torch.cuda.synchronize() and dist.barrier(). I know that the former prevents CPU thread from proceeding until the previous works are done, and the latter makes proc… --- ## [Runtime error raised in DDP when using .detach() to skip gradient computation in some DP ranks](https://discuss.pytorch.org/t/runtime-error-raised-in-ddp-when-using-detach-to-skip-gradient-computation-in-some-dp-ranks/224413)
**Author:** [@Mistzz](https://discuss.pytorch.org/u/Mistzz)\ **Replies:** 2\ **Last updated:** [January 28, 2026, 4:44am UTC](https://discuss.pytorch.org/t/runtime-error-raised-in-ddp-when-using-detach-to-skip-gradient-computation-in-some-dp-ranks/224413 "2026-01-28T04:44:40Z")
Describe I’m using DDP training network based on Transformer on several nodes. There is a part in my the MHA layer code: residual = hidden\_states hidden\_states = self.input\_layernorm(hidden\_states) … --- ## [FSDP2 vs DDP gradient mismatch on Embeddings (Flex Attention + Compile)](https://discuss.pytorch.org/t/fsdp2-vs-ddp-gradient-mismatch-on-embeddings-flex-attention-compile/224420)
**Author:** [@Akshat\_Singh](https://discuss.pytorch.org/u/Akshat_Singh)\ **Replies:** 0\ **Last updated:** [January 27, 2026, 11:12pm UTC](https://discuss.pytorch.org/t/fsdp2-vs-ddp-gradient-mismatch-on-embeddings-flex-attention-compile/224420 "2026-01-27T23:12:17Z")
Hey all, I’m hitting a weird precision issue trying to cross-check my FSDP2 run against a DDP run. I’m training a Decoder model using flex\_attention and torch.compile. When I compare the gradients between FSDP2 and DDP… [Next page](https://discuss.pytorch.org/c/distributed/12.md?page=1)