# torchtitan **URL:** https://discuss.pytorch.org/c/distributed/torchtitan/44.md [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) --- ## [About the torchtitan category](https://discuss.pytorch.org/t/about-the-torchtitan-category/209322)
**Author:** [@albanD](https://discuss.pytorch.org/u/albanD)\ **Replies:** 0\ **Last updated:** [September 9, 2024, 6:08pm UTC](https://discuss.pytorch.org/t/about-the-torchtitan-category/209322 "2024-09-09T18:08:33Z")
Use this subcategory for all discussions related to end to end composability of distributed technologies using torchtitan. --- ## [\[Distributed w/ TorchTitan\] Introducing Async Tensor Parallelism in PyTorch](https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487)
**Author:** [@yifuwang](https://discuss.pytorch.org/u/yifuwang)\ **Replies:** 13\ **Last updated:** [May 19, 2026, 1:54pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487 "2026-05-19T13:54:02Z")
with Horace He, Less Wright, Luca Wehrstedt, Tianyu Liu, Wanchao Liang TL;DR We implemented experimental async tensor parallelism support in PyTorch. We integrated it in TorchTitan and observed: Up to ~29% forward pas… --- ## [\[Distributed w/ TorchTitan\] Breaking Barriers: Training Long Context LLMs with 1M Sequence Length in PyTorch Using Context Parallel](https://discuss.pytorch.org/t/distributed-w-torchtitan-breaking-barriers-training-long-context-llms-with-1m-sequence-length-in-pytorch-using-context-parallel/215082)
**Author:** [@XWu](https://discuss.pytorch.org/u/XWu)\ **Replies:** 13\ **Last updated:** [April 22, 2026, 7:25am UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-breaking-barriers-training-long-context-llms-with-1m-sequence-length-in-pytorch-using-context-parallel/215082 "2026-04-22T07:25:13Z")
with Chien-Chin Huang (@fegin), Less Wright (@lessw2020), Tianyu Liu (@tianyu), Will Constable (@wconstab), Gokul Nadathur (@gnadathur) TL;DR We implemented pass-KV Ring Attention for Context Parallel in PyTorch We… --- ## [\`AveragedModel\` and FSDP2](https://discuss.pytorch.org/t/averagedmodel-and-fsdp2/224812)
**Author:** [@Chenchao\_Zhao](https://discuss.pytorch.org/u/Chenchao_Zhao)\ **Replies:** 0\ **Last updated:** [April 15, 2026, 4:16pm UTC](https://discuss.pytorch.org/t/averagedmodel-and-fsdp2/224812 "2026-04-15T16:16:06Z")
Is there a way to combine AveragedModel with modern ND parallel or at least FSDP2 with DTensor? --- ## [CPU thread slow to enqueue GPU and communication kernels](https://discuss.pytorch.org/t/cpu-thread-slow-to-enqueue-gpu-and-communication-kernels/223546)
**Author:** [@Andrew\_Aikawa](https://discuss.pytorch.org/u/Andrew_Aikawa)\ **Replies:** 2\ **Last updated:** [October 20, 2025, 9:02pm UTC](https://discuss.pytorch.org/t/cpu-thread-slow-to-enqueue-gpu-and-communication-kernels/223546 "2025-10-20T21:02:52Z")
I’ve been having an issue doing llama 8b pre-training FSDP 2 with an on-prem single H200x8 bare metal instance, where I’m getting very jittery performance from inexplicably slow cpu ops that take a couple seconds before … --- ## [How to apply selective activation checkpointing on \_grouped\_mm](https://discuss.pytorch.org/t/how-to-apply-selective-activation-checkpointing-on-grouped-mm/223226)
**Author:** [@go\_brrr](https://discuss.pytorch.org/u/go_brrr)\ **Replies:** 0\ **Last updated:** [September 20, 2025, 11:02pm UTC](https://discuss.pytorch.org/t/how-to-apply-selective-activation-checkpointing-on-grouped-mm/223226 "2025-09-20T23:02:37Z")
From the selective activation checkpointing blog, I understood that we can save compute intensive operations while recompute operations that are memory intensive. How do I add \_grouped\_mm to ops\_to\_save below? Does dire… --- ## [Capture training graph with collectives via TorchTitan](https://discuss.pytorch.org/t/capture-training-graph-with-collectives-via-torchtitan/220885)
**Author:** [@amodab01](https://discuss.pytorch.org/u/amodab01)\ **Replies:** 8\ **Last updated:** [August 15, 2025, 6:15pm UTC](https://discuss.pytorch.org/t/capture-training-graph-with-collectives-via-torchtitan/220885 "2025-08-15T18:15:21Z")
Hello I am currently training a llama1b with Torchtitan and hoping to capture the training graph(s) with the various collectives when I work with different parallelism combinations. Is there a way to capture those? I tr… --- ## [Question about GPU memory usage when using pipeline parallelism training under larger micro batch count](https://discuss.pytorch.org/t/question-about-gpu-memory-usage-when-using-pipeline-parallelism-training-under-larger-micro-batch-count/221886)
**Author:** [@wangmingyuan](https://discuss.pytorch.org/u/wangmingyuan)\ **Replies:** 4\ **Last updated:** [July 30, 2025, 3:20am UTC](https://discuss.pytorch.org/t/question-about-gpu-memory-usage-when-using-pipeline-parallelism-training-under-larger-micro-batch-count/221886 "2025-07-30T03:20:24Z")
I am using torchtian with FSDP2 + PP(1F1B) to train llama3-8b, however, I found that as the micro batch count increasing, the GPU memory usage will increase rapidly (from 42.27GB to 64.57GB) on the last pp stage. That’s … --- ## [\[Distributed w/ TorchTitan\] FLUX is Here: Experience Diffusion Model Training on TorchTitan](https://discuss.pytorch.org/t/distributed-w-torchtitan-flux-is-here-experience-diffusion-model-training-on-torchtitan/221119)
**Author:** [@jianiw](https://discuss.pytorch.org/u/jianiw)\ **Replies:** 0\ **Last updated:** [June 27, 2025, 6:29pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-flux-is-here-experience-diffusion-model-training-on-torchtitan/221119 "2025-06-27T18:29:27Z")
With Tianyu Liu (@tianyu), Chien-Chin Huang (@fegin), Wei (Will) Feng, Gokul Nadathur (@gnadathur) TL;DR We enabled FLUX.1 model suite large-scale pre-training on TorchTitan with FSDP and HSDP support, showcasing diffu… --- ## [Tensor parallel numeric mismatch](https://discuss.pytorch.org/t/tensor-parallel-numeric-mismatch/220684)
**Author:** [@oahz48](https://discuss.pytorch.org/u/oahz48)\ **Replies:** 1\ **Last updated:** [June 18, 2025, 12:06am UTC](https://discuss.pytorch.org/t/tensor-parallel-numeric-mismatch/220684 "2025-06-18T00:06:50Z")
Hi, I am applying the tensor parallel to a submodule self.ffn = nn.Sequential( nn.Linear(dim, ffn\_dim), nn.GELU(approximate="tanh"), nn.Linear(ffn\_dim, dim), ) the parallel plan is "ffn": PrepareModuleInpu… --- ## [\[Distributed w/ TorchTitan\] Semi synchronous training using TorchFT](https://discuss.pytorch.org/t/distributed-w-torchtitan-semi-synchronous-training-using-torchft/219874)
**Author:** [@H-Huang](https://discuss.pytorch.org/u/H-Huang)\ **Replies:** 0\ **Last updated:** [May 8, 2025, 7:18pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-semi-synchronous-training-using-torchft/219874 "2025-05-08T19:18:52Z")
with Howard Huang, Tristan Rice, Chien-Chin Huang Summary We recently integrated TorchFT into torchtitan and have added support for semi synchronous training on TorchTitan through the LocalSGD and DiLoCo algorithms. We … --- ## [PyTorch Tensor Parallel](https://discuss.pytorch.org/t/pytorch-tensor-parallel/219647)
**Author:** [@Chenchao\_Zhao](https://discuss.pytorch.org/u/Chenchao_Zhao)\ **Replies:** 0\ **Last updated:** [May 1, 2025, 12:59am UTC](https://discuss.pytorch.org/t/pytorch-tensor-parallel/219647 "2025-05-01T00:59:51Z")
Is torch implementation of DTensor materially different from the MegatronLM TP? In Megatron it’s recommended to set CUDA\_DEVICE\_MAX\_CONNECTIONS to 1 to enable TP communication overlap. Is it also true for torch DTensor T… --- ## [Dcp.save straight to cloud storage](https://discuss.pytorch.org/t/dcp-save-straight-to-cloud-storage/217961)
**Author:** [@Sam\_Shleifer](https://discuss.pytorch.org/u/Sam_Shleifer)\ **Replies:** 5\ **Last updated:** [April 15, 2025, 5:43pm UTC](https://discuss.pytorch.org/t/dcp-save-straight-to-cloud-storage/217961 "2025-04-15T17:43:00Z")
Is there an example somewhere of doing dcp.save straight to cloud storage, like s3 or google cloud storage? --- ## [How to avoid casting DTensor to Tensor before calling a custom operator (a CUDA kernel)](https://discuss.pytorch.org/t/how-to-avoid-casting-dtensor-to-tensor-before-calling-a-custom-operator-a-cuda-kernel/217195)
**Author:** [@terunofuji](https://discuss.pytorch.org/u/terunofuji)\ **Replies:** 1\ **Last updated:** [April 2, 2025, 7:59pm UTC](https://discuss.pytorch.org/t/how-to-avoid-casting-dtensor-to-tensor-before-calling-a-custom-operator-a-cuda-kernel/217195 "2025-04-02T19:59:16Z")
I am doing tensor parallel training with DTensor. In my training code, I have a custom operator gmm, which is linked to a CUDA kernel in the backend. Currently, I am casting DTensor to Tensor before the operation and cas… --- ## [\[Distributed w/ TorchTitan\] Training with Zero-Bubble Pipeline Parallelism](https://discuss.pytorch.org/t/distributed-w-torchtitan-training-with-zero-bubble-pipeline-parallelism/214420)
**Author:** [@H-Huang](https://discuss.pytorch.org/u/H-Huang)\ **Replies:** 0\ **Last updated:** [December 19, 2024, 10:24pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-training-with-zero-bubble-pipeline-parallelism/214420 "2024-12-19T22:24:30Z")
with Howard Huang, Will Constable, Ke Wen, Jeffrey Wan, Haoci Zhang, Dong Li, Weiwei Chu TL;DR In this post, we’ll dive into a few key innovations in torch.distributed.pipelining that make it easier to apply pipeline pa… --- ## [\[Distributed w/ TorchTitan\] Optimizing Checkpointing Efficiency with PyTorch DCP](https://discuss.pytorch.org/t/distributed-w-torchtitan-optimizing-checkpointing-efficiency-with-pytorch-dcp/211250)
**Author:** [@fegin](https://discuss.pytorch.org/u/fegin)\ **Replies:** 0\ **Last updated:** [October 7, 2024, 5:31pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-optimizing-checkpointing-efficiency-with-pytorch-dcp/211250 "2024-10-07T17:31:08Z")
with Lucas Pasqualin, Iris Zhang, Less Wright, Pradeep Fernando, Will Constable TLDR We integrated PyTorch Distributed Checkpointing (DCP) into TorchTitan, enabling efficient distributed checkpointing. We implemented a… --- ## [\[Distributed w/ Torchtitan\] Enabling Float8 All-Gather in FSDP2](https://discuss.pytorch.org/t/distributed-w-torchtitan-enabling-float8-all-gather-in-fsdp2/209323)
**Author:** [@weifengpy](https://discuss.pytorch.org/u/weifengpy)\ **Replies:** 0\ **Last updated:** [September 9, 2024, 6:47pm UTC](https://discuss.pytorch.org/t/distributed-w-torchtitan-enabling-float8-all-gather-in-fsdp2/209323 "2024-09-09T18:47:49Z")
with Andrew Gu, Wanchao Liang, Driss Guessous, Vasiliy Kuznetsov, Brian Hirsh TL;DR We focus on float8 because it speeds up large GEMMs on H100s and saves network bandwidth with reduced message size. We enabled float8 …