distributed

torchtitan Use this subcategory for all discussions related to end to end composability of distributed technologies using torchtitan. distributed-rpc

Topic	Replies	Views	Activity
About the distributed category distributed	1	2799	January 20, 2021
In NVIDIA container environments, PyTorch's NCCL allreduce operation exhibits extremely poor performance distributed	2	19	August 22, 2025
Torch.distributed.dcp.save does not save on all ranks distributed	2	12	August 22, 2025
Dynamo logs for distributed distributed	0	5	August 22, 2025
Support for Overlapping AllGather and ReduceScatter in FSDP distributed	0	12	August 20, 2025
Multi-GPU training hangs: Watchdog caught collective operation timeout distributed	15	15620	August 18, 2025
Capture training graph with collectives via TorchTitan torchtitan	8	116	August 15, 2025
Using DistributedDataParallel with dataloader num_workers > 0 distributed	2	3803	August 15, 2025
In multi-processing, when one process exits unexpectedly, how to get others out of hang? distributed	0	14	August 13, 2025
FSDP.HYBRID_SHARD leads to parameter inconsistency between two DP replicas distributed	0	16	August 6, 2025
Purpose and communication of set reshard_after_forward=int in fsdp2 distributed	0	20	August 6, 2025
Variable batch size in Multi-GPU trainings distributed	3	39	July 31, 2025
Question about GPU memory usage when using pipeline parallelism training under larger micro batch count torchtitan	4	83	July 30, 2025
Can I shard a subset of weights and replicate others in FSDP2? distributed	0	17	July 30, 2025
Gradient not accumulated across nodes in deepspeed code distributed	2	66	July 27, 2025
Ddp training and eval question distributed	2	22	July 26, 2025
Continued pre-training large models with FSDP2? distributed	2	47	July 26, 2025
FullyShardedDataParallel hangs depending on wrap policy for Llama-3.2-1B distributed	1	60	July 26, 2025
Why would functional and non-functional broadcast use `src` with different semantics? distributed	1	29	July 26, 2025
Why does init_device_mesh() or DeviceMesh() have to be called globally? distributed	3	56	July 22, 2025
Work vs. Future sync primitives for Distributed Torch backends distributed	1	46	July 21, 2025
[DCP] how to load dcp ckpts? distributed	3	31	July 21, 2025
Dist.all_gather with uneven tensor sizes distributed	1	58	July 20, 2025
Does elastic torch support model parallelism distributed	1	31	July 20, 2025
FSDP2 and gradient w.r.t. inputs distributed	1	32	July 20, 2025
Gathering dictionaries of DistributedDataParallel distributed	11	4035	July 17, 2025
Copying params between 2 identically sharded (FSDP) networks distributed	2	57	July 16, 2025
Split backward into multiple gpus distributed	2	58	July 16, 2025
NCCL timeout when reducing batch size distributed	1	55	July 15, 2025
How to Efficiently Gather Python Objects Across GPUs Without GPU-to-CPU-to-GPU-to-CPU Overhead in torch.distributed? distributed	1	34	July 11, 2025