distributed

torchtitan Use this subcategory for all discussions related to end to end composability of distributed technologies using torchtitan. distributed-rpc

Topic	Replies	Views	Activity
About the distributed category distributed	1	2830	January 20, 2021
In a multi-GPU DDP environment, if the loss on one rank is NaN while the others are normal, could this cause the all-reduce to hang? distributed	1	26	November 12, 2025
RPC cannot run in jetson orin because of the specific uuid of orin distributed-rpc	3	40	November 11, 2025
Distributed Training causes model to output NaN values after resuming from snapshot distributed	0	18	November 7, 2025
Pipeline Parallelism performance with distributed-rpc on Jetson Nano devices distributed-rpc	3	1119	November 6, 2025
Problem: Pipeline Parallelism with distributed-rpc on Jetson Nano devices distributed-rpc	1	200	October 28, 2025
FSDP2 and gradient w.r.t. inputs distributed	2	71	October 28, 2025
Using Symmetric Memory One Shot All Reduce distributed	1	649	October 27, 2025
Tensor parallelism in image models like Unet distributed	4	484	October 27, 2025
Windows DDP on RTX 50-series only: use_libuv was requested but PyTorch was built without libuv support (works on 40/20-series) distributed-rpc	0	111	October 25, 2025
CPU thread slow to enqueue GPU and communication kernels torchtitan	2	50	October 20, 2025
Get `state_dict` from `DataDistributedParallel` model while other thread is running `backward` distributed	0	19	October 19, 2025
Suggested design for multiprocess federated learning distributed	1	462	October 13, 2025
Use fsdp training, 80 h800 gpu can run success, but 160 h800 gpu oom distributed	0	11	October 11, 2025
I am running the below code, which is wrong, but still the torch run command runs without any errors? How do I debug this? distributed	3	50	October 6, 2025
Using debugpy with DDP results in driver leaking GPU memory distributed	1	30	October 2, 2025
Model.to(device) vs. tensor.to(device) distributed	3	202	September 28, 2025
Does torch support custom stream for nccl commucation now? distributed	5	84	September 28, 2025
"Cannot allocate memory" for multinode training distributed	2	33	September 23, 2025
How to apply selective activation checkpointing on _grouped_mm torchtitan	0	56	September 20, 2025
DistributedDataParallel init hangs distributed	1	196	September 20, 2025
Process stuck by the dist.barrier() using DDP after dist.init_process_group distributed	2	473	September 20, 2025
Proper way to call torch.distributed.send/recv distributed	4	73	September 18, 2025
Understanding relation of FSDP and TP distributed	0	43	September 16, 2025
Support for Ulysses/Ring distributed attention for long-context training (32k) for 32B dense models distributed	0	101	September 15, 2025
WebDataset Multi-GPU Single-Node distributed	3	214	September 15, 2025
DDP overwriting a buffer with random values distributed	1	28	September 15, 2025
DDP: model not synchronizing across gpu's distributed	7	5401	September 14, 2025
Low-level errors when retrying training after OOMs distributed	3	78	September 12, 2025
Proper way to combine Tensor subclass with FSDP distributed	2	44	September 8, 2025