distributed

torchtitan Use this subcategory for all discussions related to end to end composability of distributed technologies using torchtitan. distributed-rpc

Topic	Replies	Views	Activity
About the distributed category distributed	1	2676	January 20, 2021
Adding/removing new trainers on the single node to elastic training distributed	0	4	February 21, 2025
Segfault during torch.save distributed	2	14	February 21, 2025
Two models in distributed data parallel distributed	0	7	February 20, 2025
Handling signals in distributed train loop distributed	0	6	February 20, 2025
[Distributed w/ TorchTitan] Breaking Barriers: Training Long Context LLMs with 1M Sequence Length in PyTorch Using Context Parallel torchtitan	1	3263	February 20, 2025
Does FSDP shard frozen layers? distributed	0	5	February 19, 2025
CPU memory in FSDP lora merging distributed	0	13	February 14, 2025
Memory leak when using RPC for pipeline parallelism distributed-rpc	17	2511	February 13, 2025
Too much GPU memory usage for input/model size? distributed	0	17	February 13, 2025
PyTorch Not Automatically Utilizing Multiple GPUs distributed	4	48	February 13, 2025
Cannot find NCCL libnccl-net.so file distributed	5	3639	February 11, 2025
Code is getting struck due to async between the gpus in distributed setup distributed	1	20	February 11, 2025
Finding the cause of RuntimeError: Expected to mark a variable ready only once distributed	27	22998	February 10, 2025
Activation Checkpointing wrapper failed with torch.jit.save distributed	3	43	February 10, 2025
CLIP Model Batching When Batch Size Limited distributed	1	167	February 10, 2025
How to save checkpoint when using FSDP2? distributed	0	29	February 9, 2025
DDP training hangs on one rank during backward on H100s distributed	3	190	February 8, 2025
FP8 training with torchao but without torchtitan distributed	2	140	February 7, 2025
SLURM srun vs torchrun: Difference numbers of spawn processes distributed	2	908	February 7, 2025
NCCL Timeout only on H100s, not other hardware distributed	3	95	February 6, 2025
DDP with SLURM hangs distributed	1	347	February 6, 2025
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper__index_select) while using Dataparallel class distributed	11	7750	February 6, 2025
FSDP for multi-gpu encounters ValueError: Inconsistent compute device and `device_id` on rank 1: cuda:0 vs cuda:1 distributed	0	57	February 5, 2025
How to get data on the same device as fsdp model during training? distributed	0	20	February 4, 2025
FSDP clarifying questions distributed	0	55	February 3, 2025
Torch.multiprocessing: pass metadata or class wrapper for shared memory CUDA tensor? distributed	0	8	January 31, 2025
Distributed training got stuck every few seconds distributed	14	3711	January 29, 2025
Wrapping a DDP module inside a simple Module distributed	0	10	January 29, 2025
InfiniBand Vs TCP distributed	1	88	January 29, 2025