# distributed-rpc **URL:** https://discuss.pytorch.org/c/distributed/distributed-rpc/20.md [Latest](https://discuss.pytorch.org/latest.md) · [Categories](https://discuss.pytorch.org/categories.md) --- ## [About the distributed-rpc category](https://discuss.pytorch.org/t/about-the-distributed-rpc-category/66315)
**Author:** [@smth](https://discuss.pytorch.org/u/smth)\ **Replies:** 0
--- ## [Ease development by running computations on remote GPU](https://discuss.pytorch.org/t/ease-development-by-running-computations-on-remote-gpu/121002)
**Author:** [@meandmymind](https://discuss.pytorch.org/u/meandmymind)\ **Replies:** 11\ **Last updated:** [October 4, 2026, 9:08pm UTC](https://discuss.pytorch.org/t/ease-development-by-running-computations-on-remote-gpu/121002 "2026-10-04T21:08:54Z")
Hello! For development I use local machine with no GPU and have a remote machine with GPU. I like to debug my code via IDE tools but also want to have access to gpu. Using something a-la vscode over ssh is kinda slow … --- ## [RPC cannot run in jetson orin because of the specific uuid of orin](https://discuss.pytorch.org/t/rpc-cannot-run-in-jetson-orin-because-of-the-specific-uuid-of-orin/223904)
**Author:** [@mavs](https://discuss.pytorch.org/u/mavs)\ **Replies:** 3\ **Last updated:** [November 11, 2025, 2:57am UTC](https://discuss.pytorch.org/t/rpc-cannot-run-in-jetson-orin-because-of-the-specific-uuid-of-orin/223904 "2025-11-11T02:57:47Z")
When run RPC demo in jetson orin, the uuid issue were shown as below: tensorpipe/channel/cuda\_ipc/context\_impl.cc:65 "uuidStr.substr(0, 4) != "GPU-"Couldn’t obtain valid UUID for GPU #0 from CUDA driver. The uuid of je… --- ## [Pipeline Parallelism performance with distributed-rpc on Jetson Nano devices](https://discuss.pytorch.org/t/pipeline-parallelism-performance-with-distributed-rpc-on-jetson-nano-devices/149519)
**Author:** [@minh\_nn](https://discuss.pytorch.org/u/minh_nn)\ **Replies:** 3\ **Last updated:** [November 6, 2025, 5:59am UTC](https://discuss.pytorch.org/t/pipeline-parallelism-performance-with-distributed-rpc-on-jetson-nano-devices/149519 "2025-11-06T05:59:30Z")
Hi, i’m a student and currently working on pipelining models on edge devices. My current setup is: 1 Master linux machine, 2 Jetson Nano as workers. I used the example on: https://pytorch.org/tutorials/intermediate/di… --- ## [Problem: Pipeline Parallelism with distributed-rpc on Jetson Nano devices](https://discuss.pytorch.org/t/problem-pipeline-parallelism-with-distributed-rpc-on-jetson-nano-devices/204031)
**Author:** [@Nishant\_Madan](https://discuss.pytorch.org/u/Nishant_Madan)\ **Replies:** 1\ **Last updated:** [October 28, 2025, 12:10pm UTC](https://discuss.pytorch.org/t/problem-pipeline-parallelism-with-distributed-rpc-on-jetson-nano-devices/204031 "2025-10-28T12:10:04Z")
Hi, i’m a student and currently working on pipelining models on edge devices. My current setup is: 1 Master linux machine, 2 Jetson Nano as workers. I used the example on: https://pytorch.org/tutorials/intermediate/di… --- ## [Windows DDP on RTX 50-series only: use\_libuv was requested but PyTorch was built without libuv support (works on 40/20-series)](https://discuss.pytorch.org/t/windows-ddp-on-rtx-50-series-only-use-libuv-was-requested-but-pytorch-was-built-without-libuv-support-works-on-40-20-series/223698)
**Author:** [@Swati\_sd](https://discuss.pytorch.org/u/Swati_sd)\ **Replies:** 0\ **Last updated:** [October 25, 2025, 3:07am UTC](https://discuss.pytorch.org/t/windows-ddp-on-rtx-50-series-only-use-libuv-was-requested-but-pytorch-was-built-without-libuv-support-works-on-40-20-series/223698 "2025-10-25T03:07:02Z")
Goal Run PatchCore (FAISS + PyTorch) with DDP on Windows using python -m torch.distributed.run --nproc\_per\_node=2. Command set USE\_LIBUV=0 && set MASTER\_ADDR=127.0.0.1 && set MASTER\_PORT=29500 ^ && python -m torch.dist… --- ## [Using torch rpc to connect to remote machine](https://discuss.pytorch.org/t/using-torch-rpc-to-connect-to-remote-machine/168057)
**Author:** [@mndl1](https://discuss.pytorch.org/u/mndl1)\ **Replies:** 2\ **Last updated:** [June 7, 2025, 9:33pm UTC](https://discuss.pytorch.org/t/using-torch-rpc-to-connect-to-remote-machine/168057 "2025-06-07T21:33:41Z")
Hi, I have a use case where I want to train a model for which some layers are on a remote client machine, and most of the layers are on a GPU server. I understand that the torch.distributed.rpc library is there to help w… --- ## [Does DistributedOptimizer support zero\_grad and lr\_scheduling?](https://discuss.pytorch.org/t/does-distributedoptimizer-support-zero-grad-and-lr-scheduling/81359)
**Author:** [@Kunlin\_Yang](https://discuss.pytorch.org/u/Kunlin_Yang)\ **Replies:** 2\ **Last updated:** [March 27, 2025, 7:30am UTC](https://discuss.pytorch.org/t/does-distributedoptimizer-support-zero-grad-and-lr-scheduling/81359 "2025-03-27T07:30:11Z")
I try to use distributed\_rpc to implement parameter server. I would like to use zero\_grad and lr\_scheduling feature in the trainer. But it seems that the DistributedOptimizer does not support this. Is there any workaroun… --- ## [Memory leak when using RPC for pipeline parallelism](https://discuss.pytorch.org/t/memory-leak-when-using-rpc-for-pipeline-parallelism/127236)
**Author:** [@HuYang719](https://discuss.pytorch.org/u/HuYang719)\ **Replies:** 17\ **Last updated:** [February 13, 2025, 12:07pm UTC](https://discuss.pytorch.org/t/memory-leak-when-using-rpc-for-pipeline-parallelism/127236 "2025-02-13T12:07:39Z")
Hi, when I use torch.distributed.rpc to implement pipeline parallelism for transformer-based inference, the memory consumption increases with each forward pass. The code for 2 nodes is like this, First, I define two cl… --- ## [Sharing CUDA tensor between different processes and pytorch versions](https://discuss.pytorch.org/t/sharing-cuda-tensor-between-different-processes-and-pytorch-versions/215253)
**Author:** [@braindotai](https://discuss.pytorch.org/u/braindotai)\ **Replies:** 0\ **Last updated:** [January 11, 2025, 10:16am UTC](https://discuss.pytorch.org/t/sharing-cuda-tensor-between-different-processes-and-pytorch-versions/215253 "2025-01-11T10:16:36Z")
I am trying to share one Cuda tensor from one Python script to another… both scripts have different torch versions. My thinking is that if we can share the memory address and the tensor size from one process to another, … --- ## [Embedded Python can't import torch in a C++ project](https://discuss.pytorch.org/t/embedded-python-cant-import-torch-in-a-c-project/212740)
**Author:** [@Himateja\_Nallani](https://discuss.pytorch.org/u/Himateja_Nallani)\ **Replies:** 1\ **Last updated:** [December 1, 2024, 5:37am UTC](https://discuss.pytorch.org/t/embedded-python-cant-import-torch-in-a-c-project/212740 "2024-12-01T05:37:43Z")
Hi, PyTorch community! This is my first post, so please bear with me. I have been working with Network Simulator (NS-3), a C++ code base and the project was built using CMake. I am calling “import torch” using embedding… --- ## [Connect \[127.0.1.1\]:20892: Connection refused](https://discuss.pytorch.org/t/connect-127-0-1-1-connection-refused/212976)
**Author:** [@mingyu](https://discuss.pytorch.org/u/mingyu)\ **Replies:** 0\ **Last updated:** [November 14, 2024, 9:15am UTC](https://discuss.pytorch.org/t/connect-127-0-1-1-connection-refused/212976 "2024-11-14T09:15:49Z")
Hi! I have two jetson devices, A and B, both of which are connected to the same switch, with one IP address of 192.168.1.101 and the other 192.168.1.153. I want to use the RPC protocol to make two devices do a simple tas… --- ## [Getting Gloo error when connecting server and client over VPN from different systems](https://discuss.pytorch.org/t/getting-gloo-error-when-connecting-server-and-client-over-vpn-from-different-systems/191076)
**Author:** [@selineni](https://discuss.pytorch.org/u/selineni)\ **Replies:** 2\ **Last updated:** [August 15, 2024, 8:36am UTC](https://discuss.pytorch.org/t/getting-gloo-error-when-connecting-server-and-client-over-vpn-from-different-systems/191076 "2024-08-15T08:36:29Z")
i running rpc\_sync between 1 server and 2 nodes connected over VPN. when i run them as containers from single platform (like server or laptop), there is no error. but if i divide the server container and node container… --- ## [Pytorch with MPI backend](https://discuss.pytorch.org/t/pytorch-with-mpi-backend/207825)
**Author:** [@dolphin\_09](https://discuss.pytorch.org/u/dolphin_09)\ **Replies:** 1\ **Last updated:** [August 12, 2024, 4:12pm UTC](https://discuss.pytorch.org/t/pytorch-with-mpi-backend/207825 "2024-08-12T16:12:38Z")
Hi Everyone, I am trying to run MPI with Pytorch from Source for distributed runs. I am able to build, compile and instal. But post installation, i am unable to import torch. I am using OpenMPi and Pytorch latest versi… --- ## [torch.distributed.DistBackendError: NCCL error](https://discuss.pytorch.org/t/torch-distributed-distbackenderror-nccl-error/191509)
**Author:** [@Jonathan1](https://discuss.pytorch.org/u/Jonathan1)\ **Replies:** 16\ **Last updated:** [July 25, 2024, 6:31pm UTC](https://discuss.pytorch.org/t/torch-distributed-distbackenderror-nccl-error/191509 "2024-07-25T18:31:45Z")
Hello! When running a DeepSpeed training job, I get this error: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331 The job works fine on a single node but gives … --- ## [Parameter Server with RPC and NCCL](https://discuss.pytorch.org/t/parameter-server-with-rpc-and-nccl/206737)
**Author:** [@Osama\_Abu\_Hamdan](https://discuss.pytorch.org/u/Osama_Abu_Hamdan)\ **Replies:** 1\ **Last updated:** [July 25, 2024, 4:33pm UTC](https://discuss.pytorch.org/t/parameter-server-with-rpc-and-nccl/206737 "2024-07-25T16:33:57Z")
Greetings, I am trying to implement ParameterServer following this tutorial I have two observations 1- The execution is too slow. To train one epoch it takes 2-3 minutes, while in DDP with NCCL it takes 30 seconds. 2… --- ## [Distributed training on slurm cluster](https://discuss.pytorch.org/t/distributed-training-on-slurm-cluster/150417)
**Author:** [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\ **Replies:** 14\ **Last updated:** [July 16, 2024, 12:02pm UTC](https://discuss.pytorch.org/t/distributed-training-on-slurm-cluster/150417 "2024-07-16T12:02:10Z")
Sorry for the naive question but I am confused about the integration of distributed training in a slurm cluster. Do we need to explicitly call the distributed.launch when invoking the python script or is this taken care … --- ## [Set longer timeout for torch distributed training](https://discuss.pytorch.org/t/set-longer-timeout-for-torch-distributed-training/165149)
**Author:** [@ralzq01](https://discuss.pytorch.org/u/ralzq01)\ **Replies:** 5\ **Last updated:** [July 14, 2024, 7:16am UTC](https://discuss.pytorch.org/t/set-longer-timeout-for-torch-distributed-training/165149 "2024-07-14T07:16:23Z")
Hi, I wonder if there is a mechanism to synchronize all processes with unlimited waiting time. The scenario is in distributed training where one of processes in each node needs to deal with some CPU-related tasks, while… --- ## [How to implement multiprocessing with several GPUs on only one layer of neural network within the forward function](https://discuss.pytorch.org/t/how-to-implement-multiprocessing-with-several-gpus-on-only-one-layer-of-neural-network-within-the-forward-function/204868)
**Author:** [@justinS](https://discuss.pytorch.org/u/justinS)\ **Replies:** 0\ **Last updated:** [June 18, 2024, 5:52pm UTC](https://discuss.pytorch.org/t/how-to-implement-multiprocessing-with-several-gpus-on-only-one-layer-of-neural-network-within-the-forward-function/204868 "2024-06-18T17:52:46Z")
Hi All, I am running a machine learning experiment in which I need to implement parallel processing across multiple GPUs in order to speed up the forward function of the neural network. I understand the the nn.Distribute… --- ## [Concurrent P2P operation (i.e., send and recv) fail](https://discuss.pytorch.org/t/concurrent-p2p-operation-i-e-send-and-recv-fail/204437)
**Author:** [@zizhao.mo](https://discuss.pytorch.org/u/zizhao.mo)\ **Replies:** 4\ **Last updated:** [June 12, 2024, 12:41pm UTC](https://discuss.pytorch.org/t/concurrent-p2p-operation-i-e-send-and-recv-fail/204437 "2024-06-12T12:41:35Z")
Hi, I have a four-GPU group where all GPUs form a process group in torch. GPU0 and GPU1 and simultaneously send data to GPU2 and GPU3 respectively. However, only the communication between GPU0 and GPU2 is ok, the commun… --- ## [Using torch rpc with a function defined remotely](https://discuss.pytorch.org/t/using-torch-rpc-with-a-function-defined-remotely/203990)
**Author:** [@priyathamkat](https://discuss.pytorch.org/u/priyathamkat)\ **Replies:** 1\ **Last updated:** [June 4, 2024, 4:24pm UTC](https://discuss.pytorch.org/t/using-torch-rpc-with-a-function-defined-remotely/203990 "2024-06-04T16:24:32Z")
I would like to use rpc to run a function remotely. All the examples in the documentation have a method that is defined with in the same script or is from torch. I was wondering if its possible to use rpc.rpc\_sync with s… --- ## [Pytorch Distributed RPC connection using nvidia Nanos IP Addresses](https://discuss.pytorch.org/t/pytorch-distributed-rpc-connection-using-nvidia-nanos-ip-addresses/203620)
**Author:** [@Nishant\_Madan](https://discuss.pytorch.org/u/Nishant_Madan)\ **Replies:** 0\ **Last updated:** [May 28, 2024, 10:05am UTC](https://discuss.pytorch.org/t/pytorch-distributed-rpc-connection-using-nvidia-nanos-ip-addresses/203620 "2024-05-28T10:05:43Z")
I have 2 nvidia Nanos. I want to run the pretrained Pytorch Resnet sharded model on those devices. My main.py code is on GPU and I want to connect those devices using the code on the GPU for distributed inferencing. I am… --- ## [Importing RRef, rpc\_async, remote from RPC](https://discuss.pytorch.org/t/importing-rref-rpc-async-remote-from-rpc/201986)
**Author:** [@CollinsJnr](https://discuss.pytorch.org/u/CollinsJnr)\ **Replies:** 4\ **Last updated:** [May 22, 2024, 3:15pm UTC](https://discuss.pytorch.org/t/importing-rref-rpc-async-remote-from-rpc/201986 "2024-05-22T15:15:58Z")
Hey guys I just started using RPC yesterday for SEED\_RL and I have a problem importing RRef, rpc\_async, remote from torch.distributed.rpc if I run from torch.distributed.rpc import RRef, rpc\_async, remote It keeps sa… --- ## [Use DDP to train a single model, on a single GPU, multiple processes](https://discuss.pytorch.org/t/use-ddp-to-train-a-single-model-on-a-single-gpu-multiple-processes/202821)
**Author:** [@copythatpasta](https://discuss.pytorch.org/u/copythatpasta)\ **Replies:** 0\ **Last updated:** [May 15, 2024, 7:13pm UTC](https://discuss.pytorch.org/t/use-ddp-to-train-a-single-model-on-a-single-gpu-multiple-processes/202821 "2024-05-15T19:13:22Z")
I am at a loss here. I would like to maximize the number of steps my model takes within a specific time block, and so instead of filling the batch size to max out the VRAM, I want to run multiple processes for this singl… --- ## [Error for run a ready project with pytorch](https://discuss.pytorch.org/t/error-for-run-a-ready-project-with-pytorch/161174)
**Author:** [@setare](https://discuss.pytorch.org/u/setare)\ **Replies:** 14\ **Last updated:** [May 9, 2024, 2:05pm UTC](https://discuss.pytorch.org/t/error-for-run-a-ready-project-with-pytorch/161174 "2024-05-09T14:05:26Z")
I am working on a ready project https://github.com/microsoft/TAP This project is about text visual question answering. My system has multi GPU .Cuda and GPU are available in project (01:00.0 VGA compatible controller: … --- ## [How to Adapt DDP Pipeline Tutorial for Multi-Node Training](https://discuss.pytorch.org/t/how-to-adapt-ddp-pipeline-tutorial-for-multi-node-training/198597)
**Author:** [@jasonkrone](https://discuss.pytorch.org/u/jasonkrone)\ **Replies:** 1\ **Last updated:** [March 27, 2024, 12:29am UTC](https://discuss.pytorch.org/t/how-to-adapt-ddp-pipeline-tutorial-for-multi-node-training/198597 "2024-03-27T00:29:33Z")
Anyone have advice on how to adapt the ddp\_pipeline tutorial for multi-node training? I’ve been using torchrun for multi-node DistributedDataParallel training; however, the ddp\_pipeline example relies on mp.spawn. I tr… --- ## [Unexpected Behavior with torch.distributed.isend and irecv in Asynchronous Communication](https://discuss.pytorch.org/t/unexpected-behavior-with-torch-distributed-isend-and-irecv-in-asynchronous-communication/199511)
**Author:** [@zhanghb55](https://discuss.pytorch.org/u/zhanghb55)\ **Replies:** 0\ **Last updated:** [March 25, 2024, 6:54am UTC](https://discuss.pytorch.org/t/unexpected-behavior-with-torch-distributed-isend-and-irecv-in-asynchronous-communication/199511 "2024-03-25T06:54:13Z")
I am attempting to utilize asynchronous send (isend) and receive (irecv) for non-blocking communication in a PyTorch distributed setup. My understanding of asynchronous operations implies that torch.distributed.isend sho… --- ## [Problem abount fsdp training. How to select cudatoolkit version of nvidia-nccl-cu12?](https://discuss.pytorch.org/t/problem-abount-fsdp-training-how-to-select-cudatoolkit-version-of-nvidia-nccl-cu12/198259)
**Author:** [@takumi\_OSHITA](https://discuss.pytorch.org/u/takumi_OSHITA)\ **Replies:** 8\ **Last updated:** [March 6, 2024, 7:48am UTC](https://discuss.pytorch.org/t/problem-abount-fsdp-training-how-to-select-cudatoolkit-version-of-nvidia-nccl-cu12/198259 "2024-03-06T07:48:14Z")
Here is the translated text: "Hello, while conducting distributed learning using FSDP, I encounter the following error between saving weights and loading the saved checkpoint: torch.distributed.DistBackendError: NCCL e… --- ## [What port/s does DDP use?](https://discuss.pytorch.org/t/what-port-s-does-ddp-use/197948)
**Author:** [@TheVictor\_777](https://discuss.pytorch.org/u/TheVictor_777)\ **Replies:** 0\ **Last updated:** [February 29, 2024, 2:09pm UTC](https://discuss.pytorch.org/t/what-port-s-does-ddp-use/197948 "2024-02-29T14:09:46Z")
Hello, I have repeatedly attempted to train a model across two nodes using DDP and torchrun. For some unknown reason, my client node times out after 15 minutes of no connection. Thus far, I have been unable to resolve th… --- ## [RPC for model parallelism increase GPU memory usage](https://discuss.pytorch.org/t/rpc-for-model-parallelism-increase-gpu-memory-usage/197460)
**Author:** [@patataman](https://discuss.pytorch.org/u/patataman)\ **Replies:** 1\ **Last updated:** [February 27, 2024, 11:11am UTC](https://discuss.pytorch.org/t/rpc-for-model-parallelism-increase-gpu-memory-usage/197460 "2024-02-27T11:11:44Z")
Hello, I’m using RPC for applying model parallelism and I don’t see any kind of reduction in the memory usage. Even more, the memory usage is doubled! This is the code I’m executing with RPC + Torchrun to use 3 nodes (1… [Next page](https://discuss.pytorch.org/c/distributed/distributed-rpc/20.md?page=1)