Hi! I have two jetson devices, A and B, both of which are connected to the same switch, with one IP address of 192.168.1.101 and the other 192.168.1.153. I want to use the RPC protocol to make two devices do a simple task (device A passes an int variable to device B, and device B does a multiplication of this variable and outputs it). Both devices have 1.11.0 version of torch, and below is the code for both of my devices.
A Equipment:
import torch
import torch.distributed.rpc as rpc
import torch.distributed as dist
def send_data_to_device_b():
a = 3
print(f"Device A sending data: a = {a}")
rpc.rpc_sync("device_b", add_and_print, args=(a,))
def init_rpc():
print(“Initializing RPC on device A…”)
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:20892”, world_size=2, rank=0)
rpc.init_rpc("device_a", rank=0, world_size=2)
print("Sending data to device B...")
send_data_to_device_b()
print("Shutting down RPC...")
rpc.shutdown()
if name == “main”:
init_rpc()
B equipment:
import torch
import torch.distributed.rpc as rpc
import torch.distributed as dist
def add_and_print(a):
b = 4
print(f"Device B received: a = {a}, b = {b}. The result of a + b is: {a + b}")
def init_rpc():
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:29520”, world_size=2, rank=1)
rpc.init_rpc("device_b", rank=1, world_size=2)
rpc.shutdown()
if name == “main”:
init_rpc()
I started the code on device B first, and then started the code on device A, and device A returned the following error message.
Initializing RPC on device A…
Traceback (most recent call last):
File “rpc_a.py”, line 30, in
init_rpc()
File “rpc_a.py”, line 16, in init_rpc
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:29520”, world_size=2, rank=0)
File “/home/nvidia/anaconda3/envs/py3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 602, in init_process_group
default_pg = _new_process_group_helper(
File “/home/nvidia/anaconda3/envs/py3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 703, in _new_process_group_helper
pg = ProcessGroupGloo(prefix_store, rank, world_size, timeout=timeout)
RuntimeError: [/media/nvidia/NVME/pytorch/JetPack_5.0/pytorch-v1.11.0/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.1.1]:20892: Connection refused
And device B has been waiting for the connection and nothing has returned. I confirmed that the firewalls of both devices are off and can ping each other at the same time.
What is the problem and how to fix it?
Any suggestion would be very useful!