# When I use 1024 nodes in rpc, I meet RuntimeError "listen: Address already in use" **URL:** \ **Category:** distributed-rpc\ **Created:** [September 9, 2020, 9:49am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642 "2020-09-09T09:49:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![yueyilia](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/32.png) [@yueyilia](https://discuss.pytorch.org/u/yueyilia)\ **Post date:** [September 9, 2020, 9:49am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/1 "2020-09-09T09:49:55Z")
```auto rpc.init_rpc('env_{}'.format(rank), rank=rank, world_size=opt.world_size, rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(rpc_timeout=100)) ``` if I use 128 nodes, it works. but when I use 1024 nodes (32 servers \* 32 processes or 16 servers \* 64 processes), I meet ```auto ... RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:184] listen: Address already in use ``` my environment is pytorch 1.6.0, python 3.7, cuda 10.1. is there anyone have met it before ? ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [September 9, 2020, 5:50pm UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/2 "2020-09-09T17:50:24Z")
Hey @yueyilia, I haven’t seen this error before. Which version of PyTorch are you using? ---
**Author:** ![yueyilia](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/32.png) [@yueyilia](https://discuss.pytorch.org/u/yueyilia)\ **Post date:** [September 10, 2020, 1:56am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/3 "2020-09-10T01:56:49Z")
my environment is pytorch 1.6.0, python 3.7, cuda 10.1. ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [September 10, 2020, 2:45am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/4 "2020-09-10T02:45:16Z")
Does [tensorpipe RPC backend](https://pytorch.org/docs/master/rpc.html#tensorpipe-backend) work in this case? It is still experimental, but we plan to make it the default backend and focus on it in future releases. Regarding the error, I suspect this is a limitation in Gloo. I don’t have enough resource to verify this locally. If possible, can you try if [`init_process_group`](https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group) and [`all_reduce`](https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce) work with 1024 nodes? ---
**Author:** ![yueyilia](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/32.png) [@yueyilia](https://discuss.pytorch.org/u/yueyilia)\ **Post date:** [September 10, 2020, 3:20am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/5 "2020-09-10T03:20:13Z")
I replace process group with tensorpipe, I also meet ```auto ... RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:184] listen: Address already in use ``` It seems that tensorpipe still depends on gloo. I run init\_process\_group and all\_reduce, and the error is the same. ---
**Author:** ![yueyilia](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/df788c/32.png) [@yueyilia](https://discuss.pytorch.org/u/yueyilia)\ **Post date:** [September 10, 2020, 8:24am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/6 "2020-09-10T08:24:10Z")
It seems that process group needs too many ports. It will establish a connection between every two nodes, so that each server needs 32\*1024 ports (32 servers \* 32 processes) to establish a tcp connection. Do you have any plans to optimize it? ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [September 10, 2020, 2:10pm UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/7 "2020-09-10T14:10:34Z")
> It seems that tensorpipe still depends on gloo. It’s true for now, but only for initialization and shutdown. We will remove its dependency on gloo in future releases. (hopefully in v1.8) > It seems that process group needs too many ports. It will establish a connection between every two nodes, so that each server needs 32\*1024 ports (32 servers \* 32 processes) to establish a tcp connection. Do you have any plans to optimize it? Good catch! I am not aware of any plan to improve gloo for this. cc @pbelevich ---
**Author:** ![pritamdamania87](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/32/28891_2.png) [@pritamdamania87](https://discuss.pytorch.org/u/pritamdamania87)\ **Post date:** [September 17, 2020, 9:29pm UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/8 "2020-09-17T21:29:17Z")
For the RPC Framework it seems like this is happening since Gloo creates a tcp connection for all combination of processes in the group. I’m wondering if this can be avoid in TensorPipe where the TCP connections are created on demand and kept in a pool for reuse. Typically in a RPC environment, we’re not talking to all the nodes in the group at the same time. @yueyilia Could you add some details about your use case for RPC here? Are all the nodes (1024) communicating with all the other nodes in your application at the same time? Is it possible to run 1 process per server in your application to get around this in the short term? If GIL is currently the bottleneck, there is some amount of TorchScript support in the RPC framework that might help getting round GIL. ---
**Author:** ![osalpekar](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/5f9b8f/32.png) [@osalpekar](https://discuss.pytorch.org/u/osalpekar)\ **Post date:** [September 17, 2020, 11:10pm UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/9 "2020-09-17T23:10:31Z")
cc: @lcw re creating TCP connections on demand in TensorPipe ---
**Author:** ![lcw](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/b5e925/32.png) [@lcw](https://discuss.pytorch.org/u/lcw)\ **Post date:** [September 18, 2020, 11:10am UTC](https://discuss.pytorch.org/t/when-i-use-1024-nodes-in-rpc-i-meet-runtimeerror-listen-address-already-in-use/95642/10 "2020-09-18T11:10:23Z")
TensorPipe would fare better if indeed your topology (i.e., the links you actually use) are a significantly smaller subset than the complete graph. (For example, if you use a server-client pattern). In other words, if your graph is sparse. For dense (near-complete) graphs TensorPipe will perform even worse than Gloo because each pipe will internally use multiple TCP connections, whereas Gloo uses only one. The reason you’re currently unable to use TensorPipe is because, indeed, it uses Gloo internally for the join step. We’ve been wanting to get rid of this for a while but it’s hard, because the RPC agent’s join must do a barrier, and it’s easier to do it through a library that already does collectives (namely Gloo) rather than re-implement it. We could use the c10d::Store instead of the ProcessGroup for that, but currently the Store isn’t powerful enough. @osalpekar was thinking of refactoring it though so maybe then we could do this change. See [https://github.com/pytorch/pytorch/issues/42879](https://github.com/pytorch/pytorch/issues/42879) and [https://github.com/pytorch/pytorch/issues/41614](https://github.com/pytorch/pytorch/issues/41614) for more context.