How to make rpc work with WSL

frankdong · September 16, 2020, 3:00am

I’m trying below sample on multiple machines:

pytorch/examples/blob/master/distributed/rpc/parameter_server/rpc_parameter_server.py

import argparse
import os
import time
from threading import Lock

import torch
import torch.distributed.autograd as dist_autograd
import torch.distributed.rpc as rpc
import torch.multiprocessing as mp
import torch.nn as nn
import torch.nn.functional as F
from torch import optim
from torch.distributed.optim import DistributedOptimizer
from torchvision import datasets, transforms

# --------- MNIST Network to train, from pytorch/examples -----


class Net(nn.Module):
    def __init__(self, num_gpus=0):

This file has been truncated. show original

When I tried on 2 linux machine in same domain, this sample works fine. But when I try on 2 WSL machine, I can’t make these 2 WSL machine connect with each other, both machines blocking waiting on rpc init.

Tried the " WSL 2 TPC NETWORK FORWARDING" workaround from below post, rpc still can’t connect with each other:

So I’m wondering if there is any working sample/setting for WSL? Any suggestions on how to trouble the issue is also highly appreciated, thanks.

mrshenli · September 16, 2020, 6:28pm

We haven’t tests WSL (I assume you mean Windows Subsystem for Linux), and are not sure what are the gaps if there are any.

cc @ptrblck do you know who is familiar with WSL?

Not sure if this can help, I would try to verify if the following command resolves to the correct interface. If not, set GLOO_SOCKET_IFNAME explicitly.

getent hosts `hostname`

frankdong · September 17, 2020, 3:22am

@mrshenli Thanks for your reply.
Yes, we are using Windows Subsystem for Linux as Windows not support distributed yet. Although I see an open PR for this (but we don’t know if rpc will be support in this PR): https://github.com/pytorch/pytorch/pull/42897
And yes, hostname can resolve correctly. One thing that different on linux machine and WSL machine we are aware of is WSL machine share ip with host windows system but need port forwarding from windows to WSL. We already tried this solution that add port forwarding but seems rpc still can’t connect.
https://github.com/microsoft/WSL/issues/4150

So question is say rank0 is listening on one port 12560 for example, is this the only port will be used on rank0 machine during rpc initialization and connection?
Also, is there any way to output detailed information from rpc during initialization so we can find more details here. Ideally we will be able to know what address/port rank0 is listening on and where other ranks are connecting to, whether there is some mismatch on the address or other things cause the rpc initialization hangs.

ptrblck · September 17, 2020, 5:00am

Unfortunately, I’m not familiar with it and don’t know a specific user who might be.
Maybe @maxiluk or @peterjc123 would know more.

mrshenli · September 17, 2020, 3:04pm

Yep, MSFT experts are helping us adding Windows support to PT Distributed. The first step focuses on DDP only, but we do plan to cover all features in the distributed package in future releases.

So question is say rank0 is listening on one port 12560 for example, is this the only port will be used on rank0 machine during rpc initialization and connection?

I see. No, that port is only used for rendezvous during initialization. RPC backend will grab a port for each pair of workers, which are not visible to users.

cc @lcw any suggestion on how RPC can work with port forwarding?

frankdong · September 17, 2020, 7:04pm

Do we know the port range so we can have a test

lcw · September 18, 2020, 9:12am

I’m not super familiar with port forwarding on WSL, but I assume it does some sort of NAT, right? In that case, I suspect it would be enough if only the listening socket(s) on each server had a fixed port which is correctly forwarded? (as then all other sockets that are accepted will have a random port but the NAT would be aware of them and would handle them)

Unfortunately, at this moment, also the port of the listening socket(s) is picked at random (well, not at random, but we’re letting the kernel give us back an arbitrary available port). We don’t support a way for the user to specify a port. Which I think means there’s no real way for you to set up port forwarding before launching the application.