Does torch.distributed support the ipv6 address?

khalil · June 12, 2020, 3:24am

Hi All,
Now I import the torch.distributed package to train my model in some nodes. And I want to use the CPU to train it so I use gloo as the backend.

When I use this package to do distributed training, I should set some Common environment variables like GLOO_SOCKET_IFNAME and MASTER_ADDR.I use the ifconfig command to print the interface, it just like this.

I want to know how to set these environment variables to running python -m torch.distributed.launch correctly.Thanks!!!

mrshenli · June 12, 2020, 2:48pm

I haven’t tried myself, but I think it should work as the code does seem to handle ipv6:

github.com

pytorch/pytorch/blob/db81e13d6b061fe7b219a6ef4d05e8b4223dd0c3/torch/lib/c10d/Utils.cpp#L205


}

int connect(
    const std::string& address,
    PortType port,
    bool wait,
    const std::chrono::milliseconds& timeout) {
  struct ::addrinfo hints, *res = NULL;
  std::memset(&hints, 0x00, sizeof(hints));
  hints.ai_flags = AI_NUMERICSERV; // specifies that port (service) is numeric
  hints.ai_family = AF_UNSPEC; // either IPv4 or IPv6
  hints.ai_socktype = SOCK_STREAM; // TCP

  // `getaddrinfo` will sort addresses according to RFC 3484 and can be tweeked
  // by editing `/etc/gai.conf`. so there is no need to manual sorting
  // or protcol preference.
  int err =
      ::getaddrinfo(address.data(), std::to_string(port).data(), &hints, &res);
  if (err != 0 || !res) {
    throw std::invalid_argument(
        "host not found: " + std::string(gai_strerror(err)));

Can you try setting GLOO_SOCKET_IFNAME to eth0, MASTER_ADDR to its inet6 string, and MASTER_PORT to, say, 23456?

khalil · June 13, 2020, 2:17am

Thank you, mrshenli. I tried setting CLOO_SOCKET_IFNAME to eth0 and MASTER_ADDR to inet6 string, it returned a connection error(Timeout). Then I found gn0 might be an Infiniband Interface, so I set CLOO_SOCKET_IFNAME to gn0 and MASTER_ADDR to ipv4 address.

From the code, I think the MASTER_ADDR can be set to ipv6, thanks for your help!