# \`torch.distributed.barrier\` used in multi-node distributed data-parallel training **URL:** \ **Category:** distributed\ **Created:** [July 18, 2020, 3:59pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711 "2020-07-18T15:59:46Z")\ **Posts on this page:** 9\ **Page:** 2
**Author:** ![leimao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leimao/32/26860_2.png) [@leimao](https://discuss.pytorch.org/u/leimao)\ **Post date:** [July 21, 2020, 3:31am UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/22 "2020-07-21T03:31:25Z")
Thank you very much @iffiX. I will try `gloo` tomorrow. Best, Lei ---
**Author:** ![leimao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leimao/32/26860_2.png) [@leimao](https://discuss.pytorch.org/u/leimao)\ **Post date:** [July 21, 2020, 7:46pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/23 "2020-07-21T19:46:25Z")
@iffiX @mrshenli I just got time to test the `gloo` backend. It seems that the training could be run without significant problems. However, I do have concerns. I found the number of processes is 7 on each node despite the fact that I requested using 4 GPU on each node. ```auto +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 53447 C /opt/conda/bin/python 1511MiB | | 0 53448 C /opt/conda/bin/python 803MiB | | 0 53449 C /opt/conda/bin/python 803MiB | | 0 53450 C /opt/conda/bin/python 803MiB | | 1 53448 C /opt/conda/bin/python 1511MiB | | 2 53449 C /opt/conda/bin/python 1511MiB | | 3 53450 C /opt/conda/bin/python 1511MiB | +-----------------------------------------------------------------------------+ ``` The GPU memory usages are not even as well. ```auto $ nvidia-smi Tue Jul 21 19:49:09 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.126.02 Driver Version: 418.126.02 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:06:00.0 Off | 0 | | N/A 38C P0 49W / 163W | 3933MiB / 32480MiB | 11% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000000:07:00.0 Off | 0 | | N/A 38C P0 46W / 163W | 1522MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 | | N/A 38C P0 46W / 163W | 1522MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 | | N/A 37C P0 48W / 163W | 1522MiB / 32480MiB | 9% Default | +-------------------------------+----------------------+----------------------+ | 4 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 | | N/A 36C P0 42W / 163W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 | | N/A 38C P0 43W / 163W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 | | N/A 38C P0 43W / 163W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 | | N/A 37C P0 41W / 163W | 11MiB / 32480MiB | 0% Default | +-------------------------------+----------------------+----------------------+ ``` Can you guys explain what’s happening here? Regarding the `nccl` backend problem, I currently don’t have time to troubleshoot at more lower level. But I believe it is a bug, either in the `nccl` library or in the PyTorch implementation. Thank you. Best, Lei ---
**Author:** ![iffiX](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/32/24443_2.png) [@iffiX](https://discuss.pytorch.org/u/iffiX)\ **Post date:** [July 22, 2020, 12:18am UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/24 "2020-07-22T00:18:53Z")
This is not an error, as you can see: ```auto 0 53448 C /opt/conda/bin/python 803MiB 1 53448 C /opt/conda/bin/python 1511MiB ``` Their PID are the same, it seems that DDP will spawn an additional process for all “secondary processes”, except the “primary process”, probably for receiving tensors etc. “803MiB” should be the base kernel memory usage, if you spawn any process using cuda in pytorch. Actions such as moving a model to gpu, creating a tensor on gpu will invoke cuda. see this issue for detail explainations: [issue](https://github.com/pytorch/pytorch/issues/12873) I can also replicate this behavior on my machine, so don’t worry about it: ![Screenshot from 2020-07-22 08-10-32](https://discuss.pytorch.org/uploads/default/original/3X/2/2/222dab29187cc8c379eb7e42258dd9948d242792.png) The replication script is a slightly modified version from the DDP tutorial: ```auto import torch import torch.distributed as dist import torch.multiprocessing as mp import torch.nn as nn import torch.optim as optim from torch.nn.parallel import DistributedDataParallel as DDP import os os.environ["MASTER_ADDR"]="localhost" os.environ["MASTER_PORT"]="9003" def example(rank, world_size): # create default process group dist.init_process_group("gloo", rank=rank, world_size=world_size) # create local model model = nn.Linear(10, 10).to(rank) # construct DDP model ddp_model = DDP(model, device_ids=[rank]) # define loss function and optimizer loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_model.parameters(), lr=0.001) while True: # forward pass outputs = ddp_model(torch.randn(20, 10).to(rank)) labels = torch.randn(20, 10).to(rank) # backward pass loss_fn(outputs, labels).backward() # update parameters optimizer.step() def main(): world_size = 2 mp.spawn(example, args=(world_size,), nprocs=world_size, join=True) if __name__ ==" __main__": main() ``` ---
**Author:** ![leimao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leimao/32/26860_2.png) [@leimao](https://discuss.pytorch.org/u/leimao)\ **Post date:** [July 22, 2020, 6:06am UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/25 "2020-07-22T06:06:20Z")
Thank you @iffiX. I used to always use `nccl`. In my impression, I remember the GPU memory occupancy is always the same for each GPU on each node. ---
**Author:** ![iffiX](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/32/24443_2.png) [@iffiX](https://discuss.pytorch.org/u/iffiX)\ **Post date:** [July 22, 2020, 6:28am UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/26 "2020-07-22T06:28:49Z")
Then it is implementation related. ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [July 22, 2020, 3:33pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/27 "2020-07-22T15:33:52Z")
Sorry for being late to the discussion. > I saw you were using `ddp_model.load_state_dict` to load model parameters. Is this method untested and unfavored? Right, we don’t have tests for saving.loading DDP models yet, IIUC. Let me create an issue to track. > So the second node got halted in DDP constructor does have a broadcast op, I believe that’s where it is halted: > Looking at the log, some ranks proceed beyond 2.1 while others are waiting at 2.1, which suggest there is a desync across all processes. Curious, why there is no output for Location 0 at rank 0? Is it just because the print for Location 0 is actually in the if clause? For the log, can you try also print `dist.get_world_size()`, and then use `dist.get_rank()` instead of local rank? Let’s verify if the launching script did anything wrong. > I found the number of processes is 7 on each node despite the fact that I requested using 4 GPU on each node. Looks like other processes (local\_rank != 0) also created CUDA context and allocated some tensor on `cuda:0`. You can avoid this by setting `CUDA_VISIBLE_DEVICES` variable for each subprocess, either directly in command line or in the program before loading any cuda logic. See [Running on specific GPU device](https://discuss.pytorch.org/t/running-on-specific-gpu-device/89841) Note that after this change, you will also need to change all `f'cuda:{local_rank}'` to `cuda:0` as each process now only sees one device. ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [July 22, 2020, 3:42pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/28 "2020-07-22T15:42:10Z")
> [@leimao](#): > > I just got time to test the `gloo` backend. It seems that the training could be run without significant problems. hmm, this is weird. Gloo backend works means that all ranks and world sizes are configured properly. Let’s still double check using `dist.get_world_size()` and `dist.get_rank()`. If this is the case, then the broadcast in DDP might not be the place that caused the hang. Do you have access to PyTorch python files in your local env? Can you try adding some print to wrap this? > ---
**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\ **Post date:** [July 22, 2020, 3:48pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/29 "2020-07-22T15:48:43Z")
@leimao one more question regarding your test env. Would I be correct if I assume you have two 8-GPU machines, and you are using the first 4 GPUs (cuda:0-3) on those two machines, and you have exclusive access to those GPUs? ---
**Author:** ![leimao](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leimao/32/26860_2.png) [@leimao](https://discuss.pytorch.org/u/leimao)\ **Post date:** [July 23, 2020, 4:00pm UTC](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/30 "2020-07-23T16:00:21Z")
Yes. I have 8 GPUs on each node, but I just used 4 of them. I could have been using all of them. [Previous page](https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711.md?page=1)