# Distributed training failed without errors

**URL:** <https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229>\
**Category:** distributed\
**Created:** [September 14, 2020, 12:55pm UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229 "2020-09-14T12:55:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Feywell](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feywell/32/7936_2.png) [@Feywell](https://discuss.pytorch.org/u/Feywell)\
**Post date:** [September 14, 2020, 12:55pm UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229/1 "2020-09-14T12:55:00Z")

</div>

I just change

> [GitHub - ZPdesu/SEAN: SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020, Oral)](https://github.com/ZPdesu/SEAN) to distributed training.

And I get a weird error.  
It is no error，when I use one gpu.  
But, It will stop without printing errors when I use multi gpus.  
It will stop after Every 13 epoches by two gpus.

What could be the cause of such a problem？

my environments:  
ubuntu: 16.04  
gpu：nvidia-2080ti  
cuda: 10.1 / 10.2  
pytorch: 1.6.0 / 1.7.0  
nccl: 2.4.8 / 2.7.6  
python:3.6 / 3.7

---

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [September 14, 2020, 6:54pm UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229/2 "2020-09-14T18:54:54Z")

</div>

Hey @Feywell

By “change [https://github.com/ZPdesu/SEAN](https://github.com/ZPdesu/SEAN) distributed training”, which distributed training API are you referring to (e.g., DistributedDataParallel, c10d, RPC)?

Could you please share the code that uses distributed APIs?

---

<div class="post-metadata">

**Author:** ![Feywell](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feywell/32/7936_2.png) [@Feywell](https://discuss.pytorch.org/u/Feywell)\
**Post date:** [September 15, 2020, 2:14am UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229/3 "2020-09-15T02:14:25Z")

</div>

I just use DistributedDataParallel like this:

```auto
    if opt.distributed:
        cudnn.benchmark = True
        opt.device = "cuda"

        torch.cuda.set_device(opt.local_rank)
        torch.distributed.init_process_group(backend="nccl",
                                             init_method="env://") 
 
        synchronize()

```

And model :

```auto
         if opt.distributed:
            self.pix2pix_model = torch.nn.parallel.DistributedDataParallel(self.pix2pix_model,
                                                              device_ids=[opt.local_rank],
                                                              output_device=opt.local_rank,
                                                              find_unused_parameters=True)
            self.pix2pix_model_on_one_gpu = self.pix2pix_model.module

```

---

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [September 15, 2020, 3:02am UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229/4 "2020-09-15T03:02:30Z")

</div>

The initialization looks correct to me.

> `self.pix2pix_model_on_one_gpu = self.pix2pix_model.module`

Question: why retrieving the local model from DDP model?

> It will stop after Every 13 epoches by two gpus.

You mean the program crashes without any error message? How did you launch the two DDP processes?

---

<div class="post-metadata">

**Author:** ![Feywell](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/feywell/32/7936_2.png) [@Feywell](https://discuss.pytorch.org/u/Feywell)\
**Post date:** [September 15, 2020, 4:23am UTC](https://discuss.pytorch.org/t/distributed-training-failed-without-errors/96229/5 "2020-09-15T04:23:37Z")

</div>

This line just be used to save model:

> <https://github.com/ZPdesu/SEAN/blob/04c7536ff3fecd2d1a09c9ae046a1144636033a5/trainers/pix2pix_trainer.py#L23>

The program will crash in differenct epoches by different number’s gpus without error message.  
But It is ok in one gpu.  
I use pytorch launch function:  
`python -m torch.distributed.launch --nproc_per_node=$NGPUS train.py`