# How to use "break" in DistributedDataParallel training

**URL:** <https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296>\
**Category:** distributed-rpc\
**Created:** [July 7, 2020, 10:13pm UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296 "2020-07-07T22:13:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jetcai1900](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a6a055/32.png) [@jetcai1900](https://discuss.pytorch.org/u/jetcai1900)\
**Post date:** [July 7, 2020, 10:13pm UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/1 "2020-07-07T22:13:59Z")

</div>

I am using DistributedDataParallel to train the model on multiple GPUs. If I would like to stop the process early, how could I achieve it? Thanks.

---

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [July 7, 2020, 10:30pm UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/2 "2020-07-07T22:30:34Z")

</div>

Is this about uneven inputs on different processes? See:

1. [https://github.com/pytorch/pytorch/issues/33148](https://github.com/pytorch/pytorch/issues/33148)
2. [https://github.com/pytorch/pytorch/issues/38174](https://github.com/pytorch/pytorch/issues/38174)

If all processes know when to exit, simply break the loop would work. The tricky case is when one processes breaks the loop but other processes proceed as mentioned in the above two issues.

---

<div class="post-metadata">

**Author:** ![jetcai1900](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a6a055/32.png) [@jetcai1900](https://discuss.pytorch.org/u/jetcai1900)\
**Post date:** [July 7, 2020, 11:36pm UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/3 "2020-07-07T23:36:22Z")

</div>

> [@mrshenli](#):
>
> If all processes know when to exit, simply break the loop would work. The tricky case is when one processes breaks the loop but other processes proceed as mentioned in the above two issues.

Indeed this is what I meet. one process breaks the loop while others continue. The condition when the process breaks is the loss in eval dataset increases (overfitting). Do you have any ideas? Thanks.

---

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [July 8, 2020, 1:25am UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/4 "2020-07-08T01:25:01Z")

</div>

Ideally, we should address this in DDP and close [https://github.com/pytorch/pytorch/issues/38174](https://github.com/pytorch/pytorch/issues/38174). Before that takes place, you can use `all_reduce` synchronize some signal across all processes. See [Multiprocessing - Barrier Blocks all Processes?](https://discuss.pytorch.org/t/multiprocessing-barrier-blocks-all-processes/80345/10)

One thing to note is that, this might have perf impacts, especially when the model is light and its forward pass runs faster than communicating the signal.

---

<div class="post-metadata">

**Author:** ![jetcai1900](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/a6a055/32.png) [@jetcai1900](https://discuss.pytorch.org/u/jetcai1900)\
**Post date:** [July 8, 2020, 3:13am UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/5 "2020-07-08T03:13:42Z")

</div>

Thanks for your help. Probably I would set a fixed epoch number to address this, which is simple thought is not optimal.

---

<div class="post-metadata">

**Author:** ![Rakshith\_V](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rakshith_v/32/46078_2.png) [@Rakshith\_V](https://discuss.pytorch.org/u/Rakshith_V)\
**Post date:** [June 2, 2022, 4:52am UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/7 "2022-06-02T04:52:57Z")

</div>

flag\_tensor = torch.zeros(1).to(device)  
if local\_rank = 0:  
----------#Conditions for breaking the loop :  
-------------------flag\_tensor += 1  
dist.all\_reduce(flag\_tensor,op=ReduceOp.SUM)  
if flag\_tensor == 1:  
----------print(“Training stopped”)  
----------break

---

<div class="post-metadata">

**Author:** ![gianlucarloni](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gianlucarloni/32/60985_2.png) [@gianlucarloni](https://discuss.pytorch.org/u/gianlucarloni)\
**Post date:** [June 2, 2023, 9:15am UTC](https://discuss.pytorch.org/t/how-to-use-break-in-distributeddataparallel-training/88296/8 "2023-06-02T09:15:59Z")

</div>

Thank you for your simple yet elegant solution. It worked. I wanted my training script to end when meeting an early stopping criterion on the validation loss, and now all the processes are correctly ended.