# Stucks on 8gpu training setting

**URL:** <https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788>\
**Category:** distributed\
**Created:** [January 14, 2021, 2:40am UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788 "2021-01-14T02:40:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![pizza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pizza/32/33380_2.png) [@pizza](https://discuss.pytorch.org/u/pizza)\
**Post date:** [January 14, 2021, 2:40am UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/1 "2021-01-14T02:40:21Z")

</div>

Hello,

I am using Pytorch (Pytorch Lightning Framework) to train the Text to Text Transformer model ([google/mt5-base at main](https://huggingface.co/google/mt5-base/tree/main)).

I trained them on 1, 4, 5, 8 gpu environment using DDP.  
However, all of 8gpu and 5gpu training attempts, are stuck and failed at a specific point in a specific epoch (54).

This is the last log before stuck, as it seems, its end of an epoch, so I assume that training is stuck due to data loading for next epoch in 8gpu or 5gpu environment.

This issue also occurred regardless of `num_worker` in `DataLoader` or different batch\_size (32, 16)

```auto
Epoch 54: 100%|█████████▉| 2921/2931 [43:38<00:08, 1.12it/s, loss=.., v_num=0]
Epoch 54: 100%|█████████▉| 2925/2931 [43:41<00:05, 1.12it/s, loss=.., v_num=0]
Validating: 99%|█████████▉| 280/282 [02:32<00:01, 1.59it/s]e[A
Epoch 54: 100%|██████████| 2931/2931 [44:01<00:00, 1.11it/s, loss=.., v_num=0]

```

Any comment or suggestion would be appreciated.

Thank you.

---

<div class="post-metadata">

**Author:** ![pritamdamania87](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/32/28891_2.png) [@pritamdamania87](https://discuss.pytorch.org/u/pritamdamania87)\
**Post date:** [January 15, 2021, 3:13am UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/2 "2021-01-15T03:13:34Z")

</div>

Do you have even number of batches across all GPUs? If not, the training could get stuck since DDP performs a collective sync across all GPUs in the backward pass. If you have uneven batches, you can try using the [join](https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.join) API to get around this.

---

<div class="post-metadata">

**Author:** ![pizza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pizza/32/33380_2.png) [@pizza](https://discuss.pytorch.org/u/pizza)\
**Post date:** [January 15, 2021, 2:07pm UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/3 "2021-01-15T14:07:55Z")

</div>

@pritamdamania87,

First of all, thank you for your reply.

- If I understand your comment correctly, It seems I both had training with even and uneven batch size across all GPUs.
- The log shows 2931 which is the uneven number of batches assigned for a single GPU. however, in another training session, a even number of batches (like 1466) have also experienced the same issue.
- Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?
- I will check to join API try to get around this issue of course.

Thank you.!

---

<div class="post-metadata">

**Author:** ![pritamdamania87](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/32/28891_2.png) [@pritamdamania87](https://discuss.pytorch.org/u/pritamdamania87)\
**Post date:** [January 21, 2021, 3:24am UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/4 "2021-01-21T03:24:38Z")

</div>

> - Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?

You would probably have to add some logging code to your application which records the number of batches on each rank as part of the training loop.

---

<div class="post-metadata">

**Author:** ![pizza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pizza/32/33380_2.png) [@pizza](https://discuss.pytorch.org/u/pizza)\
**Post date:** [February 2, 2021, 2:37pm UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/5 "2021-02-02T14:37:07Z")

</div>

@pritamdamania87

Thank you for your comment 🙂  
I am now still in debug, and experienced the same error on DataParallel setting (without hang; just training loop stops at the end of 54 epoch) 😢

Is this good evidence for the issue?

---

<div class="post-metadata">

**Author:** ![pizza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pizza/32/33380_2.png) [@pizza](https://discuss.pytorch.org/u/pizza)\
**Post date:** [February 25, 2021, 3:32pm UTC](https://discuss.pytorch.org/t/stucks-on-8gpu-training-setting/108788/6 "2021-02-25T15:32:17Z")

</div>

This issue is caused by my code mistake.  
I accidentally give a small (54) to max epoch value, after adjust the max epoch value, works fine on DP or CPU environment.  
DDP also hangs after the max epoch which seems is interpretable behavior, but not normal behavior. (currently I don’t know why.)