# Code stuck at cuda.synchrize()

**URL:** <https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162>\
**Category:** distributed\
**Created:** [February 14, 2022, 10:27pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162 "2022-02-14T22:27:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![amsword](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amsword/32/8461_2.png) [@amsword](https://discuss.pytorch.org/u/amsword)\
**Post date:** [February 14, 2022, 10:27pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162/1 "2022-02-14T22:27:39Z")

</div>

The code hits hanging issue and the following stack shows it is stuck at cuda.synchronize(). Is there any idea of what is going on?

pytorch = 1.9.0, cuda=11.1

```auto
(gdb) py-bt
Traceback (most recent call first):
  <built-in method _cuda_synchronize of module object at remote 0x7f0b8bc0d360>
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/ __init__.py", line 446, in synchronize
    return torch._C._cuda_synchronize()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/utils/timer.py", line 165, in start
    torch.cuda.synchronize()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2372, in forward
    load_optimizer_states=True,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1563, in _call_impl
    >>> net = nn.Sequential(l, l)
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1531, in inner_loop_step
    loss_dict = self.model_engine(self.dict_data)
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1672, in inner_loop
    losses /= self.gradient_accumulate
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1758, in do
    self.optimizer.zero_grad()
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 1743, in do_train
    port=self.cfg.dist_url_tcp_port,
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 4443, in train
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 1540, in ensure_train
    arguments={'iteration': start_iter},
  File "/tmp/code/act/src/qd/pipeline.py", line 679, in pipeline_train_eval_multi
    pip.ensure_train()
  File "src/qd/qd_common.py", line 3349, in execute_func
  File "src/qd/qd_common.py", line 4338, in <module>
(gdb) 

```

---

<div class="post-metadata">

**Author:** ![mrshenli](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/32/12220_2.png) [@mrshenli](https://discuss.pytorch.org/u/mrshenli)\
**Post date:** [February 15, 2022, 4:46pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162/2 "2022-02-15T16:46:14Z")

</div>

Looks like there is a deadlock in the code. There could be multiple reasons:

1. Running concurrent NCCL communications. NCCL only allows one CUDA communicator to access the device a a time. If, say pipeline and ZeRO in deepspeed launches concurrent comms, it could hang.
2. User-code comm deadlock. Suppose there are two processes, X and Y. X calls commA → `cuda.synchronize()` → commB, while Y calls commB → `cuda.synchronize()` → commA. It could deadlock, as both process requires the subsequent comm op after `cuda.synchronize()` to unblock.

If you didn’t launch comm ops in your user code, could you please create an issue in deepspeed repo and discuss with deepspeed experts?

---

<div class="post-metadata">

**Author:** ![aljipa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aljipa/32/47227_2.png) [@aljipa](https://discuss.pytorch.org/u/aljipa)\
**Post date:** [December 12, 2022, 5:02pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162/3 "2022-12-12T17:02:40Z")

</div>

Is there a built-in mechanism to help debug these communication deadlocks? E.g. with thread dumps we can see which thread is waiting/holding individual locks. Or is adding logging the only option?

---

<div class="post-metadata">

**Author:** ![agu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/agu/32/49913_2.png) [@agu](https://discuss.pytorch.org/u/agu)\
**Post date:** [December 13, 2022, 9:44pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162/4 "2022-12-13T21:44:32Z")

</div>

Maybe setting `TORCH_DISTRIBUTED_DEBUG=DETAIL` can help?  
[https://pytorch.org/docs/master/distributed.html#torch-distributed-debug](https://pytorch.org/docs/master/distributed.html#torch-distributed-debug)

---

<div class="post-metadata">

**Author:** ![aljipa](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aljipa/32/47227_2.png) [@aljipa](https://discuss.pytorch.org/u/aljipa)\
**Post date:** [February 15, 2024, 3:07pm UTC](https://discuss.pytorch.org/t/code-stuck-at-cuda-synchrize/144162/5 "2024-02-15T15:07:40Z")

</div>

Thanks Andrew, I didn’t know this existed.