# How to know if an operation is blocking/synchronizing or not?

**URL:** <https://discuss.pytorch.org/t/how-to-know-if-an-operation-is-blocking-synchronizing-or-not/182685>\
**Category:** Uncategorized\
**Created:** [June 22, 2023, 1:42pm UTC](https://discuss.pytorch.org/t/how-to-know-if-an-operation-is-blocking-synchronizing-or-not/182685 "2023-06-22T13:42:06Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![skandermoalla](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/skandermoalla/32/61542_2.png) [@skandermoalla](https://discuss.pytorch.org/u/skandermoalla)\
**Post date:** [June 22, 2023, 1:42pm UTC](https://discuss.pytorch.org/t/how-to-know-if-an-operation-is-blocking-synchronizing-or-not/182685/1 "2023-06-22T13:42:06Z")

</div>

Where do we find documentation on whether an operation such as `cuda_tensor.item()`, `cuda_tensor.numel()`, `cuda_tensor.to("cpu")`, is blocking or not? ([CUDA semantics — PyTorch 2.0 documentation](https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution)).

Thanks!

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 22, 2023, 3:12pm UTC](https://discuss.pytorch.org/t/how-to-know-if-an-operation-is-blocking-synchronizing-or-not/182685/2 "2023-06-22T15:12:39Z")

</div>

You can set `torch.cuda.set_sync_debug_mode(mode)` to “warn” or “error” and execute the desired line of code.

---

<div class="post-metadata">

**Author:** ![skandermoalla](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/skandermoalla/32/61542_2.png) [@skandermoalla](https://discuss.pytorch.org/u/skandermoalla)\
**Post date:** [June 22, 2023, 3:37pm UTC](https://discuss.pytorch.org/t/how-to-know-if-an-operation-is-blocking-synchronizing-or-not/182685/3 "2023-06-22T15:37:48Z")

</div>

Awesome!!

```python
Python 3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.set_sync_debug_mode("warn")
/opt/conda/envs/implicit-pg/lib/python3.10/site-packages/torch/cuda/ __init__.py:762: UserWarning: Synchronization debug mode is a prototype feature and does not yet detect all synchronizing operations (Triggered internally at /opt/conda/conda-bld/pytorch_1682343967769/work/torch/csrc/cuda/Module.cpp:830.)
  torch._C._cuda_set_sync_debug_mode(debug_mode)
>>> t = torch.arange(10, device="cuda")
>>> z = t**2
>>> z.numel()
10
>>> x = z.float().mean()
>>> m = x.item()
<stdin>:1: UserWarning: called a synchronizing CUDA operation (Triggered internally at /opt/conda/conda-bld/pytorch_1682343967769/work/c10/cuda/CUDAFunctions.cpp:148.)
>>> 

```