# How multi-threading is managed within ddp?

**URL:** <https://discuss.pytorch.org/t/how-multi-threading-is-managed-within-ddp/130896>\
**Category:** distributed\
**Created:** [September 2, 2021, 12:02am UTC](https://discuss.pytorch.org/t/how-multi-threading-is-managed-within-ddp/130896 "2021-09-02T00:02:15Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![sbelharbi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/f475e1/32.png) [@sbelharbi](https://discuss.pytorch.org/u/sbelharbi)\
**Post date:** [September 7, 2021, 11:38pm UTC](https://discuss.pytorch.org/t/how-multi-threading-is-managed-within-ddp/130896/4 "2021-09-07T23:38:55Z")

</div>

@ptrblck any idea why this is happening?  
this is the reason i was looking for a cuda extension in the [other](https://discuss.pytorch.org/t/c-cuda-custom-function-runtimeerror-cuda-error-invalid-device-function/131085) thread.

this is a cpu c++ code, and has nothing to do with the other thread.

basically, the function `compute_on_cpu()` calls the c++ function:

> <https://github.com/meng-tang/rloss/blob/1caa759e568db2c7209ab73e73ac039ea3d7101c/pytorch/wrapper/bilateralfilter/bilateralfilter.cpp#L42>

i call this function in pytorch after wrapping it in python using swig.  
this c++ function creates a parallel region. i assume that each thread will deal with a sample in the minibatch in the lop.

this is actually fast (70ms).  
when using ddp+2gpus, the multi-threadins does not seem to work (call 500ms == time when using only 1 thread).

the maximum threads on that machine is 48.  
batch size 32.

any idea why?

thank you very much for your help!

---

_[View the full topic](https://discuss.pytorch.org/t/how-multi-threading-is-managed-within-ddp/130896)._