# How to minimize the gaps between kernels on GPU?

**URL:** <https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841>\
**Category:** C++\
**Created:** [November 12, 2024, 4:57am UTC](https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841 "2024-11-12T04:57:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zhewen\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhewen_hu/32/73939_2.png) [@Zhewen\_Hu](https://discuss.pytorch.org/u/Zhewen_Hu)\
**Post date:** [November 12, 2024, 4:57am UTC](https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841/1 "2024-11-12T04:57:40Z")

</div>

I’m currently training multiple small models across several CUDA streams to achieve parallelism with Libtorch. However, each kernel execution is very short (lasting only a few microseconds) and scattered across the timeline, which means they rarely overlap or parallelize effectively. I’ve attached a screenshot showing how dispersed the kernel executions are.

I’m wondering if it is due to the overhead of launching each kernel? And if so, is there any way to minimize the gaps between these kernels to improve parallelism? Any suggestions would be appreciated!

 ![Snipaste_2024-11-11_22-54-35](https://discuss.pytorch.org/uploads/default/original/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2.png)

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 12, 2024, 11:14pm UTC](https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841/2 "2024-11-12T23:14:50Z")

</div>

If your workload suffers from slow kernel launches, you could check if you could use CUDAGraphs for your use case either [directly](https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs) or via `torch.compile`.

---

<div class="post-metadata">

**Author:** ![Zhewen\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhewen_hu/32/73939_2.png) [@Zhewen\_Hu](https://discuss.pytorch.org/u/Zhewen_Hu)\
**Post date:** [November 12, 2024, 11:31pm UTC](https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841/3 "2024-11-12T23:31:47Z")

</div>

Thanks for the help. I am using Libtorch, does it also have torch.compile? I couldn’t find the document of it for Libtorch

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 12, 2024, 11:39pm UTC](https://discuss.pytorch.org/t/how-to-minimize-the-gaps-between-kernels-on-gpu/212841/4 "2024-11-12T23:39:06Z")

</div>

No, I don’t think `torch.compile` or any of its stack is exposed as a pure C++ API. You could try to `torch.export` your model as described [here](https://pytorch.org/docs/stable/export.html).