# Reduce time to first kernel when using CUDA graphs

**URL:** <https://discuss.pytorch.org/t/reduce-time-to-first-kernel-when-using-cuda-graphs/214310>\
**Category:** Uncategorized\
**Created:** [December 17, 2024, 1:19pm UTC](https://discuss.pytorch.org/t/reduce-time-to-first-kernel-when-using-cuda-graphs/214310 "2024-12-17T13:19:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![SzymonOzog](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/szymonozog/32/74552_2.png) [@SzymonOzog](https://discuss.pytorch.org/u/SzymonOzog)\
**Post date:** [December 17, 2024, 1:19pm UTC](https://discuss.pytorch.org/t/reduce-time-to-first-kernel-when-using-cuda-graphs/214310/1 "2024-12-17T13:19:28Z")

</div>

I’ve been profiling the inference stack that I’m using against [vLLM](https://github.com/vllm-project/vllm) and I found that in their case after calling graph replay, the first kernel gets executed almost instantly(left), whereas in my code(right) it is towards the end of `graph.replay`:

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/0/8/08d201b851ec9db2bfa411e7306bea547f7c42aa.jpeg)

Does anyone have any insight on what might cause this big delay? Graph launch takes 1-2ms so it’s a big slowdown for me

---

<div class="post-metadata">

**Author:** ![soulitzer](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/839c29/32.png) [@soulitzer](https://discuss.pytorch.org/u/soulitzer)\
**Post date:** [December 19, 2024, 4:23am UTC](https://discuss.pytorch.org/t/reduce-time-to-first-kernel-when-using-cuda-graphs/214310/2 "2024-12-19T04:23:11Z")

</div>

The delay should just be the cuda graph kernel launch, which seems expected, e.g. the kernel executes after it has launched. One way you can get overlap (like what you see in vLLM) is to launch a different kernel before.