I have been experimenting with CUDA Graph reuse for LLM inference and turned the work into a small open source Python package called GPUOpt Runtime.
It captures repeated greedy decode operations, validates the generated tokens before trusting a new graph shape, and reuses compatible graphs through a cache. If capture or validation fails, it falls back to regular eager SDPA instead of returning questionable output.
Current Qwen2.5 0.5B results for 64 generated tokens:
| GPU | Eager SDPA | CUDA Graph | Speedup | Validation |
|---|---|---|---|---|
| RTX 5070 | 60.06 tok/s | 281.18 tok/s | 4.68x | 64/64 tokens |
| A100 1g.10gb MIG | 70.55 tok/s | 130.14 tok/s | 1.84x | 64/64 tokens |
The first request still pays the capture and validation cost. The improvement is for later requests with compatible shapes.
Version 0.1.1 currently focuses on NVIDIA CUDA, Hugging Face causal models, unpadded prompts, and greedy decoding. I am looking for feedback and results from other GPUs and models.
Install:
pip install gpuopt-runtime
PyPI: gpuopt-runtime · PyPI
I would especially appreciate feedback on the graph lifecycle, KV cache handling, safety fallback, and what model or workload support would be most useful next.