Batch-1 LLM decoding in PyTorch + Triton, approximately at parity with llama.cpp on an RTX 4060: looking for independent reproductions

projects
1

Problem. Batch-1 decoding is bound by weight bandwidth, but Hugging Face generate() also leaves the GPU idle between kernels. On Qwen3-1.7B with my int8 kernel, the GPU idled about 2 ms per token.

Approach. A weight-only int8 Triton GEMV (a torch.library custom op), plus a sync-free decoder that replays one CUDA graph per token, streaming tokens through pinned memory. Measured steps: 91.3 tok/s with generate() → 96.8 with the sync-free CUDA-graph loop → 105.1 with the int8 kernel configuration chosen by end-to-end timing (the microbenchmark winner was slower inside the model) → 111.9 with a KV cache sized to the request.

Result (Qwen3-1.7B, batch 1, greedy, 256 new tokens, 4 fresh-process rounds, RTX 4060, Windows 11):

  • 111.9 tok/s, approximately at parity with the tested llama.cpp Q8_0 configuration (Ollama defaults): 1.02×, 95% CI [1.015, 1.051], below my 5% bar for calling it a win
  • 1.33× torchao int8 weight-only, 1.89× HF bf16 with a compiled static cache

Correctness. KernelAgent-Ada, the repo’s Triton kernel search, checks every candidate against fp64 ground truth, mutation-tests its own evaluator (12/12 deliberately wrong kernels rejected) and re-measures winners end to end. That check cut a 1.1–1.6× attention-kernel win to about 1% of the decode step.

Limitations. One GPU, Windows only, no independent reproduction yet. Batch 1 and greedy only. Slower first token than llama.cpp (54 vs 33 ms). llama.cpp ran with Ollama defaults, not a tuned build; vLLM and TensorRT-LLM were not compared.

Could you reproduce it? I’m looking for runs on Linux or on an NVIDIA GPU other than an RTX 4060 (compute capability 8.0+, 8 GB+):

  1. Set up the environment from docs/REPRODUCE_ON_ANOTHER_MACHINE.md in the repo (Python 3.11, torch 2.11 / CUDA 12.8, transformers 5.21).
  2. Run bash research/release/reproduce.sh (about an hour; --quick compares int8 vs bf16 only). It packs the logs and raw results into one archive.
  3. Post the archive in the repo’s GitHub Discussions, category “Hardware results” (the pinned post there explains how).

Privacy: before uploading, please open the archive and check the logs, env.json, pip_freeze.txt and nvidia_smi_q.txt for anything personal (user names, paths, host names). The script removes serial numbers, the list of running programs and home-directory paths, but it can’t catch everything.

Failed or contradicting results are as useful as matching ones, and criticism of the method is welcome here or on GitHub.

Repository (MIT): GitHub - BagelsFun/efficientgpt: Weight-only int8 + sync-free CUDA-graph decoder for Qwen3-1.7B in PyTorch/Triton: 111.9 tok/s on an RTX 4060, parity with llama.cpp Q8_0, 1.33x torchao int8, 1.89x HF bf16. Plus KernelAgent-Ada, a verification-first Triton kernel search. All raw evidence included. Fork of nanoGPT. · GitHub
74-second overview video: https://youtu.be/SKWrki4NHn4

Disclosure: built with an AI coding assistant. I checked every number against the raw runs, which are in the repository.

2

Batch-1 decode is weight-bandwidth bound, so a parity claim is only interpretable with the bandwidth number attached: bytes read per token (quantized weights plus KV traffic) times tokens/s, measured against the card’s peak DRAM bandwidth. If a run already sits near 85-90% of peak, no kernel change closes the remaining gap and the comparison is measuring the memory system rather than the kernels. Two controls that make decode comparisons reproducible across machines: fresh-process rounds with pinned GPU clocks, because consumer cards downclock between rounds and that drift is often larger than a 1.02x margin, and an equal-bytes check, where both sides read the same weight bytes per token, otherwise a quantization win and a runtime win get conflated. Which number will the reproduction posts report - tok/s alone, or tok/s plus achieved weight bandwidth, so a different card can tell real parity from a bandwidth coincidence?

3

Thanks for the detailed feedback. I agree that tokens/s alone isn’t enough to establish meaningful performance parity.
I’ll extend the reproduction report to include both tokens/s and estimated achieved memory bandwidth, with explicit calculations of quantized weight traffic and KV-cache traffic per generated token.
I’ll also investigate fresh-process repeatability, GPU clock variability, and equal-byte comparisons to distinguish runtime improvements from quantization differences.
I haven’t yet validated the bandwidth figures, so I won’t claim a specific percentage of peak DRAM bandwidth until the measurements support it.
Once the additional benchmarks are complete, I’ll share the methodology, raw results, and limitations so others can independently reproduce the findings.
Thanks for highlighting these important controls!