[RFC-0036] Zero-GC 64-Byte Cache-Aligned Flat Arena for Speculative Decoding Verification

C++
1

Hi PyTorch community,

We recently submitted PyTorch Core RFC-0036 proposing an optional zero-runtime-allocation, 64-byte cache-aligned (alignas(64)) C++20 flat arena memory pattern:
:backhand_index_pointing_right: RFC PR #110: RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena (Aventine Labs LLC) by markbgilbert · Pull Request #110 · pytorch/rfcs · GitHub

Motivation

In high-throughput LLM serving utilizing speculative decoding (and edge inference via ExecuTorch), candidate token verification frequently encounters subtle host-side CPU overhead. When verification loops traverse pointer-chasing objects, unaligned structures, or trigger allocator contention, the host CPU stalls and forces the accelerator/GPU to idle waiting on verification.

Empirical Hardware Benchmark

To demonstrate the mechanical sympathy of this layout, we published a standalone reproducible benchmark running on consumer x86-64 hardware:
:backhand_index_pointing_right: Benchmark Repo: GitHub - markbgilbert/aegis-zero-gc-benchmark: Empirical 1-Billion Operation Zero-GC 64-Byte Cache-Aligned Memory Arena Benchmark (C++20 / AVX-512) · GitHub

Key empirical results:

  • 1,000,000,000 operations executed in 0.649 seconds (1.54 Billion ops/sec)
  • Sub-nanosecond latency: 0.649 ns per operation
  • True zero-allocation: 23 KB total heap delta, 0 GC/allocator pauses

We would love to gather feedback, architectural critiques, and thoughts from the compiler team (torch.compile Inductor) and ExecuTorch maintainers on integrating this pattern into PyTorch host verification loops.