# \[RFC-0036\] Zero-GC 64-Byte Cache-Aligned Flat Arena for Speculative Decoding Verification

**URL:** <https://discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/225427>\
**Category:** C++\
**Created:** [September 17, 2026, 5:56pm UTC](https://discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/225427 "2026-09-17T17:56:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Gilbert](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mark_gilbert/32/80038_2.png) [@Mark\_Gilbert](https://discuss.pytorch.org/u/Mark_Gilbert)\
**Post date:** [September 17, 2026, 5:56pm UTC](https://discuss.pytorch.org/t/rfc-0036-zero-gc-64-byte-cache-aligned-flat-arena-for-speculative-decoding-verification/225427/1 "2026-09-17T17:56:56Z")

</div>

Hi PyTorch community,

We recently submitted **PyTorch Core RFC-0036** proposing an optional zero-runtime-allocation, 64-byte cache-aligned (`alignas(64)`) C++20 flat arena memory pattern:  
👉 **RFC PR #110:** [RFC-0036: Zero-GC 64-Byte Cache-Aligned Flat Arena (Aventine Labs LLC) by markbgilbert · Pull Request #110 · pytorch/rfcs · GitHub](https://github.com/pytorch/rfcs/pull/110)

### Motivation

In high-throughput LLM serving utilizing speculative decoding (and edge inference via ExecuTorch), candidate token verification frequently encounters subtle host-side CPU overhead. When verification loops traverse pointer-chasing objects, unaligned structures, or trigger allocator contention, the host CPU stalls and forces the accelerator/GPU to idle waiting on verification.

### Empirical Hardware Benchmark

To demonstrate the mechanical sympathy of this layout, we published a standalone reproducible benchmark running on consumer x86-64 hardware:  
👉 **Benchmark Repo:** [GitHub - markbgilbert/aegis-zero-gc-benchmark: Empirical 1-Billion Operation Zero-GC 64-Byte Cache-Aligned Memory Arena Benchmark (C++20 / AVX-512) · GitHub](https://github.com/markbgilbert/aegis-zero-gc-benchmark)

Key empirical results:

- **1,000,000,000 operations executed in 0.649 seconds** (1.54 Billion ops/sec)
- **Sub-nanosecond latency:** 0.649 ns per operation
- **True zero-allocation:** 23 KB total heap delta, 0 GC/allocator pauses

We would love to gather feedback, architectural critiques, and thoughts from the compiler team (`torch.compile` Inductor) and ExecuTorch maintainers on integrating this pattern into PyTorch host verification loops.