Copying byte‑exact key‑value entries into a frozen language model delivers a dramatic performance boost without any weight updates. The trick is simple: stash a verified KV state once, then graft it into fresh inference contexts, and the model reproduces the original logits perfectly while consuming far fewer tokens.
Before KV‑cache grafting, the frozen Gemma‑4‑12B already sat at a respectable 80 % on AIME 2025. “A frozen Gemma-4-12B, using its own best inference-time configuration … reaches 80.0% (24 of 30), above its 77.5% model-card anchor” [1].
Grafting the cached library lifts that same model to 93.3 %, a 13.3‑point absolute gain that outstrips even its 31B sibling’s published 89.2 % anchor. “Frozen Gemma-4-12B alone: 80.0%. Frozen Gemma-4-12B plus the grafted library: 93.3%.” [1]
The gain translates into massive token savings: eight problems that the base model never solved within a 401,026‑token budget are answered from cached solutions in just 61 decode tokens, a 6,574‑fold reduction and roughly 8,700× less energy. “answered … in 61 total decode tokens, a factor of 6,574 fewer tokens and about 8,700x less energy” [1].
The result hinges on a pinned deterministic configuration and a proprietary engine that guarantees byte‑exactness; the paper reports that under a pinned deterministic configuration the grafted logits have zero KL divergence; deviations from this setup are not evaluated in the study, and scaling the approach to arbitrary architectures remains untested. The paper notes that the library must be curated for each target domain; whether this approach can replace systematic fine‑tuning for broader coverage remains an open question.
If KV‑cache grafting holds across tasks, every lightweight deployment should treat a verified KV store as an upgrade path rather than an optional add‑on. Re‑running AIME or similar benchmarks with and without the graft will quickly reveal whether the technique supersedes traditional retrieval augmentation for frozen models.
Top comments (0)