This morning I noticed my local coding agent answering way faster than it used to, and I couldn't explain why. I don't like speedups I can't explain, so we benchmarked it instead of guessing. What came out of that is the biggest single improvement my local setup has ever gotten, and a bug report that probably applies to your setup too if you run Gemma-family models on Apple Silicon.
The setup
I maintain claude-code-local, a repo for running coding agents against local models on a Mac — no cloud, no API key. The original approach pointed Claude Code (the CLI) at a local MLX server through a proxy. It works, and it's still in the repo. But Claude Code was designed for cloud models: its system prompt is tens of thousands of tokens and parts of it change every turn. A local model pays for that twice — once prefilling a huge prompt, and again because a prompt whose head keeps changing defeats KV-cache reuse completely.
So we built the obvious alternative: a small native engine, about 900 lines of Python on mlx-lm. Fixed ~550-token system prompt, the same tools (bash, read, write, edit, glob, grep), and a KV cache that gets trimmed to the shared prefix each turn so only the new tokens are ever prefilled.
The bug
Benchmarking the engine surfaced something I didn't expect. Short conversations were fast, exactly as designed — 0.3 seconds to first token. But past a certain conversation length, every turn suddenly cost 6.5 to 7.2 seconds, as if the cache didn't exist. It wasn't gradual. It was a cliff.
The cliff turned out to be Gemma's sliding-window attention. Gemma-family models give five out of every six layers a RotatingKVCache capped at the window size — 1024 tokens on Gemma 4. The moment your transcript outgrows the window, those rotating caches report themselves as untrimmable, and mlx-lm's prompt-cache reuse silently dies. Every turn re-prefills the entire transcript. The longer your session, the worse it gets — which means the failure lands exactly where caching matters most, and there's no error, no warning, nothing. It just gets slow.
If you're building a Gemma-based agent on mlx-lm, check for this. You probably have it right now.
The fix
Give every layer a plain KVCache instead. That sounds like it should change the model's output, but it doesn't: the sliding-window attention mask is what enforces the window. The cache type only decides what gets stored. We verified this the honest way — greedy decoding, same conversation, stock caches vs plain caches, and the outputs were byte-identical on every turn.
The numbers, on a Gemma 4 31B (4-bit) with a 4,500-token conversation:
| time to first token | |
|---|---|
| stock rotating cache | 6.5–7.2 s |
| plain KV cache | 0.36 s |
That's roughly 20× less waiting per turn, on the same model and the same MacBook. The trade is that KV memory now grows with the transcript instead of capping at the window, so the engine shows a live context meter, and an env var restores stock behavior if you'd rather have the memory ceiling.
But did removing the big harness make it dumber?
Fair question — Claude Code's giant prompt exists for a reason, just not for a 31B model. We built a 12-task eval (create-and-run scripts, fix a failing test, rename across files, escape-heavy file content, precise edits, CSV work — all machine-checked by actually running the results, temperature 0). Qwen 3 Coder went 12 for 12 with the bare 550-token prompt. Gemma went 11 for 12, and its one failure was instructive: asked to write a file full of quotes and backslashes, it piped the content through shell echo, and sh's echo silently collapsed the backslashes. A five-line prompt rule — never write file contents through the shell, always use the write/edit tools — took it to 12 for 12 with no regressions.
So no. For models this size, less harness turned out to be more capability, as long as the few rules you do include are aimed at failures you actually observed.
Where it all lives
Everything shipped today in the repo — the engine, the fix, the benchmark script (bench/agent_bench.py) so you can reproduce the numbers on your own machine, and the write-up. Existing Claude Code launchers are untouched; this is a second path, not a replacement. Credit where it's due: the prompt-cache trim fix contributed in PR #46 is what made the deeper rotating-cache problem visible at all.
Local AI on a Mac keeps surprising me. The models were already good. The gap has been in the plumbing — and the plumbing bugs are small, findable, and fixable.
matt
Top comments (0)