64 s
Local easy-suite time vs 122 s cloud
40
Checks every judge must pass first
$0
Per token on my own hardware
🕵️I caught my own judge lying
My benchmarks are filesystem-judged: the model does real work in a sandbox — create the file, fix the failing test, rename a function across files — and a script checks the artifacts afterward. The model’s opinion of its own work is never consulted. I thought that made the scores bulletproof.
Then I built a validation gate: before any judge is allowed to score a model, it has to pass a known-good reference solution and fail a plausible known-bad one. Twenty tasks, forty checks. Nineteen judges passed. One didn’t.
The hard-suite cache task (build an LRU cache with per-item TTL) had a probe that asserted an eviction order its own spec forbids. I wrote a spec-perfect solution by hand and the judge failed it. Which means both models in my published video had been scored down by a bug in my test. The video said 6/8 and 6/8. The truth, under the repaired judge: 7/8 for Qwen3-Coder and a perfect 8/8 for Qwen3.6. My broken test made both models look worse — that’s the lucky direction of this failure, and it still wasn’t good enough to leave up.
Judges are code. Code has bugs. If a benchmark has never tested its own judges, it’s a random number generator with a nice table.
🏟️So I built the league
Agent-12 is a public leaderboard for the question r/LocalLLaMA asks every single week: which model can actually be my agent on my own hardware? Not trivia scores. Twenty sandboxed agent tasks — 12 everyday chores, 8 hard reasoning problems — judged only by what lands on disk. Temperature 0, fixed step caps, fresh sandbox and fresh conversation per task, hash-pinned test files so a model that “fixes” a failing test by editing the test scores zero.
And the part I now consider non-negotiable: every judge is validated against known-good and known-bad reference solutions before it scores anything, and every model runs the full gauntlet three to four times. A lucky run counts for nothing. The repo — tasks, judges, runner, validation gate, methodology — is open, and any agent CLI can plug into the same tasks through a generic adapter.
🥊Day one: five local fighters, one cloud yardstick
All local rows ran on one Apple M5 with 128 GB unified memory, through my lean ~550-token terminal engine.
| Model | Easy (12) | Hard (8) | tok/s | Notes |
|---|---|---|---|---|
| Qwen3.6-35B 👑 | 12/12 | 8/8 | 46 | Perfect in every sweep. Champion. |
| Qwen3-Coder-30B | 12/12 | 7/8 | 84 | Fastest. Fails the same parser task every run. |
| Gemma 4 31B | 11–12/12 | 7–8/8 | 26 | Scores wander between runs. |
| DeepSeek V4 Flash (2-bit, 0731) | 12/12 | 8/8 | 8 | 284B MoE. Perfect, but patient work. |
| DeepSeek V4 Flash (original) | Beaten by the 0731 update in every category. Deleted — 81 GB freed. The league has cuts. |
The repeat runs earn their keep. Qwen3-Coder failing the same task three sweeps in a row is a reproducible fingerprint you’d never see in a single-run benchmark. Gemma’s misses moving around between runs is a stability finding in its own right.
😮The row that surprised me
I ran Claude — the frontier cloud model — through the same engine, same tasks, same judges, as a clearly-labeled reference point. It went a perfect 20/20. So did my local champion. And on wall-clock, the local model won: 64s vs 122s on the easy suite, 125s vs 131s on the hard suite (cloud times include network round-trips — which is the honest end-to-end experience of using it).
That’s not a claim, it’s a row on a public board with the runner that produced it.
⚙️The harness is half the story
One more result, because it’s the reason this board exists. The same Qwen3.6 that goes 20/20 in my lean terminal engine spent an hour in a 32,000-token thinking spiral inside a heavyweight agent harness (7,800-token system prompt, 66 tools) and produced zero files. Same weights. Same machine. The harness was the only variable.
The ecosystem is currently running the other direction — plugging local models into big cloud-grade harnesses and blaming the model when it chokes. There’s fresh academic backing for harness-model matching mattering this much (AgentMeter, arXiv 2606.21140) — but it was measured on cloud CLIs. Nobody was measuring it on hardware you can buy. Now somebody is.
🔮What happens next
Every new model that drops gets a row and a fight video. The board lives at nicedreamzapp.github.io/agent12, the code at github.com/nicedreamzapp/agent12. If you think your favorite model belongs on it, tell me — it’ll get the same twenty tasks, the same validated judges, and the same number of chances to prove it wasn’t luck.
👑A 35B model on the Mac on my desk tied the cloud on score and beat it on time, at zero dollars per token.
Originally published at Nice Dreamz Wholesale. Run AI locally on your own hardware with claude-code-local, open source and no cloud required. More at nicedreamzwholesale.com/software.
Top comments (0)