Some days ago I published the first round: one harness, seven GGUFs under llama.cpp plus one model under the Strata engine, four dimensions, the receipts of every call. Since then the machine took three weeks of llama.cpp commits (0.6.0-dev 4413a19 to 0.9.0-dev b1c902f) and a Strata engine update (0.1.44 to 0.1.57). Same fleet, same harness, same pinned sampling — temperature 0.2, top-p 0.95, top-k 40, min-p 0.05, seed 42, KV cache q8_0, --jinja, one slot. So I re-measured everything, expecting the ranking to survive. It did not.
The fleet, then and now
| model | think | tools | code | agent | speed | global | was |
|---|---|---|---|---|---|---|---|
| minicpm-v-4.7-35b-a3b q4_k_m | on | 100% | 84% (10 excl.) | 92% | 23ms | 0.87 | 0.85 |
| qwen36-35b-distill-mxfp4_moe | off | 100% | 71% | 93% | 16ms | 0.87 | 0.84 |
| minicpm-v-4.7-35b-a3b q4_k_m | off | 100% | 80% (2 excl.) | 90% | 21ms | 0.86 | 0.85 |
| qwen3.8-27b-ud-q3_k_xl | off | 100% | 81% | 94% | 36ms | 0.83 | 0.82 |
| ornith-1.5-35b-a3b-apex-mtp | off | 100% | 74% | 91% | 26ms | 0.82 | 0.82 |
| qwen3.8-27b-ud-q3_k_xl | on | 100% | 90% (3 excl.) | 96% | 686ms | 0.81 | 0.80 |
| qwen3.8-27b-heretic-q4_k_m | on | 100% | 97% (7 excl.) | 97% | 1053ms | 0.84 | 0.80 |
| gemma-4-26b-a4b-it-ud-q5_k_xl | on | 94% | 79% | 89% | 690ms | 0.75 | 0.74 |
| ornith-1.5-35b-a3b-apex-mtp | on | 94% | 87% (1 excl.) | 96% | 412ms | 0.79 | 0.77 |
| strata-iq3_xxs (Strata 0.1.57) | on | 94% | 87% (4 excl.) | 94% | 398ms | 0.78 | 0.70 |
| qwen36-35b-distill-mxfp4_moe | on | 94% | 74% (4 excl.) | 89% | 244ms | 0.73 | 0.72 |
| gemma-4-26b-a4b-it-ud-q5_k_xl | off | 89% | 83% | 89% | 32ms | 0.80 | 0.80 |
| strata-iq3_xxs (Strata 0.1.57) | off | 89% | 83% | 90% | 36ms | 0.80 | 0.80 |
| qwen3.8-27b-heretic-q4_k_m | off | 100% | 81% | 93% | 66ms | 0.80 | 0.81 |
ON rows ran with generation caps 1024/8192 and OFF rows with the fleet defaults 128/512; the gap between those two is the first finding. Single-run pass, seed 42: every number here is one measurement where October's were the mean of three, and the ± columns return with the 3-seed pass. The turbofcfusion-neo-coder row is record only — its quant is no longer on disk.
The mistake worth your time: caps
The first thing I re-measured collapsed. Agent scores fell to 58% and 64% where October said 96% and 97%, with the models answering empty in turn one. The receipts said why: the thinking chain burned the entire 512-token turn budget before any tool call, and the episode carried on with a hole in it. The October fleet ran ON rows at 1024/8192; this pass first ran them at the harness defaults of 128/512.
Nothing about the models had degraded; the harness conditions had. Re-run at the fair caps, the collapse inverted — UD-Q3 back to 96% agent, Heretic to 97% — and ON went from a tax to a win almost everywhere. If you benchmark agentic models, take this away: a thinking model measured with direct-answer caps measures the harness, not the model. The failure mode (empty replies, one turn, no error) looks exactly like a model regression unless the receipts record it. Mine did; that is the only reason this paragraph exists.
What moved with the engine
Four things stood out.
Thinking stopped being only a hard-code tool
In October, ON bought quality only on the hard tier. This time ON improved six of seven models. Heretic-on posts the fleet's best code and agent, 97% on both, and MiniCPM-on ties the top of the board at 0.87 while keeping its 23ms turns — still the only model whose thinking is free.
Strata gained the most
0.70 to 0.78 on engine 0.1.57 with the fair caps; tools went 78 to 94% and agent 77 to 94%. The OFF row stayed at 0.80, so the engine work landed specifically in thinking mode.
OFF stayed put except tools
llama.cpp 0.9 kept every OFF global within ±0.03 of October, and most rows moved less than the run-to-run spread. The movement sat in tool calling: qwen36 went 94 to 100% with turns at 16ms, while Heretic trades five points of code for a 19ms faster turn.
The top is now a tie between modes
October's ties were between OFF rows. This time MiniCPM-on 0.87 equals qwen36-off 0.87, so the pick is by use: qwen36-off for 16ms turns, MiniCPM-on for 84% code with receipts.
How much context does a model burn to pass?
The question a reader left on the first round: the agent loop re-sends the whole conversation every turn, so two models with the same pass rate can cost very differently in sent tokens. Until now the harness counted what the model emitted (out_tokens) and not what it read. This round every row carries context receipts — in_tokens per call for tools and code, tokens per agent session summed over the episode's turns, averaged and median across the fleet's 72 sessions:
| model | think | tools in/call | code in/call | agent session avg | median |
|---|---|---|---|---|---|
| qwen36-35b-distill-mxfp4_moe | off | 305 / 294 | 187 / 115 | 4,839 | 3,789 |
| minicpm-v-4.7-35b-a3b q4_k_m | off | 305 / 294 | 187 / 115 | 4,502 | 3,643 |
| qwen3.8-27b-ud-q3_k_xl | off | 305 / 294 | 187 / 115 | 5,150 | 3,913 |
| ornith-1.5-35b-a3b-apex-mtp | off | 305 / 294 | 187 / 115 | 5,155 | 4,120 |
| qwen3.8-27b-heretic-q4_k_m | off | 305 / 294 | 187 / 115 | 5,404 | 3,817 |
| gemma-4-26b-a4b-it-ud-q5_k_xl | off | 104 / 86 | 190 / 116 | 4,412 | 3,048 |
| strata-iq3_xxs (Strata 0.1.57) | off | 289 / 281 | 187 / 115 | 4,663 | 3,512 |
| qwen36-35b-distill-mxfp4_moe | on | 303 / 292 | 185 / 113 | 6,086 | 4,200 |
| minicpm-v-4.7-35b-a3b q4_k_m | on | 303 / 292 | 185 / 113 | 6,186 | 4,465 |
| qwen3.8-27b-ud-q3_k_xl | on | 341 / 330 | 223 / 151 | 9,057 | 6,130 |
| ornith-1.5-35b-a3b-apex-mtp | on | 303 / 292 | 185 / 113 | 10,758 | 6,353 |
| strata-iq3_xxs (Strata 0.1.57) | on | 326 / 317 | 223 / 151 | 10,101 | 6,058 |
| qwen3.8-27b-heretic-q4_k_m | on | 341 / 330 | 223 / 151 | 8,500 | 4,710 |
| gemma-4-26b-a4b-it-ud-q5_k_xl | on | 102 / 84 | 188 / 114 | 11,060 | 7,740 |
Three things the receipts say.
Per-call burn is flat
Roughly 290-340 tokens in per tool call and 185-220 per code call everywhere, because those prompts are the same tasks. All the variance lives in the agent loop, where the conversation grows.
Thinking is a context tax too
Same model, same tasks, ON against OFF: +26% session tokens on qwen36, +57% on Heretic, +76% on UD-Q3, +117% on Strata, +151% on gemma. The latency table already showed one cost of thinking; the receipts show the second.
Quality per burned token is its own ranking
qwen36-off delivers the joint-top 0.87 at 4.8K tokens per session, and MiniCPM-on the same 0.87 at 6.2K — the cheap thinker. At the other end, gemma-on burns the most context in the fleet (11.1K average, 7.7K median) for 0.75, and Strata-on pays 10.1K for 0.78. The engine work bought its points expensively.
What I would tell you to do
Same as last time, with one amendment. Measure on your own hardware, read the exclusions, treat close rows as ties, and when a score moves, check the caps before you believe the model. An engine upgrade does not change what a model knows; it changes how the plumbing counts. The fleet still has no single winner: tools and cheap turns belong to qwen36-off, code and agent now belong to Heretic-on, speed-agnostic quality to MiniCPM, and long-form dispatch to Strata, which just closed most of the gap its first round left open.
Scoreboard and live report: jordimarsal.github.io/ggufbench. Code, tasks and receipts: github.com/jordimarsal/ggufbench. The first round, for context: which local model for coding agents.
Top comments (0)