DEV Community: Jordi Marçal

Which local model should a coding agent run? Round 2: llama.cpp 0.9

Jordi Marçal — Fri, 06 Nov 2026 20:37:27 +0000

Some days ago I published the first round: one harness, seven GGUFs under llama.cpp plus one model under the Strata engine, four dimensions, the receipts of every call. Since then the machine took three weeks of llama.cpp commits (0.6.0-dev 4413a19 to 0.9.0-dev b1c902f) and a Strata engine update (0.1.44 to 0.1.57). Same fleet, same harness, same pinned sampling — temperature 0.2, top-p 0.95, top-k 40, min-p 0.05, seed 42, KV cache q8_0, --jinja, one slot. So I re-measured everything, expecting the ranking to survive. It did not.

The fleet, then and now

model	think	tools	code	agent	speed	global	was
minicpm-v-4.7-35b-a3b q4_k_m	on	100%	84% (10 excl.)	92%	23ms	0.87	0.85
qwen36-35b-distill-mxfp4_moe	off	100%	71%	93%	16ms	0.87	0.84
minicpm-v-4.7-35b-a3b q4_k_m	off	100%	80% (2 excl.)	90%	21ms	0.86	0.85
qwen3.8-27b-ud-q3_k_xl	off	100%	81%	94%	36ms	0.83	0.82
ornith-1.5-35b-a3b-apex-mtp	off	100%	74%	91%	26ms	0.82	0.82
qwen3.8-27b-ud-q3_k_xl	on	100%	90% (3 excl.)	96%	686ms	0.81	0.80
qwen3.8-27b-heretic-q4_k_m	on	100%	97% (7 excl.)	97%	1053ms	0.84	0.80
gemma-4-26b-a4b-it-ud-q5_k_xl	on	94%	79%	89%	690ms	0.75	0.74
ornith-1.5-35b-a3b-apex-mtp	on	94%	87% (1 excl.)	96%	412ms	0.79	0.77
strata-iq3_xxs (Strata 0.1.57)	on	94%	87% (4 excl.)	94%	398ms	0.78	0.70
qwen36-35b-distill-mxfp4_moe	on	94%	74% (4 excl.)	89%	244ms	0.73	0.72
gemma-4-26b-a4b-it-ud-q5_k_xl	off	89%	83%	89%	32ms	0.80	0.80
strata-iq3_xxs (Strata 0.1.57)	off	89%	83%	90%	36ms	0.80	0.80
qwen3.8-27b-heretic-q4_k_m	off	100%	81%	93%	66ms	0.80	0.81

ON rows ran with generation caps 1024/8192 and OFF rows with the fleet defaults 128/512; the gap between those two is the first finding. Single-run pass, seed 42: every number here is one measurement where October's were the mean of three, and the ± columns return with the 3-seed pass. The turbofcfusion-neo-coder row is record only — its quant is no longer on disk.

The mistake worth your time: caps

The first thing I re-measured collapsed. Agent scores fell to 58% and 64% where October said 96% and 97%, with the models answering empty in turn one. The receipts said why: the thinking chain burned the entire 512-token turn budget before any tool call, and the episode carried on with a hole in it. The October fleet ran ON rows at 1024/8192; this pass first ran them at the harness defaults of 128/512.

Nothing about the models had degraded; the harness conditions had. Re-run at the fair caps, the collapse inverted — UD-Q3 back to 96% agent, Heretic to 97% — and ON went from a tax to a win almost everywhere. If you benchmark agentic models, take this away: a thinking model measured with direct-answer caps measures the harness, not the model. The failure mode (empty replies, one turn, no error) looks exactly like a model regression unless the receipts record it. Mine did; that is the only reason this paragraph exists.

What moved with the engine

Four things stood out.

Thinking stopped being only a hard-code tool

In October, ON bought quality only on the hard tier. This time ON improved six of seven models. Heretic-on posts the fleet's best code and agent, 97% on both, and MiniCPM-on ties the top of the board at 0.87 while keeping its 23ms turns — still the only model whose thinking is free.

Strata gained the most

0.70 to 0.78 on engine 0.1.57 with the fair caps; tools went 78 to 94% and agent 77 to 94%. The OFF row stayed at 0.80, so the engine work landed specifically in thinking mode.

OFF stayed put except tools

llama.cpp 0.9 kept every OFF global within ±0.03 of October, and most rows moved less than the run-to-run spread. The movement sat in tool calling: qwen36 went 94 to 100% with turns at 16ms, while Heretic trades five points of code for a 19ms faster turn.

The top is now a tie between modes

October's ties were between OFF rows. This time MiniCPM-on 0.87 equals qwen36-off 0.87, so the pick is by use: qwen36-off for 16ms turns, MiniCPM-on for 84% code with receipts.

How much context does a model burn to pass?

The question a reader left on the first round: the agent loop re-sends the whole conversation every turn, so two models with the same pass rate can cost very differently in sent tokens. Until now the harness counted what the model emitted (out_tokens) and not what it read. This round every row carries context receipts — in_tokens per call for tools and code, tokens per agent session summed over the episode's turns, averaged and median across the fleet's 72 sessions:

model	think	tools in/call	code in/call	agent session avg	median
qwen36-35b-distill-mxfp4_moe	off	305 / 294	187 / 115	4,839	3,789
minicpm-v-4.7-35b-a3b q4_k_m	off	305 / 294	187 / 115	4,502	3,643
qwen3.8-27b-ud-q3_k_xl	off	305 / 294	187 / 115	5,150	3,913
ornith-1.5-35b-a3b-apex-mtp	off	305 / 294	187 / 115	5,155	4,120
qwen3.8-27b-heretic-q4_k_m	off	305 / 294	187 / 115	5,404	3,817
gemma-4-26b-a4b-it-ud-q5_k_xl	off	104 / 86	190 / 116	4,412	3,048
strata-iq3_xxs (Strata 0.1.57)	off	289 / 281	187 / 115	4,663	3,512
qwen36-35b-distill-mxfp4_moe	on	303 / 292	185 / 113	6,086	4,200
minicpm-v-4.7-35b-a3b q4_k_m	on	303 / 292	185 / 113	6,186	4,465
qwen3.8-27b-ud-q3_k_xl	on	341 / 330	223 / 151	9,057	6,130
ornith-1.5-35b-a3b-apex-mtp	on	303 / 292	185 / 113	10,758	6,353
strata-iq3_xxs (Strata 0.1.57)	on	326 / 317	223 / 151	10,101	6,058
qwen3.8-27b-heretic-q4_k_m	on	341 / 330	223 / 151	8,500	4,710
gemma-4-26b-a4b-it-ud-q5_k_xl	on	102 / 84	188 / 114	11,060	7,740

Three things the receipts say.

Per-call burn is flat

Roughly 290-340 tokens in per tool call and 185-220 per code call everywhere, because those prompts are the same tasks. All the variance lives in the agent loop, where the conversation grows.

Thinking is a context tax too

Same model, same tasks, ON against OFF: +26% session tokens on qwen36, +57% on Heretic, +76% on UD-Q3, +117% on Strata, +151% on gemma. The latency table already showed one cost of thinking; the receipts show the second.

Quality per burned token is its own ranking

qwen36-off delivers the joint-top 0.87 at 4.8K tokens per session, and MiniCPM-on the same 0.87 at 6.2K — the cheap thinker. At the other end, gemma-on burns the most context in the fleet (11.1K average, 7.7K median) for 0.75, and Strata-on pays 10.1K for 0.78. The engine work bought its points expensively.

What I would tell you to do

Same as last time, with one amendment. Measure on your own hardware, read the exclusions, treat close rows as ties, and when a score moves, check the caps before you believe the model. An engine upgrade does not change what a model knows; it changes how the plumbing counts. The fleet still has no single winner: tools and cheap turns belong to qwen36-off, code and agent now belong to Heretic-on, speed-agnostic quality to MiniCPM, and long-form dispatch to Strata, which just closed most of the gap its first round left open.

Scoreboard and live report: jordimarsal.github.io/ggufbench. Code, tasks and receipts: github.com/jordimarsal/ggufbench. The first round, for context: which local model for coding agents.

Which local model should a coding agent run?

Jordi Marçal — Tue, 03 Nov 2026 14:09:47 +0000

UPDATE 2026-11-06: I re-measured this fleet on llama.cpp 0.9 and Strata 0.1.57 — with fair generation caps, thinking now pays almost everywhere, Heretic-on posts the best code+agent of the fleet, and Strata-on gains +0.08. Full table and the measurement mistake behind it: follow-up

One harness, same conditions, four dimensions, and the raw receipts of every call — this round, seven GGUFs under llama.cpp plus one model under the Strata engine, each in both thinking modes.

The question

I run coding agents on my own hardware. llama.cpp on two RTX 5080s, models from my models folder, and one question: which GGUF, at which quant, should I load before I start working?

Public leaderboards measure something else. Most of their tasks are in models' training data, they score single-shot answers, and they say nothing about whether a model can call a tool with well-formed arguments, survive a multi-call loop, read a folder, edit a file, and verify its own change. That is the work an agent does all day, so I stopped reading those tables for this decision and built ggufbench.

The rules that make the numbers mean something

Every model runs with identical sampling: temperature 0.2, top-p 0.95, top-k 40, min-p 0.05, seed 42, KV cache q8_0, --jinja, one slot. Every sampler value is pinned explicitly, because a GGUF carries its own sampling metadata — this fleet alone ships three different top_k settings (20, 64, and none at all) — and an unpinned value ends up different per model. The settings are fixed in two places, the server flags and the request body, so config drift on the server cannot leak into a comparison.

Every model keeps its own profile file. That profile applies last and can override the baseline, so models compete in their best real configuration and not in a lab rig I invented. The harness fixes the infrastructure flags, and what each model ended up running with is recorded in its result row.

Exclusions are annotated with a reason. A model that runs out of time or answers with prose where a code block was expected does not silently vanish from the average; it shows up in the report, counted.

The harness uses only the Python standard library. No dependency tree to rot while the benchmark waits.

Every task stores three runs, seeds 42, 43 and 44, and the scoreboard reports mean ± standard deviation. When a row says 0.85 ±1, that spread is part of the answer.

What the scoreboard said

The current pool runs set v5: 18 tool tasks, 72 code tasks across four languages (32 easy, 20 hard, 20 expert), 72 multi-step agent tasks (13 of them multi-file projects) and a 32K output budget for code. The rows below are the current pool, task set v5, generated from the live scoreboard, so they are never a snapshot of a night that has passed.

model	think	tools	code	agent	speed	global
minicpm-v-4.7-35b-a3b q4_k_m	on	100% ±0	82% ±4 (8 excl.)	89% ±0	23ms	0.85 ±1
minicpm-v-4.7-35b-a3b q4_k_m	off ★	100% ±0	83% ±1 (2 excl.)	90% ±2	26ms	0.85 ±0
qwen36-35b-distill-mxfp4_moe	off ★	94% ±0	74% ±3	92% ±2	19ms	0.84 ±1
qwen3.8-27b-ud-q3_k_xl	off	100% ±0	79% ±4	90% ±1	36ms	0.82 ±1
ornith-1.5-35b-a3b-apex-mtp-quality	off ★	100% ±0	75% ±2	92% ±1	32ms	0.82 ±1
qwen3.8-27b-heretic-q4_k_m	off ★	100% ±0	86% ±2	92% ±2	85ms	0.81 ±1
qwen3.8-27b-neo-coder-max-mtp q4_k_s	off	100% ±0	79% ±2	91% ±1	46ms	0.81 ±0
qwen3.8-27b-ud-q3_k_xl	on	94% ±0	93% ±1 (1 excl.)	96% ±1	629ms	0.80 ±0
gemma-4-26b-a4b-it-ud-q5_k_xl	off ★	89% ±0	86% ±1	87% ±2	33ms	0.80 ±0
strata-iq3_xxs (Strata engine)	off ★	91% ±3	84% ±1	91% ±2	40ms	0.80 ±1
qwen3.8-27b-heretic-q4_k_m	on	94% ±0	90% ±2	97% ±1	1060ms	0.80 ±0

★ = each model's optimal thinking mode by data: quality within ±0.02 of the other mode and no extra wall clock; exact ties go to off.

Four things stood out.

The top rows are ties

The top ten rows span five hundredths of a point — 0.85 down to 0.80 — and neighbouring rows sit at most one or two hundredths apart, the same size as the run-to-run spread printed beside each score. The global score ranks a fleet; it does not certify that row 1 beats row 7. Anyone ranking two close rows by their second decimal is quoting noise.

Thinking mode is a tax that only pays in hard code

I measured it as separate rows instead of folding it into one number. Thinking ON multiplies turn latency — every ON row but MiniCPM's runs 280–1060 ms against 19–85 ms OFF (MiniCPM's ON lands at 23 ms) — and it costs quality in every model as well, with MiniCPM the only tie. It also concentrates the exclusions: across the fleet's 1,512 code runs per mode, OFF excludes 6 and ON excludes 101, most of them answers with no code block where one was expected (fully excluded tasks: 2 against 14).

Where it does buy something is the hard code tier: UD-Q3 climbs from 79% to 93% on code overall (92% hard, 91% expert), Ornith +10 points over its own OFF, Strata +7, Heretic +4 — and Heretic's agent loop reaches 97%, the fleet's best. On the tasks that need it, thinking is worth ten times the wait; everywhere else it is a tax with no visible return.

Easy code tasks don't discriminate

With thinking off the entire fleet lands between 91% and 100% on the easy tier, while hard and expert live between 47% and 87%. Those two levels are where the separation is — especially the expert ones, built from multi-bug debugging with strict output contracts (return the full corrected file, exact rounding, defensive copies). The scoreboard folds all three levels into one code column; the per-level breakdown it records (by_level) is where the arguments happen.

The engine made no measurable difference

I ran the same harness against a different engine, the community MoE streaming engine BigMoeOnEdge, to see whether the engine decides the score. It did not: 0.920 against 0.923 on the same 72 agent tasks, same model, same quant, at the same throughput (48 tok/s against 47). For a 27B on two 5080s, streaming experts from NVMe is not the bottleneck.

Two conditions, so the comparison is honest. The engine ran at a 16K context, because a 32K-wide prefill graph wants a 19.2 GB compute buffer and does not fit a 16 GB card. And it has no knob for the KV cache type, so the KV was the engine's default rather than the q8_0 the rows above pin. The seed is a session parameter there, so it was set per run and the session reloaded.

It did surface a bug in that engine's router hook. A generated turn that its chat parser cannot turn into text or a tool call comes back as an empty assistant message: 31 tokens generated, the turn dropped, and the episode carries on with a hole in it. Twelve of those in the three runs, and two bash tasks flip from pass to fail because of it. llama.cpp recovers where the other one drops the turn, so the same model and the same prompts produce no empty reply there. Filed with a backtrace: issue #334.

What I would tell you to do

Pick by dimension. In this table, tools belong to MiniCPM (100%), hard code to UD-Q3 with thinking (92% hard, 91% expert), the agent loop to Heretic with thinking (97%), and per-turn speed to qwen36 off (19 ms). No row wins everything.

Measure on your own hardware with your own profiles. A quant that is comfortable on two 5080s is a slideshow on one, and the result changes with the KV cache configuration.

Read the exclusions. A model that fails to format a tool call once in a while is not slightly worse than one that never misses. In an agent loop, it stops.

Treat close rows as ties. Ten rows inside five hundredths of a point form one group. I choose inside it by latency and by how a model behaves when a call fails, not by the third decimal of a score. Do not look for the bottleneck in the engine first: the same harness against a different engine moved the score by three thousandths and the throughput by one token per second.

The fine print

The tasks are written for this harness, the reference solutions never leave the machine, and the models under test are the consumer-GPU quants people actually run. Scores answer "which GGUF should I run for agentic work", not "how smart is this model". Two thinking-ON rows (gemma, Strata) were measured under tighter tool-loop caps than the rest and are flagged in the report as owed a fair pass; the thirteen models retired along the way live in its Retired tab, record only. The table above is generated from the live scoreboard, not typed in, and the engine comparison ran under the conditions stated in its section. Full method in docs/SPEC.md of the repository.

Scoreboard and live report: jordimarsal.github.io/ggufbench. Code, tasks and receipts: github.com/jordimarsal/ggufbench.

RAG without an eval gate is a demo.

Jordi Marçal — Mon, 19 Oct 2026 13:37:13 +0000

I run a portfolio site where every claim about my work has to survive its own eval. This essay is the why behind that discipline: a RAG demo is not a system, and the distance between the two is a golden set plus a CI gate. It was first published on jordimp.net — this cross-post keeps that page as the canonical home.

The project in question, CodebaseRAG, is an open-source RAG engine over a codebase. Numbers below are its real, committed floors — including the embarrassing parts.

The problem: recall nobody measures

Most RAG projects ship a demo. You ask a question, the retriever returns five chunks, a model writes a confident paragraph, and everyone nods. Nobody can say whether the right document was in those five chunks — because nobody wrote down what “right” means and nobody measures it. The demo works until a chunking change, a new embedding model or a different vector store quietly moves quality, and the only signal is a vague feeling that answers got worse.

That is the real failure mode: not hallucination as moral panic, but retrieval that degrades with nobody watching. A demo optimises for the happy path in front of an audience; a system optimises for a number it can regress against. The distance between the two is an eval — and CodebaseRAG exists to close it.

The golden set: turning “right” into forty checkable pairs

An eval is only as honest as its questions. The golden set is a public list of question/answer pairs drawn from the repository itself: a question a developer would actually ask, paired with the answer and where it lives. It holds ≥40 pairs — enough to catch a real regression, small enough that every pair can be reviewed by hand and kept honest.

You do not need a labelling platform. You need somewhere to write questions down and the discipline to keep them. They live next to the code, so a retriever change and its updated expectations travel in the same review.

Three entries, paraphrased from the public set, show the shape:

Which abstraction does the query use case depend on, and why is it not a vector-store type? It depends on a retriever port; adapters implement it, so the core never imports a pgvector or Qdrant class.

What should CI do when mean recall@5 drops below the committed floor? Fail the build. The floor is a committed number, and moving it is an explicit, ADR-gated decision — not an edit that makes CI green.

Can the eval run with no network? Yes. Embeddings arrive through a port with two adapters — Ollama locally, Anthropic when hosted — so the same golden set runs offline.

The examples are not meant to be hard. They are meant to have defensible answers, so a wrong one is a failing test rather than a disagreement.

The CI gate: what fails, what does not

A gate is only useful if it fails loudly and predictably. This one measures mean recall@5, mean reciprocal rank and nDCG@5 over the golden set, and fails the build when mean recall@5 drops below the committed baseline of 0.409.

What fails: any change that pushes mean recall@5 under 0.409. That is the entire rule. CI runs the same deterministic eval as a laptop, so a red build points at a retrieval change, not a flaky environment.

What does not fail matters too. A model change that keeps recall@5 at or above the floor passes even when MRR and nDCG@5 move — the gate guards the metric that was committed to, not every number on the dashboard. A refactor that moves code without moving retrieval passes. And the gate does not judge answer fluency: this is a retrieval eval, and pretending it measures generation quality would be the same dishonesty as having no eval at all.

Why hexagonal: swapping Qdrant must not rewrite the core

You measure retrieval, so what you measure must not be welded to what you deploy. CodebaseRAG is hexagonal: the query use case depends on ports, and pgvector, Qdrant, Ollama and Anthropic sit behind them as adapters.

The test is concrete. Swapping Qdrant for pgvector — or running fully local on Ollama — must not rewrite the core. If changing a vector store forces edits across the use cases, the eval is measuring an accident of the storage layer, and every future swap re-opens whether the metrics still mean anything.

Keeping the core free of frameworks and transports buys more than tidiness: it keeps the eval deterministic, exercised against in-memory or container-backed adapters with no network required. A committed floor is only trustworthy when the code beneath it does not change shape every time the infrastructure does.

Current baseline and what it does not claim

The committed baseline is a mean recall@5 of 0.409 (MRR 0.231, nDCG@5 0.277) over a golden set of at least forty pairs. Read it as a floor, not a result. 0.409 is not SOTA and this is not a claim that retrieval is solved. It is the number the project committed to defend, chosen low enough to stay stable and high enough to catch a real drop.

What the number does not claim: that the golden set is exhaustive, that a green build means good answers, or that the embedding model is the best available. Raising the floor is a recorded decision with a justification, never a vanity metric pasted into a README.

What it does claim is smaller and more useful: on every change, the repository can tell you whether retrieval got worse, and prove it from a public golden set. That is the line between a RAG demo and a RAG system. A demo shows you an answer; a system lets you disagree with it.

Cross-posted from jordimp.net/en/blog/rag-eval-gate, where it lives with its revision history. The engine is open source: jordimarsal/codebaserag. If you enjoyed the honest numbers, the same site runs a RAG concierge whose gen model survived a 16-candidate benchmark before shipping — receipts included.