DEV Community: Adrian Cockcroft

Retort Thinking Level Results: What Opus Actually Does With the Extra Time

Adrian Cockcroft — Fri, 31 Jul 2026 15:59:03 +0000

This is a snapshot of the analysis that is updated whenever new experiments warrant it at:
https://github.com/adrianco/retort/blob/main/levels-blog.md

Published 2026-07-31 · updated 2026-07-31 — Adrian Cockcroft

--effort low|medium|high|xhigh|max is the newest and largest cost lever in retort, and the least understood. versions-blog.md established that it costs; this page is about what the model does with the time, read out of the archived agent logs rather than inferred from the totals.

The short version: below the top of the dial Opus 5 writes more. At the top it stops writing and starts revising — re-reading its own code, re-running its own tests, editing what it already wrote. That switch is where the cost curve leaves the time curve, and on the routine task it buys nothing the gate can see.

Data: adrianco/experiment-55, Claude Opus 5 and GPT-5.6 Terra, python and go, five effort levels, 2 replicates on the routine task (bookshop) and 1 on the hard task (brazil). Every figure below is recomputed from the per-run databases and the archived _agent_stdout.log files.

The dial, end to end

Claude Opus 5 on the routine task, python, mean of 2 replicates per level:

`effort`	wall	cost	turns	tool calls	tokens	src lines	tests	`test_coverage`	`requirement_coverage`
low	2.0 min	$0.76	16	15	452,552	192	10	0.98	1.00
medium	3.1 min	$1.04	21	20	648,077	219	18	0.98	1.00
high	5.9 min	$1.68	29	28	1,083,234	458	60	0.98	1.00
xhigh	11.9 min	$2.55	42	40	1,714,098	574	51	0.99	1.00
max	35.1 min	$19.21	175	220	15,814,628	944	78	0.99	1.00

Everything scales monotonically — time, cost, turns, tokens, lines, tests. The one column that never moves is the one the project scores on: requirement_coverage is 1.00 at every level. The task is fully implemented in two minutes for 76 cents, and it is still exactly fully implemented 33 minutes and $19 later.

Note the curve shapes differ. Wall clock grows 17× from low to max; cost grows 25×; tokens grow 35×. Cost outruns time because the work per turn intensifies, not just the number of turns.

The behavioural switch: from writing to revising

The totals don't say what changed. The tool mix does. Same runs, mean tool calls per run:

`effort`	Write	Edit	Edit:Write	Read	Bash
low	5	2	0.30	1	8
medium	5	4	0.80	1	10
high	12	3	0.24	2	11
xhigh	18	6	0.33	2	14
max	26	64	2.43	36	86

Through xhigh the pattern is stable and unremarkable: Opus writes files, runs the tests a handful of times, and stops. Writes outnumber edits roughly 3:1. More effort simply means more of the same — high writes 12 files where low wrote 5.

At max the shape of the work changes. Edits outnumber writes 2.4:1, Reads jump 18× (2 → 36), and Bash calls jump 6× (14 → 86). It is no longer producing a solution; it is auditing one it already has — re-reading its own modules, re-running the suite, adjusting. Tool calls per turn also rise from ~0.95 (one action per turn, all the way up the dial) to 2.02, meaning it starts batching parallel actions.

This is where the self-imposed extras appear, and only here. Across the ten python runs, mutation testing, subagent delegation and a hand-written OpenAPI document occur exclusively in the two max runs — never at low through xhigh. That matches what the slowest passing run turned out to contain: a seven-module package, unprompted ruff linting, and mutation testing, for a books CRUD API.

The switch point tracks task difficulty, not the dial position. On the hard task the same flip happens two levels earlier:

brazil, python (n=1)	wall	cost	turns	Edit:Write
low	18.7 min	$8.14	75	0.93
medium	25.9 min	$13.94	126	1.91
high	59.9 min	$45.34	231	2.11

On bookshop, Opus revises only at max. On brazil it is already revising at medium. The dial doesn't set the behaviour directly — it sets a budget, and the model switches into revision mode once it has spent enough to have something worth revising.

Does any of it show up in a measured response?

Only where there was headroom. Compare the same dial in go, where the test-coverage scorer is not already pinned at the ceiling:

`effort`	python `test_coverage`	go `test_coverage`
low	0.98	0.73
medium	0.98	0.74
high	0.98	0.76
xhigh	0.99	0.87
max	0.99	0.88

Python starts at 0.98 and has nowhere to go — the extra 33 minutes move it 0.01. Go starts at 0.73 and gains 15 points, a real and monotonic improvement. So the honest answer is not "effort buys nothing", it is: effort buys coverage exactly where coverage is missing, and nothing where it isn't.

Everything else is flat or worse. requirement_coverage is 1.00 at all ten cells. Maintainability does not improve with effort — on python it reads 0.96 at low and 0.89 at max, because 944 lines are simply more to maintain than 192.

The same five names mean different things per vendor

Run the identical sweep on GPT-5.6 Terra and the dial barely registers:

`effort`	Terra cost	Terra tokens	Opus 5 cost	Opus 5 tokens	cost ratio
low	$0.16	182,387	$0.76	452,552	5×
medium	$0.12	118,504	$1.04	648,077	9×
high	$0.16	165,724	$1.68	1,083,234	11×
xhigh	$0.17	157,287	$2.55	1,714,098	15×
max	$0.29	278,882	$19.21	15,814,628	67×

Terra's whole range is $0.12–$0.29 — its max costs 2.4× its medium. Opus 5's max costs 25× its low. Both score 1.00 in all ten cells.

The two CLIs accept the same five words, which makes the factor look comparable. It isn't: max is a mild nudge on one vendor and a regime change on the other. Any cross-vendor comparison has to pin the level explicitly and still report cost, because matching the name does not match the behaviour.

What to actually do with the dial

On routine work, leave it at low. Ten of ten cells scored 1.00; low did it in 2 minutes for 76 cents. Everything above it is paying for revision of an already-correct answer.
Raise it when a measured response has headroom — the go coverage gain is real. That is a reason to go to xhigh, and the gain is already there at xhigh rather than max.
Treat max as a different mode, not one more notch. It is where the model starts auditing itself, where cost goes super-linear, and where it does work nobody asked for.

Caveats

n is small. Two replicates per bookshop cell, one per brazil cell. The go max row is n=1 (13.3 min, $4.21) and lands below its own xhigh (18.4 min, $6.70) — almost certainly the missing replicate rather than a real inversion, so the go max figures are not used for any claim above.

The brazil sweep is incomplete. Opus 5 ran low/medium/high before the account hit its usage limit; xhigh and max never ran. The brazil table is the shape of the first three levels only, and the two most expensive cells are missing.

One measurement trap, recorded because it bit this analysis. A Claude Code JSONL log can contain more than one result record — 2 of the 24 Opus runs here do. Reading num_turns from the last record alone undercounts: the 45.6-minute max run reports 94 that way when the true total is 116 + 94 = 210. An earlier version of tasks-blog.md and experiments-blog.md published the 94; both are corrected. Directly counted tool_use blocks (256 for that run) are the more reliable measure and are what the tool-mix table above uses.

Retort - The Optimal AI Coding Stack

Adrian Cockcroft — Thu, 30 Jul 2026 23:41:14 +0000

This snapshot of the current optimal stack changes all the time as new experiments complete. If you disagree or have different configurations or models available, run your own experiments with Retort and send a pull request...

https://github.com/adrianco/retort/blob/main/optimal-blog.md

If you want to consume the current recommendation directly by tooling there is also a json formatted file at https://github.com/adrianco/retort/blob/main/optimal.json

Living document — last updated 2026-07-30 (first published 2026-07-14). This records **what to run today: the leading stacks, and the exact configuration each one needs. It is not a history. Superseded stacks and rejected configurations are not discussed here; they are retired, and retirement is the point.

What this document is

A stack is the whole thing you actually deploy, not just a model:

language × model × quantization × serving layer × agent × context engine × sampling × prompt

Retort scores stacks on pass-proportion — run a stack N times on a real task and count the fraction whose output fully implements the spec: every requirement on a fixed checklist, tests that actually execute, verified by an independent evaluator. A run that misses one requirement is a failure, not a 0.9. Read it as the probability that a single unattended run comes out completely correct.

That harsh bar is deliberate, because it is the number that decides whether you can let a stack work unattended.

Lifecycle: how a stack gets here, and how it leaves

This document is the human-readable view of a lifecycle the tool already runs. Each stage is a retort command, so entries here are derived, not curated by taste:

CANDIDATE ──► SCREENING ──► TRIAL ──► PRODUCTION ──► RETIRED

Stage	What it means	How it's decided
Candidate	a new model, agent or configuration appears	`retort intake --factor model --level <new>` augments the existing design rather than restarting it
Screening	Resolution III — do its main effects matter at all?	`retort run --phase screening` → `retort analyze`
Trial	Resolution IV/V — interactions estimated	`retort promote --from screening --to trial` (gate: p-value)
Production	listed in this document — the recommended stack for its niche	`retort promote --to production` (gate: posterior confidence); `retort maturity` scores readiness
Retired	dominated on every metric by a newer stack — removed, not demoted	`retort report pareto` identifies who is still non-dominated

The gates are configuration, not opinion — they live in each workspace.yaml:

promotion:
  screening_to_trial:   { p_value: 0.10 }
  trial_to_production:  { posterior_confidence: 0.80 }

A stack earns a place here only by leading on an axis a developer actually chooses along — a language, a task size, a cost or latency budget. Being new is not a qualification. When it leads on none, retort report pareto shows it dominated and it comes out. The list stays short on purpose.

Configurations that reliably degrade a stack are eliminated outright and recorded as forbidden settings (below). New model releases — frontier and local — trigger a re-qualification pass, so this document is expected to churn.

Start here: the cheapest (model × thinking level) that clears each cell

A stack is a model and a thinking level. Measured on one cell across four Claude versions × five effort levels, every one of the 21 combinations passed — and the bill spanned 16×, from \$0.42 to \$6.75, for identical output. The dial moves cost more than the version does, and it moves the result not at all. So the recommendation is a pair, never a model alone.

The machine-readable form of the table below is committed at optimal.json (regenerate with retort report optimal --routing-json optimal.json) so other tools — the metaharness router, CI, your own scripts — can consume the routing decision directly rather than scraping prose.

Language	Routine → cloud	pass	$	Routine → local	Hard → cloud	pass	$
c	Opus 4.8 @ `default` n=1	1.00	$1.28	—	Fable 5 @ `default` n=1	1.00	$10.59
clojure	Opus 4.7 @ `default` n=6	1.00	$1.06	—	Opus 5 @ `default` n=1	1.00	$2.55
cpp	Opus 4.8 @ `default` n=1	1.00	$1.08	—	Opus 5 @ `default` n=1	1.00	$17.50
csharp	Opus 4.8 @ `default` n=1	1.00	$0.65	—	Sonnet 5 @ `default` n=3	1.00	$9.96
elixir	Opus 4.8 @ `default` n=3	1.00	$0.85	—	Fable 5 @ `default` n=1	1.00	$13.21
erlang	Opus 4.8 @ `default` n=3	1.00	$1.35	—	Fable 5 @ `default` n=1	1.00	$12.06
go	GPT-5.6 Luna @ `default` n=3	1.00	$0.08	Qwen3-Coder-Next 80B @ `default` n=3	Opus 4.8 @ `default` n=6	1.00	$5.30
java	Opus 4.7 @ `default` n=6	1.00	$0.92	—	Fable 5 @ `default` n=1	1.00	$12.01
objc	Opus 4.8 @ `default` n=1	1.00	$1.52	—	Fable 5 @ `default` n=1	1.00	$13.30
python	GPT-5.6 Luna @ `default` n=3	1.00	$0.06	Qwen3-Coder-Next 80B @ `default` n=3	Opus 4.7 @ `default` n=3	1.00	$4.64
rust	Opus 4.8 @ `default` n=6	1.00	$0.71	—	Fable 5 @ `default` n=3	1.00	$9.63
swift	Opus 4.8 @ `default` n=1	1.00	$1.25	—	Fable 5 @ `default` n=1	1.00	$9.24
typescript	Opus 4.8 @ `default` n=7	1.00	$0.65	Qwen3-Coder-Next 80B @ `default` n=3	Opus 4.7 @ `default` n=3	1.00	$4.14

Reading the effort column. Almost every cell says default, and that is an honest statement rather than a recommendation: thinking level has only ever been varied on one cell (python × bookshop, exp-49). Everywhere else the corpus has exactly one measured level, so the router reports the level those runs actually used and does not imply a comparison nobody made. Where the dial was swept, the finding was consistent — low cost ~1.6× less than the CLI default at identical reliability, and the default is not the cheap end. Expect these to shift toward low as the sweep widens.

A null means nothing measured clears the bar for that cell — not "untested". And read the n: several cells are n=1, where a 1.00 is much weaker evidence than a 1.00 at n=9.

✅ One bar for every stack: 1.00. Local stacks used to qualify at 0.50, on the view that a \$0 stack is worth a lower bar if you are watching it. That contradicted the metric's own definition — the probability a single **unattended* run comes out completely correct*, where a single sub-1.0 run is a fail — and it let a coin flip outrank a perfect stack purely on price (python-on-hard-task recommended the local 35B at 0.50). Being free is already expressed in the cost column; it should not also lower the standard.

Raising local to 1.00 did not push local out of the recommendations — it picked a better local stack: routine python and go moved from the 35B (0.87 / 0.85) to the 80B at 1.00, still \$0. The double standard had been hiding the more reliable free option. Hard-task python moved from the 35B at 0.50 to Opus 4.7 at 1.00, which is the honest answer.

The leading stacks

Reliability, cost and time are all reported per task size — routine and hard are different jobs, and a stack that is cheap-and-certain on routine work can be neither on a hard one. The routine reliability here is a cross-language blend and is the least useful number in this document — it hides a stack's weak languages inside its strong ones (local Qwen passes Python/Go but not Rust; Opus 4.8 dips on Java). Use it only as a rough sort; the per-language success-rate matrix below is the number to actually decide on. (Hard reliability is single-task, measured on Python/Go.)

Stack	Reliability (routine · hard)	Cost (routine · hard)	Time (routine · hard)
Claude Opus 5	1.00 · 1.00	$3.23 · $21.67	546 s · 2630 s
Claude Fable 5	1.00 · 1.00	$1.58 · $10.47	166 s · 1090 s
GPT-5.6 Terra (codex)	1.00 · —	$0.22 · —	155 s · —
GPT-5.6 Luna (codex)	0.67 · —	$0.09 · —	153 s · —
Claude Sonnet 5	1.00 · 0.93	$1.10 · $7.64	237 s · 1252 s
Claude Opus 4.8	0.98 · 0.59	$0.93 · $3.27	258 s · 608 s
Claude Opus 4.7	1.00 · 0.40	$0.92 · $2.95	165 s · 500 s
Qwen3.6-35B-A3B (local, $0)	0.86 · 0.25	$0.00 · $0.00	355 s · 1542 s
Qwen3-Coder-Next 80B (local, $0, ctx 0.9)	1.00 · 0.00	$0.00 · $0.00	604 s · 2014 s

(Table generated from master.db by retort report optimal — do not hand-edit between the markers.) Each local stack's routine number is scoped to the languages it is recommended for (35B: Python/Go; 80B: Python/Go/TypeScript) — the full per-language truth, including the languages they fail, is in the matrix below. On the hard task local models are now measured and both do poorly — 35B 0.25, 80B 0.00 (see the per-stack bullets). Rust local is unqualified (80B 0.33, near-misses).

⚠️ Local + hard task: the wall holds unattended, but a repair loop gets through

The 0.00 below is the unattended, first-attempt number and it stands — exp-50 re-ran it and got 0/6 on first attempts, matching exp-39 exactly.

What exp-50 adds: with retort's self-repair second chance — the cell re-seeded with its own code plus the evaluation's critique — the 80B reached full req-coverage on 3 of 6 runs. Every pass was a second attempt; none was unattended. At half credit for a second-try pass that is a pass-proportion of 0.25 against a published 0.00.

Read it as: the 80B reliably gets ~11 of 12 capabilities and cannot close the last one alone, but closes it about half the time when told what is missing. "Hard tasks → cloud" remains right for unattended use; a local stack with a repair loop is a different and more promising proposition.

(The re-test was launched on a theory — a 30-turn agent cap — that turned out to be wrong: the old runs took 32–90 turns and nothing truncated them. Kept visible in the write-up.)

Pick by task size — the two columns tell different stories:

Fable 5 clears the hard task every time (1.00), in all thirteen languages. When a hard, multi-capability job has to be right unattended, this is the one — ~$10.47 and ~18 min a run, and the certainty is what you're buying.
Opus 5 also clears the hard task 1.00 in all thirteen, at ~$21.67 and ~44 min — twice the money and 2.4× the time for the same outcome. It held this slot briefly on the strength of being the only stack measured across all 13 languages; once Fable 5 was actually run on the nine it had been missing, that distinction vanished. There is no language here where Opus 5 is the necessary choice.
Sonnet 5 lands 0.93 on hard at ~15% less than Fable and comparable time — the cost-aware hard-task pick when roughly 1-in-14 misses is acceptable.
Opus 4.8 is the cheapest hard-capable cloud stack, but hard reliability is only ~0.59 — a coin flip, so budget for review-and-retry. On routine work it is ~1.00 and cheap; that's its real niche.
Opus 4.7 is a fine, cheap routine stack (1.00, ~$0.97) but weak on hard (0.40) — don't send hard work to it.
Qwen 35B local does routine Python / Go for $0 at 0.85 each — its real niche. Its leading number (0.78) is lower than either language because the blend now includes its Rust and TypeScript runs, which score 0.00 — the clearest example in this doc of why you must read the per-language matrix, not the average. On the hard task it scores 0.25 (3/12) — occasionally nails all 12 capabilities, but not reliably. Rust and TypeScript unqualified.
Qwen 80B local (Qwen3-Coder-Next) — the best local stack for Python, Go and TypeScript (context_threshold: 0.9, "full context"). The featured numbers above are now at 0.9, the clean full-9-language baseline (exp-38, n=3/language): Python 1.00, Go 1.00, TypeScript 1.00 — all three reliable, all $0. TypeScript is the unlock: it was 0.33 at ctx 0.35/0.7, and raising compaction to full context makes it pass 3/3. It's slower than the 35B (~600 s routine). Rust is 0.33 (1/3) → cloud — but note its 2 failures are near-misses (req-coverage 0.92/0.92; the code compiles and its tests pass 100%, it just misses 1–2 spec requirements), not the thrash-to-the-wall stalls of the old story (confirmed via retort diagnose+rescore+reevaluate, which caught them as scorer tooling false-failures and recovered their true near-miss scores). The other five languages → cloud: java/erlang are near-misses (0/3), and clojure/csharp/elixir score a genuine 0.00 — they can't produce working code at all (diagnosed GENUINE, not harness). On the hard task it scores 0.00, and that is now verified config-invariant — re-run at full context (exp-39, brazil at 0.9) it is still 0/6, same as 0.7 (exp-31): Python gets as close as 11/12 but never all 12, and Go actually regresses (a stall — the same late-compaction downside that bites any run that can't finish). So full context is strictly for the easy languages; it does not raise the hard-task ceiling. Rule: 80B for local Python, Go and TypeScript (at context_threshold: 0.9); Rust, the niche languages, and hard tasks → cloud.

The 80B's stall was a fixable config artifact, not a capability wall — and raising the compaction threshold is a graded lever, not a switch. The intermittent hang was lcm compaction firing too early: at the default context_threshold: 0.35 it compacts live context at ~92K, truncating the agent's working history mid-build so it loses the thread and thrashes to the wall. Raising it walks the results up:

0.35 → 0.7 kills the Go stalls: over 9 Go runs (exp-34/36) 0 stalls, Go = 8/9 = 0.89 (up from 0.67-with-2-stalls), and Python holds at 1.00 (exp-37). But TypeScript was still only 0.33 here.

0.7 → 0.9 ("full context", compact at ~236K) unlocks TypeScript. The full-9-language re-baseline at 0.9 (exp-38, n=3) gives Python 1.00, Go 1.00, TypeScript 1.00 — TS goes from 0.33 to 3/3 because at full context the agent keeps its whole working history through the longer TS build instead of being compacted mid-stream. So lcm.context_threshold: 0.9 is now the recommended config for the 80B, and the featured table reflects it (the 0.7 runs remain the larger-n Go evidence and the proof that the fix is graded).

The cost of going to 0.9: a run that can't finish thrashes longer before failing (6 M tokens on a failed Rust/niche cell) — but those languages go to cloud anyway, so it doesn't touch the recommended Python/Go/TS path. (The same lever only partly rescues the 35B on Rust — exp-35: first-ever Rust pass at 0.7, but 2/3 still stall. And it doesn't help the hard task: exp-39 re-ran brazil at 0.9 and got the same 0/6 as 0.7, with Go now hitting the late-compaction stall — full context helps the easy languages, not the hard ceiling.)

The same lever partly explains the "Rust wall" — but only partly. At 0.35 the 35B thrashes to the wall on every Rust run (clean 0.00). At 0.7 it scored its first-ever Rust pass (exp-35 rep1 = 1.00, at 113K context — exactly the regime 0.35 truncates), so Rust is not a pure capability wall. But 2 of 3 still stalled at 0.7 — unlike the 80B (0/6), the 35B on Rust is only partially rescued. Net: raising context_threshold is a real lever for local non-termination, but its strength is model- and language-dependent — a clean fix for the 80B on Go/TS, a partial one for the 35B on Rust. Rust stays → cloud.

On the Opus 4.8 hard number. 0.59 is an honest blend: a small clean run scored 1.00 (n=6) while a larger one scored 0.50 (n=36). The optimistic single-run figure is not representative — treat hard-task Opus 4.8 as a coin flip, not a sure thing.

What to run, by language and task size

Task size is the axis that matters most, more than language:

Routine — CRUD, glue, well-trodden patterns, a few interacting requirements.
Hard — a novel domain, many interacting capabilities, real data, a protocol to implement correctly (our reference: an MCP server over six datasets, twelve required capabilities).

Start from the per-language success rate, not a single headline number. This is the matrix that matters — routine pass-proportion for each language × stack, pass (n), generated from master.db. A blank cell means we have no qualified runs there. Read down a column to see where a model is weak (Opus 4.8 on Java; the 35B local passes Python/Go but scores 0.00 on Rust/TypeScript; the 80B local is strong on Python but drops on Go/TS), and across a row to pick the cheapest stack that actually passes that language:

Language	Opus 5	Fable 5	Terra	Luna	Sonnet 5	Opus 4.8	Opus 4.7	Qwen 35B local	Qwen 80B local
c	1.00 (1)	1.00 (1)	—	—	—	1.00 (1)	—	—	—
clojure	1.00 (1)	1.00 (3)	—	—	—	1.00 (6)	1.00 (6)	—	0.00 (3)
cpp	1.00 (1)	1.00 (1)	—	—	—	1.00 (1)	—	—	—
csharp	1.00 (1)	1.00 (1)	—	—	1.00 (3)	1.00 (1)	—	—	0.00 (3)
elixir	1.00 (1)	1.00 (1)	—	—	—	1.00 (3)	1.00 (3)	—	0.00 (3)
erlang	1.00 (1)	1.00 (1)	—	—	—	1.00 (3)	1.00 (3)	—	0.00 (3)
go	1.00 (10)	1.00 (3)	1.00 (10)	1.00 (3)	1.00 (3)	1.00 (7)	1.00 (6)	0.85 (27)	1.00 (3)
java	1.00 (1)	1.00 (1)	—	—	—	0.83 (6)	1.00 (6)	—	0.00 (3)
objc	1.00 (1)	1.00 (1)	—	—	—	1.00 (1)	—	—	—
python	1.00 (26)	1.00 (18)	1.00 (10)	1.00 (3)	1.00 (3)	1.00 (22)	1.00 (21)	0.87 (30)	1.00 (3)
rust	1.00 (1)	1.00 (3)	—	—	1.00 (3)	1.00 (6)	1.00 (6)	0.00 (2)	0.33 (3)
swift	1.00 (1)	1.00 (1)	—	—	—	1.00 (1)	—	—	—
typescript	1.00 (1)	1.00 (1)	—	0.00 (3)	1.00 (3)	1.00 (7)	1.00 (6)	0.00 (3)	1.00 (3)

The language split: Python, Go and TypeScript run locally for free (on the 80B at full context); every other language means Claude. The 80B (Qwen3-Coder-Next, at context_threshold: 0.9) is reliable on all three — Python 1.00, Go 1.00, TypeScript 1.00 (exp-38, n=3 each) — the last only after raising compaction to full context (it was 0.33 below that). The 35B is the faster alternative but only on Python and Go (0.85 each); it scores 0.00 on TypeScript and Rust even at its tuned config, so its cross-language average (0.85 when scoped to Python/Go; lower if you blend in the languages it can't do) is exactly why this document leads with the matrix, not an average. Rust and the five niche languages (clojure/csharp/elixir/java/erlang) still go to cloud — the 80B either near-misses (Rust 0.33, java/erlang) or can't produce working code at all (clojure/csharp/elixir score a genuine 0.00). So local has two stacks for Python/Go and one (the 80B at 0.9) that adds TypeScript.

The recommendation table distills the matrix into, per language, the cheapest model that clears routine work, the hard-task pick, and the prompt / testing method (the last is qualitative, from the prompt experiments):

Language	Routine → cheapest qualifying	Hard task	Prompt / testing method
Python	Qwen 80B local ($0) 1.00 for reliability, or 35B 0.85 for more speed	Fable 5 (1.00); Opus 4.8 cheaper but ~0.59	80B: neutral (prompt is a no-op — all pass). 35B: neutral/BDD, never ATDD. Cloud: neutral
Go	Qwen 35B ($0) 0.85, or 80B @ ctx 0.9 ($0) 1.00 — both local-viable	Fable 5; Opus 4.8 cheaper / riskier	35B: neutral or BDD, never ATDD. 80B: neutral. Cloud: neutral
TypeScript	Qwen 80B @ ctx 0.9 ($0) 1.00 (n=3) — newly local-viable, or Opus 4.8 (~$0.65)	Fable 5	80B: neutral (needs `context_threshold: 0.9`). Cloud: neutral
Rust	Opus 4.8 (~$0.71) — local 0.33 (near-misses, → cloud)	Fable 5	Cloud: neutral
Clojure	Opus 4.7 (~$1.06)	Fable 5	Cloud: neutral
Java	Opus 4.7 (~$0.92)†	Fable 5	Cloud: neutral
C#	Opus 4.8 (~$0.65)‡	Fable 5	Cloud: neutral
Elixir	Opus 4.8 (~$0.85)	Fable 5	Cloud: neutral
Erlang	Opus 4.8 (~$1.35)	Fable 5	Cloud: neutral
C ᴱ	Opus 4.8 (~$1.28, n=1)	Fable 5	Cloud: neutral
C++ ᴱ	Opus 4.8 (~$1.08, n=1)	Fable 5	Cloud: neutral
Objective-C ᴱ	Opus 4.8 (~$1.52, n=1)	Fable 5	Cloud: neutral
Swift ᴱ	Opus 4.8 (n=1)	Fable 5	Cloud: neutral

† On our routine Java runs Opus 4.8 dipped to 0.83 while 4.7 held 1.00 — small n; review Java output whichever you use. ‡ C# Opus 4.8 is n=1; Sonnet 5 (~$1.57, n=3) is the better-sampled fallback. ᴱ C / C++ / Objective-C / Swift are the newer exploratory languages (exp-43); their routine picks are n=1, so treat them as "known to work once," not as reliability estimates.

Why Fable 5 in every hard-task cell, and not Opus 5? Both clear all thirteen languages at 1.00. Fable 5 does it for $10.47 / 18.2 min per cell against Opus 5's $21.67 / 43.8 min — 2.1× cheaper and 2.7× faster on the identical nine-language subset, using 2.3× fewer agentic turns. There is no language in this dataset where Opus 5 is the necessary choice. (Opus 5 briefly held this column on the strength of being the only model measured across all 13; that was a gap in coverage of the comparison, not a capability difference — see exp-48.)

Prompt / testing method — it matters only in proportion to how weak the model is. The table above (Python routine) makes the rule concrete: the prompt is a lever on a weak model and a no-op on a strong one.

Strong models (all cloud, and the local 80B): flat line. Every methodology passes — the 80B goes 1.00 on all four, ATDD included. Pick neutral and spend nothing on methodology ceremony; it's the cheapest and loses nothing.
Weak models (the local 35B): the prompt bites. neutral/BDD 0.67, TDD 0.33, and ATDD 0.00 — a weak model can't carry ATDD's front-loaded discipline and burns the run. So on the 35B: neutral (cheapest) or BDD, and never ATDD.

The takeaway: reach for a disciplined methodology only when you're near a model's capability edge; on a model that clears the task comfortably, the prompt is ritual.

The decision procedure:

Hard task, must be right the first time, unattended? → Fable 5 (1.00). Costliest and slowest; certainty is what the premium buys.
Hard task, but cost matters and ~1-in-14 misses is tolerable? → Sonnet 5 (0.93), ~15% under Fable. Opus 4.8 is cheaper again but only ~0.59 — budget a review-and-retry loop if you use it.
Routine, local & free? (run the 80B at context_threshold: 0.9) → Python: 80B (1.00) for reliability or the 35B (0.85) for more speed. Go: 80B (1.00) or 35B (0.85) — both viable. TypeScript: 80B (1.00) — newly local-viable at full context (below 0.9 it's a coin-flip). Review the output; you'll still see a miss every few runs. (Rust, the niche languages, everything else → cloud.)
Routine, any other language? → cheapest current cloud (Opus 4.7 / 4.8, ~$1); they all reach ~1.00, so paying more buys nothing. Use the neutral prompt.

Configuration

Reliability is a property of the stack, not the model. A leading model on a bad configuration is not a leading stack. These are the settings each one requires.

Local: Qwen3.6-35B-A3B


Model	`unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit` (4-bit MLX, ~20 GB)
Serving	oMLX 0.5.0 · served from `~/models/<name>`
Agent	Hermes v0.18 with the `hermes-lcm` plugin (`context.engine: lcm`)
Context	`context_length: 262144` — set it explicitly; the default fallback is far lower
Compaction	`lcm.context_threshold: 0.9` for the 80B ("full context" — default 0.35 compacts at ~92K and causes intermittent stalls; 0.7 kills the stalls and 0.9 additionally unlocks TypeScript — see exp-34/38; env override: `LCM_CONTEXT_THRESHOLD=0.9`). The 35B is fine at 0.35.
Hardware	Apple Silicon, 64 GB. Raise the GPU wired limit: `sudo sysctl iogpu.wired_limit_mb=57344`

Sampling — this is not optional:

temperature:        0.6     # anything in 0.2–0.7; the requirement is: NOT 1.0
top_p:              0.95
top_k:              20
repetition_penalty: 1.0     # OFF. See forbidden settings.

Harness settings that decide whether the model can work at all:

playpen_root:  ~/.retort/work   # NOT the system temp dir — see forbidden settings
timeout_minutes: 60             # a high wall: local models are slow, let good work finish
stall_minutes:   25             # kill unproductive loops, not slow-but-productive runs

Operational — the local serving layer needs babysitting over long sessions:

Restart oMLX between long runs. After many hours of continuous serving, oMLX quality degrades — a normally-flawless language (Python is 21/21) starts throwing fast all-zeros fails (the model emits garbage in ~2 min). These look like a real result but are a serving artifact; the fix is to kill oMLX (the next run reloads a fresh server) — do it before any run whose result you intend to trust, and treat a sudden cluster of all-zeros on a reliable language as "restart the server," not "the model got worse."
Watch the disk — the paged-SSD cache is a trap. oMLX's --paged-ssd-cache grows to its configured cap (a 120 GB default filled a 926 GB disk to 98% in one session) and, per exp-24, it doesn't help these generation-bound runs at all — set the cap small (~5–20 GB) or disable it. A full disk makes the agent's file writes fail → false zeros. retort run now does a disk preflight (aborts under 15 GB free, warns under 40 GB).
Configure Time Machine for a benchmarking box — or it will eat the disk. On macOS/APFS, deleting a big cache doesn't free space if a Time-Machine local snapshot still pins the freed blocks (an hourly cadence pinned ~93 GB of already-deleted oMLX cache in one session). Two settings fix it, and everything large here is regenerable (models re-download from HF, the repo state lives in git), so exclude it all from backups:

  # exclude the big, regenerable, high-churn paths (‑p = sticky, survives recreation)
  sudo tmutil addexclusion -p ~/.cache/omlx-ssd     # oMLX paged-SSD prefix cache
  sudo tmutil addexclusion -p ~/.omlx/cache         # oMLX prefix-block cache
  sudo tmutil addexclusion -p ~/models              # MLX weights (re-download from HF)
  sudo tmutil addexclusion -p ~/.cache/huggingface  # HF download cache
  sudo tmutil addexclusion -p ~/.retort/work        # per-run playpens (transient)
  # reclaim space a snapshot is already pinning, now (df won't budge until you do):
  sudo tmutil thinlocalsnapshots / 100000000000 4

Also drop the snapshot cadence from hourly to daily (e.g. TimeMachineEditor) so cache churn between runs can't pile up dozens of space-pinning snapshots. Exclusions shrink the backup and per-snapshot delta; the daily cadence caps how many pin space at once; thinning clears what's already stuck.

Cloud: Claude Fable 5, Sonnet 5, Opus 4.8 / 4.7

There is no single cloud winner — the pick is set by task size (see the tables above): Fable 5 for hard work that must be right, Sonnet 5 for hard work on a budget, Opus 4.8 / 4.7 for cheap routine work.

Run the model as shipped. The stack that matters is the agent around it; no sampling tuning is required or recommended, and on cloud the prompt is a flat line — use the plain neutral prompt and don't pay for methodology ceremony.

Forbidden settings

Configurations that measurably degrade a stack. These are eliminated, not tuned.

Setting	Why it is forbidden
`repetition_penalty` > 1.0	Any repetition penalty derails an agentic tool-calling loop — the model stops converging, stalls, and produces nothing. This holds even at 1.05, and even when the model's own card recommends it: model-card sampling is tuned for single-turn generation, not multi-turn agent loops. Set it to 1.0.
`temperature: 1.0` (server default)	Costs roughly half the reliability of a local coding stack. Any value in 0.2–0.7 is fine; the precise value does not matter.
Playpen under the system temp dir (`/var/folders/...` on macOS)	Agents refuse to write to paths they consider system-owned, so the agent cannot create files in its own workspace — and a run that writes nothing scores a false zero indistinguishable from an incapable model. Keep playpens under `$HOME`.
An unrecorded stack	A pass-proportion without the stack it was measured on is not a result. Capture versions, model revision hashes, sampling, agent config, and harness settings — every run writes a `provenance.json`.

Measured data (auto-generated)

These tables are regenerated from master.db by retort report optimal — run retort report optimal --write optimal-blog.md to refresh everything between the GEN markers. The leading stacks table above is generated the same way.

Per language — cheapest stack that clears its reliability bar (routine task):

Language	Routine → cheapest qualifying stack	Reliability	n
c	Claude Opus 4.8 ($1.28)	1.00	1
clojure	Claude Opus 4.7 ($1.06)	1.00	6
cpp	Claude Opus 4.8 ($1.08)	1.00	1
csharp	Claude Opus 4.8 ($0.65)	1.00	1
elixir	Claude Opus 4.8 ($0.85)	1.00	3
erlang	Claude Opus 4.8 ($1.35)	1.00	3
go	Qwen3-Coder-Next 80B (local, $0, ctx 0.9) ($0)	1.00	3
java	Claude Opus 4.7 ($0.92)	1.00	6
objc	Claude Opus 4.8 ($1.52)	1.00	1
python	Qwen3-Coder-Next 80B (local, $0, ctx 0.9) ($0)	1.00	3
rust	Claude Opus 4.8 ($0.71)	1.00	6
swift	Claude Opus 4.8 ($1.25)	1.00	1
typescript	Qwen3-Coder-Next 80B (local, $0, ctx 0.9) ($0)	1.00	3

Prompt / testing method — the local sweep (on cloud the prompt is a flat line):

Prompt	35B pass	80B pass
neutral	0.67 (n=3)	1.00 (n=3)
BDD	0.67 (n=3)	1.00 (n=3)
TDD	0.33 (n=3)	1.00 (n=3)
ATDD	0.00 (n=3)	1.00 (n=3)

Keeping this current

Each new frontier or local model release triggers a qualification pass on the standard tasks. A stack is added only when it leads on an axis someone chooses along, and is removed when it no longer leads on any. The tables are regenerated from master.db by retort report optimal; run retort report optimal --health to check the data before trusting a refresh.

Known data-pipeline gaps the generator has to work around (it curates the qualified config in FEATURED_STACKS because master.db can't express it):

Historical local runs carry a blank model — older runs recorded agent: hermes-local but no model, so ~250 rows are attributed to a stack only via their experiment slug. Fixed going forward: the harness now always records the resolved model (stack_metadata()), so exp-29 and later land with a real model id.
No sampling / context columns — temperature, top_p, top_k, repetition_penalty are absent and max_context_tokens is populated only on the newest runs, so "the qualified config" can't be filtered from the data; the tuned experiments are named in the script instead.
experiment-11 isn't ingested — it has no retort.db (an empty/aborted experiment).

Backfilling the historical blank-model rows and re-ingesting would let the generator drop its slug curation and become a plain group-by.

Next review: on the next frontier release, or the next local model that fits 64 GB and tool-calls cleanly.

Retort Tasks: What Gets Built, and How Differently a Run Can Pass

Adrian Cockcroft — Thu, 30 Jul 2026 23:21:27 +0000

This is a snapshot of an auto-generated analysis report from https://github.com/adrianco/retort/blob/main/tasks-blog.md

Published 2026-07-30 · updated 2026-07-30 — Adrian Cockcroft

What retort actually asks an agent to build, and — for each task — the fastest and the slowest run that fully passed. Both mean shortest/longest duration_seconds among runs scoring requirement_coverage == 1.0, restricted to runs whose agent log was archived, since a record with no log can't be shown.

The slow end is worth as much attention as the fast end. Nothing fails there: every run below scores a perfect 1.00. They just take between 4× and 61× longer to get there, and the reasons differ per task — over-engineering on one, the language itself on another, and a tool that costs time on the third.

Task definitions live in tasks/ and are indexed by tasks/registry.yaml. Three of the seven registered tasks have been run at scale; the other four (react-dashboard, cli-data-pipeline, brazil-bench-neutral, funkygibbon-port) are defined but have few or no scored runs.

A caveat on records. Two faster runs exist in master.db but are not shown here: a 2.49-min brazil (exp-2) and a 0.71-min rest-api (exp-1). Both predate agent-log archiving and the mechanical gate — a sibling exp-2 "pass" carries test_coverage=0.0, which today fails automatically. They are not comparable to the runs below.

1. `rest-api-crud` — the routine task

Source: tasks/rest-api-crud/ · 712 runs · the "easy" task

Build a CRUD REST API for a books collection: POST /books, GET /books (with ?author= filter), GET/PUT/DELETE /books/{id}, plus GET /health. Data in SQLite or the language's embedded equivalent, JSON responses with correct status codes, input validation, a README, and at least 3 tests. Scored in all 13 languages.

This is the workhorse. It's deliberately unremarkable — the point is that a competent stack should score 1.00 on it every time, so it measures reliability, not capability.

Fastest logged pass — 44.5s


Stack	Claude Fable 5, `effort=low`, python, prompt=neutral
Experiment	`adrianco/experiment-49-versions-cloud` rep2
Duration	44.5 s · 7 turns · $0.86 · 105,745 tokens
Result	12/12 requirements · 6 tests pass · coverage 0.98

The entire run, condensed:

[Read]  TASK.md
[TEXT]  "I'll build a Flask + SQLite book API with tests."
[Write] app.py
[Write] test_app.py
[Write] README.md
[Write] requirements.txt
[Bash]  python3 -m venv venv && ./venv/bin/pip install -q flask pytest && ./venv/bin/python -m pytest -q
     -> ...... [100%] 6 passed in 0.19s
[TEXT]  "Done — all 6 tests pass."

One read, four writes, one verification command, done. No exploration, no iteration, no failed attempt. This is what the routine task looks like when a model simply knows the answer — and it's why the routine task no longer discriminates between frontier models on reliability, only on cost.

Slowest logged pass — 45.6 min, for the same 1.00


Stack	Claude Opus 5, `effort=max`, python, prompt=neutral
Experiment	`adrianco/experiment-55-terra-vs-opus5-effort-bookshop` rep2
Duration	45.6 min (61×) · 94 turns · $24.96 (29×) · 18,448,183 tokens (174×)
Result	12/12 requirements · 104 tests · coverage 1.00

2,351 lines against Fable 5's 194. It built a seven-module package, linted itself with ruff unprompted, spawned a subagent, and ran mutation testing — on a books CRUD API. Every one of the judge's five findings is an info noting scope beyond the spec: a PATCH endpoint, filtering and pagination, a hand-written 304-line OpenAPI document, WAL journaling, NUL-byte validation. The task asked for five endpoints, a health check and at least three tests.

It is not slop — it scores better on maintainability (0.85 vs 0.27) and idiomaticity. But the gate cannot tell the two runs apart: both are 1.00. The full breakdown, including how much variance the thinking dial adds, is in experiments-blog.md.

2. `brazil-bench` — the hard task

Source: github://brazil-bench/benchmark-template · 284 runs

Build an MCP server over six real Kaggle CSVs of Brazilian football (23,954 matches across five files with three different schemas, plus 18,207 FIFA players). Twelve pinned requirements in REQUIREMENTS.json: match queries by team / date-range / competition / season, team W-D-L records, player search and filtering, season standings computed from results, aggregate statistics, head-to-head, and automated tests.

It is hard for reasons that have nothing to do with algorithms: team names carry state suffixes and accents (São Paulo-SP vs Sao Paulo), one file uses Portuguese column names and DD/MM/YYYY dates, and the datasets overlap — the same real-world match appears in two or three files.

Fastest logged pass — 3 min 19 s


Stack	GPT-5.6 Terra (codex), `effort=medium`, python, prompt=neutral
Experiment	`adrianco/experiment-55-terra-vs-opus5-effort-brazil` rep1
Duration	199.5 s · 16 agent steps · $0.3165 · 339,649 tokens (291,840 cached)
Result	12/12 requirements · 7 tests pass · coverage 0.85

Previous best in the modern harness was 5.67 min (Opus 5, clojure); the fastest previous python brazil pass was 5.02 min. This is a real step change, not a rounding difference.

The run, condensed:

[CMD]  ls && sed -n '1,240p' TASK.md && rg --files          # read spec + repo
[CMD]  sed -n '241,520p' TASK.md ... head data/kaggle/*.csv  # read rest of spec + data shapes
[CMD]  sed -n '1,240p' prompts.txt; wc -l data/kaggle/*.csv  # count rows
[MSG]  "building a dependency-free MCP stdio server with an indexed CSV data
        layer and BDD-style tests"
[EDIT] README.md, brazilian_soccer_mcp.py, server.py, test_*.py   # ALL files, one patch

[CMD]  python -m pytest -q                    -> exit 127: command not found: python
[MSG]  "The environment provides python3 rather than python"
[CMD]  python3 -m pytest -q                   -> 7 FAILED
                                                 "Extra data: line 2 column 1"
[EDIT] brazilian_soccer_mcp.py                # fix the extended-file parser
[CMD]  python3 -m pytest -q && python3 server.py <<< '{jsonrpc initialize}...'
     -> 7 passed in 3.19s
     -> loaded 23954 matches and 18207 players
     -> top 2019: [Flamengo 38pl 90pts, Palmeiras 74, Santos 74]
     -> {"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05", ...}}
[CMD]  python3 -c "Counter by source file"    # investigate dataset overlap
[EDIT] brazilian_soccer_mcp.py                # add standings source-dedup guard
[CMD]  python3 -m pytest -q && py_compile && git diff --stat
[MSG]  "all seven BDD-style tests pass, including the known 2019 Brasileirão
        standings result, and the MCP stdio handshake/tool call succeeds"

Sixteen steps, two real failures, both diagnosed and fixed. Not a lucky one-shot.

What the solution looks like. 342 lines across 3 files, zero dependencies — Python standard library only:

brazilian_soccer_mcp.py (291 lines) — a SoccerData service plus the MCP layer.
server.py (5 lines) — entrypoint, calls serve().
test_brazilian_soccer_mcp.py (46 lines) — 7 BDD-named tests.

Was the Kaggle data really blended and loaded? Yes. All six files, verified in-run: loaded 23954 matches and 18207 players, which is exactly 10296 + 4180 + 1337 + 1255 + 6886. Five match files with three different row shapes are normalized into one frozen Match dataclass by a dispatching row-mapper:

Kind	Files	Columns it reads
`standard`	Brasileirao, Copa do Brasil, Libertadores	`datetime`, `home_team`, `away_team`, `home_goal`, `season`, `round`
`extended`	BR-Football-Dataset	`date`, `home`, `away`, `tournament`, plus corners/shots kept in an `extras` dict
`historic`	novo_campeonato_brasileiro	`Data` (DD/MM/YYYY), `Equipe_mandante`, `Gols_mandante`, `Arena`, `Vencedor`

Team names are matched through a normalized() function that strips accents, lowercases, and removes state suffixes via a regex listing all 27 Brazilian UFs plus South American country codes — so São Paulo-SP and Sao Paulo compare equal. Dates go through a multi-format parse_date. Both are direct answers to the spec's data-quality notes.

What database backend? None. There is no database and no graph store — no SQLite, no Neo4j, no networkx, no index of any kind. SoccerData holds self.matches: list[Match] and self.players: list[dict], and every query is a linear scan over all 23,954 matches, re-filtered per call. The agent's own summary called it an "indexed CSV data layer"; that is not accurate — defaultdict is imported and never used. At this data size a scan is fast enough that nothing catches it.

On "knowledge graph": the spec's overview asks for "a knowledge graph interface for Brazilian soccer data", but its Required Capabilities section — which is what the pinned REQUIREMENTS.json was authored from — specifies query categories, not storage. The word "graph" appears zero times in REQUIREMENTS.json. So this solution scores 12/12 while containing no graph whatsoever: no nodes, no edges, no traversal. It is a flat table with filters. Every brazil run in this project has been graded the same way, so the comparison between runs is fair — but the checklist does not test the spec's headline framing.

A defect the gate did not catch. Because the source files overlap in coverage (BR-Football 2014–2023, Brasileirao_Matches 2012–2022, novo_campeonato 2003–2019) and load() concatenates without deduplication, the same real-world match is counted two or three times. The agent noticed this partially — it added a guard so standings() uses only the dedicated file, but only for Brasileirão and only for seasons 2003–2019, which is why the 2019 table it printed is historically correct (Flamengo, 38 played, 90 points).

Everything else still double-counts. The run's own MCP handshake demonstrates it:

{"team": "Corinthians", "season": 2022, "venue": "home",
 "matches": 44, "wins": 28, "draws": 11, "losses": 5, "goals_for": 62}

A Brasileirão club plays 19 home league matches a season; even adding home cup and Libertadores ties, ~25 is the ceiling. 44 is the league fixtures counted twice plus the cups. The spec's own worked example says Corinthians' 2022 home record is 19 matches, 11 W, 5 D, 3 L.

The tests do not catch this because they assert accounting identities — matches == wins + draws + losses, points == wins*3 + draws — which stay true when every match is double-counted. The judge flagged the overlap as low/enhancement and scoped it to standings() alone; it is broader than that, and it is a correctness defect in team_statistics, head_to_head, search_matches and aggregate_statistics.

This is a finding about the task's checklist, not a retraction of the run. The 3m19s and the 12/12 stand as measured. But "passes the pinned requirements" and "returns correct answers" are not the same thing, and here they diverge.

Slowest logged pass — 64 minutes of Objective-C


Stack	Claude Opus 5, objc, prompt=neutral
Experiment	`adrianco/experiment-46-opus5-brazil` rep1
Duration	64.0 min · 175 turns · 174 tool calls · $31.31
Result	12/12 requirements · coverage 1.00 · `code_quality` 1.00

A different failure mode entirely: this one isn't gold-plating, it's the language. Nine thousand lines of .m and .h — a hand-rolled CSV parser, match/player models, a query engine and an MCP server — because Objective-C offers no ecosystem to lean on here. The tool mix says it plainly: 59 Edits, 42 Writes, 54 Bash, and only 2 Reads. It was not exploring. It was typing.

The cleanest way to see the cost is to hold the model and task fixed and vary only the language. Opus 5 ran all 13 on this task, and every one scored 1.00:

	clojure	python	rust	swift	java	go	cpp	erlang	c	csharp	elixir	objc
min	5.7	16.1	38.2	39.4	41.4	42.2	42.3	52.8	53.2	56.4	58.3	64.0
$	2.55	9.38	20.08	16.87	20.18	22.35	17.50	23.01	24.11	39.05	33.61	31.31

Same model, same spec, same 1.00 — an 11× spread in wall clock and 15× in cost, decided entirely by the language. (TypeScript is omitted: it passed in 59.9 min but recorded $0.00, a telemetry gap, not a free run.) This is the single strongest illustration of retort's premise that the stack is the unit of measurement, not the model.

3. `py-catalog-reservations` — the modify-existing task

Source: tasks/py-catalog-reservations/ · 18 runs

The other two tasks build from an empty directory. This one ships a working 111-line catalog/ package (models, store, loans, service facade) and its existing test suite, then asks for a reservations feature: reserve only when zero copies are available, FIFO ordering, automatic fulfilment when a copy is returned, cancellation, all exposed through the facade — without breaking the six tests that already pass (a no_regression gate).

Built for the graphify tooling factor: it needs comprehension of code the agent did not write.

Fastest logged pass — 72.8s


Stack	Claude Opus 4.8, `tooling=none`, python
Experiment	`adrianco/experiment-44-graphify-catalog` rep1
Duration	72.8 s · 15 turns · $0.43 · 302,025 tokens
Result	11/11 requirements · 14 tests pass (6 existing + 8 new) · coverage 0.99

[Read] TASK.md
[Read] catalog/models.py, catalog/store.py, catalog/loans.py,
       catalog/service.py, tests/test_catalog.py, conftest.py   # read EVERYTHING first
[TEXT] "Now I'll implement the reservations capability."
[Edit]  catalog/models.py          # add the Reservation type
[Write] catalog/reservations.py    # new service
[Edit]  catalog/service.py  (×2)   # wire into the facade
[TEXT] "Now the tests for the new behavior:"
[Write] tests/test_reservations.py
[Bash] python -m pytest -q     -> command not found: python
[Bash] python3 -m pytest -q    -> .............. [100%] 14 passed in 0.01s
[TEXT] "All 14 tests pass — the 6 existing tests plus 8 new reservation tests."

Note the shape versus the greenfield task: six reads before the first edit. On a modify-existing task the agent front-loads comprehension, then edits surgically — models.py and service.py are edited, not rewritten. That difference is the whole reason this task exists.

Slowest logged pass — 4.5 min, and free


Stack	Qwen3-Coder-Next 80B local (Hermes + oMLX), `tooling=beads`, python
Experiment	`adrianco/experiment-45-graphify-local-catalog` rep1
Duration	4.5 min (3.7×) · 1,000,059 tokens · $0.00
Result	11/11 requirements · coverage 1.00

The third failure mode is not a failure at all. This run is 3.7× slower than the cloud record and costs nothing — it ran on a laptop. For a task you run once, $0.43 and 73 seconds wins; for one you run in a loop, the free option changes the arithmetic. It is the clearest case for reading the cost column next to the clock.

One pattern worth flagging inside this experiment: the three slowest passing runs are all tooling=beads (4.5, 4.3, 4.2 min), while every none and graphify run came in faster (2.6–3.3 min). That matches the README's factor analysis, where beads measurably adds cost. At n=3 per arm it is a consistent ordering rather than a proven effect — but it is the direction the tooling factor exists to detect.

The `python` vs `python3` stumble — found here, since fixed

All three record holders — two different vendors, three different models — ran python, got command not found, and retried with python3. Each paid a step for it.

That is a property of this machine (macOS ships python3 only) being charged to the model as agent work. Writing this page is what made it visible, because it is the one place three runs sit side by side.

Pulling the thread found two bigger problems behind it. pip install was a coin flip: against a Homebrew interpreter it fails with externally-managed-environment, so whether an agent could install a dependency at all came down to whether it happened to build a venv first — some did (Fable 5 above), some wrote stdlib-only code instead (Terra above). That is a difference in the stack, silently attributed to the model. And the scorer built a different interpreter than the agent's: it reuses a venv if the agent shipped one, otherwise it creates a throwaway — so a suite could be written against one interpreter and graded on another.

Retort now provisions a venv into every python workspace before the agent starts, with python, pip and pytest already on PATH, and the scorer reuses that same venv. Provisioning happens outside the timed window, so this removes a turn without adding time. Python runs from before and after this change are not turn-count comparable — it is logged in experiments-blog.md with the other harness changes that moved numbers.

The general lesson is the one this project keeps relearning: a benchmark number always includes some of the bench.

The Retort Experiment Index: What Was Measured, and What the Harness Was Doing at the Time

Adrian Cockcroft — Thu, 30 Jul 2026 23:15:42 +0000

This is a snapshot of an auto-generated analysis report from https://github.com/adrianco/retort/blob/main/experiments-blog.md

Published 2026-07-30 · updated 2026-07-30 — Adrian Cockcroft

An index, not an argument. Retort has run 1,014 scored runs across 56 experiment groups and 13 languages; this page says what each group was for, where to read the detail, and — the part that matters most for reading an old number — what the harness itself was doing at the time. Several published figures moved because the tooling was fixed, not because a model changed.

Detail lives in docs/past-experiments.md. Current recommendations: optimal-blog.md and the machine-readable optimal.json. What the agents are asked to build, and the fastest and slowest passing run for each: tasks-blog.md. What's next: docs/future-experiments.md.

The categories

Category	The question	Experiments	Written up in
Cloud models	Which frontier model, at what cost and reliability?	`exp-1` (first grid: opus vs sonnet, 6 languages) · `exp-3/4/5/6/8` (Opus 4.6 → 4.7 → 4.8) · `exp-7` (fast mode as a factor) · `exp-10` (Fable 5) · `exp-15` (Sonnet 5) · `exp-46` (Opus 5, all 13 languages × both tasks) · `exp-48` (Fable 5 gap-fill) · `exp-53` (Codex / GPT-5.6 — first non-Claude lineage) · `exp-55` (Terra vs Opus 5 at matched thinking levels)	model, versions
Local models	Can a laptop do this, and what does it take?	`exp-12` (Qwen2.5-Coder 7B via Ollama — first local run, n=1) · `exp-16/17/18` (Qwen3-Coder → Hermes → the lcm context engine) · `exp-20` (first all-9-language local sweep) · `exp-22/23` (Qwen-Next 80B, Devstral) · `exp-27/28` (sampling factorial, then the re-baseline) · `exp-29/30/33/36/37/38` (the 80B, language by language) · `exp-47` (gpt-oss-20b — first non-Qwen local lineage)	harness, model
Config levers	Same model, different stack — what actually moves?	`exp-34/35` (lcm `context_threshold` — the biggest local lever) · `exp-27` (temperature / top_p / top_k / repetition_penalty, Res IV) · `exp-24` (KV prefix cache — null: these runs are generation-bound, not prefill-bound) · `exp-16` (context 64K → 128K → 256K) · `exp-49` (thinking level)	optimal, harness
Task difficulty	Routine REST API vs a 12-capability MCP server	`exp-2` (introduces the hard task) · `exp-25/26` (hard task, local 35B) · `exp-31/39` (hard task, 80B — config-invariant) · `exp-50` (re-test with the turn cap removed)	model
Prompt / method	Does prescribing BDD/TDD/ATDD change the outcome?	`exp-13/14` (cloud, hard task) · `exp-19` (local 35B) · `exp-32` (local 80B — does a weaker model need the scaffolding more?)	prompt
Agent & tooling	What drives the run, and what does it get to use?	`exp-11` (Gemini — first cross-agent design; scaffolded, never run) · `exp-53` (codex, where the agent factor actually landed) · `exp-44/45` (graphify knowledge-graph, frontier and local arms) · `beads` throughout	optimal
Self-repair	Does a second attempt with the evaluator's feedback help?	`exp-21` (35B) · `exp-41` (80B, iteration 2) — an inline second chance now runs by default, at half credit	model

Two candidates were rejected before running: Ornith-1.0-35B (vision-optimized, agent-hostile sampling) and Poolside Laguna XS 2.1 (architecture not in mainline serving). Both are documented with their gate-probe evidence at the end of docs/past-experiments.md, because a candidate ruled out cheaply is still a result.

Harness changes that moved published numbers

Check this before comparing a number from one experiment against another. None of these involved a model changing.

Era	Change	What it moved
exp-17 → exp-27	Three unrecorded stack variables at once — playpen under `/var` (writes silently refused: 41/48 runs in exp-27, 6/6 in exp-26), oMLX `temperature: 1.0`, and context silently 128K while both the config and provenance read 256K	Every Hermes-era local result in this range is an understated floor, not a measurement. The exp-28 re-baseline supersedes them. It also overturned the "niche-language wall" — though exp-38 later confirmed clojure/csharp/elixir are genuinely 0.00 on a fixed stack.
exp-34 → exp-38	lcm `context_threshold` 0.35 → 0.9	TypeScript on the 80B went 0.33 → 1.00. A config lever, not a capability.
exp-43	Test-command exit code as the universal pass signal; 900s scorer timeout; process-group reaping	C/C++/Swift were being scored 0 for unparsed output, and leaked servers squatted ports and false-failed later cells.
exp-46	Run timeout 60 → 120 min	Opus 5 is 3–5× slower and brazil cells average 47 min — the old wall was cutting off work in progress.
exp-46	pytest exit-code fallback	A project whose 239 tests all passed scored `test_coverage=0` because its own pytest flags suppressed the summary line.
exp-49	`effort` became a factor — and `aggregate` was silently dropping factor columns it didn't know	Everything before exp-49 ran at each CLI's default thinking level, uncontrolled; and all 63 exp-49 runs first landed in `master.db` without the very factor they were built to test.
exp-50	Hermes turn cap (30) never reconciled with the declared 200	Local runs could be truncated mid-work and scored as model failures.
exp-53	Codex telemetry + per-token pricing	Codex runs recorded 0 tokens, 0 turns and \$0 until the parser matched the real CLI event shape. Costs are now priced from published rates for every vendor, since Claude Max reports a price it doesn't bill and Codex reports nothing at all.
exp-55	`xhigh` was missing from retort's effort levels	exp-49's "full sweep" had skipped a level that exists in both CLIs.
ongoing	Attribution — local runs wrote a blank `model`; experiment labels are now `<githubid>/exp#`	~250 rows are identified by experiment slug rather than model id, and a stack predicate that enumerated local models by exclusion once counted exp-47's gpt-oss runs as the 35B's.
post-exp-55	Python workspaces get a provisioned venv (`python`, `pip` and pytest on PATH)	Every earlier python run inherited a host with no `python` — only Homebrew's `python3`. The fastest recorded run of all three tasks, across two vendors and three models, spent a turn on `command not found` and a retry. Dependency installs were also luck: `pip` against a Homebrew interpreter fails `externally-managed-environment`, so whether an agent could install anything depended on whether it happened to build its own venv. And the scorer built a different interpreter than the agent's. Python runs before and after this are not turn-count comparable.

The shape that keeps recurring: a failing model and a broken harness are identical in the scores. retort diagnose exists to separate them, and the standing rule is that a surprising zero gets reproduced by hand before it gets published. The full post-mortems are in Historical: harness bugs & the local re-baseline saga.

Conclusions that were retracted

Kept visible, because the corrections are more useful than the originals.

"Opus 5 is the only model that clears the hard task everywhere" (exp-46) — withdrawn by exp-48. Fable 5 had never been run on 9 of the 13 languages; when it was, it cleared them at roughly half the price. An unrun cell and an unpassable cell look identical in a table.
"Newer Claude versions take steadily more turns" — corrected by exp-49. Three generations sit at ~9–13 turns; the apparent trend came from one old experiment's figures, which don't replicate.
"Thinking level explains the cross-version cost gap" — retracted by the experiment run to test it. The single run that motivated it (33 turns at max) did not replicate: 14, then 18.
"gpt-oss matches the 80B on Go at 3.6× the speed" (exp-47, n=3) — removed at n=5, where Go fell to 0.80.
"The local hard-task wall is a turn cap" (exp-50's whole premise) — wrong. api_calls is 1:1 with turns, not 3:1, so nothing had been truncated. The wall is real.

Four of those five came from reading a single run as a result.

The slowest successful run

tasks-blog.md shows the fastest and slowest passing run for each task. The slow end is the more instructive one, because nothing fails there either.

Both of these are the same task — rest-api-crud, python, a books CRUD API — and both score requirement_coverage = 1.00.

	Fastest (Fable 5, `effort=low`)	Slowest (Opus 5, `effort=max`)	Ratio
Wall clock	44.5 s	45.6 min	61×
Cost	$0.86	$24.96	29×
Tokens	105,745	18,448,183	174×
Turns	7	94 (256 tool calls)	13×
Source + test lines	194	2,351	12×
Tests	6	104	17×
`requirement_coverage`	1.00	1.00	—
`test_coverage`	0.98	1.00
`maintainability`	0.27	0.85
`token_efficiency`	1.00	0.0019

The slow run is not a failure, and not slop. It scores better on maintainability, idiomaticity and coverage. It built a seven-module package, ran ruff on itself unprompted, spawned a subagent, and performed mutation testing — on a books CRUD API. Every one of the judge's five findings is an info describing scope beyond the spec: a PATCH endpoint, filtering/sorting/pagination, a hand-written 304-line OpenAPI 3.0 document served at GET /openapi.json, WAL journaling and busy-timeouts on SQLite, and validation for NUL bytes and SQLite integer bounds.

Nobody asked for any of it. The task asked for five endpoints, a health check, and at least three tests.

What actually goes wrong is that the gate cannot see the difference. requirement_coverage is 1.00 for both, so pass-proportion — the headline metric of this whole project — rates a 44-second $0.86 run and a 46-minute $25 run as identical. That is correct by its own definition and still the most misleading number on the page unless you read cost beside it. token_efficiency is the response that does separate them, by a factor of 534.

The dial also buys variance, not just cost. Within exp-55, the same Opus 5 python cell run twice:

`effort`	rep1	rep2	mean cost
low	1.9 min	2.2 min	$0.76
medium	3.0 min	3.2 min	$1.04
high	6.2 min	5.6 min	$1.68
xhigh	15.3 min	8.6 min	$2.55
max	24.5 min	45.6 min	$19.21

At low the two replicates agree within 15%; at max they differ by 1.9×, and cost by 1.9×. Ten of ten cells score 1.00. On this task the thinking dial is a 25× cost multiplier that buys nothing measurable and becomes unpredictable at the top. At n=2 the variance claim is suggestive, not established — but the cost is not in doubt. Meanwhile GPT-5.6 Terra ran the same ten cells between 1.4 and 4.0 minutes for $0.10–$0.33, also 1.00 throughout.

One more thing the log shows: the slow run spent its opening turns probing the host interpreter and discovered that the preinstalled FastAPI was broken (pydantic v1 cannot build models on Python 3.14), then chose Flask instead. That is the environment leaking into the measurement again — and it is the same class of problem as the missing python, now addressed by provisioning a clean venv per workspace.

Reading the numbers

Pass-proportion is the fraction of runs that fully implement the spec — every requirement on a pinned checklist, with tests that actually execute, verified by an independent LLM judge. A run that misses one requirement scores 0, not 0.9. Read a cell as the probability that one unattended run comes out completely correct.

Three things to check before trusting one: n (many cells are 1–3 runs, and results at n=1 have reversed repeatedly in this project), the judge (recorded per experiment in master.db since exp-53; requirement_coverage is one model's opinion, and only pools across experiments graded the same way), and the language mix (an all-language average silently compares models that were run on different sets of languages — the per-language matrix is the like-for-like view).

Testing Claude from 4.7 to Opus 5 and Fable 5 - Why Are Newer Models Slower?

Adrian Cockcroft — Mon, 27 Jul 2026 21:56:21 +0000

What follows is re-generated automatically whenever relevant experiments finish. This is the current snapshot of https://github.com/adrianco/retort/blob/main/versions-blog.md

The Unit of Choice Is Model × Thinking Level, Not Model

Published 2026-07-25 · updated 2026-07-30 — Adrian Cockcroft

For a year the question has been "which model should I use?" That question is underspecified. Every frontier model now has a thinking-level dial, and on the evidence below the dial moves cost more than the model choice does — while moving the result not at all.

One task (a "bookshop" REST API), one prompt, one judge throughout. First four Claude versions × five thinking levels, three replicates each — 21 cells, 63 runs, zero failures. Then the same sweep against OpenAI's GPT-5.6 Terra, to see whether any of it is a Claude peculiarity. It isn't, but the two vendors turn out to have built the dial to do different things.

Every cell passed. The bill spans 16×.

model	effort	turns	tokens	cost	seconds
Opus 4.8	default	9.3	245 K	$0.42	79
Opus 4.8	low	10.3	272 K	$0.57	83
Opus 4.8	high	14.0	393 K	$0.60	117
Opus 4.7	low	7.3	273 K	$0.61	56
Opus 4.8	medium	12.7	348 K	$0.66	97
Opus 4.7	medium	8.7	329 K	$0.67	65
Opus 4.7	default	10.3	397 K	$0.75	91
Opus 5	low	15.7	432 K	$0.75	114
Opus 4.7	high	11.7	451 K	$0.79	97
Opus 5	medium	18.0	436 K	$0.86	142
Fable 5	low	8.0	131 K	$0.91	49
Opus 4.8-fast	default	11.5	204 K	$0.95	125
Opus 4.7	max	14.0	588 K	$1.13	171
Fable 5	high	13.3	369 K	$1.13	109
Fable 5	medium	10.3	245 K	$1.16	75
Fable 5	default	13.0	352 K	$1.21	107
Opus 5	default	31.0	950 K	$1.38	270
Opus 4.8	max	15.3	628 K	$1.45	326
Opus 5	high	30.7	1,072 K	$1.77	399
Fable 5	max	13.7	428 K	$2.28	241
Opus 5	max	43.7	2,337 K	$6.75	1,110

Every row is pass-proportion 1.00. Not "mostly" — all 63 runs fully implemented the spec, tests executing, verified by an independent judge. The cheapest configuration costs $0.42 and the dearest $6.75: a 16× spread buying literally nothing on this task.

The dial is a bigger lever than the version

Compare the two axes directly.

Within one model, changing only the thinking level:

model	low → max	cost multiple
Opus 4.7	$0.61 → $1.13	1.9×
Opus 4.8	$0.57 → $1.45	2.5×
Fable 5	$0.91 → $2.28	2.5×
Opus 5	$0.75 → $6.75	9.0×

Between models, holding the thinking level fixed:

effort	cheapest → dearest	cost multiple
low	$0.57 (4.8) → $0.91 (Fable 5)	1.6×
default	$0.42 (4.8) → $1.38 (Opus 5)	3.3×
max	$1.13 (4.7) → $6.75 (Opus 5)	6.0×

Two things fall out. At low, the four model generations nearly converge — 1.6× apart, and all perfect. Most of what looks like "the new model is expensive" is the new model thinking harder by default, not the weights being pricier. And the two axes interact: Opus 5 responds to the dial 9×, where the older models respond 2×. The newest model is both the most expensive at its default and by far the most sensitive to the setting.

The practical consequence is concrete: Opus 5 at low ($0.75) is cheaper than Fable 5 at its default ($1.21) and half the price of Opus 5 at its own default. If you want the newest model, the dial — not the version — is what decides your bill.

Why cost tracks turns, and turns tracks the dial

The mechanism is unchanged from earlier versions of this post, and it is the part that has held up under every re-measurement.

Cost follows the number of agentic turns, not per-turn speed. Seconds-per-turn is roughly flat; what varies is how many steps the loop takes. And token cost grows faster than turns because every turn re-reads the whole accumulated conversation from cache. Opus 5 at max: 43.7 turns, and 2.34 M tokens against roughly 33 K actually generated. Cache reads are individually cheap, which is why the bill is $6.75 and not absurd — but they dominate the totals, and they scale with roughly the square of the step count.

So the dial's mechanism is simple: more thinking → more steps → quadratically more re-read context. That is why max costs 9× on the model that takes the most steps to begin with.

What the dial does not buy

Nothing measurable here. Every level, every model: 1.00.

That is a real finding and a bounded one. This is one language on a routine task, where the ceiling was already saturated — with everything at 1.00 there is no headroom in which a difference could show. Thinking level may well earn its cost on genuinely hard work, and nothing in these 63 runs tests that. It is the obvious next experiment.

The honest summary: on work your stack already handles, the dial is pure expense. On work it doesn't, we don't know yet.

The version story, corrected

An earlier version of this post described a smooth generational climb — Fable 5 10.7 turns → Opus 4.8 17.3 → Opus 5 36.0 — and built a narrative on it. Measured in one batch, with the thinking level held at the historical default, three of those generations are indistinguishable:

model	in-batch turns (n=3)	previously published	ratio	source of the old figure
Opus 4.8-fast	11.5	11.3	1.02×	exp-7
Fable 5	13.0	10.7	1.21×	exp-10
Opus 5	31.0	36.0	0.86×	exp-46
Opus 4.7	10.3	17.2	0.60×	exp-6
Opus 4.8	9.3	17.3	0.54×	exp-6

Three of five replicate closely. The two that don't both come from exp-6, the oldest source — so the "gradual climb" was substantially one old experiment's inflated middle. Corrected: 4.7, 4.8 and Fable 5 all sit around 9–13 turns; Opus 5 alone takes ~2.7× that. Not a trend across versions — one model behaving differently.

(A note on how that was found: the cross-version comparison had been assembled from experiments run months apart on different harness versions. The fix was to re-run every arm in a single batch. If you take one methodological point from this post, take that one.)

A second vendor: the dial does something completely different at OpenAI

Everything above is Claude. If "the dial costs more than the model" were really about thinking level rather than about one vendor's tuning, it should show up elsewhere — so the same experiment was run against OpenAI's GPT-5.6 Terra, driven by codex exec.

Terra is the right comparison because GPT-5.6's three tiers price onto the Claude ladder rather than against it: Luna (\$1/\$6) ↔ Sonnet · Terra (\$2.50/\$15) ↔ Opus · Sol (\$5/\$30) ↔ Fable. Opus is the strongest Claude pick in most cells today, so Terra is its price-peer. Both CLIs expose exactly the same five levels — low, medium, high, xhigh, max — so the dial can be set identically on both sides. Python and Go, n=2, 20 cells. Every one scored 1.00.

effort	Terra	Opus 5	ratio	Terra time	Opus 5 time	Opus 5 turns
low	$0.19	$0.81	4.3×	112 s	136 s	16
medium	$0.15	$1.14	7.6×	105 s	222 s	20
high	$0.18	$1.84	10.2×	132 s	401 s	30
xhigh	$0.22	$4.63	21.0×	169 s	909 s	62
max	$0.35	$14.21	40.6×	254 s	1669 s	92

Terra's most expensive setting is half the price of Opus 5's cheapest. At matched max the gap is forty-fold — for an identical, independently-judged 1.00 on the same task.

But the interesting column is the last one, because it shows the two dials are not the same instrument. Terra's agent steps stay flat across the entire range — 10 to 19 — no matter where the dial is set. Opus 5's climb from 16 to 92. Turning the dial up on Opus 5 buys more agentic iteration; turning it up on Terra appears to buy deeper reasoning inside a roughly constant number of steps.

That distinction is exactly the mechanism this post has been building toward. Cost tracks turns, and turns re-read the accumulated conversation, so a dial that adds steps compounds and a dial that doesn't, doesn't. The per-token price difference between the two models is only about 2.4×; the rest of the 40× is turns. The thesis survives contact with a second vendor — but the vendors have made opposite choices about what the dial should do.

There is a second, smaller surprise. OpenAI's flagship tier defaults to the lowest reasoning setting (Sol defaults to low, while the cheaper Terra and Luna default to medium), where Claude's default sits near high. So "newer models think harder by default" — the tempting explanation earlier in this post — is not a law about frontier models. It is a product decision, and two vendors made it in opposite directions.

This is also why the comparison had to set effort explicitly on both sides. default is not a shared operating point: comparing out-of-the-box settings would have compared two companies' packaging choices and called it a model difference, and would have shown a muddled ~9× instead of the clean 4×→40× interaction.

(Getting a trustworthy number out of Codex took four fixes first, none of which were visible without running it: the telemetry parser was written to an event shape the CLI does not emit and recorded **0 tokens, 0 turns; input and output tokens were double-counted against their own cached and reasoning subsets, inflating cost ~490%; a ChatGPT subscription reports **no cost at all, so Codex would have logged \$0 and won every cheapest-stack ranking on an unmeasured number; and Codex's turn.completed fires once per invocation rather than per step, so recording it as "turns" would have placed it at the bottom of this post's own axis looking impossibly efficient. Cost here is list-price-per-token — the same basis Claude's CLI reports and does not bill on a Max plan.)

It isn't a Claude phenomenon

The same cell run on local models, driven by the Hermes agent over oMLX on a 64 GB laptop:

stack	n	turns	replicates
Qwen3.6-35B (local)	3	12.0	10, 8, 18
Qwen3-Coder-Next 80B (local)	3	24.7	44, 17, 13

Placed on the same axis: Opus 4.8 9.3 · Opus 4.7 10.3 · 35B 12.0 · Fable 5 13.0 · 80B 24.7 · Opus 5 31.0. A 35B open-weights model on a laptop sits in the same cluster as three Claude generations, and the 80B sits with Opus 5 well above it. Two vendors, a 20× parameter range, and the same two-group split — which points at how a model was tuned to behave in an agent loop rather than at scale or architecture. With Terra above, that is now three independent lineages saying the same thing: step count is a tuning decision, not a capability tier.

Caveat worth keeping: the 80B's replicates span 44/17/13, a 3.4× spread. A single 80B run tells you very little.

What to actually do

Pick a (model, effort) pair, not a model. The pair spans 16× in cost here; the model alone explains a minority of that.
On routine work, turn the dial down. low cost 1.6× less than default on average and changed no outcome. The CLI default is not the cheap end.
If you want the newest model, run it at low. Opus 5 at low undercuts Fable 5 at its default. Opus 5 at max costs 16× the cheapest cell for the same passing app.
Check what your vendor's dial actually does. On Opus 5 it adds agentic steps and compounds; on Terra it doesn't move step count at all. Same five names, different instrument — so a rule of thumb learned on one model does not transfer.
Don't generalise any of this to hard tasks. Everything above saturated at 1.00. The dial's value, if it has one, lives where the ceiling isn't already hit — and nothing here tests that.

Method: python × bookshop × prompt=neutral, n=3 per cell, judged by an independent Opus 4.8 against a pinned requirement checklist; a run passes only if it implements the whole spec. Cost is list-price-per-token — the basis every metered stack here is recorded on, and one that does not vary with whose subscription happened to pay. Data: master.db, experiments adrianco/experiment-49-versions-cloud (Claude × effort) and adrianco/experiment-55-terra-vs-opus5-effort (cross-vendor, matched effort).

Connecting to my old dev.to account

Adrian Cockcroft — Fri, 24 Jul 2026 16:48:58 +0000

I wrote some dev.to posts when I was working at AWS, and I've lost access to the email that I used to setup the account https://dev.to/adrianco - so I'm continuing with this account https://dev.to/adrianco_54

https://dev.to/aws/scaling-aws-costs-to-match-the-business-f9k

https://dev.to/aws/why-are-services-slow-sometimes-mn3

https://dev.to/aws/if-at-first-you-don-t-get-an-answer-3e85

https://dev.to/aws/what-is-sustainability-transformation-32hi

https://dev.to/adrianco/measuring-energy-usage-5ip

The Stack Under the Model: oMLX, llama.cpp, Hermes, and Why There Are So Many

Adrian Cockcroft — Wed, 22 Jul 2026 22:49:51 +0000

Retort is a framework for comparing coding stacks, with results for versions of Claude and local tests on a 64GB M5Pro for many languages. It was developed with Claude, and I was getting confused by all the components of the harness and stack, so I asked Claude to explain what is out there and how they are being tested. That's what follows:

Most benchmarks answer "which model is best?" Retort insists that's the wrong
unit. A coding result is produced by a whole stack — and the model is only
one layer of it:

language × model × weights-format × serving engine × agent/harness × context engine × sampling × prompt

Change any layer and the numbers move. TypeScript went from failing to 1.00 on
the local 80B by raising one context-engine knob — same model, same weights.
So before the results make sense, you need a map of the layers. This post is that
map: what each piece is, where it came from, what competes with what, how they
stack, and why the zoo is so crowded. Then it explains the metaharness — the
part of Retort that turns the harness itself into a variable you can measure.

If you've seen oMLX, llama.cpp, Hermes, GGUF, omp, lcm, or
OpenRouter fly past in the other blogs and nodded along without quite knowing
what they are, start here.

The stack, top to bottom

Read this from the bottom up — each layer sits on the one below it:

┌──────────────────────────────────────────────────────────────┐
│  PROMPT / METHODOLOGY   "write tests first" · BDD · terse    │  ← how you ask
├──────────────────────────────────────────────────────────────┤
│  AGENT / HARNESS        claude-code · Hermes · gemini · omp  │  ← the ReAct loop:
│                         opencode                             │    read → edit → run → repeat
├──────────────────────────────────────────────────────────────┤
│  CONTEXT ENGINE         lcm (compaction) · sampling knobs    │  ← what the model "sees"
├──────────────────────────────────────────────────────────────┤
│  SERVING ENGINE         oMLX · llama.cpp · Ollama · cloud API│  ← turns weights into tokens
├──────────────────────────────────────────────────────────────┤
│  WEIGHTS + FORMAT       safetensors · GGUF · MLX  (a model   │  ← the numbers on disk
│                         at some quantization, e.g. 4-bit)    │
├──────────────────────────────────────────────────────────────┤
│  HARDWARE               Apple Silicon · NVIDIA · a cloud GPU │  ← what runs the math
└──────────────────────────────────────────────────────────────┘

"Which model is best" only asks about one band in the middle. Retort measures the
column. The rest of this post walks the layers where the confusing names live —
weights, serving, agent, context — and then the metaharness, which makes the
agent layer a factor you can sweep.

Layer 1: weights and their formats — safetensors, GGUF, MLX

A model is a big pile of numbers (weights). How those numbers are stored on
disk is a surprising source of fragmentation, because the file format is tied to
the engine that reads it.

safetensors — the neutral, framework-agnostic format models are usually published in (it replaced the old, unsafe pickle format). Both of the local formats below are usually converted from safetensors.
GGUF — the format used by llama.cpp. A single self-contained file that bundles the weights, the tokenizer, and metadata, at a chosen quantization (e.g. Q4_K_M ≈ 4-bit). Designed to run well on CPUs and non-CUDA GPUs.
MLX — Apple's format, produced for Apple's MLX framework. MLX is Apple's array/ML library for Apple Silicon; it exploits the Mac's unified memory (CPU and GPU share the same RAM), which is why a 64 GB Mac can hold a 42 GB model that would otherwise need a datacenter GPU.

Quantization is the other axis here: the same model shipped at 4-bit is a
quarter the size of 16-bit and runs far faster, at some accuracy cost. When you
see Qwen3-Coder-Next-4bit (≈42 GB) that's "the 80B model, MLX format, 4-bit."

Why two local formats (GGUF and MLX) for the same models? Because each is glued
to a different engine, which is the next layer.

Layer 2: the serving engine — oMLX vs llama.cpp vs Ollama vs the cloud

The serving engine is the program that loads the weights and turns your prompt
into tokens, usually exposing an OpenAI-compatible HTTP endpoint (POST /v1/chat/completions) so anything that speaks that protocol can drive it. This
is the layer with the most competitors, because it's where hardware, format, and
performance all collide.

llama.cpp — the foundational open-source local engine, written in C/C++ by Georgi Gerganov. It reads GGUF, runs on almost anything (CPU, CUDA, and Apple's Metal GPU API), and defined the format the rest of the ecosystem converged on. Most local tooling is either llama.cpp or a wrapper around it. Its server binary is llama-server; Retort's llamacpp backend drives it with --jinja so the model's own chat/tool template is applied.
Ollama — a friendly daemon + model registry built on top of llama.cpp (ollama run qwen). It made local models one-command easy and is how most people first run one. Retort's earliest local attempt used it (see below) and moved on.
oMLX — a local server for MLX-format models on Apple Silicon, OpenAI-compatible on 127.0.0.1:8080. It's the MLX-world counterpart to llama.cpp: same job (weights → tokens over HTTP), different format and a Metal/MLX-native path tuned for Macs. Retort restarts it at each experiment boundary and records the effective settings in provenance.json.
cloud APIs — for Opus, Gemini, GPT, etc. you don't run a serving engine at all; a provider runs it and you rent tokens. Same OpenAI-shaped protocol, so the layers above don't care whether the tokens came from your laptop or a datacenter.

What competes with what: llama.cpp (GGUF) and oMLX (MLX) are the two serious
local engines, split mostly by format and hardware heritage — GGUF/llama.cpp is
the cross-platform default, MLX/oMLX is the Apple-Silicon-native path. Ollama
competes on ease, not capability (it is llama.cpp underneath). Retort now
supports both local engines via a serving.backend: omlx | llamacpp switch,
precisely because neither dominates: some models ship only as MLX, some only as
GGUF, and some new architectures land in one engine months before the other.
That "which engine has this architecture yet?" gap is a recurring reason a
promising model can't be tested — it's a serving-layer problem, not a model one.

Layer 3: the agent / harness — claude-code, Hermes, gemini, opencode, omp

A served model just answers messages. To build software it needs a loop that
reads files, writes edits, runs the tests, reads the failures, and tries again —
the agentic loop (often "ReAct": reason → act → observe, repeat). That loop is
the agent / harness, and it's a real, swappable layer: the same model behaves
differently under different harnesses because they differ in how they present
tools, when they retry, and how they manage the conversation.

The harnesses Retort supports:

claude-code — Anthropic's Claude CLI. Retort's default, and it doubles as the impartial spec-gate judge that grades every run (so an independent model scores all agents fairly, even non-Claude ones).
gemini — Google's Gemini CLI, used for gemini-* models. The agent usually follows from the model id: list a Claude id and you get claude-code, a Gemini id and you get the Gemini CLI.
Hermes — the harness Retort uses to drive local models (via oMLX). It's the featured local path: Hermes runs the ReAct loop and manages context (its context engine is lcm, below) while oMLX serves the tokens. Note: here "Hermes" is the agent, not a model — the model underneath is Qwen.
opencode — another open agent harness Retort can drive.
omp — oh-my-pi, an early community-contributed local harness (it talked to llama.cpp, not Ollama). It's documented as a legacy path — the first, honest dead-end on a 24 GB Mac before the Hermes + oMLX stack became the featured one.

Why the agent is its own layer (and its own zoo): each vendor ships the
harness tuned for its own model, and open harnesses exist to run any model. They
genuinely differ — a weak model can pass under a forgiving harness and fail under
a strict one — which is exactly why "which harness?" deserves measurement rather
than assumption. That's what the metaharness (last section) is for.

Layer 4: the context engine and sampling — the knobs that quietly decide everything

Two smaller layers sit between the agent and the model, and they've caused more
wrong conclusions in this project than any model choice.

The context engine (lcm) decides what the model actually sees. An agentic coding session quickly grows past the model's context window, so the engine compacts older turns to make room. Hermes's engine is lcm, and its key knob is context_threshold — the fraction of the window at which it compacts. At the default 0.35 it compacts at ~92K tokens and the 80B intermittently stalls; raised to 0.9 ("full context") the same model runs Python, Go, and TypeScript at 1.00. Same weights, same engine — a different threshold. This is the single clearest example of a non-model layer deciding the result.
Sampling — temperature, top_p, top_k, repetition_penalty: how randomly the next token is picked. Retort learned the hard way that the oMLX default temperature = 1.0 roughly halved local reliability, and that a repetition_penalty ≠ 1.0 (even a value a model's own card recommends) can derail the multi-turn tool loop into stalls. These are recorded per run and held fixed on purpose.

The lesson threaded through the whole project: a set-but-unverified knob in one
of these quiet layers produces confident, wrong results. Retort's provenance
records the effective value of each, because the config file's value and the
value the model actually ran at have diverged more than once.

So why are there so many?

The zoo isn't accidental. Several independent forces each spawn alternatives:

Force	What it splits	Example
Hardware lineage	serving engine + format	Apple-Silicon/unified-memory → MLX/oMLX; cross-platform/CUDA → GGUF/llama.cpp
Format lock-in	which engine can even load a model	a model published only as MLX can't run on llama.cpp, and vice-versa
New architectures	which engine supports them first	a brand-new model's arch may land in one engine months before the other — the usual reason a candidate is "blocked"
Ease vs control	wrapper vs raw engine	Ollama (easy) wraps llama.cpp (control); one command vs every flag
Vendor vs open	the agent/harness	claude-code/gemini ship per-vendor; Hermes/opencode/omp run any model
Cost & privacy	local vs cloud	a 64 GB Mac runs the 80B for \$0 and offline; the cloud rents frontier capability per token
Quantization	size vs accuracy	the same model at 4-bit / 6-bit / 8-bit trades RAM and speed against quality

None of these layers dominates, so none of the others disappear. That's the
whole reason Retort measures the stack instead of the model: the interesting
question is usually "which combination is best for my language, task, and
budget?" — and the answer moves as new engines, formats, and harnesses land.

Where Retort's featured stacks sit today: the free local path is
Qwen3-Coder-Next 80B, MLX 4-bit, served by oMLX, driven by Hermes with lcm at
context_threshold 0.9 on a 64 GB Mac — Python/Go/TypeScript at 1.00 for \$0.
The cloud path is Claude Opus 4.8 via claude-code for the languages and hard
tasks the local stack can't yet clear. Same protocol, different columns.

The metaharness: making the harness itself a variable

Everything above treats the harness as a fixed choice per run. But the harness is
a bundle of strategies — and those strategies plausibly move results as much as
the model does. The retort_metaharness layer
(a documented, experimental side-branch) makes the agentic-orchestration harness
a first-class factor you can sweep, so a statistical analysis can say how much
of any lift is the harness versus the raw model.

It doesn't reinvent Retort — it composes the same design generator, ANOVA, and
Pareto engine — but it adds three new factors on top of the usual language/model:

`harness_config` — the orchestration strategy (the headline factor)

level	what it does
base-ReAct	plain single-agent reason→act→observe loop. The control: no tricks.
self-consistency-N	sample N independent solutions and majority/judge-select the best. An accuracy lever that costs N× the tokens (default N=5).
routed	a cheap model drafts; a frontier model takes over only on low-confidence steps. A cost lever — aims for comparable reliability at lower \$.
+agenticow-memory	ReAct plus a copy-on-write memory that persists agent state across steps/replicates — tests whether memory changes the outcome.
+darwin-evolved-genome	ReAct driven by a harness "genome" (prompt + tool policy) tuned by an evolutionary loop — tests whether evolution moves the needle.

`scaffold` — the reasoning structure wrapped around each attempt

none · plan-and-solve (plan first, then execute) · reflexion (attempt,
self-critique, retry).

`model` — the raw model, spanning cheap→frontier

deepseek-v4-pro · glm-5.2 · opus-4.8 · gpt-5.2, reached through
OpenRouter (a unified API that routes one request format to many providers),
chosen to spread from cheap to frontier so the analysis can separate a model
effect from a harness effect.

What it would actually test

Because a fractional-factorial design crosses all of these at once (rather than
changing one thing at a time), the ANOVA can attribute the variance in a metric to
model vs harness vs scaffold vs language + their interactions, and report which
effects are cleanly estimated versus confounded. Concretely, it answers questions
the model-only grid structurally can't:

Of the lift from +agenticow-memory, how much is the memory branching versus just the underlying model being good?
Does routed actually hold reliability while cutting cost — i.e., does it sit on the accuracy-vs-\$ Pareto front?
Is self-consistency-5 worth 5× the tokens, or a rounding error on a task the model already nails?
Is a reasoning scaffold a real lever or a ritual — and does that depend on model strength? (Retort already found the sibling result that the prompt is a lever only in proportion to how weak the model is; the metaharness generalizes that from prompt to full orchestration.)

The honest caveats: it's cloud-only (OpenRouter, metered — the
self-consistency × frontier × replicates corner gets expensive) and the real
orchestration logic lives in an external solver the adapter shells out to, so
without that solver only a \$0 stub runs. That's why it's a documented
side-branch rather than a headline result — it stays one until a first screening
run shows the harness variance is real enough to promote. The staged plan lives in
docs/future-experiments.md.

Takeaway

There are "so many" harnesses and engines because at least six independent forces
— hardware, format, new architectures, ease-vs-control, vendor-vs-open, and
cost-vs-privacy — each keep their own alternatives alive, and no layer has a
winner that retires the others. The practical consequence is the thesis of this
whole repo: don't benchmark the model, benchmark the stack. The model is one
band in a tall column, and the quiet layers under and around it — the serving
engine, the context threshold, the sampling defaults, and the orchestration
strategy — routinely decide the result. The metaharness is Retort turning the last
of those, the orchestration layer, from an assumption into a measurement.