I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.
I never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.
The result
One sweep. Twenty coding tasks, three harness shapes, real models, $0.99.
| harness | calls/task | score | cost | latency |
|---|---|---|---|---|
| one drafter | 1 | 95% | $0.031 | 2.2s |
| planner → drafter | 2 | 90% | $0.264 | 9.1s |
| planner → two drafters → judge | 4 | 80% | $0.692 | 18.3s |
Adding the scaffolding made it worse and cost 22× more. The four-call panel beat the single drafter on zero of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.
The part I care about more
Here is what the tool actually said about that:
Only 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the harnesses.
The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.
95% versus 80% looks like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.
So the honest reading is narrower and more useful:
- There is no evidence the panel helps on this suite.
- It costs 22× more and takes 8.4× longer, which is measured, not inferred.
- Whether it is genuinely worse needs more tasks than twenty.
Those are three different statements. Most eval tooling collapses them into a ranking.
Why the suite size is the real constraint
It isn't step size — the 15-point gap is three times the 5-point resolution. It's that the two shapes only disagreed on three tasks. Everything else tied, and ties are exactly what a paired test throws away. Twenty tasks is simply too few to generate enough disagreements for any test to work with. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.
This is the same lesson my drift board taught me earlier this week, when four "regressions" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.
How it works
A harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.
Scoring is deterministic: fixed predicates execute the generated code and return a verdict, and no model grades anything. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.
Worth being precise about what that does and doesn't buy. The grading is deterministic — the same output always scores the same. The generation is not: I set no temperature and no seed, and this sweep used one run per config, so a 5-point move between two runs of the same shape sits inside sampling noise. That is an argument for more tasks and more runs, and it is a second reason the tool won't call a winner here.
The comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.
Your key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.
One vendor detail worth writing down, because it cost me an hour: api.openai.com answers the CORS preflight with the right headers and then omits access-control-allow-origin on the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what anthropic-dangerous-direct-browser-access is for. Testing only the preflight with curl -X OPTIONS shows success and is misleading.
Where this sits (so I don't oversell it)
Harness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.
What this cost me
$0.99 and about ten minutes — the sweep runs strictly sequentially, so 60 runs at those latencies is 592 seconds of model time before anything else — to find out that the architecture I had been assuming was better is, on this evidence, not better and definitely more expensive.
One caveat on the 22×: that ratio is at Sonnet 5's introductory pricing, which runs through 2026-08-31. After that the gap gets wider, not narrower.
I would rather know.
Run it yourself: https://egnaro9.github.io/never-touch-ai/sweep.html — draw a harness and sweep it. It runs free on mock substrates with no key at all.
Source: https://github.com/egnaro9/never-touch-ai
Raw result: results/sweep_2026-07-25.json — the numbers above are computed from it, so you can check them.
Deeper write-up: the field note — graph execution model, the sign test, and the two bugs the live run surfaced.
Built by Erik Hill · https://egnaro9.github.io
Top comments (0)