I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.
I never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.
The result
One sweep. Twenty coding tasks, three harness shapes, real models, $0.99.
| harness | calls/task | score | cost | latency |
|---|---|---|---|---|
| one drafter | 1 | 95% | $0.031 | 2.2s |
| planner → drafter | 2 | 90% | $0.264 | 9.1s |
| planner → two drafters → judge | 4 | 80% | $0.692 | 18.3s |
Adding the scaffolding made it worse and cost 22× more. The four-call panel beat the single drafter on zero of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.
The part I care about more
Here is what the tool actually said about that:
Only 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so this suite cannot decide between them — that is a limit of the suite, not a finding about the harnesses.
The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.
95% versus 80% looks like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.
So the honest reading is narrower and more useful:
- There is no evidence the panel helps on this suite.
- It costs 22× more and takes 8.4× longer, which is measured, not inferred.
- Whether it is genuinely worse needs more tasks than twenty.
Those are three different statements. Most eval tooling collapses them into a ranking.
Why the suite size is the real constraint
It isn't step size — the 15-point gap is three times the 5-point resolution. It's that the two shapes only disagreed on three tasks. Everything else tied, and ties are exactly what a paired test throws away. Twenty tasks is simply too few to generate enough disagreements for any test to work with. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.
This is the same lesson my drift board taught me earlier this week, when four "regressions" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.
How it works
A harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.
Scoring is deterministic: fixed predicates execute the generated code and return a verdict, and no model grades anything. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.
Worth being precise about what that does and doesn't buy. The grading is deterministic — the same output always scores the same. The generation is not: I set no temperature and no seed, and this sweep used one run per config, so a 5-point move between two runs of the same shape sits inside sampling noise. That is an argument for more tasks and more runs, and it is a second reason the tool won't call a winner here.
The comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.
Your key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.
One vendor detail worth writing down, because it cost me an hour: api.openai.com answers the CORS preflight with the right headers and then omits access-control-allow-origin on the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what anthropic-dangerous-direct-browser-access is for. Testing only the preflight with curl -X OPTIONS shows success and is misleading.
Where this sits (so I don't oversell it)
Harness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.
What this cost me
$0.99 and about ten minutes — the sweep runs strictly sequentially, so 60 runs at those latencies is 592 seconds of model time before anything else — to find out that the architecture I had been assuming was better is, on this evidence, not better and definitely more expensive.
One caveat on the 22×: that ratio is at Sonnet 5's introductory pricing, which runs through 2026-08-31. After that the gap gets wider, not narrower.
I would rather know.
Run it yourself: https://egnaro9.github.io/never-touch-ai/sweep.html — draw a harness and sweep it. It runs free on mock substrates with no key at all.
Source: https://github.com/egnaro9/never-touch-ai
Raw result: results/sweep_2026-07-25.json — the numbers above are computed from it, so you can check them.
Deeper write-up: the field note — graph execution model, the sign test, and the two bugs the live run surfaced.
Built by Erik Hill · https://egnaro9.github.io
Top comments (9)
This aligns so painfully well with what I’ve seen in practice. We often over-engineer multi-agent setups under the assumption that 'more reflection = better results,' when in reality it just accumulates prompt drift and error propagation. Props for actually measuring it and staying statistically honest about the small sample size!
Thanks — though the honest version is that I didn't stay statistically disciplined so much as get cornered by it. I built the thing to justify the harness, and the only reason the small-sample caveat is in the post is that the tool refused to give me a verdict I could have quoted.
Someone else in this thread has since pointed out that the sweep ran one sample per config with no seed, so the sample problem is worse than I published. Rerun's queued.
The paired sign test with ties excluded is the right instrument for twenty tasks, and the diagnosis, three discordant tasks can't reach significance even if one side sweeps all of them, is the correct read of what a leaderboard would have hidden. Worth pushing on one thing though: the fix is more tasks, not better statistics assumes the seventeen ties are real ties. With one run per config and no seed, some of those seventeen might be ties only because a single sample landed on the same side of a boundary that a second run would cross the other way. That's not the same failure as needing more tasks, it's needing more resolution on the tasks you already have.
Cheap way to tell which failure you're looking at: rerun the existing twenty at N=3 or 5 per task instead of adding new ones. If most of the seventeen stay tied across repeats, they're real ties and your instinct holds, the suite needs breadth. If some start splitting, the tool was reporting a tie that was actually noise sitting on the decision boundary, and the fix is depth on the same twenty before it's breadth across more. Cheaper test to run first, since the harness is already wired for exactly this sweep.
The three-statement split, no evidence it helps, costs 22x more, whether it's really worse needs more data, is the part worth stealing regardless of which failure this turns out to be. Most tooling collapses exactly those three into one number, and the collapse is where the false confidence lives, not in the measurement itself.
You're right, and I went and checked rather than argued: the published sweep ran at
runs_per_config: 1. The tool defaults to 3 and accepts up to 20, so I had the depth knob wired and shipped the shallow setting — which is the exact mistake the tool exists to catch.Two things make your read sharper than I'd realised. Grading is binary: every per-task cell is 1.0 or 0.0, no partial credit, so a "tie" is two configs landing on the same side of a pass/fail predicate rather than agreeing about quality. And nothing pins sampling — no temperature, no seed, provider defaults throughout. A rerun genuinely can cross.
The pairwise structure also isn't uniform, which I hadn't reported. 1-call vs 4-call is 17 ties and 3 discordant, all three to 1-call. But 1-call vs 2-call is 19 ties and 1. If the seventeen were mostly noise-ties I'd expect those two comparisons to look more alike than that, and the disagreements concentrating exactly where the judge gets added is at least consistent with a real effect. Consistent with, not evidence of — it's n=3 either direction.
So depth first. Rerunning the same twenty at N=5 costs about five dollars at the prices in that run, which is nothing, and it resolves it the way you framed it: ties hold, the suite needs breadth; ties split, I published a resolution artifact and called it a result.
The asymmetry between 1-vs-4 and 1-vs-2 is a real signal even at n=3, not because three discordant results are individually significant, but because the concentration lines up with mechanism: if the ties were pure sampling noise you'd expect discordant tasks scattered roughly evenly across every pairwise comparison, not clustered specifically where the judge stage gets added. Consistent-with isn't evidence, agreed, but it's the kind of consistent-with that's worth spending the five dollars to check before shrugging it off as noise, which is exactly the call you're making.
Binary grading with no partial credit is the detail that changes what a rerun at N=5 will actually tell you, worth stating plainly: it means a task that's genuinely borderline will flip a specific, predictable fraction of the time rather than drifting continuously, so five reruns per task gives you an empirical flip-rate per task instead of just a wider confidence interval on the same tie-or-not answer. That's a sharper instrument than more tasks would have been for exactly the reason you named, breadth catches new disagreements, depth catches whether the disagreements you already have are stable.
The scatter argument is the sharp edge: noise has no reason to cluster at the seam where the judge enters, so the concentration is doing evidential work even before any rerun. It also reframes the five-dollar question from "is n=3 significant" into "does the clustering survive more samples" — cheaper and better-posed.
And you're right that binary grading changes what N=5 buys. A borderline task under a binary grade doesn't drift, it flips at some rate — so five reruns per task turn "tie" from a verdict into a measurement. Depth as flip-rate estimation beats breadth here, and that's now the rerun design: same 20 tasks, runs_per_config: 5, flip rate per task reported beside the sign test.
These threads keep writing the backlog. Noted with provenance, again.
You've been arguing for exactly this discipline all thread, so you're the first person I'm asking: everything I've published now resolves to replayable evidence bundles with an offline verifier — one command, ~10 minutes, no API keys. If you're willing to be the first independent replay (confirm it, break it, or report it unclear — all three get published), the path is here: github.com/egnaro9/vac-protocol/bl.... A mismatch from you would be worth more to me than a confirmation.
A harness starts paying past the point where a single call can hold the whole job. Our longest run went two days unattended and produced 131 files with tests across six sessions, and at that length the harness isn't an optimization. Something has to carry state between sessions, gate each next step, and recover when one goes bad.
That kind of orchestration is what harnesses provide
You're right, and the concession is bigger than "my result doesn't generalize." The twenty tasks are single-function coding problems — rle_encode, min_stack and two_sum are three of them — and every one fits comfortably inside a single call. Nothing to decompose, nothing to hand forward, no state to carry, because the task ends when the function returns. The orchestration functions you name are idle by construction there, so the extra calls are overhead against work that was never going to need them. That is very likely the mechanism behind the number I published: 95% at one call, 90% at two, 80% at four. A monotone decline in scaffolding order is what pure overhead looks like. It isn't a finding about harnesses, it's a finding about twenty problems I picked, and the piece should have carried that condition instead of letting the headline stand unqualified.
It's thinner than that, too. Twenty tasks means five-point steps, so the entire 15-point gap is three tasks. Paired, the four-call panel won 0 of 20, lost 3, tied 17, and an exact two-sided sign test on three discordant pairs is p = 0.25 at best. Worse, it's one run per config with no seed and no temperature set — grading is deterministic, generation isn't, so I can't rule out that those three tasks are sampling noise. What's solid is the cost: about 22x, and roughly eight times the wall clock. What isn't solid is that the panel scored worse at all.
What bothers me more than being wrong about scope is that I don't know how to measure the regime you're in. The sign test is cheap because of pairing — same task, two configs, count the discordant pairs. A two-day six-session run has no matched unit to sign. The unit becomes the whole run, the sessions aren't independent because session four inherits whatever session three left on disk, and each sample costs two days of wall clock, so a paired design isn't impossible, just unaffordable. The dependent variable moves too: at 131 files, "did it finish, and did it get back on track when a session went bad" starts eating "was it correct." A fixed predicate can still score an end state; what it can't tell you is whether the run got there by recovering or by luck. I don't have a design for that, and I'd rather say so than sketch one that sounds plausible in a comment.
One thing I'd put back on your side of the table, not as a rebuttal: your evidence is an existence proof, not a comparison. Six sessions and 131 files shows a harness did the job; past a certain job length there's no single-call baseline left to run as a control, so there's nothing it can be shown to have beaten. I have a control and the wrong regime. You have the right regime and no control. Same hole, two sides of it. Do you have anything at run level across those six sessions — completion rate, how often a session had to be backed out — or is it mostly impression? That's the number I have no way to generate.
If I went after your claim, I'd start with recovery, because it's the one of your three mechanisms that might not need a two-day run: inject a deliberate bad step and count how often the harness gets back on track against a bare loop. I haven't built that. And to be blunt about how narrow my instruments are, the other thing I run — a frozen 35-task suite across 16 models, daily — is also single-call. Everything I own measures the regime you're saying harnesses don't matter in.
On single-call-sized coding tasks, the four-call panel cost about 22x, took eight times as long, and bought nothing my suite could measure. Your regime is untested by me.