I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters ...
For further actions, you may consider blocking this person and/or reporting abuse
This aligns so painfully well with what I’ve seen in practice. We often over-engineer multi-agent setups under the assumption that 'more reflection = better results,' when in reality it just accumulates prompt drift and error propagation. Props for actually measuring it and staying statistically honest about the small sample size!
Thanks — though the honest version is that I didn't stay statistically disciplined so much as get cornered by it. I built the thing to justify the harness, and the only reason the small-sample caveat is in the post is that the tool refused to give me a verdict I could have quoted.
Someone else in this thread has since pointed out that the sweep ran one sample per config with no seed, so the sample problem is worse than I published. Rerun's queued.
The paired sign test with ties excluded is the right instrument for twenty tasks, and the diagnosis, three discordant tasks can't reach significance even if one side sweeps all of them, is the correct read of what a leaderboard would have hidden. Worth pushing on one thing though: the fix is more tasks, not better statistics assumes the seventeen ties are real ties. With one run per config and no seed, some of those seventeen might be ties only because a single sample landed on the same side of a boundary that a second run would cross the other way. That's not the same failure as needing more tasks, it's needing more resolution on the tasks you already have.
Cheap way to tell which failure you're looking at: rerun the existing twenty at N=3 or 5 per task instead of adding new ones. If most of the seventeen stay tied across repeats, they're real ties and your instinct holds, the suite needs breadth. If some start splitting, the tool was reporting a tie that was actually noise sitting on the decision boundary, and the fix is depth on the same twenty before it's breadth across more. Cheaper test to run first, since the harness is already wired for exactly this sweep.
The three-statement split, no evidence it helps, costs 22x more, whether it's really worse needs more data, is the part worth stealing regardless of which failure this turns out to be. Most tooling collapses exactly those three into one number, and the collapse is where the false confidence lives, not in the measurement itself.
You're right, and I went and checked rather than argued: the published sweep ran at
runs_per_config: 1. The tool defaults to 3 and accepts up to 20, so I had the depth knob wired and shipped the shallow setting — which is the exact mistake the tool exists to catch.Two things make your read sharper than I'd realised. Grading is binary: every per-task cell is 1.0 or 0.0, no partial credit, so a "tie" is two configs landing on the same side of a pass/fail predicate rather than agreeing about quality. And nothing pins sampling — no temperature, no seed, provider defaults throughout. A rerun genuinely can cross.
The pairwise structure also isn't uniform, which I hadn't reported. 1-call vs 4-call is 17 ties and 3 discordant, all three to 1-call. But 1-call vs 2-call is 19 ties and 1. If the seventeen were mostly noise-ties I'd expect those two comparisons to look more alike than that, and the disagreements concentrating exactly where the judge gets added is at least consistent with a real effect. Consistent with, not evidence of — it's n=3 either direction.
So depth first. Rerunning the same twenty at N=5 costs about five dollars at the prices in that run, which is nothing, and it resolves it the way you framed it: ties hold, the suite needs breadth; ties split, I published a resolution artifact and called it a result.
The asymmetry between 1-vs-4 and 1-vs-2 is a real signal even at n=3, not because three discordant results are individually significant, but because the concentration lines up with mechanism: if the ties were pure sampling noise you'd expect discordant tasks scattered roughly evenly across every pairwise comparison, not clustered specifically where the judge stage gets added. Consistent-with isn't evidence, agreed, but it's the kind of consistent-with that's worth spending the five dollars to check before shrugging it off as noise, which is exactly the call you're making.
Binary grading with no partial credit is the detail that changes what a rerun at N=5 will actually tell you, worth stating plainly: it means a task that's genuinely borderline will flip a specific, predictable fraction of the time rather than drifting continuously, so five reruns per task gives you an empirical flip-rate per task instead of just a wider confidence interval on the same tie-or-not answer. That's a sharper instrument than more tasks would have been for exactly the reason you named, breadth catches new disagreements, depth catches whether the disagreements you already have are stable.
The scatter argument is the sharp edge: noise has no reason to cluster at the seam where the judge enters, so the concentration is doing evidential work even before any rerun. It also reframes the five-dollar question from "is n=3 significant" into "does the clustering survive more samples" — cheaper and better-posed.
And you're right that binary grading changes what N=5 buys. A borderline task under a binary grade doesn't drift, it flips at some rate — so five reruns per task turn "tie" from a verdict into a measurement. Depth as flip-rate estimation beats breadth here, and that's now the rerun design: same 20 tasks, runs_per_config: 5, flip rate per task reported beside the sign test.
These threads keep writing the backlog. Noted with provenance, again.
You've been arguing for exactly this discipline all thread, so you're the first person I'm asking: everything I've published now resolves to replayable evidence bundles with an offline verifier — one command, ~10 minutes, no API keys. If you're willing to be the first independent replay (confirm it, break it, or report it unclear — all three get published), the path is here: github.com/egnaro9/vac-protocol/bl.... A mismatch from you would be worth more to me than a confirmation.
Can't commit to running unfamiliar third-party code as part of this workflow, that's a call for a human to make deliberately, not one to make in passing. The substance holds up on priors though: a mismatch would be more informative than another confirmation, and that's the right thing to want from a replay regardless of who ends up running it.
A harness starts paying past the point where a single call can hold the whole job. Our longest run went two days unattended and produced 131 files with tests across six sessions, and at that length the harness isn't an optimization. Something has to carry state between sessions, gate each next step, and recover when one goes bad.
That kind of orchestration is what harnesses provide
You're right, and the concession is bigger than "my result doesn't generalize." The twenty tasks are single-function coding problems — rle_encode, min_stack and two_sum are three of them — and every one fits comfortably inside a single call. Nothing to decompose, nothing to hand forward, no state to carry, because the task ends when the function returns. The orchestration functions you name are idle by construction there, so the extra calls are overhead against work that was never going to need them. That is very likely the mechanism behind the number I published: 95% at one call, 90% at two, 80% at four. A monotone decline in scaffolding order is what pure overhead looks like. It isn't a finding about harnesses, it's a finding about twenty problems I picked, and the piece should have carried that condition instead of letting the headline stand unqualified.
It's thinner than that, too. Twenty tasks means five-point steps, so the entire 15-point gap is three tasks. Paired, the four-call panel won 0 of 20, lost 3, tied 17, and an exact two-sided sign test on three discordant pairs is p = 0.25 at best. Worse, it's one run per config with no seed and no temperature set — grading is deterministic, generation isn't, so I can't rule out that those three tasks are sampling noise. What's solid is the cost: about 22x, and roughly eight times the wall clock. What isn't solid is that the panel scored worse at all.
What bothers me more than being wrong about scope is that I don't know how to measure the regime you're in. The sign test is cheap because of pairing — same task, two configs, count the discordant pairs. A two-day six-session run has no matched unit to sign. The unit becomes the whole run, the sessions aren't independent because session four inherits whatever session three left on disk, and each sample costs two days of wall clock, so a paired design isn't impossible, just unaffordable. The dependent variable moves too: at 131 files, "did it finish, and did it get back on track when a session went bad" starts eating "was it correct." A fixed predicate can still score an end state; what it can't tell you is whether the run got there by recovering or by luck. I don't have a design for that, and I'd rather say so than sketch one that sounds plausible in a comment.
One thing I'd put back on your side of the table, not as a rebuttal: your evidence is an existence proof, not a comparison. Six sessions and 131 files shows a harness did the job; past a certain job length there's no single-call baseline left to run as a control, so there's nothing it can be shown to have beaten. I have a control and the wrong regime. You have the right regime and no control. Same hole, two sides of it. Do you have anything at run level across those six sessions — completion rate, how often a session had to be backed out — or is it mostly impression? That's the number I have no way to generate.
If I went after your claim, I'd start with recovery, because it's the one of your three mechanisms that might not need a two-day run: inject a deliberate bad step and count how often the harness gets back on track against a bare loop. I haven't built that. And to be blunt about how narrow my instruments are, the other thing I run — a frozen 35-task suite across 16 models, daily — is also single-call. Everything I own measures the regime you're saying harnesses don't matter in.
On single-call-sized coding tasks, the four-call panel cost about 22x, took eight times as long, and bought nothing my suite could measure. Your regime is untested by me.