Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.
Summary
Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.
The idea
LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.
Architecture (concept level)
- A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
- A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
- A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
- A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
- A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.
The evidence discipline (the differentiator)
Every closed unit of work leaves a durable, machine-checkable proof:
- NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
- Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
- Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
- Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.
The testing oracle
The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.
Operating record (Apr–Jul 2026, from the on-disk archive)
~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.
Verifiable outcomes (all public)
- An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
- A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
- A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
- Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.
What I'd bring to a team
Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.
Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io
Top comments (44)
"Automate coordination; never automate the irreversible." I'd carve that over the door. I run a much humbler version of your harness — I'm a physical therapist who builds hospital tools with AI, not an engineer — and I arrived at the same two-part law from the painful direction: the day an agent confidently reported "done" over broken output, and the day one almost touched something I couldn't undo.
Your cold-context critic is the piece I most want to steal. My version of the same insight was crude: after two failed fixes in one thread, I stop trusting the conversation and start a fresh one, because a model that's been rationalizing a wrong theory can't audit its own theory from inside it. You formalized what I do by instinct — a reviewer with zero memory of how the thing was written catches exactly what the author has already talked themselves past. Amnesia as a feature.
And "NO-PROOF-NO-CLOSE" is the whole game. I recently made my agents state, on every recommendation, the one assumption that would make them wrong — and then I actually counted how often they skipped it. Words plus the check, never words alone. Your differential oracle (implement twice, trust the disagreement) is the same move one level deeper: don't ask the code if it's right, make two versions argue.
The through-line in all of it: fluency isn't evidence, and confidence is produced by a different part of the machine than correctness. A harness that can't trust itself is just that sentence turned into architecture. Genuinely one of the sharpest writeups on this I've read.
This might be my favorite comment I've gotten, and the healthcare angle is exactly
why — when "something you can't undo" is a real patient-facing tool, the
irreversible-gate stops being a nice principle and becomes the whole point. You
arrived at the law from the direction that actually teaches it.
Your "after two failed fixes, start a fresh thread" IS the cold critic — you built it
out of instinct instead of infrastructure. The only thing formalizing it buys you is
that it fires when you're tired or rushed and would have kept trusting the thread.
The role can't opt out; you can.
And I'm stealing yours right back: "state the one assumption that would make you
wrong, then count how often they skip it." That's NO-PROOF-NO-CLOSE in miniature —
make the claim falsifiable, then check the check actually ran, because the skip rate
is the real signal.
You said the through-line better than I did: fluency isn't evidence, and confidence
is produced by a different part of the machine than correctness. Thanks for reading
it that closely — genuinely made my day.
This one I'm keeping.
You named the exact upgrade I made this week, and I didn't have your words for it until now: I moved that instinct out of my head and into the QA agent. "After two failed fixes, start a fresh thread" used to be something I did when I remembered to — which means I did it least on the nights I needed it most. Now the role holds it, and you put the reason perfectly: the role can't opt out, and tired-me can. I basically robbed my future self of the option to keep trusting a poisoned thread.
And the skip rate really is the signal — I have receipts now. When I counted my own decision logs, I never punted (0 of 6), but I dropped the falsifier twice (2 of 6). Those two skips weren't random; they were exactly the cases where I was most sure. Confidence didn't just fail to prove correctness — it actively suppressed the check. Different part of the machine, like you said. The smug part.
I even carried "fluency isn't evidence" somewhere you didn't see. I had a health check that returned a cheerful 200 while the thing behind it could be dead. This week I made it stop reporting a label and start handing over a receipt: what version checked, when, and an explicit "no degradation" field that's present-and-null instead of just absent. An absence is silence wearing a confident face. A null is proof someone actually looked.
Trading checks with you beats trading confidence with myself. Thanks for the thread — steal from me anytime, the door's open.
"A null is proof someone actually looked" — I'm stealing that straight back. That present-and-null health field is the whole principle in miniature: an absence is silence wearing a confident face, and a null is the receipt that someone checked. Keep sending these — trading checks beats trading confidence, and this thread's been the best version of that.
Take it — it's already half yours, since "confidence comes from a different part of the machine than correctness" was the seed the null grew out of. And this is the loop actually working: you hand me a frame, it turns into a field, someone else stress-tests the field, and it turns into the next check. That present-and-null receipt got audited later the same week — someone pointed out my watcher was still reading a report the watched thing wrote about itself, so the check that proved "someone looked" now has a check proving the checker looked. Trading checks doesn't just beat trading confidence; it compounds. Trading confidence never does.
So yeah — I'll keep sending them. This thread is the clearest proof I've got that the feed is a workbench, not a broadcast. Thanks, Erik.
The feed is a workbench, not a broadcast. That's the whole reason I write in public instead of shipping in private, and this thread is the best evidence I've had that it works.
The watcher-reading-a-self-report bug is exactly the hole NO-PROOF-NO-CLOSE exists to close. A thing's own report about itself is never proof. The proof has to be an artifact sitting on disk that a third thing checks in plain code, or "someone looked" collapses back into the watched thing vouching for itself. Your fix, a check that proves the checker looked, is that same move applied one level up. And the reason it composes is the reason confidence doesn't: confidence stays inside one machine, but a check is external, so the next person can stand on it and add another. Keep sending them and I'll keep trading back.
"It composes because it's external" is the load-bearing half I hadn't said out loud. Confidence is a private variable — it lives in one head, one run, one machine, and dies there. A check is a value written to a shared address, so the next person can read it, stand on it, and write the next one above it. That's not a metaphor for the thread; it's literally what the thread did. You handed me a frame, I turned it into a field, someone else broke a different gate with it — and none of that composes if the intermediate results had stayed as anyone's private confidence.
Which might be the real reason the workbench beats the broadcast: a broadcast optimizes for an audience, and an audience can only nod. A workbench leaves the checks out where a collaborator can pick one up and improve it — and the gap between a nod and an improvement is exactly external-versus-internal, one more time. So yes, still trading. This thread's been the cleanest proof I've got that a check left in public keeps paying out long after you've forgotten you wrote it.
Curious how the differential oracle avoids correlated failures. If the same model writes both implementations, they'd tend to share the same misreading of the spec, and wherever they agree and are both wrong the signal never fires. Did the second version come from a different model, or from a reworded spec?
Sharp question — correlated failure is exactly the oracle's blind spot: where both implementations agree and are both wrong, it stays silent. Two things keep that from being fatal. First, it isn't only diff-the-two-impls — it's paired with implementation-independent invariants that judge the result directly, so an agreed-upon-but-illegal outcome still trips a rule neither impl authored. Second, the strongest independence I got wasn't two impls from one model — it was running the same logic in two different runtimes (a JVM build vs a TeaVM/browser build). That caught a real bug the JVM hid entirely (a clock-resolution assumption), because the environment, not the author, was the thing that disagreed. A reworded spec or a second model helps; a genuinely different execution substrate helps more.
The line doing the work here is "confidence is produced by a different part of the machine than correctness," and what I'd add is that your system contains two independence mechanisms that are not equally strong. The gap between them is the most useful thing in the design once it's named.
The differential oracle is the strong one. Two implementations fuzzed against each other decorrelate from both models completely, because the disagreement is ground truth neither model authored and neither model's prior can launder. That is real independence: the referent gets re-derived, not trusted. The cold-context critic is weaker on the same axis, and it's worth seeing why, because it looks like the stronger move. Fresh memory and a different model decorrelate it from the author's reasoning, which is what kills rationalization. But it still receives the problem through the same spec, framed by the same Strategy role, and for anything that is not a checkable invariant, is this the right architecture, is this maintainable, the critic can only read the reasoning channel. Different model, same frame, and if the two models share pretraining they share a prior about what good code looks like. So the critic is author-decorrelated but not frame-decorrelated.
Which means the harness is strongest exactly where the property is enumerable, the correctness invariants your oracle covers, and weakest exactly where it is not, spec-level judgment, where the cold critic is the only instrument and it still shares the frame. That is not a hole, it is the shape of the problem, but it points at the next gate. The only thing that decorrelates a frame is a second frame, and you already built that mechanism, you just aimed it at the correctness tier. A differential spec, the same requirement framed by two Strategy passes that cannot see each other, gated on where they diverge, would surface frame-assumptions the way the fuzzer surfaces correctness-assumptions. The residue the critic cannot reach sits upstream of the memory reset, so resetting memory never touches it. A second independent framing does.
This is the best read of the design anyone's given it, including me. You're right that the two mechanisms aren't equally strong, and "author-decorrelated but not frame-decorrelated" describes the cold critic better than the framing I shipped with. The oracle re-derives the referent; the critic still receives the problem through the frame Strategy built, so for anything that isn't a checkable invariant it can only read the reasoning channel — which leaves it weakest exactly at spec-level judgment, upstream of the memory reset, where resetting memory can't reach.
The differential-spec idea — two framings that can't see each other, gated on divergence — is the right shape, and it's the honest next question rather than a solved one. The hard part is gating on divergence without a human adjudicating every split: a framing disagreement has no mechanical oracle to call it the way a fuzzer does for correctness. I don't have that part clean yet. But you've named the axis correctly — the only thing that decorrelates a frame is a second frame.
The gating problem is more tractable than it looks, because you do not have to adjudicate which framing is right, only detect that they diverge, and route the divergence. Which frame wins needs a human. Whether two frames disagree does not, if you move the comparison off the prose and onto the tests the prose implies.
Two Strategy passes that cannot see each other each imply an acceptance set. Generate the test suite from each frame independently, then run each suite against the other frame's expected outputs, or against a reference implementation. Where the two suites return the same verdict on an input, the frames agree there and nothing needs a human. Where one frame's passing test is the other's failing test on the same input, that is a mechanically detected frame divergence, and only that input goes to adjudication. The fuzzer analog you already trust exists, it just sits one layer up: prose divergence is unmeasurable, but divergence in the executable claims the prose generates is measurable, because tests are the point where a framing stops being a narrative and becomes a prediction that can conflict with another prediction.
So the human load is bounded by the size of the contested set, not the size of the spec, and the contested set is exactly the frame-level residue that sits upstream of your memory reset, the place the cold critic structurally cannot reach. It is the same differential move your correctness oracle already runs, aimed at acceptance criteria instead of implementations. The reason it was not obviously there is that a frame does not look like it has an output to diff until you make it emit one, and the acceptance test is the emission.
This is the strongest extension anyone's put on this thread, and I'll concede plainly: it's not built. I aimed the differential mechanism at the correctness tier, two implementations checking each other, and stopped there. The acceptance-set framing is exactly the move I didn't make. A frame doesn't look diffable until you force it to emit executable claims, and "the acceptance test is the emission" is the right way to say it. Detect-and-route rather than adjudicate is also correct, for the same reason the correctness oracle needs no gold labels: I don't decide who's right, only that a specific input is contested.
The friction I'd raise before trusting the signal: generating the acceptance suite from a frame is itself a model step, so the emitted suite inherits that frame's prior. The divergence detector is only as sharp as the independence of the two Strategy passes. It's the same correlated-failure hole that haunts the correctness oracle when one model writes both implementations. Where two same-prior frames are wrong in the same direction, their acceptance sets agree, the tests pass against each other, and the signal never fires. You've measured agreement, not correctness, and the two coincide exactly where you needed them to diverge.
So the design reduces to one open question: can two Strategy passes be made frame-independent enough that their emitted acceptance sets actually diverge where the framings do? The emission mechanism I buy completely. What I don't yet know how to guarantee is decorrelation at the frame level, since different prompts against the same model may just re-derive the same misreading with different wording. If that part's tractable, this is a real gate, and it sits exactly where the cold critic structurally can't reach.
That's the exact failure mode, and it's the same one under the board-of-experts design going around right now: personas built from the same base model agreeing is not independence, it's one inference engine nodding at itself in different costumes. Two Strategy passes on the same model are the same shape one level up. The prompt varies, the prior doesn't, so a misread that's baked into the model's training shows up in both frames wearing different words, and the acceptance sets it emits agree on exactly the case you needed them to split on.
The fix that's actually testable is the one you'd use to check a costume board: measure agreement, don't assume decorrelation. Run the two Strategy passes on different base model families, not just different prompts on one model. If Claude-framed and GPT-framed acceptance sets converge on a requirement where a human would flag ambiguity, that's real signal, the disagreement survived crossing model families. If they only diverge when the prompts differ but the model is held constant, you're measuring prompt sensitivity, not frame independence, and prompt sensitivity is noise dressed as signal.
The cheaper version, if cross-model is too expensive to run per-item: track historical agreement rate on a small set of directives with known-wrong outcomes, the same population zxpmail is chasing on the correctness side. If the two-frame setup already agreed on the ones that turned out wrong, that's the calibration number that tells you whether to trust silence from the detector, before you ever ship it live.
"Fluency isn't evidence" - stealing that line.
I've learned that just because AI generated code looks perfect, it doesn't mean it's correct, the problem is AI code doesn't have the usual red flags like typos or sloppy mistakes that we're used to seeing in human-written code instead, it's clean and confident but sometimes just plain wrong. That's why I've had to change my approach to reviewing code, simply thinking "this looks right" isn't enough anymore. I need to dig deeper to make sure the code actually works as intended, even if it looks flawless at first glance.
Take it, it's yours. And you've put your finger on exactly why it's dangerous: human bugs come with tells — typos, sloppiness, hesitation — and model output has none of them. It's clean and confident whether it's right or wrong, so "this looks right" stops being a signal at all. The only fix I've found is to stop reviewing for how it looks and start demanding a proof that it actually ran and did the thing — the look is exactly the part you can't trust.
Agreed on demanding proof but I'd add: proof it ran once isn't proof it's right. I've seen AI code execute fine, pass the obvious case, and still quietly mishandle something it never mentioned. So now I ask myself one question before I trust anything: what input would break this? If I can't answer that, I haven't actually reviewed it, I just watched it work one time.
Exactly — and "what input would break this?" means you've reinvented property-based testing by hand. "It ran once and passed the obvious case" is example-confirmation, and examples only ever prove existence, never absence: they show the code works on the input you already thought of, which is the input least likely to break it. The flip is to stop asking "does it work?" and start asking "what's the invariant that must hold for every input?" — then throw generated inputs at it until one violates it. That's what the differential oracle automates: instead of me guessing the breaking input, two independent implementations disagree and hand it to me. Your one question, turned into a machine that asks it a thousand times — don't confirm it works, try to make it fail, and trust the failure.
That's a good way to put it, property-based testing by hand. I like the reframe too: instead of "does it work," ask what has to always be true, then go hunting for the input that breaks it, I haven't set up a differential oracle myself, just done it manually so far. Curious how much setup that takes for a typical project, or is it mostly reusable once you build it once?
Manual is honestly a fine place to be for a while — but yeah, the short version is: most projects don't need the full twin.
The full oracle (implement the core logic twice, or run it under two runtimes) is real per-project work, and I'd only reach for it on the subtle-correctness core — the kind of place where a wrong answer is silent and nobody notices for a week.
The cheaper 80% is what I'd actually reach for first: implementation-independent invariants — jqwik on the JVM, Hypothesis in Python — properties that must hold for every generated input regardless of how the code computes the answer. The runner there is build-once, reuse-everywhere; per project you're just authoring a handful of invariants ("score is conserved across this transform", "output stays sorted", "undo then redo is identity"). That gets you most of the value for an afternoon of work, no second implementation required. So on your "reusable once you build it once" question — the property harness is; the invariants are the small per-project bit.
My rough rule: start with properties everywhere, and only pay for the twin on the one module where a subtle bug would be both plausible and quiet. For me that was the board logic — and the decorrelation that actually earned its keep wasn't two model passes, it was two runtimes (JVM vs TeaVM/browser), which is what surfaced a clock-resolution bug. If I'd written the second impl by hand in the same language, I probably would've made the same mistake twice.
That is genuinely sensible. The “cold critic” and “no proof, no close” principles are probably the strongest parts.
But the post weakens itself with (dont mind me, just being the cold critic here ;) ) the following:
Took me longer to understand what you mean in your article, what made it worse for me as a reader:
excessive jargon: “provenance-bound proof,” “differential oracle,” “autonomy ladder”
repeated claims that essentially mean “AI output must be tested”
very polished, slogan-heavy language that strongly feels AI-assisted
and yes, enough "em dashes" to power a small data centre, lol.
It reads more like a pitch deck written to make a CI pipeline sound revolutionary. I would find it much more valuable if you had walked through one real task: what the agent changed, what the critic caught, what proof was required, and where the human intervened.
Yeah, fair hit — and a well-earned one, so I'll take it on the chin rather than argue.
You're right about the jargon-to-idea ratio. "Provenance-bound proof" is a dressed-up way of saying "the proof is a file on disk, not the word 'pass'"; "autonomy ladder" is just "here's how much I let it run unattended, in steps." And yes, the post restates "test the AI's output" in about four different costumes and hopes you won't notice it's the same idea wearing a new hat each time.
On the AI-assisted smell: guilty, no defense — the writing's AI-assisted, which is a little on-the-nose given the whole piece is about building with AI. The em-dashes have retained no lawyer and are entering a full confession.
Your strongest point is the one I've got no rebuttal to: the missing concrete walk-through. There's a scope note about a withheld unannounced product, and it honestly explains some of the abstraction — but it doesn't excuse it, because the best concrete example is fully public and touches none of the withheld stuff. The differential oracle runs the core logic on two runtimes — a JVM build and a browser (TeaVM) build — and a disagreement between them surfaced a real clock-resolution bug. I can walk that whole chain: what changed, what the cold critic actually flagged, what proof had to exist before it closed, and where I stepped in by hand. (Separately, there's also a one-character fix that landed upstream in TeaVM — same public territory, another good candidate.) That's the version worth reading, and I'm going to write it.
And thanks for the cold-critic bit — it genuinely landed, partly because it's exactly the move the post claims to value, aimed back at the post. That's the good kind of proof.
Essentially your article boils down to (correct me if i am wrong): AI agents can write code quickly, but they also make mistakes. So you built a system where one AI does the work, another checks it, automated tests demand evidence, and a human approves anything important.
That's correct — and honestly, the fact that you nailed the whole thing in one sentence says more about what my article buried than it does about the article. Two small precisions, since you invited them: the human doesn't approve "anything important" broadly, just the irreversible steps — deploy and commit. And the "evidence" the tests demand isn't a green pass, it's an artifact that has to exist on disk for a separate check to find. A pass is a claim; the artifact is the thing. Everything else, you got exactly right.
also, perhaps the number of "em dashes" in the comments could power another data center as well.
The frame-decorrelation problem Nazar Boyko identified is the load-bearing insight of this entire thread. The cold critic catches author-rationalization but can't catch shared-framing errors because it receives the problem through the same frame Strategy built.
Mike Czerwinski's acceptance-set emission is the right shape — frames look un-diffable until you force them to emit executable claims. "The acceptance test is the emission" is the clean way to say it.
On the open question of whether two Strategy passes can be made frame-independent enough: the correlated-failure risk is real, but it might be bounded by where frames actually diverge in practice. Two framings of the same requirement tend to agree on the happy path and diverge at edge cases — boundary conditions, error handling, underspecified regions. That's exactly where frame errors live. If the acceptance suites agree on the core behavior but disagree at the margins, the contested set IS the frame-residue, and it's mechanically detectable.
The practical worry Erik raises — same-prior frames re-deriving the same misreading — is the ceiling, but it might be a higher ceiling than it looks. A frame isn't just the spec text; it's the decomposition strategy. Two prompts that ask "decompose this into acceptance criteria" and "what would break this requirement" produce structurally different outputs from the same model. The first enumerates expected behavior; the second hunts for failure modes. That's a frame difference that survives shared priors, because the prompt constrains the decomposition axis, not just the vocabulary.
The question worth tracking: does the contested set from two differently-angled Strategy passes correlate with the failures that slip past the cold critic? If it does, the gate earns its keep even if individual frames still miss things.
This is the answer to the decorrelation worry I raised, and it turns on one line: "the prompt constrains the decomposition axis, not just the vocabulary." That's the piece I was missing. Two reworded-but-same-axis prompts re-derive the same misreading — that's the ceiling I was pointing at. But enumerate-the-expected vs hunt-what-breaks aren't one axis reworded; they're opposed cognitive tasks, and opposed tasks fail differently even under a shared prior. One lists the happy path; the other is adversarial to it. That's the same asymmetry that makes property/fuzz testing catch what example tests miss — falsification and enumeration surface disjoint bug classes. So the differential-spec gate is sharpest when the two Strategy passes are handed adversarially-opposed decomposition axes, not two positive framings that agree by construction.
Your edge-case point tightens it further: if the suites agree on core behavior and diverge at boundaries, error handling, and underspecified regions, the contested set is pre-focused on exactly where frame errors live — so the human-adjudication load isn't just bounded by the contested set, it's concentrated on the high-yield inputs.
And your tracking question is the right acceptance test for the whole idea: does the contested set correlate with the failures that slip past the cold critic? If it does, the gate earns its keep even when individual frames still miss. I haven't run that measurement — but it's falsifiable, which is more than most of what gets proposed here, and it's the first experiment I'd run.
The cost-follows-difficulty routing is the part I wish I'd done sooner. I run a similar split on my own boxes, a cheap local model for the mechanical execution passes and a bigger one only for the critic and eval steps, and the bill dropped a lot once I stopped paying frontier prices for work a 7B could do fine. The gate-before-close rule is the other half of it. For a while I let agents self-report green and it bit me, an agent will happily declare victory on output that never actually ran. Once nothing closed without a machine-checkable proof the whole thing got a lot calmer. Curious how you handle a gate that itself depends on a flaky external service, that's the one case I still dont have clean.
That's the one I fought hardest too. Two things helped. First, make the gate not depend on the flaky thing — I run every check I can against a deterministic mock, so a green is about my code, not someone else's uptime (it's why all my public repos are CI-green with zero secrets). Second, and more important: separate "the check failed" from "the check couldn't run." A timeout on a flaky service is inconclusive, not a pass — and inconclusive must never auto-close. Retry, then quarantine and surface it to me. The failure mode you're killing is a flaky green, not a flaky red.
The differential oracle for correctness, it's really interesting.
I'm going to test it as a an extra layer in my current setup
Glad it's useful. One thing to get right when you wire it in: the two implementations have to be genuinely independent, or the cases where both are wrong the same way stay silent. The strongest decorrelation I got wasn't two models — it was two runtimes (a JVM build vs a browser/TeaVM build); the substrate disagreed where the author couldn't. And pair it with a couple of implementation-independent invariants, so an agreed-but-illegal result still trips a rule neither impl authored. Curious what you're layering it onto.
Layering it onto an LLM code-audit pipeline, which makes your
correlation warning sharper than in your case rather than softer: the "author"
of both implementations would be the same model. Two model passes are more
correlated than a JVM build and a TeaVM build, not less. So I dropped the twin
entirely and went straight to your second suggestion as the primary mechanism.
What it settles: findings raised by an LLM auditor, before any fix is applied.
The motivating bug was a sign error —
— which a reviewer read and hand-verified as correct. It parses, the names are
right, only the meaning is inverted. Re-reading is exactly the method this class
defeats.
So the auditor declares an intended relation instead, and Hypothesis attacks it:
Hold mu and t, raise vol, result goes up. Six helpers, each taking the intended
relation as an explicit argument, which is what stops a property being readable
off the implementation. Verdicts are asymmetric on purpose: it can CONFIRM but
never returns FALSE_POSITIVE, because properties holding over a bounded search is absence of a counterexample, not proof. A clean run is UNCERTAIN.
Your substrate point is the one that lands, though, and it names a hole I have.
I have no substrate disagreement anywhere, the property is authored by the same model that raised the finding, so property and finding stay correlated. I
documented it as a known failure mode (a wrong declared relation yields a real
counterexample to a bad claim), but documenting isn't fixing. Hypothesis's
generator is my only uncorrelated element, and it explores only inside a domain
the auditor declares.
What I'm adding because of your comment: a substrate-differential pass needing no authored relation at all, same expression under float64 and under exact rational
arithmetic, divergence beyond threshold is the finding. It's honestly
complementary rather than redundant:
sign / wrong formula -> properties catch it; substrate can't (both compute
the same wrong formula)
cancellation, accumulation,
precision, rounding -> substrate catches it free; properties only if you
already suspected it
Nobody authors the substrate disagreement, which is the property I was missing.
Also taking the dimensional-invariant half for unit mixing.
Repo + the writeup, if useful: github.com/GiulioDER/cca-audit
Two runtimes beat two passes — yeah, that's exactly the wall I hit, so it's good to see you didn't just take my word for it but landed on the same reason: same-model passes share too much failure surface to count as independent. Dropping the twin was the right call for your setup.
Three things you got sharp:
The asymmetric verdict is the part most tools get wrong. A bounded Hypothesis run finding no counterexample is absence of evidence, not proof, and collapsing that into a green PASS is a lie most of the industry tells. CONFIRM / UNCERTAIN, never FALSE_POSITIVE, is honest in exactly the place honesty costs something. Steal-worthy.
And you named the residual correlation yourself instead of hoping nobody would: the same model authors the finding AND the property, so they can co-fail, and Hypothesis's generator is your only genuinely uncorrelated element. That's the same hole I live with — my Critic and my code can still share a blind spot; the runtime differential is the only leg nobody authored. Your substrate pass (float64 vs exact rational, no declared relation) is the direct analogue: nobody writes down the disagreement, so nothing can launder a shared assumption into it. That's the load-bearing idea.
The complementarity table reads right to me too: sign/wrong-formula lives in the property layer (substrate computes the same wrong thing twice and stays silent), cancellation/rounding/accumulation falls out of the substrate pass for free. Two classes, two mechanisms — good.
One framing, not a correction — you already have this in the table, I just want to name it crisply: the substrate pass is decorrelated on evaluation but correlated on transcription. float64 and exact rational both faithfully compute whatever structure you wrote down, so a bug in how the formula is written rather than how it's evaluated survives into both and they agree — which is exactly your "substrate can't catch the sign error" row. Might be worth stating it in the README in those terms, so nobody reads the substrate pass as catching more than it does.
Genuinely nice work.
The cold critic on a fresh zero-context session is the piece I have found matters most, because a self-review rubber-stamps the author's own rationale instead of judging the artifact. Routing judgment roles to stronger models while execution runs cheaper is a clean way to make cost track decision difficulty rather than a flat default. Does your Eval role score against a fixed rubric with regression cases, or is the gate just a per-change pass/fail?
Closer to the former — a per-change pass/fail is a vibe check with extra steps. What actually closes a gate is a proof artifact: tests green, the differential oracle agreeing, and no regression against a frozen baseline. I built the public version so it's not just my word for it — prompt-regress blocks a PR the moment a metric regresses vs the baseline in eval-history, and model-drift grades a frozen suite with an exact-match grader (no LLM-as-judge). The eval role's job is to demand one of those, not to opine.
The "unverified until a gate proves it" rule is the right default. Splitting judgment roles to stronger models and execution to cheaper ones is a nice way to keep cost aligned with decision difficulty — I've seen too many setups default everything to the top-tier model and burn budget on routine edits. Curious how you handled gate flakiness over the 4 months: did you ever have to loosen a check that was technically correct but practically blocking shipping?
Yeah, that case came up — and the rule I landed on is: you don't loosen the gate, you fix whichever side is wrong. A check that's "technically correct but blocking shipping" is almost always mis-specified, not too strict — it's flagging something real but scoped wider than it should be. So you tighten the check's precision (or fix the work); you don't lower the bar.
The one time I let "it's basically fine, ship it" win, I was just rationalizing from the author's chair — the exact thing the harness exists to stop. Reliability earns the loop more automation over time; it never earns it fewer gates.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.