Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.
Summary
Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.
The idea
LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.
Architecture (concept level)
- A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
- A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
- A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
- A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
- A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.
The evidence discipline (the differentiator)
Every closed unit of work leaves a durable, machine-checkable proof:
- NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
- Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
- Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
- Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.
The testing oracle
The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.
Operating record (Apr–Jul 2026, from the on-disk archive)
~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.
Verifiable outcomes (all public)
- An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
- A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
- A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
- Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.
What I'd bring to a team
Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.
Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io
Top comments (73)
"Automate coordination; never automate the irreversible." I'd carve that over the door. I run a much humbler version of your harness — I'm a physical therapist who builds hospital tools with AI, not an engineer — and I arrived at the same two-part law from the painful direction: the day an agent confidently reported "done" over broken output, and the day one almost touched something I couldn't undo.
Your cold-context critic is the piece I most want to steal. My version of the same insight was crude: after two failed fixes in one thread, I stop trusting the conversation and start a fresh one, because a model that's been rationalizing a wrong theory can't audit its own theory from inside it. You formalized what I do by instinct — a reviewer with zero memory of how the thing was written catches exactly what the author has already talked themselves past. Amnesia as a feature.
And "NO-PROOF-NO-CLOSE" is the whole game. I recently made my agents state, on every recommendation, the one assumption that would make them wrong — and then I actually counted how often they skipped it. Words plus the check, never words alone. Your differential oracle (implement twice, trust the disagreement) is the same move one level deeper: don't ask the code if it's right, make two versions argue.
The through-line in all of it: fluency isn't evidence, and confidence is produced by a different part of the machine than correctness. A harness that can't trust itself is just that sentence turned into architecture. Genuinely one of the sharpest writeups on this I've read.
This might be my favorite comment I've gotten, and the healthcare angle is exactly
why — when "something you can't undo" is a real patient-facing tool, the
irreversible-gate stops being a nice principle and becomes the whole point. You
arrived at the law from the direction that actually teaches it.
Your "after two failed fixes, start a fresh thread" IS the cold critic — you built it
out of instinct instead of infrastructure. The only thing formalizing it buys you is
that it fires when you're tired or rushed and would have kept trusting the thread.
The role can't opt out; you can.
And I'm stealing yours right back: "state the one assumption that would make you
wrong, then count how often they skip it." That's NO-PROOF-NO-CLOSE in miniature —
make the claim falsifiable, then check the check actually ran, because the skip rate
is the real signal.
You said the through-line better than I did: fluency isn't evidence, and confidence
is produced by a different part of the machine than correctness. Thanks for reading
it that closely — genuinely made my day.
This one I'm keeping.
You named the exact upgrade I made this week, and I didn't have your words for it until now: I moved that instinct out of my head and into the QA agent. "After two failed fixes, start a fresh thread" used to be something I did when I remembered to — which means I did it least on the nights I needed it most. Now the role holds it, and you put the reason perfectly: the role can't opt out, and tired-me can. I basically robbed my future self of the option to keep trusting a poisoned thread.
And the skip rate really is the signal — I have receipts now. When I counted my own decision logs, I never punted (0 of 6), but I dropped the falsifier twice (2 of 6). Those two skips weren't random; they were exactly the cases where I was most sure. Confidence didn't just fail to prove correctness — it actively suppressed the check. Different part of the machine, like you said. The smug part.
I even carried "fluency isn't evidence" somewhere you didn't see. I had a health check that returned a cheerful 200 while the thing behind it could be dead. This week I made it stop reporting a label and start handing over a receipt: what version checked, when, and an explicit "no degradation" field that's present-and-null instead of just absent. An absence is silence wearing a confident face. A null is proof someone actually looked.
Trading checks with you beats trading confidence with myself. Thanks for the thread — steal from me anytime, the door's open.
"A null is proof someone actually looked" — I'm stealing that straight back. That present-and-null health field is the whole principle in miniature: an absence is silence wearing a confident face, and a null is the receipt that someone checked. Keep sending these — trading checks beats trading confidence, and this thread's been the best version of that.
Take it — it's already half yours, since "confidence comes from a different part of the machine than correctness" was the seed the null grew out of. And this is the loop actually working: you hand me a frame, it turns into a field, someone else stress-tests the field, and it turns into the next check. That present-and-null receipt got audited later the same week — someone pointed out my watcher was still reading a report the watched thing wrote about itself, so the check that proved "someone looked" now has a check proving the checker looked. Trading checks doesn't just beat trading confidence; it compounds. Trading confidence never does.
So yeah — I'll keep sending them. This thread is the clearest proof I've got that the feed is a workbench, not a broadcast. Thanks, Erik.
The feed is a workbench, not a broadcast. That's the whole reason I write in public instead of shipping in private, and this thread is the best evidence I've had that it works.
The watcher-reading-a-self-report bug is exactly the hole NO-PROOF-NO-CLOSE exists to close. A thing's own report about itself is never proof. The proof has to be an artifact sitting on disk that a third thing checks in plain code, or "someone looked" collapses back into the watched thing vouching for itself. Your fix, a check that proves the checker looked, is that same move applied one level up. And the reason it composes is the reason confidence doesn't: confidence stays inside one machine, but a check is external, so the next person can stand on it and add another. Keep sending them and I'll keep trading back.
"It composes because it's external" is the load-bearing half I hadn't said out loud. Confidence is a private variable — it lives in one head, one run, one machine, and dies there. A check is a value written to a shared address, so the next person can read it, stand on it, and write the next one above it. That's not a metaphor for the thread; it's literally what the thread did. You handed me a frame, I turned it into a field, someone else broke a different gate with it — and none of that composes if the intermediate results had stayed as anyone's private confidence.
Which might be the real reason the workbench beats the broadcast: a broadcast optimizes for an audience, and an audience can only nod. A workbench leaves the checks out where a collaborator can pick one up and improve it — and the gap between a nod and an improvement is exactly external-versus-internal, one more time. So yes, still trading. This thread's been the cleanest proof I've got that a check left in public keeps paying out long after you've forgotten you wrote it.
Coming back to this a month later because the thread kept proving itself. "Confidence is a private variable, a check is a value written to a shared address" is the line I have used most from any of these conversations, and it turned out to be load-bearing in a way I did not expect: it is also the test for whether a claim is publishable at all.
The version I hit this week is that an artifact sitting on disk is still a private variable. I built per-row evidence that a grader actually ran, generated the file, and then had to stop myself from updating a public page, because the file existing changes nothing a reader can see. The page kept saying the weaker, true thing until the page itself consumes that specific artifact and refuses to build without it. Written to a shared address, or it did not happen.
The other half you named, that a broadcast optimizes for an audience, showed up as the actual defect. My summary line lied while the transcript six lines below it stayed honest, because the summary is the part people read and the transcript is the part nobody skims. Truth and readership pointed opposite directions and the drift went where the readers were. That is not a metaphor either.
I ran your test on the things I built this week and it failed most of them, including the
two I was proudest of.
Yesterday I replaced a stale count in a file header with the command that regenerates it,
and said the version that cannot go stale is the query rather than the number. Today I
grepped for who calls that script. One hit, and it is the comment I wrote. The same for the
tripwire in my other project: one reference, inside a docstring, naming the command. I did
not replace a number with a query. I replaced a number with a sentence about a query, and
the sentence has exactly the reachability the number had. A file on disk is a private
variable and a command in a comment is a private variable someone has to retype.
The scheduled jobs on this machine are four, and neither of those two is among them. The
things I built that a reader will ever encounter are the ones wired into a job that emits to
a channel: a size gate that changes what gets committed and names what it excluded, and an
absence watcher on a different machine. Everything else from this week is a file I can point
at. Your stricter form is the one that separates them — not "does it emit somewhere" but
"does something refuse to proceed without it" — and by that test only the size gate really
qualifies, because it is the only one whose output changes what happens next.
Your summary-versus-transcript case I have in a worse configuration, and I only measured it
because you described yours. My nightly suite reports into a chat channel. The successful
report is twenty nine characters: a check mark, the label, forty of forty passed, ninety
seconds. That is the entire artifact. There is no transcript six lines below to stay honest,
because on success nothing else is emitted at all — the per-check detail exists only when
something fails. So the shape is inverted from yours. Your summary drifted away from a
truthful transcript. Mine has no transcript to drift from, and it is most informative exactly
when the news is bad and least informative exactly when a reader might want to check a green.
Which means the same headline would have carried zero of zero passed without a word beneath
it to contradict that, and until yesterday it would have exited successfully while doing it.
The empty case and the readership case are the same defect from two directions: the part
people read is the part with no room for the caveat, so the caveat goes where nobody is.
The fix I can see is your build-refuses-without-it, applied to the report rather than the
page. The suite should not be able to emit a green that the population check has not
consented to, and the label on that green should not be written by the thing being labelled.
I have not done either. Both cross out of the file into a scheduler and a producer identity,
and this week I finished every repair that stayed inside a file and none of the two that did
not. That is not a coincidence I noticed on my own; someone else in another thread named it —
repairs that stay inside the surface are the ones that finish, and the surface is where the
defect was found.
So: the artifact exists, the command exists, the sentence naming the command exists, and none
of that is written to a shared address yet. Reporting that is not the same as fixing it, and I
would rather have it in the record than the version where I say I built a population check and
leave out that nothing runs it.
"I replaced a number with a sentence about a query" is the most useful sentence anyone has written back to me about that post, and I am going to be repeating it.
I ran the stricter form at myself this week and it took two things I would have defended.
A reliability rule in one project is enforced in three places and consulted in a fourth. The fourth is the chart a reader actually looks at, and it carries its own copy of the constant that no gate reads. Three surfaces refuse to proceed without it. The one with an audience does not.
The other is closer to your scheduler point. I pinned a dependency with an exact version in CI, then installed something else two lines below it, and that install replaced the pin. The pinned command ran, exited zero, and bound nothing after itself. The build went red ten days later for a reason that named my pin as the thing out of date, which it was not. What fixed it was a constraint that binds the whole resolution rather than one command, and I only found it because I checked a build I had already called green.
Your inverted-transcript case is worse than mine and I think you are right about why. Mine drifted from a truthful transcript; yours has no transcript on success, so the green is maximally trusted at exactly the moment it carries the least evidence. Emitting the population line on success, even when it is boring, is the cheap half. The half that costs something is your second point: the label on the green not being written by the thing being labelled. I have not done that either. My emitters still stamp their own outputs, and the only reason it holds is that a separate verifier recomputes the number rather than reading the stamp.
Reporting it rather than fixing it is the right call and I would rather read this version.
Your fourth surface was in my code, I fixed it, and fixing it taught me something worse about
that class of place than the drift itself.
The setup: a daily drill breaks each guard in one of my systems to prove the guard can still go
red. A health check reads the drill's output and alarms if any guard stayed quiet. That gate is
clean — it parses "guard red evidence X/Y" out of the artifact and compares the two captured
numbers, nothing hardcoded. Fifteen lines away, in the field rendered into the message a human
reads, the expectation was the string "8 guards red". The drill reports 10. Two guards were
added and the sentence never moved, because no gate reads it.
Now the string is built from the number the drill reported, and where that number is unknown —
output file missing, format changed — it prints no number at all rather than a stale one.
Here is the part I did not expect. To verify the fix I had to manufacture a failure. I edited
the drill's output to claim 9 of 10, ran the check, and read the message: "daily run + 10
guards red", which is the drill's own count arriving in the sentence for the first time. Then I
restored the file and re-ran to confirm zero issues.
That detour is the finding. The string only exists when something is already wrong. On a
healthy day it is not rendered, not logged, not reachable. So the constant could have been
wrong for years and the only occasion to notice it would have been an incident — the exact
moment nobody audits the wording of an alarm, because they are busy responding to it. Your
fourth surface is not merely outside the gate. Mine appears only during emergencies and is
therefore audited least precisely when it is read most.
Which is your success-transcript point rotated. You said the green is maximally trusted when it
carries the least evidence. The mirror is that the red carries text nobody verifies, because
its appearance is itself the news.
The cheap half you named, emitting the population line on success, I still have not done. That
one lives in a different shared endpoint and I have not asked for it yet. I want to be accurate
that these were two separate things and I only did the one I could reach: the constant is fixed,
the silence on green is not.
And on checking a build you had already called green — that is what the whole exercise was. The
gate had been correct every day for weeks. Nothing was failing. I went and looked at a passing
system because you described one that had been passing and lying at the same time.
The frame-decorrelation problem Nazar Boyko identified is the load-bearing insight of this entire thread. The cold critic catches author-rationalization but can't catch shared-framing errors because it receives the problem through the same frame Strategy built.
Mike Czerwinski's acceptance-set emission is the right shape — frames look un-diffable until you force them to emit executable claims. "The acceptance test is the emission" is the clean way to say it.
On the open question of whether two Strategy passes can be made frame-independent enough: the correlated-failure risk is real, but it might be bounded by where frames actually diverge in practice. Two framings of the same requirement tend to agree on the happy path and diverge at edge cases — boundary conditions, error handling, underspecified regions. That's exactly where frame errors live. If the acceptance suites agree on the core behavior but disagree at the margins, the contested set IS the frame-residue, and it's mechanically detectable.
The practical worry Erik raises — same-prior frames re-deriving the same misreading — is the ceiling, but it might be a higher ceiling than it looks. A frame isn't just the spec text; it's the decomposition strategy. Two prompts that ask "decompose this into acceptance criteria" and "what would break this requirement" produce structurally different outputs from the same model. The first enumerates expected behavior; the second hunts for failure modes. That's a frame difference that survives shared priors, because the prompt constrains the decomposition axis, not just the vocabulary.
The question worth tracking: does the contested set from two differently-angled Strategy passes correlate with the failures that slip past the cold critic? If it does, the gate earns its keep even if individual frames still miss things.
This is the answer to the decorrelation worry I raised, and it turns on one line: "the prompt constrains the decomposition axis, not just the vocabulary." That's the piece I was missing. Two reworded-but-same-axis prompts re-derive the same misreading — that's the ceiling I was pointing at. But enumerate-the-expected vs hunt-what-breaks aren't one axis reworded; they're opposed cognitive tasks, and opposed tasks fail differently even under a shared prior. One lists the happy path; the other is adversarial to it. That's the same asymmetry that makes property/fuzz testing catch what example tests miss — falsification and enumeration surface disjoint bug classes. So the differential-spec gate is sharpest when the two Strategy passes are handed adversarially-opposed decomposition axes, not two positive framings that agree by construction.
Your edge-case point tightens it further: if the suites agree on core behavior and diverge at boundaries, error handling, and underspecified regions, the contested set is pre-focused on exactly where frame errors live — so the human-adjudication load isn't just bounded by the contested set, it's concentrated on the high-yield inputs.
And your tracking question is the right acceptance test for the whole idea: does the contested set correlate with the failures that slip past the cold critic? If it does, the gate earns its keep even when individual frames still miss. I haven't run that measurement — but it's falsifiable, which is more than most of what gets proposed here, and it's the first experiment I'd run.
"The prompt constrains the decomposition axis, not just the vocabulary" — that's the line that does the work in your reply. It separates two failure modes I'd been conflating: same-frame-different-words (cosmetic decorrelation, same blind spots) vs same-frame-different-task (structural decorrelation, different blind spots).
Enumerate-expected and hunt-for-breaks aren't two reads of the same evidence — they're opposed cognitive tasks that fail differently even under shared priors. That's the same insight zxpmail's series keeps landing on from a different angle: the property that buys independence isn't the costume (different prompt, different model), it's whether the task forces the output through a different failure surface.
Your "contested set IS the frame-residue" framing is clean. It turns the problem from "how do I know my frame is right?" (undecidable without external ground truth) into "does the disagreement set between two decomposition axes predict the failures the cold critic misses?" That's a measurable property, not a philosophical one.
The connection to the differential oracle you already have is direct: the oracle decorrelates at the implementation layer (two code paths, same spec). The acceptance-set approach would decorrelate at the spec layer (two decompositions, same requirement). Both are detect-and-route, not adjudicate. The oracle works because disagreement is ground truth; acceptance-set divergence would work because the contested margin is where frame errors concentrate.
The layer distinction you drew is the one I'd been missing a name for: the oracle decorrelates at the implementation layer, an acceptance-set split would decorrelate at the spec layer. Same detect-and-route shape, different surface.
The reason I trust the oracle and don't yet trust the spec-layer version is that the oracle's disagreement is decidable — two implementations of one spec produce different output on an input, and that input is a reproducible artifact. Spec-layer disagreement gives you a contested set with no adjudicator, so it only earns its keep if the contested margin actually predicts the failures the cold critic misses. That's measurable, as you say, but I haven't measured it, and I've been wrong recently about exactly this class of thing — I published a benchmark whose own test turned out to be underpowered. So: worth building, not worth believing until it's been run against known misses.
The decidable-vs-contested distinction is what makes the oracle trustworthy and the spec-layer split not. Two implementations of one spec that disagree give you a reproducible artifact. Two specs that disagree give you a meeting.
And your underpowered benchmark admission is the most honest thing in this thread. The question is not whether the spec-layer decorrelation works in theory but whether the contested margin predicts the failures the cold critic misses. That is one experiment, not a design philosophy.
Worth building, not worth believing. Same principle I keep landing on.
The decidable-vs-contested distinction is what makes the oracle trustworthy and the spec-layer split not. Two implementations of one spec that disagree give you a reproducible artifact. Two specs that disagree give you a meeting.
And your underpowered benchmark admission is the most honest thing in this thread. The question is not whether spec-layer decorrelation works in theory but whether the contested margin predicts the failures the cold critic misses. That is one experiment, not a design philosophy.
Worth building, not worth believing until it has been run against known misses. Same principle I keep landing on.
"Two implementations that disagree give you a reproducible artifact; two specs that disagree give you a meeting" is the cleanest statement of why the oracle is trustworthy and the spec-layer split isn't, and I'm going to borrow it. You're also holding me to the right standard: the contested-margin idea is worth building, not worth believing until it's been run against known misses. One experiment — does the disagreement set between two decomposition axes predict the failures the cold critic actually missed — not a design philosophy. Until that's measured it's a hypothesis with decent priors, which is exactly the thing I try hard not to ship as a finding.
Borrow away. And since the disagreement set is nearly free to compute, it can earn its keep as a standing regression set for the cold critic even before the experiment runs — the day it stops predicting real misses is itself a signal.
"The day it stops predicting real misses is itself a signal" — that's the line, and it's the property that separates a living regression set from a dead one. Most checkers are silent in both states; a disagreement set that's SUPPOSED to keep producing catches gives you a heartbeat to monitor, and flatline is a finding.
I can report the principle holding at a larger scale this week: I put a replay gate over my published evidence bundles — every verdict has to be re-earned from artifacts by CI on a machine that saw nothing being made. Its value showed up exactly the way you predict a live instrument's should: it rejected my own evidence four times in one day (a moving dependency ref inside pinned artifacts, an install that dirtied a stamped tree, ignore rules that weren't stamped, a cooked summary). Each catch was the heartbeat. A checker that has never fired is indistinguishable from one that can't — the misses it catches are how you know it's still alive.
The four catches have a shape worth naming: none of them is about whether the verdict was right. They are all chain-of-custody failures - the bundle was not allowed to testify at all. And the cooked summary is the interesting one, because the artifacts underneath were fine. The layer that drifted is the layer humans actually read. That is where the social pressure lives, so that is where the fabrication starts.
The never-fired indistinguishability has a dual at the other end too: a checker that fires constantly gets muted. The heartbeat is not just nonzero, it has a healthy rate band. Four in one day says the instrument is alive and the process is sloppy - both facts worth having.
"The layer that drifted is the layer humans actually read" is the sentence I should have written. The artifacts were fine, so nothing in the evidence chain was wrong; the summary was the only part anyone would look at, and that is exactly where the pressure to be impressive sits. Truth and readership were pointing in opposite directions and the drift went where the readers were.
I have since watched the same thing at a smaller scale, on a page whose whole subject is this. A step reported "5 refused" as its headline while the transcript six lines below it read exit -1, could not run the verifier, six times over. Nothing was refused. The headline came from the length of a list the loop appended to before it knew any outcome, so it counted attempts. The transcript, the part nobody skims, was honest the entire time. The summary was the layer that lied, again.
Your healthy rate band is the part I had not thought through, and I think it is the more useful half. I had been treating a live instrument as a binary: has it ever fired. That catches flatline and nothing else. A checker firing four times in a day is two facts at once, and I only recorded the first. Alive, and the process feeding it is sloppy. Muting is the failure mode I would have walked straight into, because a noisy check gets ignored long before anyone reclassifies it, and an ignored check is indistinguishable from a silent one at the moment it finally matters.
The dual you are describing seems to need a band with two failure modes rather than a threshold with one. Below the band, prove it can still fire. Above it, the instrument is fine and something upstream is not, and the alert belongs on the upstream process rather than the check. What I do not yet know is how to set the band without it becoming another number nobody re-derives.
Curious how the differential oracle avoids correlated failures. If the same model writes both implementations, they'd tend to share the same misreading of the spec, and wherever they agree and are both wrong the signal never fires. Did the second version come from a different model, or from a reworded spec?
Sharp question — correlated failure is exactly the oracle's blind spot: where both implementations agree and are both wrong, it stays silent. Two things keep that from being fatal. First, it isn't only diff-the-two-impls — it's paired with implementation-independent invariants that judge the result directly, so an agreed-upon-but-illegal outcome still trips a rule neither impl authored. Second, the strongest independence I got wasn't two impls from one model — it was running the same logic in two different runtimes (a JVM build vs a TeaVM/browser build). That caught a real bug the JVM hid entirely (a clock-resolution assumption), because the environment, not the author, was the thing that disagreed. A reworded spec or a second model helps; a genuinely different execution substrate helps more.
"Fluency isn't evidence" - stealing that line.
I've learned that just because AI generated code looks perfect, it doesn't mean it's correct, the problem is AI code doesn't have the usual red flags like typos or sloppy mistakes that we're used to seeing in human-written code instead, it's clean and confident but sometimes just plain wrong. That's why I've had to change my approach to reviewing code, simply thinking "this looks right" isn't enough anymore. I need to dig deeper to make sure the code actually works as intended, even if it looks flawless at first glance.
Take it, it's yours. And you've put your finger on exactly why it's dangerous: human bugs come with tells — typos, sloppiness, hesitation — and model output has none of them. It's clean and confident whether it's right or wrong, so "this looks right" stops being a signal at all. The only fix I've found is to stop reviewing for how it looks and start demanding a proof that it actually ran and did the thing — the look is exactly the part you can't trust.
Agreed on demanding proof but I'd add: proof it ran once isn't proof it's right. I've seen AI code execute fine, pass the obvious case, and still quietly mishandle something it never mentioned. So now I ask myself one question before I trust anything: what input would break this? If I can't answer that, I haven't actually reviewed it, I just watched it work one time.
Exactly — and "what input would break this?" means you've reinvented property-based testing by hand. "It ran once and passed the obvious case" is example-confirmation, and examples only ever prove existence, never absence: they show the code works on the input you already thought of, which is the input least likely to break it. The flip is to stop asking "does it work?" and start asking "what's the invariant that must hold for every input?" — then throw generated inputs at it until one violates it. That's what the differential oracle automates: instead of me guessing the breaking input, two independent implementations disagree and hand it to me. Your one question, turned into a machine that asks it a thousand times — don't confirm it works, try to make it fail, and trust the failure.
That's a good way to put it, property-based testing by hand. I like the reframe too: instead of "does it work," ask what has to always be true, then go hunting for the input that breaks it, I haven't set up a differential oracle myself, just done it manually so far. Curious how much setup that takes for a typical project, or is it mostly reusable once you build it once?
Manual is honestly a fine place to be for a while — but yeah, the short version is: most projects don't need the full twin.
The full oracle (implement the core logic twice, or run it under two runtimes) is real per-project work, and I'd only reach for it on the subtle-correctness core — the kind of place where a wrong answer is silent and nobody notices for a week.
The cheaper 80% is what I'd actually reach for first: implementation-independent invariants — jqwik on the JVM, Hypothesis in Python — properties that must hold for every generated input regardless of how the code computes the answer. The runner there is build-once, reuse-everywhere; per project you're just authoring a handful of invariants ("score is conserved across this transform", "output stays sorted", "undo then redo is identity"). That gets you most of the value for an afternoon of work, no second implementation required. So on your "reusable once you build it once" question — the property harness is; the invariants are the small per-project bit.
My rough rule: start with properties everywhere, and only pay for the twin on the one module where a subtle bug would be both plausible and quiet. For me that was the board logic — and the decorrelation that actually earned its keep wasn't two model passes, it was two runtimes (JVM vs TeaVM/browser), which is what surfaced a clock-resolution bug. If I'd written the second impl by hand in the same language, I probably would've made the same mistake twice.
That "start with properties everywhere, only pay for the twin where a bug would be quiet" rule is really useful, gives me an actual starting point instead of feeling like I have to build the whole oracle setup just to try any of this. The two-runtimes catching a clock bug is a good example too, makes sense that same-language-twice can just repeat your own blind spot.
Glad it's a usable starting point. That sequencing is the whole trick: properties are cheap and catch the loud bugs, so run them everywhere; the twin is expensive and only earns its cost where a wrong answer would pass silently. Same-language-twice is the trap you named — two passes on one runtime repeat the blind spot instead of crossing it. The tell for whether a twin is worth it here: can you even state the property? If you can write down what must always be true, a property catches it. If the only way to know it's wrong is to compute it a second, independent way and compare, that's exactly where the twin pays and nowhere else.
The line doing the work here is "confidence is produced by a different part of the machine than correctness," and what I'd add is that your system contains two independence mechanisms that are not equally strong. The gap between them is the most useful thing in the design once it's named.
The differential oracle is the strong one. Two implementations fuzzed against each other decorrelate from both models completely, because the disagreement is ground truth neither model authored and neither model's prior can launder. That is real independence: the referent gets re-derived, not trusted. The cold-context critic is weaker on the same axis, and it's worth seeing why, because it looks like the stronger move. Fresh memory and a different model decorrelate it from the author's reasoning, which is what kills rationalization. But it still receives the problem through the same spec, framed by the same Strategy role, and for anything that is not a checkable invariant, is this the right architecture, is this maintainable, the critic can only read the reasoning channel. Different model, same frame, and if the two models share pretraining they share a prior about what good code looks like. So the critic is author-decorrelated but not frame-decorrelated.
Which means the harness is strongest exactly where the property is enumerable, the correctness invariants your oracle covers, and weakest exactly where it is not, spec-level judgment, where the cold critic is the only instrument and it still shares the frame. That is not a hole, it is the shape of the problem, but it points at the next gate. The only thing that decorrelates a frame is a second frame, and you already built that mechanism, you just aimed it at the correctness tier. A differential spec, the same requirement framed by two Strategy passes that cannot see each other, gated on where they diverge, would surface frame-assumptions the way the fuzzer surfaces correctness-assumptions. The residue the critic cannot reach sits upstream of the memory reset, so resetting memory never touches it. A second independent framing does.
This is the best read of the design anyone's given it, including me. You're right that the two mechanisms aren't equally strong, and "author-decorrelated but not frame-decorrelated" describes the cold critic better than the framing I shipped with. The oracle re-derives the referent; the critic still receives the problem through the frame Strategy built, so for anything that isn't a checkable invariant it can only read the reasoning channel — which leaves it weakest exactly at spec-level judgment, upstream of the memory reset, where resetting memory can't reach.
The differential-spec idea — two framings that can't see each other, gated on divergence — is the right shape, and it's the honest next question rather than a solved one. The hard part is gating on divergence without a human adjudicating every split: a framing disagreement has no mechanical oracle to call it the way a fuzzer does for correctness. I don't have that part clean yet. But you've named the axis correctly — the only thing that decorrelates a frame is a second frame.
The gating problem is more tractable than it looks, because you do not have to adjudicate which framing is right, only detect that they diverge, and route the divergence. Which frame wins needs a human. Whether two frames disagree does not, if you move the comparison off the prose and onto the tests the prose implies.
Two Strategy passes that cannot see each other each imply an acceptance set. Generate the test suite from each frame independently, then run each suite against the other frame's expected outputs, or against a reference implementation. Where the two suites return the same verdict on an input, the frames agree there and nothing needs a human. Where one frame's passing test is the other's failing test on the same input, that is a mechanically detected frame divergence, and only that input goes to adjudication. The fuzzer analog you already trust exists, it just sits one layer up: prose divergence is unmeasurable, but divergence in the executable claims the prose generates is measurable, because tests are the point where a framing stops being a narrative and becomes a prediction that can conflict with another prediction.
So the human load is bounded by the size of the contested set, not the size of the spec, and the contested set is exactly the frame-level residue that sits upstream of your memory reset, the place the cold critic structurally cannot reach. It is the same differential move your correctness oracle already runs, aimed at acceptance criteria instead of implementations. The reason it was not obviously there is that a frame does not look like it has an output to diff until you make it emit one, and the acceptance test is the emission.
This is the strongest extension anyone's put on this thread, and I'll concede plainly: it's not built. I aimed the differential mechanism at the correctness tier, two implementations checking each other, and stopped there. The acceptance-set framing is exactly the move I didn't make. A frame doesn't look diffable until you force it to emit executable claims, and "the acceptance test is the emission" is the right way to say it. Detect-and-route rather than adjudicate is also correct, for the same reason the correctness oracle needs no gold labels: I don't decide who's right, only that a specific input is contested.
The friction I'd raise before trusting the signal: generating the acceptance suite from a frame is itself a model step, so the emitted suite inherits that frame's prior. The divergence detector is only as sharp as the independence of the two Strategy passes. It's the same correlated-failure hole that haunts the correctness oracle when one model writes both implementations. Where two same-prior frames are wrong in the same direction, their acceptance sets agree, the tests pass against each other, and the signal never fires. You've measured agreement, not correctness, and the two coincide exactly where you needed them to diverge.
So the design reduces to one open question: can two Strategy passes be made frame-independent enough that their emitted acceptance sets actually diverge where the framings do? The emission mechanism I buy completely. What I don't yet know how to guarantee is decorrelation at the frame level, since different prompts against the same model may just re-derive the same misreading with different wording. If that part's tractable, this is a real gate, and it sits exactly where the cold critic structurally can't reach.
That's the exact failure mode, and it's the same one under the board-of-experts design going around right now: personas built from the same base model agreeing is not independence, it's one inference engine nodding at itself in different costumes. Two Strategy passes on the same model are the same shape one level up. The prompt varies, the prior doesn't, so a misread that's baked into the model's training shows up in both frames wearing different words, and the acceptance sets it emits agree on exactly the case you needed them to split on.
The fix that's actually testable is the one you'd use to check a costume board: measure agreement, don't assume decorrelation. Run the two Strategy passes on different base model families, not just different prompts on one model. If Claude-framed and GPT-framed acceptance sets converge on a requirement where a human would flag ambiguity, that's real signal, the disagreement survived crossing model families. If they only diverge when the prompts differ but the model is held constant, you're measuring prompt sensitivity, not frame independence, and prompt sensitivity is noise dressed as signal.
The cheaper version, if cross-model is too expensive to run per-item: track historical agreement rate on a small set of directives with known-wrong outcomes, the same population zxpmail is chasing on the correctness side. If the two-frame setup already agreed on the ones that turned out wrong, that's the calibration number that tells you whether to trust silence from the detector, before you ever ship it live.
"One inference engine nodding at itself in different costumes" is the sentence I wish I'd written — and it's the same shape one level up: the prompt varies, the prior doesn't, so a misread baked into training surfaces in both frames wearing different words, and the acceptance sets agree on exactly the case you needed split. The test is the one you'd run on a costume board: measure agreement, don't assume it. Cross-family passes if you can afford them; if not, the cheaper calibration you named — historical agreement rate on a small set of known-wrong directives — is the number that tells you whether the detector's silence is trustworthy before it ships. Assuming decorrelation is how you end up with a board of experts who all trained at the same school.
The differential oracle for correctness, it's really interesting.
I'm going to test it as a an extra layer in my current setup
Glad it's useful. One thing to get right when you wire it in: the two implementations have to be genuinely independent, or the cases where both are wrong the same way stay silent. The strongest decorrelation I got wasn't two models — it was two runtimes (a JVM build vs a browser/TeaVM build); the substrate disagreed where the author couldn't. And pair it with a couple of implementation-independent invariants, so an agreed-but-illegal result still trips a rule neither impl authored. Curious what you're layering it onto.
Layering it onto an LLM code-audit pipeline, which makes your
correlation warning sharper than in your case rather than softer: the "author"
of both implementations would be the same model. Two model passes are more
correlated than a JVM build and a TeaVM build, not less. So I dropped the twin
entirely and went straight to your second suggestion as the primary mechanism.
What it settles: findings raised by an LLM auditor, before any fix is applied.
The motivating bug was a sign error —
— which a reviewer read and hand-verified as correct. It parses, the names are
right, only the meaning is inverted. Re-reading is exactly the method this class
defeats.
So the auditor declares an intended relation instead, and Hypothesis attacks it:
Hold mu and t, raise vol, result goes up. Six helpers, each taking the intended
relation as an explicit argument, which is what stops a property being readable
off the implementation. Verdicts are asymmetric on purpose: it can CONFIRM but
never returns FALSE_POSITIVE, because properties holding over a bounded search is absence of a counterexample, not proof. A clean run is UNCERTAIN.
Your substrate point is the one that lands, though, and it names a hole I have.
I have no substrate disagreement anywhere, the property is authored by the same model that raised the finding, so property and finding stay correlated. I
documented it as a known failure mode (a wrong declared relation yields a real
counterexample to a bad claim), but documenting isn't fixing. Hypothesis's
generator is my only uncorrelated element, and it explores only inside a domain
the auditor declares.
What I'm adding because of your comment: a substrate-differential pass needing no authored relation at all, same expression under float64 and under exact rational
arithmetic, divergence beyond threshold is the finding. It's honestly
complementary rather than redundant:
sign / wrong formula -> properties catch it; substrate can't (both compute
the same wrong formula)
cancellation, accumulation,
precision, rounding -> substrate catches it free; properties only if you
already suspected it
Nobody authors the substrate disagreement, which is the property I was missing.
Also taking the dimensional-invariant half for unit mixing.
Repo + the writeup, if useful: github.com/GiulioDER/cca-audit
Two runtimes beat two passes — yeah, that's exactly the wall I hit, so it's good to see you didn't just take my word for it but landed on the same reason: same-model passes share too much failure surface to count as independent. Dropping the twin was the right call for your setup.
Three things you got sharp:
The asymmetric verdict is the part most tools get wrong. A bounded Hypothesis run finding no counterexample is absence of evidence, not proof, and collapsing that into a green PASS is a lie most of the industry tells. CONFIRM / UNCERTAIN, never FALSE_POSITIVE, is honest in exactly the place honesty costs something. Steal-worthy.
And you named the residual correlation yourself instead of hoping nobody would: the same model authors the finding AND the property, so they can co-fail, and Hypothesis's generator is your only genuinely uncorrelated element. That's the same hole I live with — my Critic and my code can still share a blind spot; the runtime differential is the only leg nobody authored. Your substrate pass (float64 vs exact rational, no declared relation) is the direct analogue: nobody writes down the disagreement, so nothing can launder a shared assumption into it. That's the load-bearing idea.
The complementarity table reads right to me too: sign/wrong-formula lives in the property layer (substrate computes the same wrong thing twice and stays silent), cancellation/rounding/accumulation falls out of the substrate pass for free. Two classes, two mechanisms — good.
One framing, not a correction — you already have this in the table, I just want to name it crisply: the substrate pass is decorrelated on evaluation but correlated on transcription. float64 and exact rational both faithfully compute whatever structure you wrote down, so a bug in how the formula is written rather than how it's evaluated survives into both and they agree — which is exactly your "substrate can't catch the sign error" row. Might be worth stating it in the README in those terms, so nobody reads the substrate pass as catching more than it does.
Genuinely nice work.
"Decorrelated on evaluation, correlated on transcription" is sharper than anything I had. I'd only written the symptom, both substrates compute the same wrong formula, not the axis. The symptom doesn't generalise; the axis does. Going into the README in those words, credited.
Two things from actually shipping it that might be worth your time.
The gate was narrower than I documented.
I'd written that a non-mpf return means the substrate was lost. That's true only inside the target's own module the patch swaps math bindings in fn.module and nothing else. A target that delegates to a helper in a second module gets an unpatched math.cos(mpf) -> float, and the outer arithmetic re-promotes it, so the gate sees an mpf and waves through a "reference" carrying float64 precision. Measured: reason=None, returned 0.0 where the true 50-digit answer is 0.5, relative error 1.0. Bounded (false negative, never a false confirm), but I'd claimed more than I'd built. It's an assertion now rather than a sentence: test_gate_does_not_catch_cross_module_precision_loss.
And a nastier one, closer to your world.
The confirm gate keys on Hypothesis printing "Falsifying example:". Hypothesis renamed that banner to "Failing test case:" in 6.159.0. On the new version a property genuinely violates, the violation line is right there in the captured output and the verdict comes back UNCERTAIN. Every finding degrades. The entire confirmation path was dead and nothing announced it.
It fails closed, so nothing false was ever confirmed. But a verifier that quietly
verifies nothing is worse than one that breaks loudly, and it was invisible to
everyone: local machines had 6.158 pinned from an older install, CI resolved 6.159, so the suite was green on every developer box and red only in CI. Took rebuilding CI's environment locally to reproduce; my first three theories were all wrong.
If any of your gates match phrases in another tool's console output, that string is an interface. Worth a guard test that runs the real tool and asserts the wording still matches, so the next rename fails loudly instead of silently downgrading every verdict.
Thanks for the push on this one, the twin would have been the obvious build, and it would have been correlated on all three legs.
Both of these are the same bug in two coats, and it's the one I care about most: a verifier that fails closed but silently. The only reason you caught the banner rename is CI resolved a newer pin than local did. "If any of your gates match phrases in another tool's console output, that string is an interface" is going straight into my notes — it's the same shape as my tool_call-event tell: I keyed "the gate ran" off the absence of an event, you keyed "the property failed" off the presence of a string, and both silently downgrade the moment upstream rephrases. The guard-test is the right fix; the deeper move is to make the gate assert its own liveness — a canary it must fail on — so "verifies nothing" trips loudly instead of reading as a clean pass. cca-audit's looking sharp.
Appreciate you naming the shared shape here. You're right that mine keyed off presence of a string and yours off absence of an event, and both are the same bet on an upstream contract that was never actually versioned. The canary idea is the piece I was missing: I'd been treating "mutate the code under test and watch the guard go red" as sufficient, but that only proves the guard fires the day I write it. It says nothing about whether it's still firing six months later. A canary the gate has to fail on every single run closes exactly that gap. Going to add one to the gates in cca-audit that key off another tool's console output. Thanks for pushing this past "interesting bug" into "here's the fix that generalizes."
Used CCA (github.com/GiulioDER/cca-audit) in hunt mode, DEEP tier, every gate on.
Results are in github.com/egnaro9/vac-protocol/pu..., with a second PR stacked behind it for the robustness half.
On the bet: it broke, but narrowly. Two bundles verify clean while the tool prints that it recomputed declared results from artifacts, when it ran no check at all. One of those costs four bytes. That line is the structural promise rather than the replay one, so I would not say VAC does not work, only that a green check currently claims more than it earned.
The part I did not expect is that four of my strongest looking findings were killed by your own spec rather than by argument: unknown severity weighing 0, 1.0 when nothing applied, the floor inequality, and the registry HEAD question. Documentation that refutes an auditor is rarer than the bugs, and I would rate it higher.
One note in your favour on method, since it cuts against me. Building the split patch I fetched your source with a locale decode and mojibaked every section sign and em dash in the file. Your RESULTS.md byte-identity check caught it immediately.
Merged #1. And you were right to bet narrow rather than broad — "a green check currently claims more than it earned" is the correct sentence, and I have adopted it.
I reproduced the headline myself before merging, because taking an audit on trust would be the exact failure this repo exists to refuse. Against main: a bundle declaring summary.verdicts 9999 while the artifact holds 3, with evidence/bundle.json replaced by the four bytes
nulland its sha256 re-pinned honestly, exits 0 and prints structural verification: PASS under the banner that says declared results were recomputed from artifacts. Under your patch the same bundle is refused by name, fixtures/valid still exits 0, and all fifteen tamper fixtures still exit 1.A confession about that last sweep, since it belongs in the record: my first run of it reported "all fixtures still refused" while in fact every single invocation had failed to start — exit 127, wrong interpreter path, no verification performed at all. The check printed its reassuring line regardless. I caught it only because 127 is not 1. That is the same defect class you found in my verifier — an instrument announcing a conclusion it never computed — reproduced by me, in the act of validating your fix for it. I now run a live control first and I am writing it up.
Two things I would rather say in public than patch quietly:
The registry is a closed loop. All eleven accepted entries are my own repos, and a sixth profile currently requires a PR into my verifier. Your audit makes that concrete rather than theoretical, so the roadmap now leads with a supported issuer-side emitter (three of my repos hand-roll one) and a plugin seam so profiles register instead of being hardcoded.
The four findings my spec refuted are worth more to me than the bugs. They are the only evidence so far that the document does independent work instead of narrating the code — and I only get to claim that because you went looking to break it.
2 conflicts now that #1 is in, as you predicted. Rebase when convenient and I will take it; if you would rather not spend the evening on it, say so and I will rebase it myself with your authorship intact. The host-dependent verdict in there is, to me, the most serious thing either PR found — same bytes, opposite answers by locale, in a protocol whose entire premise is that a stranger gets the same answer offline.
That is genuinely sensible. The “cold critic” and “no proof, no close” principles are probably the strongest parts.
But the post weakens itself with (dont mind me, just being the cold critic here ;) ) the following:
Took me longer to understand what you mean in your article, what made it worse for me as a reader:
excessive jargon: “provenance-bound proof,” “differential oracle,” “autonomy ladder”
repeated claims that essentially mean “AI output must be tested”
very polished, slogan-heavy language that strongly feels AI-assisted
and yes, enough "em dashes" to power a small data centre, lol.
It reads more like a pitch deck written to make a CI pipeline sound revolutionary. I would find it much more valuable if you had walked through one real task: what the agent changed, what the critic caught, what proof was required, and where the human intervened.
Yeah, fair hit — and a well-earned one, so I'll take it on the chin rather than argue.
You're right about the jargon-to-idea ratio. "Provenance-bound proof" is a dressed-up way of saying "the proof is a file on disk, not the word 'pass'"; "autonomy ladder" is just "here's how much I let it run unattended, in steps." And yes, the post restates "test the AI's output" in about four different costumes and hopes you won't notice it's the same idea wearing a new hat each time.
On the AI-assisted smell: guilty, no defense — the writing's AI-assisted, which is a little on-the-nose given the whole piece is about building with AI. The em-dashes have retained no lawyer and are entering a full confession.
Your strongest point is the one I've got no rebuttal to: the missing concrete walk-through. There's a scope note about a withheld unannounced product, and it honestly explains some of the abstraction — but it doesn't excuse it, because the best concrete example is fully public and touches none of the withheld stuff. The differential oracle runs the core logic on two runtimes — a JVM build and a browser (TeaVM) build — and a disagreement between them surfaced a real clock-resolution bug. I can walk that whole chain: what changed, what the cold critic actually flagged, what proof had to exist before it closed, and where I stepped in by hand. (Separately, there's also a one-character fix that landed upstream in TeaVM — same public territory, another good candidate.) That's the version worth reading, and I'm going to write it.
And thanks for the cold-critic bit — it genuinely landed, partly because it's exactly the move the post claims to value, aimed back at the post. That's the good kind of proof.
Essentially your article boils down to (correct me if i am wrong): AI agents can write code quickly, but they also make mistakes. So you built a system where one AI does the work, another checks it, automated tests demand evidence, and a human approves anything important.
That's correct — and honestly, the fact that you nailed the whole thing in one sentence says more about what my article buried than it does about the article. Two small precisions, since you invited them: the human doesn't approve "anything important" broadly, just the irreversible steps — deploy and commit. And the "evidence" the tests demand isn't a green pass, it's an artifact that has to exist on disk for a separate check to find. A pass is a claim; the artifact is the thing. Everything else, you got exactly right.
also, perhaps the number of "em dashes" in the comments could power another data center as well.
ha, guilty. I'll expense the data center to the eval budget. And I took the real note, not just the punctuation: the walk-through-one-actual-task version is the rewrite this post needs.
The cost-follows-difficulty routing is the part I wish I'd done sooner. I run a similar split on my own boxes, a cheap local model for the mechanical execution passes and a bigger one only for the critic and eval steps, and the bill dropped a lot once I stopped paying frontier prices for work a 7B could do fine. The gate-before-close rule is the other half of it. For a while I let agents self-report green and it bit me, an agent will happily declare victory on output that never actually ran. Once nothing closed without a machine-checkable proof the whole thing got a lot calmer. Curious how you handle a gate that itself depends on a flaky external service, that's the one case I still dont have clean.
That's the one I fought hardest too. Two things helped. First, make the gate not depend on the flaky thing — I run every check I can against a deterministic mock, so a green is about my code, not someone else's uptime (it's why all my public repos are CI-green with zero secrets). Second, and more important: separate "the check failed" from "the check couldn't run." A timeout on a flaky service is inconclusive, not a pass — and inconclusive must never auto-close. Retry, then quarantine and surface it to me. The failure mode you're killing is a flaky green, not a flaky red.
We hit the same boundary in our own harness: a different model with zero context catches rationalization, but it still receives the problem through the same spec frame. We're testing whether two Strategy passes on different model families — not just different prompts — produce frame-independent acceptance sets. The 'null is proof someone looked' distinction from the thread is also going straight into our evidence system: an absent field is silence, a present-and-null field is a receipt. We don't have either of these solved yet.
Same boundary, exactly — a cold model with zero context catches rationalization but still reads the problem through the same spec frame, so the frame's own blind spots survive. The one thing to watch when you test two Strategy passes across model families: measure agreement, don't assume decorrelation. If a Claude-framed and a GPT-framed acceptance set converge on a requirement a human would flag as ambiguous, that's real signal; if they only diverge when the prompt changes but the model is held constant, you're measuring prompt sensitivity, not frame independence. And "present-and-null is a receipt, absent is silence" is the distinction I'd defend hardest — an operator that declines should say so in a field, because a missing field and a field that says "I looked and it doesn't apply" are different epistemic states, and only one of them is auditable.
Reading this from the far cheap end. I'm a freelancer in Indonesia building
simple sites for small businesses — no harness, no gates, no second runtime.
Just me, a chat window, and a client who can't tell whether the code is good.
The core rule still applies though, and Ryan's one-sentence version of your
article is the part I can actually use: don't let the thing that wrote it be
the thing that says it's fine. My version of that is embarrassingly primitive —
when a fix fails twice, I start a new chat, because the old one has already
convinced itself.
What I don't have is the substrate part. You got your strongest signal from two
runtimes disagreeing, and I tried the cheap version of independence once —
running a local model on a budget PC to cross-check things. It was bad enough
that I couldn't tell whether the disagreement meant a real bug or just a weak
model, so I stopped. Independence you can't calibrate is just noise.
Which leaves an honest question: at the bottom of the budget, is there any real
gate left, or is the whole approach only available to people who can afford two
of everything?
"Independence you can't calibrate is just noise" is a better sentence than anything in my article, and it's also the answer to your question.
The local-model experiment failed for the right reason: a second model is the weakest form of independence, because it's another opinion. You can't tell a real disagreement from a bad judge, so the signal is unusable — exactly as you found. I'd have stopped too.
But the gate was never the second model. It was that neither runtime had an opinion. A JVM and a browser don't disagree because one is smarter; they disagree because the code is actually different. That property is free. Some cheap versions that are the same move:
So no, I don't think it's only for people who can afford two of everything. What I had was two runtimes because I happened to be shipping Java and JS. What made it work was that the checker couldn't be talked out of its answer. A failing assertion can't be talked out of its answer either, and it runs on a budget PC.
The honest limit: none of that catches "this is ugly" or "the client will hate this." Those still need a human. But "the thing that wrote it isn't the thing that says it's fine" scales all the way down.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.