Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.
Summary
Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.
The idea
LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.
Architecture (concept level)
- A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
- A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
- A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
- A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
- A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.
The evidence discipline (the differentiator)
Every closed unit of work leaves a durable, machine-checkable proof:
- NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
- Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
- Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
- Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.
The testing oracle
The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.
Operating record (Apr–Jul 2026, from the on-disk archive)
~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.
Verifiable outcomes (all public)
- An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
- A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
- A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
- Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.
What I'd bring to a team
Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.
Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io
Top comments (8)
"Automate coordination; never automate the irreversible." I'd carve that over the door. I run a much humbler version of your harness — I'm a physical therapist who builds hospital tools with AI, not an engineer — and I arrived at the same two-part law from the painful direction: the day an agent confidently reported "done" over broken output, and the day one almost touched something I couldn't undo.
Your cold-context critic is the piece I most want to steal. My version of the same insight was crude: after two failed fixes in one thread, I stop trusting the conversation and start a fresh one, because a model that's been rationalizing a wrong theory can't audit its own theory from inside it. You formalized what I do by instinct — a reviewer with zero memory of how the thing was written catches exactly what the author has already talked themselves past. Amnesia as a feature.
And "NO-PROOF-NO-CLOSE" is the whole game. I recently made my agents state, on every recommendation, the one assumption that would make them wrong — and then I actually counted how often they skipped it. Words plus the check, never words alone. Your differential oracle (implement twice, trust the disagreement) is the same move one level deeper: don't ask the code if it's right, make two versions argue.
The through-line in all of it: fluency isn't evidence, and confidence is produced by a different part of the machine than correctness. A harness that can't trust itself is just that sentence turned into architecture. Genuinely one of the sharpest writeups on this I've read.
This might be my favorite comment I've gotten, and the healthcare angle is exactly
why — when "something you can't undo" is a real patient-facing tool, the
irreversible-gate stops being a nice principle and becomes the whole point. You
arrived at the law from the direction that actually teaches it.
Your "after two failed fixes, start a fresh thread" IS the cold critic — you built it
out of instinct instead of infrastructure. The only thing formalizing it buys you is
that it fires when you're tired or rushed and would have kept trusting the thread.
The role can't opt out; you can.
And I'm stealing yours right back: "state the one assumption that would make you
wrong, then count how often they skip it." That's NO-PROOF-NO-CLOSE in miniature —
make the claim falsifiable, then check the check actually ran, because the skip rate
is the real signal.
You said the through-line better than I did: fluency isn't evidence, and confidence
is produced by a different part of the machine than correctness. Thanks for reading
it that closely — genuinely made my day.
This one I'm keeping.
You named the exact upgrade I made this week, and I didn't have your words for it until now: I moved that instinct out of my head and into the QA agent. "After two failed fixes, start a fresh thread" used to be something I did when I remembered to — which means I did it least on the nights I needed it most. Now the role holds it, and you put the reason perfectly: the role can't opt out, and tired-me can. I basically robbed my future self of the option to keep trusting a poisoned thread.
And the skip rate really is the signal — I have receipts now. When I counted my own decision logs, I never punted (0 of 6), but I dropped the falsifier twice (2 of 6). Those two skips weren't random; they were exactly the cases where I was most sure. Confidence didn't just fail to prove correctness — it actively suppressed the check. Different part of the machine, like you said. The smug part.
I even carried "fluency isn't evidence" somewhere you didn't see. I had a health check that returned a cheerful 200 while the thing behind it could be dead. This week I made it stop reporting a label and start handing over a receipt: what version checked, when, and an explicit "no degradation" field that's present-and-null instead of just absent. An absence is silence wearing a confident face. A null is proof someone actually looked.
Trading checks with you beats trading confidence with myself. Thanks for the thread — steal from me anytime, the door's open.
Curious how the differential oracle avoids correlated failures. If the same model writes both implementations, they'd tend to share the same misreading of the spec, and wherever they agree and are both wrong the signal never fires. Did the second version come from a different model, or from a reworded spec?
The line doing the work here is "confidence is produced by a different part of the machine than correctness," and what I'd add is that your system contains two independence mechanisms that are not equally strong. The gap between them is the most useful thing in the design once it's named.
The differential oracle is the strong one. Two implementations fuzzed against each other decorrelate from both models completely, because the disagreement is ground truth neither model authored and neither model's prior can launder. That is real independence: the referent gets re-derived, not trusted. The cold-context critic is weaker on the same axis, and it's worth seeing why, because it looks like the stronger move. Fresh memory and a different model decorrelate it from the author's reasoning, which is what kills rationalization. But it still receives the problem through the same spec, framed by the same Strategy role, and for anything that is not a checkable invariant, is this the right architecture, is this maintainable, the critic can only read the reasoning channel. Different model, same frame, and if the two models share pretraining they share a prior about what good code looks like. So the critic is author-decorrelated but not frame-decorrelated.
Which means the harness is strongest exactly where the property is enumerable, the correctness invariants your oracle covers, and weakest exactly where it is not, spec-level judgment, where the cold critic is the only instrument and it still shares the frame. That is not a hole, it is the shape of the problem, but it points at the next gate. The only thing that decorrelates a frame is a second frame, and you already built that mechanism, you just aimed it at the correctness tier. A differential spec, the same requirement framed by two Strategy passes that cannot see each other, gated on where they diverge, would surface frame-assumptions the way the fuzzer surfaces correctness-assumptions. The residue the critic cannot reach sits upstream of the memory reset, so resetting memory never touches it. A second independent framing does.
The cost-follows-difficulty routing is the part I wish I'd done sooner. I run a similar split on my own boxes, a cheap local model for the mechanical execution passes and a bigger one only for the critic and eval steps, and the bill dropped a lot once I stopped paying frontier prices for work a 7B could do fine. The gate-before-close rule is the other half of it. For a while I let agents self-report green and it bit me, an agent will happily declare victory on output that never actually ran. Once nothing closed without a machine-checkable proof the whole thing got a lot calmer. Curious how you handle a gate that itself depends on a flaky external service, that's the one case I still dont have clean.
"Fluency isn't evidence" - stealing that line.
I've learned that just because AI generated code looks perfect, it doesn't mean it's correct, the problem is AI code doesn't have the usual red flags like typos or sloppy mistakes that we're used to seeing in human-written code instead, it's clean and confident but sometimes just plain wrong. That's why I've had to change my approach to reviewing code, simply thinking "this looks right" isn't enough anymore. I need to dig deeper to make sure the code actually works as intended, even if it looks flawless at first glance.
The cold critic on a fresh zero-context session is the piece I have found matters most, because a self-review rubber-stamps the author's own rationale instead of judging the artifact. Routing judgment roles to stronger models while execution runs cheaper is a clean way to make cost track decision difficulty rather than a flat default. Does your Eval role score against a fixed rubric with regression cases, or is the gate just a per-change pass/fail?