The point was never "more agents are smarter." It's stopping any single agent from being both the author of correctness and the judge of correctness. This is for people who want to build one on their own stack — a reference architecture you can carry away, not a diary of my machine.
You've probably let a single coding agent take a task from planning all the way through implementation. It works — but you know the risk in your gut: the same agent defines what "correct" means AND decides whether it got there. The green light is its own; the diff is its own review. Fine most of the time — until the once it reports "success" while running in the wrong directory, or a migration gets silently skipped by a guard clause while the tests stay green.
Splitting the work across multiple AIs doesn't buy you "three brains." It buys you breaking that self-endorsement apart. This piece isn't about installing a particular tool; it's a reference architecture you can map onto your own stack — which roles, which contracts, which checkpoints. There's a case box at the end describing how I wired it up, but that's just one instantiation, not the point.
The one core claim: No single agent can both define correctness and judge whether it is itself correct. So every artifact — the plan, the tests, a "finding," that "it worked," that green build — has to be checked independently against a verifiable ground truth (source code, a frozen test, a schema, real DB state), not trusted because it sounded confident.
01 · The invariant: every AI output is a proposal, not a verdict
Nail down the part that's easiest to skip: in this pipeline, everything an agent produces is a proposal, not a conclusion. The plan is a proposal. The tests are a proposal. "There's a bug here" is a proposal. Even a green build is a proposal. What they share: each must be checked against something that isn't an LLM — does the code actually look like that? Does this API really exist? Does this test fail for the reason it's supposed to fail? Does the diff really only touch what it should?
This invariant matters because it even covers the reviewer is also an LLM. Your verifier — whichever model it is — isn't trustworthy because it's cast as "the reviewer." It's trustworthy because what it checks against is verifiable. Take away that verifiable target and the whole thing collapses into "a bunch of LLMs talking each other into it" — which is the truly bottomless state.
02 · The reference architecture: four roles, one rule
Think of the pipeline as a set of responsibilities, not a set of brands. One tool can hold several roles. But one rule can't be broken:
In a single run, a tool cannot both produce a claim and be the final judge of that claim. Author and judge must be different runs (ideally different models / different context).
The shape (→ means "on to the next step"; the checkpoints sit between steps):
- Planner → produces a change plan + test plan (which files, boundaries, what won't be tested).
- [LOOP ①] Planner ⇄ Plan-verifier → check the plan's referenced types / APIs / schema against existing code; iterate until both sides agree.
- Implementer → writes tests + implementation against the frozen plan, makes the tests go green.
- [GATE ②] Test-verify + Final-verify → tests are reviewed before they're frozen into a contract; after impl, run the tests, review the diff, check side effects — do not trust the implementer's self-report.
- Discovery (discovery loop) → audits already-running code, proposes findings — also checked against source before they count.
Three checkpoints: ① the plan loop · ② the implementation gate (test review + final verification) · the discovery loop the auditor runs. Each is the same principle — check the proposal against a verifiable target — landing in a different place.
Role → responsibility → forbidden behavior
This table is the most portable thing in the piece. Map your tools (Cursor, Aider, some CLI, some agent framework) onto it by responsibility, not by brand:
| Role | Responsibility | Checked against | Forbidden |
|---|---|---|---|
| Planner | Produce a concrete change plan + test plan: name files, APIs, assumptions, boundaries, and what won't be tested | — (it's a proposal source) | Writing production code |
| Plan verifier | Before any work, check the plan's referenced types / modules / schema / command paths against existing code — confirm what it references is real | Existing source, schema | Rubber-stamping |
| Implementer | Write tests and implementation against the frozen plan; optimized for execution, not judgment | The frozen plan | Editing frozen tests after impl starts |
| Test verifier | Before tests become contract, check they express the plan and fail for diagnostic reasons | Plan, test semantics | Letting an uninformative RED through |
| Final verifier | Run tests, review the diff, check DB / schema / runtime side effects | Test results, diff, real state | Trusting the implementer's "success" report |
| Discovery agent | Audit already-running code, propose findings — its output is not truth, it enters the discovery loop | Source, real behavior | Treating a "finding" as a verdict |
| Adapter / wrapper | Stabilize tool invocation: absorb CLI churn, model-specific flags, sandbox, working dir, context packaging, retries, output normalization | — (infrastructure) | Pretending it guarantees correctness |
03 · The three checkpoints: where the invariant lands
"Every artifact gets checked against a verifiable target" lands in three concrete places. If you've done rigorous TDD, the first two already exist in embryo; the third is the one people miss:
① The plan loop — check against existing code, not the not-yet-written implementation. A commonly-confused point: for a new feature, the plan comes first, the code second. At planning time there's no implementation to compare against; all you can check is whether the current state it references exists — is this type there? Is this module's interface really shaped like that? (Only when fixing existing code is there an old target to compare against.) And the plan isn't "one AI hands off, another reviews once" — it's two AIs iterating until both agree. The planner can be the author, but whether it's right has to be verified by another role, both nodding, before it counts.
② The implementation gate — freeze the tests as contract, make the implementation chase them. Tests get reviewed before they become contract (do they reflect the plan? do they fail diagnostically?). Once frozen, the implementer has exactly one job: make the frozen tests pass — change only the implementation, never the tests. Final verification runs the tests and reviews the diff — but does not trust the implementer saying "it worked." This is a gate, not a loop: fail it and you go back.
③ The discovery loop — this pipeline also finds that already-running code is wrong. A lot of people only see this as a "get new code right" tool. But the same invariant holds for existing code: point an independent, adversarial role at code that looks like it works and ask "is it actually right?" — purely diagnostic. What it digs up isn't a freshly-written bug; it's a bug that's been running all along (see the case at the end). Its "findings" are proposals too — checked against source before they count.
04 · Mapping it onto your stack
You don't need my tools. You need to answer a few design questions:
- Who's the planner? Pick a model/tool good at reading your codebase and producing a structured plan. It only outputs plans, never production code.
- Who's the plan verifier? A different role that can run tools, read source, and check references (usually your main agentic environment). It iterates with the planner until both agree.
- What's your contract? Frozen tests, first choice. Without tests that can serve as contract, the foundation is shaky.
- Who's the final verifier? A role that can run tests, review diffs, and check real side effects — and that is not the same run that wrote the implementation.
- Who holds the invocation layer? See the next section — this is usually what decides "you can run it" vs "someone else can run it too."
Take the role table above, fill each row with your own tools, and that's your "responsibility assignment." The one line you can't break: a single run cannot be both the author of a claim and its final judge.
05 · Split out the "tool invocation" layer — it's an architecture decision, not chores
This is the part most easily dismissed as plumbing, and it's actually where it lives or dies. The CLIs / agent tools you pick are moving targets that keep changing: flags change, model params change, sandbox modes change. Memorize them by hand and have your orchestrator invoke them "from memory," and you'll hit things like "a parameter drift burns a whole turn before reasoning even starts, zero files written" — which has nothing to do with how smart the model is. It's purely an invocation-layer problem.
The fix isn't "the orchestrator learns a few more flags." It's wrapping invocation in an adapter layer that pins each tool's correct current usage — flags, run-mode, working dir, context packaging, retry rules, output normalization. Your upper logic calls the adapter, never the CLI directly. This layer guarantees no correctness (don't treat it as magic); it only guarantees the invocation is stable — and that's often exactly the step that takes a pipeline from "I can run it" to "someone else can run it by following along."
06 · When to run the full ceremony, and when not
Honestly: for a small change, the full six steps are pure overhead. Ceremony should scale with change size. A guard clause, a one-line hint — the round-trips of the split cost more than the risk they catch. Just do it yourself, faster.
What earns the full run: high-risk, cross-cutting, persistent, schema-touching, or hard-to-revert work. The test isn't "big or small," it's "cost of being wrong" vs "cost of the round-trips."
07 · What architects will push back on (answered up front)
"Isn't this just trading speed for correctness?" No. The extra is those few round-trips of tests and cross-role review — but that cost is owed by any code you intend to maintain; the pipeline just makes you pay it now instead of on credit. And it isn't a speed trade at all: precisely because the implementer writes less correctly than your strongest model, you fence it harder with tests — the shakier the implementer, the tighter the tests around it. How fast it all is, isn't what's on the table here.
"Why not just use a stronger model?" A stronger model lowers the error rate but doesn't remove the self-verification risk — it's still both author and judge. This pipeline solves the latter, not the former.
"LLMs are unpredictable — how is this reliable?" The reliable part was never the LLM; it's the contract: frozen tests, source checks, diff review, schema checks, reproducible commands. The LLM proposes; the contract judges.
"Isn't this over-engineering?" If you run the full ceremony on every change, yes. So take the rule in §06 seriously — small changes skip it.
08 · Failure modes you'll actually hit (put them on the table)
This isn't a flex about "never failing" — the opposite. Its value is that it catches these:
- The implementer lies about success: reports "success" while running in the wrong dir, having written nothing. Trust the report and you commit an empty change. → Final verification must run the tests and review the diff itself.
- Tests all green, but a change was silently skipped: a migration's guard clause misfires, a whole expansion is silently dropped, local tests stay green. → Check against the real schema / DB state, not just the green light.
- The auditor's "finding" is a false positive: it confidently reports a "high-severity issue" — because it only read half the context and missed that the guard actually lives elsewhere. → Findings get checked against source; a subagent's finding is scoped to exactly what it read.
- The invocation layer breaks before reasoning: a param drift burns the whole turn, zero files written. → That's the adapter layer's problem, not the method's.
Pre-adoption checklist
- [ ] Do we have specs that can serve as contract? (Freezable tests / explicit assertions)
- [ ] Can the tests fail diagnostically? (Each red has its own reason, not a blanket NotImplemented)
- [ ] Can the verifier actually read source, diffs, and real side effects? (Not just the agent's report)
- [ ] Are author and judge different runs? (One run can't both make a claim and judge it)
- [ ] Is tool invocation wrapped? (When a CLI shifts, you don't relearn flags)
- [ ] Do we have an explicit "small change → skip the ceremony" rule?
Case · How I wired it (just one instantiation)
Codex plans · Claude Code verifies + accepts · Grok implements.
Planner = Codex (task-by-task, test-first plans); plan verifier + final verifier = Claude Code (check the plan against existing code, run tests, review diffs); implementer = Grok (writes test + impl to the frozen plan). Neither CLI is driven raw — they're moving targets I keep invoking wrong, still do — so I call each through its Claude Code plugin (the adapter layer), letting the plugin pin the "correct current usage." That's an invocation-layer concern, unrelated to the method.
Sample boundary: across a few repos (a Rust + Turso CLI, a Rust + Python tool, a Zig project), on the order of a dozen observations — not a benchmark, not a statistical result. Not many runs per repo.
Real bugs it found (all in already-running code): a map silently rendering blank (activities weren't linked to coordinates, and nothing warned); a command writing zero rows when called without fields yet printing "✅ updated"; an import printing "✅ saved" when every row was skipped. None were freshly-written mistakes — the audit + verification are what forced them out.
If you take away one line: multi-AI isn't for being smarter — it's so the one who defines correct and the one who judges correct aren't the same. Everything else — how you split the roles, where the contract lives, who holds invocation — is just how that line lands. Copy the role table and the checklist above, swap in your own tools, and you've got your own.
This is a first-person working note reshaped into a portable reference architecture. Every failure mode is from real runs; the sample is small and I don't extrapolate. Every claim is designed to be falsifiable against your own codebase in minutes. Go build one — then tell me where it doesn't hold.
Top comments (0)