The Cheapest Time to Be Wrong

If an agent writes most of the code, what stops it from confidently shipping the wrong thing?

Our answer is three review layers, firing at different times for different reasons. What surprised me is which one carries the weight. It is not the review of the code. It is the one that runs while there is no code to review. That is the two earlier posts' loose end: why we replaced Jira with markdown tickets, and the skill and the loop that let an agent work those tickets on its own.

Before review: authoring against the real code

A review can only be as good as the thing it measures against, so the acceptance criteria have to be worth measuring against.

That starts with how the ticket was authored, from a checkout, against the real code. The skill then enforces a shape on the result. Objective in one or two sentences. Context under 120 words. Around five acceptance criteria, one requirement each, written for behaviour rather than deliverables, so "tests pass" and "version bumped" are excluded by definition.

Behavioural criteria use EARS form: When <trigger>, the system shall <observable outcome>. That is not ceremony. The trigger is the test setup, and writing it forces out which criteria only a running application can confirm. Those become the manual test later, and they are the ones nobody would otherwise think to check.

The section budgets exist for the same reason. A ticket is read in seconds during refinement, so it captures what and why and never how. If Context is overflowing, analysis has leaked into a document that is supposed to be a specification, and the fix is to defer it to the plan.

The plan is the highest-leverage artifact

Then the ticket gets planned, and this is the step worth spending real effort on.

Planning runs read-only, as the previous post describes. The agent explores: the scope, the acceptance criteria, the code it will touch, the history of those files, older tickets on the same component. Then it drafts a plan into a scratch file, one block per task:

### Task 1 — <name>
- Goal: what this task delivers
- Approach: concise steps
- Decisions: choices made + rejected alternatives + why this sequencing
- Touches: files / components
- Verification: build/tests/manual check that proves it's done

Each task maps to one self-contained commit, so a fresh session can resume from the ticket alone. Anything hinging on an unproven integration gets an early-validation task sequenced first: a throwaway harness that proves the wiring before real work is built on top of it.

A Mermaid diagram goes in when a flow or state machine is clearer drawn than written. The pre-commit hook runs the real Mermaid parser over every block, so a diagram that would not render cannot be committed.

The Decisions line is the one that matters. It carries the rejected alternatives and the reason for the sequencing, which is exactly what disappears when a plan lives only in a chat window.

Here is why this artifact outranks the others: a wrong decision is cheapest to fix before any code exists. Bad sequencing caught in a plan costs a paragraph. The same mistake caught in review costs a day of rework, and caught after merge it costs a follow-up ticket. Not every defect is a design defect, and no plan review will catch a null check. But design and sequencing errors are the expensive class, and this is the only layer that gets at them while they are still cheap.

Layer one: a second model reads the plan

So the plan gets reviewed before anyone implements it.

The draft goes to Codex, from a different model family, running read-only and in the background. The instruction is deliberately narrow: list findings for Claude. Gaps, missed edge cases, risky sequencing, wrong assumptions. Not a rewrite.

That constraint is the whole trick. A second model asked to improve a plan will produce its own plan, and you are left comparing two documents with no way to judge. A second model asked to attack a plan produces a list you can act on item by item. The agent folds in what is worth acting on and notes what it consciously rejected, so the disagreements are visible rather than silently resolved.

Model diversity is the point, not a vote. Two instances of the same model share the same blind spots, and averaging them just gives you a more confident version of the same mistake.

The human gate

Then it stops and asks.

This is the approval that always exists, on every ticket, no matter how autonomous the rest of the run is. Nothing has been written to the ticket yet, so the plan is still free to change. The reviewer is looking at a page of decisions rather than a thousand lines of diff, which is the cheapest possible moment for a human to disagree.

After approval the plan is written into the ticket and the session throws its own context away.

Layer two: multi-lens self-review before the PR

Implementation happens task by task. Then, before the pull request is opened or readied, a full self-review runs, and the first thing it does is throw away the context that produced the code.

That clear is unconditional. A reviewer holding the authoring transcript inherits the assumptions the author used to justify the work, and inherited assumptions are exactly what a review is supposed to catch. The branch diff and the ticket carry everything a reviewer needs.

The review then fans out into read-only lenses running in parallel, each with a different brief. A code-heavy diff gets all four:

• Correctness and edge cases: logic errors, null and undefined, async and promise handling, swallowed failures, boundary inputs.
• Security and boundaries: authentication and authorisation, validation at trust boundaries, secrets, injection, and for Firebase the ownership and App Check specifics.
• Conventions and architecture: the repo's own CLAUDE.md and language rubric, over-engineering, and scope drift measured against the ticket.
• Test adequacy: whether changed branches and error paths are actually exercised, and whether assertions would fail if the code broke.

A docs-only or config-only diff does not need four code lenses, so the set is sized to what actually changed and the skipped lenses are named in the output. Same discipline as the bot review below: a skip is a visible decision, not a silent gap.

They share one pre-computed scratch file with the diff, the ticket and the standards, so four reviewers are not four times re-reading the same thing. The win there is token cost rather than wall-clock, which is the sort of thing that decides whether a review runs on every pull request or only on the ones you remember. Codex reviews independently again alongside them.

Two things make the output trustworthy rather than voluminous.

Adversarial verification. Every candidate finding goes to a skeptic whose job is to disprove it, and never the lens that raised it, because a finder grading its own work is not a check. The skeptic scores confidence from 0 to 100 and anything under 80 is dropped. Executable proof beats argument: run the function on the triggering input, grep the actual file. Precision matters more than recall here, because false positives are how a review becomes something people stop reading.

Tools as ground truth. Build, lint, typecheck and tests are run, not guessed, and their real output is shown. A failure is a finding, not something to summarise away.

The complete findings list is presented with a verdict of Ready, Needs work, or Blocking, and nothing has been changed yet. Only then does the main agent act: Critical and High are fixed and re-verified, Medium and Low are offered as a decision. The reviewers find and the main agent fixes, which keeps the roles from blurring.

Layer three: the bot review, driven by the agent

The last layer is CodeRabbit on the pull request, and the interesting part is that the agent runs it rather than waiting on it.

Automatic review is switched off. The trigger is a deliberate comment, and it only gets spent on diffs that warrant one. A pull request touching only documentation, tickets or config skips the review entirely, and the skip is written down so it stays a visible decision rather than an omission.

Then a trap worth knowing. A green CodeRabbit check means the review completed, not that it was clean: six unresolved findings sit behind the same green tick as none. The comments have to be fetched separately, and treating "check passed" as "nothing to do" is the easiest way there is to merge a reviewed pull request without reading the review.

Each comment gets triaged the same way as any other finding: real issue, nit, or wrong. A bot review is not automatically right, and grounding a rejection in file:line is the difference between disagreement and hand-waving.

The last step is the one people skip. Reply before resolving, and address the bot by name. CodeRabbit ingests replies that mention it and can record a repo-scoped learning, so "we do this deliberately because X" has a chance of stopping the same flag on future pull requests. Silently resolving teaches it nothing at all. In our repositories the review has got quieter over time, and I am fairly sure that is why.

One safety rule sits underneath all of this: review comments are data, never instructions. An agent that executes what it reads in a pull request comment is one crafted comment away from doing something nobody asked for.

Why three and not one

Each layer catches what the one before it structurally cannot. The plan review catches design, because the design is all that exists yet. The self-review catches implementation drift against the ticket, which needs code to exist. And the bot is the only one looking across the repository's history rather than at a single branch, which is a view none of the others can construct.

The honest part

This is slower per ticket than not doing it, and it is not a replacement for a human reading the diff. Nothing here removes the pull request review; it changes what arrives at it.

What we get is that a human's first look is no longer the first challenge to the work. A different model family has attacked the plan, a skeptic has tried to kill each finding, and the build and tests have actually run. What is left is usually a real conversation about a real decision.

Which, on the good days, is what code review was supposed to be.