The agent that writes the code cannot read the tests that grade it

TL;DR:

Letting an AI coder review its own code and tests leads to compromised code quality, test-softening, and false positives. This article introduces a practical, multi-agent governance layer inspired by the Ralph loop but hardened with two core constraints:

1. The Locked Oracle: The coding agent is completely blocked from reading or editing the test suite, which is frozen before coding begins.
2. The Cross-Model Checker: Code review and testing are strictly delegated to a different model family to significantly reduce blind spots and self-verification bias.

Press enter or click to view image in full size

The Correcting Loop Architecture: The Director freezes the tests in advance (Locked Oracle). The Builder codes blindly without test access, and a cross-model Reviewer evaluates the diff against human Gates to prevent bias

Developers experimenting with auto AI coding, often hits the wall of self-verification bias, where the agent quietly softens your tests just and happily claims success. I have just shared a loop-based AI builder I’ve been using internally for some time. This builder gives me the best results from anything I have tested before in solving this problem.

The strongest impact that I see is in significantly reducing the ‘faux tests’ problem i.e. tests which are softened so that the app can be marked completed. An additional strong impact is on the PRD compliance, where I now get way fewer occurrences of the AI-dev process ‘forgetting’ parts of the requirements.

It relies heavily on loop concepts for automated AI coding, with some twists that I believe result in a more solid development process. You can basically look at it as a Locked-oracle Cross-model variant of the Ralph loop (an iterative AI-coding pattern).

It also borrows heavily from several other ideas floating around the AI dev sphere and combines them into something that has worked well for me in developing multiple smaller-scale projects.

These ideas, specified below, are not mine! I happily copied them from others, and you’re welcome to do the same with the model I’m sharing here.

Since the AI sphere has not yet reached the stage of naming consensus, I will use the names I believe are most common. I could be wrong.

The ideas that I borrowed

Ralph loop

From Ralph I’ve borrowed the most. Especially the part of looping with a refreshing session context in each unit of work in the loop, and the usage of git commits as both source-of-truth and checkpoints.

I did NOT take from Ralph its eternal loop, since I believe too many code/review iterations are a good indication of a broken process that needs to be stopped and reviewed.

I also did not take its single-agent self-review, which I think is inherently wrong for a valid code review. Instead I used:

Locked Oracle

Here I provide strong limitations to what the rolling dev loop is allowed to access.

The rule is simple: the builder may read all code, even code which appears unrelated to its current task. But it cannot read or modify the frozen acceptance suite used for system verification and later acceptance. Those are written once as part of system design, and are then locked before the first line of application code exists.

By doing this I solve two common problems facing test design:

1. Test-biased code, where the builder, knowing what the test checks for, creates a non-complete solution that still manages to pass the tests.
2. Test softening, where the process slowly relaxes the tested conditions to the point where the test no longer presents the hard demand set by the PRD for correctness.

With this addition I have way more assurance that a green test suite actually means the product is behaving as it should, rather than the agent having adjusted its own success criterion.

The lock is not an instruction in a rules file. It’s a deny rule plus a pre-tool hook, evaluated outside the model, on paths the model cannot influence.

# EXIT CODE SEMANTICS — the easiest way to disable this guard accidentally:
# exit 0  allow the tool call
# exit 2  BLOCK the tool call; stderr is returned to the model
# exit 1  does NOT block in Claude Code; the tool call proceeds.
# Using the conventional Unix failure code here silently disables the guard.

Cross-Model checker

Classic Ralph has one agent and no independent verifier. The same agent writes the code, writes the tests, and decides it is done. After using this approach for some time I dropped it. I simply could not rely on its correctness.

Instead, I found it more valid to replace it with a check model where each review/test session is done by a new agent instance belonging to a different model than that of the builder.

Now why is this important?

Because self-verification is the root of evil. The same model that generated the code always carries some ‘blind spot’ into the testing phase, where the considerations and context used to write the code find their way into scoring it. For this reason you would never assign the human coder to be the code reviewer.

Using a different model family (e.g., Claude building, GPT reviewing, or Gemini testing) brings a fresh set of reasoning and training biases, making it way more likely to intercept errors.

The fact that each reviewer agent is restarted, hence provided with clean context, also strengthens our review phase, by preventing ‘context rot’, i.e. a context so bogged with prior tasks that you can no longer rely on it to provide reasonable answers to the current task. Restarting the agents indeed solves this problem, admittedly at the cost of a higher learning curve when introducing each reviewer instance to the current code.

Inner/Outer loop model

The outer loop iterates execution steps to exhaustion, and an inner loop is deployed to verify by fix & trial the correctness of each step’s execution.

I have made multiple attempts not to use this model, which looked overly complex to me, but never managed to get it right using a single loop. I tend to think the reason is inherent: that you cannot really automate dev tasks without these two levels of looping.

Frozen judge model

The catalog is written before a single line of application code exists. That timing is what makes the lock enforceable rather than aspirational. A catalog written after the code exists is already shaped by that code, and by then there is nothing left to freeze.

Differential Agent Skills

This model relies on 3 different entities:

• Director for overall orchestration and high-level review
• Builder for coding
• Reviewer for code review compared to design docs

Each of those is provided with different skills required for their role.

Not every role needs a strong model

The three roles do not need equal model quality, and treating them as if they do is the most expensive mistake available here.

Builder quality matters least. Every defect it produces gets caught by a frozen oracle it cannot influence. A weaker builder writes worse first drafts and burns more repair rounds, but it cannot ship something broken past the judge. That is what makes a cheap builder safe in this setup when it would not be safe in an ordinary one.

Director quality matters most. The Director writes the judge, and nothing downstream checks the judge except a human reading it once. A weak Director produces a shallow test catalog, the loop runs green all the way through, and the gap only surfaces during human testing. There is no automated recovery from that.

Reviewer quality matters in between, and its model family matters more than its raw capability. A reviewer from the builder’s family shares the builder’s blind spots and misses the same cases.

One practical note: don’t pick the builder model by benchmark scores. Measure repair rounds per step on a throwaway project. One or two is fine. Three or more consistently means the model is failing the spec rather than the gate, and a cheap model that needs three passes is not cheap.

The cost saving from this allocation is real, but it’s a side effect. Even if tokens were free, the family separation between builder and reviewer would still be required.

Why this works for me

This model has several benefits that make it great for my needs, which are rapid development of small-to-medium projects.

It audits the PRD before anything else

It runs extensive tests on the product book (PRD) and effectively blocks development until the PRD is corrected to be both self-contained and easily breakable into development steps. In general, ‘AI-dev friendly’.

It contains a double-loop mode

The entire project gets automatically broken into several execution steps so that each step is manageable in a single AI builder session with no significant context window decay.

Note that there’s an inherent plus in giving the AI the task of doing the breaking (I could have done it myself): AI should know best the task length for keeping the context window effective.

After each step completion, the process delegates to a newly created agent of a different model for ‘corrective review’. Corrective in the sense that the process expects the review to return with a changelist and handles the process of review, request-for-changes, new-code-review.

While the decision to force the reviewer to be of a different model than the builder is self-explanatory (you don’t want the job reviewer to think like the worker), the issue of restarting a new reviewer agent for each step is something I was not sure about. There are lots of benefits in keeping a reviewer context up and aware of all aspects of the entire project. At the end I went with restarting the reviewer so as to keep its context window clean, and in doing so got good enough results.

It never weakens a test to make it pass

It forces an aggressive policy of testing each stage AND of never weakening tests just so they pass. A failed test means an action item to be fixed in code.

As part of these testings, the automatic process will deploy a local realistic test env (DB in Docker container, node server, more components may be added) and test runs realistic scenarios locally.

It minimizes human intervention

It is very good at minimizing human intervention in the loop (it was built for it). The only human gates defined here are:

• Gate A, where a human approves processed design/architecture docs, project steps, and directory structure.
• Gate B, where a human reviews and approves the implementation of the 1st step.
• Gate C, where development plus local testing is basically completed and the human reviews the complete system (and in real-world issues a change request…).

Even this level of human intervention can be reduced. Just add a short prompt stating that the agent:

“should work in full automation mode and circumvent any product book review and permission requests but must not reduce the product’s quality, and especially should retain working in steps and performing extensive per-step review by a different model”

Here the agent is basically on its own, so this should be limited to internal/throw projects.

It defaults to boring

The model defaults to a very basic manner of development where the ‘solid simple’ is almost always selected.

Backend will run on Node, DB layer will be an agnostic RDBMS, mobile apps will be written over Capacitor, but if app defs do not allow for it, will fall through to React Native, and only if impossible (e.g. complex animations, extensive mobile OS API needed) will work in native code.

All code, excluding scripts, mobile native if needed, etc., will be written in TypeScript, which adds extra simplicity to the model. It also adds resilience, since in my experience the no. 1 ‘code-level bug’ in AI-generated code is getting the types wrong.

It scales without a rewrite

While keeping it simple, the model was designed to accommodate an increase in app usage at a very low cost.

The backend is instructed to be stateless so as to make horizontal scaling trivial. DB layer is built with the most beneficial indexes already declared and is agnostic, hence should allow switching to a stronger DB with very few changes. And Node itself is very efficient and scale-friendly.

All this is to say: if and when success strikes, you will probably not need to rewrite even for way more users than you started with.

Same pattern, different domain: Solidity auditing

This same pattern is even stronger for smart contract security. Before any pragma solidity\ line, a Director (say Opus) drafts the security spec: reentrancy guards, access control, overflow checks. Now these become the locked test catalog.

Now a cheap builder (say Sonnet) writes the contract. A reviewer from a different model family (GPT-4o) audits with Slither + the frozen spec. The builder never sees the audit checklist, so it cannot soften its own guards.

Used on one helper contract (< 900 LoC), the Reviewer caught a spec mismatch at Gate A that would have cost a full rewrite later.

Two notes before you try it

1. This is a starting point, not a polished product. The core ideas are what I’m sharing — the implementation is yours to adapt. That said, the repo has a demo/\ folder with a working example. Just clone and run them and you should be good to go.

2. Know the ceiling. Works well up to ~20k LOC. Beyond that, context window limits kick in. Extensible — I just haven’t needed to yet.

Get the code: CorrectingLoopManagedFlow

Found this useful? Star the repo, drop a comment, or share how you manage your autonomous AI coding loops.

Originally published on Medium