DEV Community

Iliya Oblakov
Iliya Oblakov

Posted on

Four unreviewed commits, one already merged: how my Claude Code and Codex setup got its rules

One night in October I found four test-only commits in my own projects that no reviewer had ever read. One was already merged. The other three were waiting in the merge queue.

Nothing broke. Codex confirmed all four before the queue resumed. That night is the reason the rules in my setup now live in scripts instead of in my head.

This is how the setup came together, one failure at a time, and what you can copy from it today.

Why one engine can't sign off its own work

The session that wrote a change is the worst place to decide whether that change is finished. It already believes it is.

So I split the jobs. Claude Code writes the code, and Codex reviews it read-only. I picked a reviewer from a different company because it has different blind spots. On one pull request in October the first Codex review came back with 2 blockers and 8 should-fix findings. That single review is why the loop exists.

The reviewer has to be read-only. A reviewer that can fix what it finds is a second author, and nobody reviewed that second author. Codex runs in a read-only sandbox, so every fix, small ones too, goes back to Claude Code as a fail list, and the new commit gets reviewed again.

What a second engine catches

Four findings from Codex reviews of my own pull requests, with project details removed. Each came with the file, the line and a way to make it fail.

1) A CI step turned screenshot capture errors into report rows but never counted them as problems. A timeout could finish with every report written, zero problems and a green build.
2) A guard meant to keep a publishing script away from the wrong database read only one shell variable. A target chosen through the environment file got past it.
3) A new exemption for media links ran before the scheme and host checks, so links to a local dev server and ftp addresses passed an audit built to reject them.
4) A stricter validation step crashed on one malformed field in an API response that the old code had reported cleanly, which would have stopped the rest of the audit run.

The first two were blockers.

The night the review became a file

Since that night, the review script saves the Codex verdict in a file named after the exact commit it reviewed. A new commit has no verdict until it's reviewed, so an ACCEPT from an earlier commit can't cover code nobody read. With REQUIRE_REVIEW=1 the merge script refuses any commit without its own ACCEPT file.

The script is strict about what counts as a verdict. It reads only the reviewer's final message, and it accepts the result only when Codex exited 0, the checkout didn't change during the review and the final message has a single Verdict: line. A crash, a cut-off answer and a checkout that moved mid-review all end with exit 1 and no verdict file. Two Verdict: lines count as none, because the script has no way to know which one the reviewer meant.

The bill that moved CI onto my machine

One private repository spent about 2,100 GitHub Actions minutes in its first five days of hosted CI, on a free tier of 2,000 minutes a month. Since then every check runs on my machine first, and work is pushed only after it passes there.

The local gate runs your tests on the same commit Codex reviewed. It writes a pass record only after a full pass on a tree that was clean before and after the run. It refuses a dirty tree because a check could pass on a file that isn't in the commit. The merge script refuses to push without that record for the exact commit, then merges with GitHub's sha parameter, so GitHub refuses the merge if the branch head moved in between.

My own gate runs 40 to 43 checks. Run one group after another, a full pass took 35 to 40 minutes. Split into five parallel lanes, it takes about 11 minutes on that machine.

The test that changed which model does what

On 12 October I gave both engines the same task: a Tetris implementation against 44 hidden tests, three runs each. Claude Code with Opus 5.5 at xhigh and Codex with GPT-6.1-Sol at xhigh both passed 44/44 in all three runs. The median times were 21:17 and 18:13. Opus cost 8.9 times as much.

So work goes by what a mistake costs. Bounded implementation with a clear brief goes to Sonnet. Haiku gets search, extraction and listing, where a wrong answer is cheap to spot. xhigh and max effort go to design-heavy slices, the first review of a pull request and hard debugging. The Tetris run is one task, so those last rows keep xhigh until they get their own rerun.

What the setup can't do

These checks cannot make the code correct. If Claude Code weakens a test in the same commit that breaks the code, the test gate passes that commit, and catching it falls to the reviewer. The review prompt says new tests must fail on the old behaviour and nothing gets weakened without a stated reason, and Codex checks the diff against that rule. A review can miss things.

A wrong ACCEPT looks the same as a right one. That's why ACCEPT only unblocks the pull request in my setup, and I do the merge myself.

The recorded demo of the kit 1.0.2 scripts on 23 October shows the refusing side. It ran on a throwaway project with real Codex reviews and DRY_RUN merges, and the merge script refused all seven attempts because the gate log was missing, the gate had not passed, or the verdict was missing or not ACCEPT. The demo never reached ACCEPT and nothing was merged.

Copy this into your own repo

1) Keep one rules file both engines load. Claude Code imports it into CLAUDE.md with an @ line, and a small script builds AGENTS.md for Codex from the same source.
2) Run Claude Code in its own git worktree, away from your main branch.
3) Run Codex with a read-only sandbox, and make its final message carry a single verdict line, Verdict: ACCEPT or Verdict: CHANGES REQUIRED.
4) Save that verdict in a file named after the commit hash.
5) Run your tests on the same commit, on a clean tree, and record a pass only after a full pass.
6) Let a script refuse the merge when the current commit lacks either file.

Get the full loop

The Two Engines Kit is this loop packed so one script installs it. You get one rulebook both engines load, the cross-review and gate scripts, a merge script that refuses without a pass record for the exact commit (and, with REQUIRE_REVIEW=1, without the saved ACCEPT), and GUIDE.md, which explains each piece and the mistake that produced it. The installer previews every change and asks y/N, and uninstall.sh puts your old files back byte for byte. It needs both Claude Code and Codex, supports WSL2, Linux and macOS, and is tested on WSL2 with Ubuntu 24.04.

I've been building it since the first Codex and Claude CLIs. Get the Two Engines Kit at https://rayastats.com/en/two-engines-kit

Top comments (1)

Collapse
 
itsvicmoreno profile image
Vic Moreno •

Naming the verdict file after the commit hash is the part most review loops skip, and it is exactly what makes an old ACCEPT unable to cover new code. On the gap you list (a test weakened in the same commit), a cheap mechanical check is to count assertions per commit and fail the gate when the count drops without a stated reason. Did you try anything like that before handing it to the reviewer?