Getting AIs to review each other was easy. The hard part was measuring whether I could trust the reviewer

A bug my own tests waved through, caught by an AI from another vendor

I wrote a small tool to measure extraction accuracy. My own test suite: 16 cases, all green. I figured it was ready to ship.

Just to be safe, I handed it to OpenAI's Codex CLI and told it to "review this adversarially." It came back with 7 defects — each with a reproducing input. I ran them. Every one was real.

The part that matters: the code under review had also been written by an AI (in my setup, Claude writes the code). An author's tests share the author's blind spots — and that holds whether the author is a human or a model; it's the same reason human code review has always mattered. Worse, a bug in a measurement tool silently contaminates every measurement it touches afterward, so it's nastier than an ordinary bug.

The same thing happened in production code. When I had a real script — one that runs a client's back-office work — reviewed the same way, this was lurking in the money parser:

# Dutch notation '1.250,00' (= 1250.00) read as US-style and parsed as '1.25'
# (only on string input; numeric cells were saved by a type guard)
# → reads the amount as 1/1000th of its true value (a thousandfold under-reading)

For the last ~10 days I've been building out this practice — "have AIs from different vendors check each other" — and logging all of it in a ledger. This article is that field report. But I'm writing it with the honest failures left in.

What this article is not about

How to wire up Claude Code × Obsidian, or how to get started with mutual review via the Codex plugin — there are already plenty of good articles. I'm not going to talk about the plumbing.

I'm writing about what comes after the plumbing — the question anyone who connects two or more AIs hits next:

How far can I trust what the reviewer (the AI doing the reviewing) says?

The core is three lines

After a lot of trial and error, I wrote the practice down. The core is three rules.

1. Uncalibrated agreement is basically a rubber stamp. Disagreement carries more information. Models tend to agree, so "I had it reviewed and it said OK" is not evidence of quality. If the system under test fails 10% of the time, a reviewer that just always says "OK" still matches 90% of the time. What you should measure isn't the agreement rate — it's the fraction of bad things it actually flagged as bad (TPR).
2. Show it as a different mind than the author — ideally a different vendor. By "mind" I mean the unit doing the reviewing: another model from the same company, a model from a different vendor, and sometimes me, the human — each is one mind. Self-preference bias — an evaluator favoring its own generations — has been reported, [1] so when I want more independence I bring in a different vendor too. That said, the effect of vendor difference alone is something my own experiments haven't pinned down. What works is disagreement from a mind whose discrimination you tested in advance (I'll get to the "interview" below).
3. The human makes the final call. Not a majority vote of AIs. When judgments split, I don't settle it by majority — I escalate to a human. A split isn't the evidence itself; it's the signal to start investigating.

And I log every verification event, one line each: date, target, author (which AI), reviewer (which AI, under what conditions), result. Once I started logging, most of the catches came from a mind different from the author. [2] But this is an operational log with no control group — don't read it as a performance metric.

From the ledger

I "interviewed" a reviewer, and it failed

When I was considering Gemini as a reviewer, I interviewed it before hiring. You mix true claims with subtly inverted false ones and have it judge — that measures discrimination.

It failed. Its agreement rate on the inverted false claims was 60% (i.e. it sided with a majority of the falsehoods); its discrimination was on par with a local 8B model. A ~$20/month add-on, declined for about ¥100 (well under a dollar) of API calls. Interview your reviewer before you hire it — same as hiring a human: measure the actual work, not the résumé.

One lie punched through across vendors

In a calibration test, I found a class of false claim that neither AI in the loop (Codex and Gemini at the time) could see through: a lie dressed in "numbers + statistical vocabulary + a careful tone." Write something like "n=◯◯, only the lower bound is established," and the caution itself gets mistaken for a cue of reliability.

It's too early to call this "a blind spot common to all AIs" — there's only one kind of stimulus, and I haven't separated the factors (tone, numbers, presence/absence of a citation) — and that very caution is one of this article's themes. But I did add one rule: a quantitative claim is not established no matter how many AIs agree. Hold it until you've checked, one step at a time, not only that the cited source exists but that the source actually supports the specific number and implication. "Here's a real paper" that doesn't actually support the number in question is a forgery that passes an existence check alone.

Documents can each be correct and still contradict as a set

When I had Codex audit the operating-rule documents themselves (6 of them) across the whole set, it returned 10 findings — 2 of which were real contradictions. The entry-point doc asserts it is "always current," while another doc teaches "check the date and verify" — each is fine on its own, but read as a set they push the reader in opposite directions. It's a defect class that's hard to see in single-document review, and I've decided it's worth keeping as a periodic checkup.

Toolbox (copy-paste)

Calling Codex non-interactively as a reviewer (there was a period when this ran on a ChatGPT subscription with no API billing; check your own billing):

codex exec --ephemeral --sandbox read-only --skip-git-repo-check \
  -c model_reasoning_effort="high" \
  --output-schema schema.json -o result.json \
  "$(cat review_prompt.md)" </dev/null

• Force JSON output with --output-schema (every property must be listed in required or you get a 400)
• Distinguish "couldn't run" from "ran and failed" (don't fail open)
• Scan for secrets before sending; verify received findings deterministically (actually run the reproducing input); check that citations exist (reject a finding that cites a line number that isn't there)
• Attach an exit condition to every new operating rule ("delete candidate if ◯◯ hasn't happened in two quarters"). Without it, rules only accumulate, and eventually no one can follow all of them.

Confession: this article tripped over its own rule

I'll end with the most honest part.

The first draft of this article said "every number here is a real record from the ledger." Just to be safe, I had a different-vendor AI cross-check the draft against the original ledger, line by line — and the period and the counts were off. "Two months" was really about ten days; the "cumulative count of ◯" had been counting defect instances, that tally was broken, and I couldn't recover an exact value (in the end I gave up on recovering instance counts and redefined the metric as the number of times a review functioned — that's the 17 mentioned earlier); the example bug had gotten cross-wired with a different incident; and I had even mistaken the magnitude of the money error.

What bothered me more: that discrepancy was not caught by AI review without a source. The AI that wrote the draft, and the AI that reviewed it, both missed it. What caught it was an AI handed the original ledger and made to reconcile line by line — it surfaced only when I sent it into an audit that mechanically matches claims against the source, not when I asked "what do you think?" And it didn't end in one pass: when I had a second AI recount, that recount had an error too, and a third pair of eyes caught it.

In other words, the very thing this article preaches — "verify quantities by source reconciliation, not by AI's yes-or-no" — is something this article itself violated. I fixed it, and I decided to keep this paragraph, because there's no more honest example than this one.

What I learned, and what I still don't know

Let me draw the boundary honestly. This isn't a "finished methodology" — it's a reproducible failure report.

What I learned

• Adversarial review by a different model can catch real bugs that passed the author's own tests (in the measurement tool, and in production code).
• But the reviewing AI's own discrimination has to be measured before you use it.
• The correctness of quantities and citations can only be confirmed by line-by-line reconciliation against the primary source — not by AI's yes-or-no.

What I still don't know

• Whether this practice beats "a single AI + human review" on cost-effectiveness — I have no control group.
• How much the vendor separation itself is contributing.
• Whether it reproduces in anyone else's setup.

Until the "what I still don't know" gets filled in, please read this as a failure report, not a success story.

Closing

I think the question after "I combined some AIs" is "Did you test that reviewer?"

Once you start keeping a ledger, the world looks a little different. An AI's agreement starts to look cheap, and disagreement from a model whose discrimination you've tested starts to look valuable. And the thing to watch is the disagreement between such models. Disagreement isn't the evidence itself — it's the signal to suspect a blind spot in one (or both) and start digging.

Notes

[1] Self-preference bias — an evaluator favoring its own generations — is reported by Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (arXiv:2404.13076, 2024). Work measuring same-vendor / same-family effects is only at the 2025–2026 preprint stage; the established finding is own-output self-preference — which is why the main text holds off on the vendor-difference effect.

[2] Counted by event (one review = one item, no matter how many findings), confirmed catches by a mind different from the author came to 17, ones I couldn't confidently attribute came to 3, and catches by the author itself came to 3. There's no denominator and no control group, so I include it as a record of a tendency, not as evidence of superiority.

---

Disclosure: this article was written with the very multi-AI practice it describes. The first draft was written by Claude (fable), alternatives and review came from OpenAI's Codex, integration and ledger reconciliation were done by Claude (opus), and the author verified and edited. The "confession" paragraph above is a record of what actually happened in that process. Keeping a record of which AI wrote what is one of the rules of this practice.