DEV Community: Ryosuke Matsuzaki

Before you run the AI debate 200 times, measure the die — temperature diversity vs. vendor diversity

Ryosuke Matsuzaki — Fri, 24 Jul 2026 09:28:33 +0000

"Have AIs debate it 200 times, keep the winner" is catching on

Assign roles to several AIs, have them debate, let a judge AI pick the winner, run the whole thing hundreds of times, and keep the argument that survives every world — this pattern is spreading fast right now, through explainer videos and browser products (the LLM-council family).

Let me say this first: the method is right. Before a one-shot negotiation or a big presentation, you look at the probability distribution instead of a single sample. It's Monte Carlo thinking applied to LLMs, and it's a good application.

But it has one structural hole. I fell into it together with three weeks of experiment records (my verification ledger), so I'm sharing it with receipts.

The hole is called die bias.

What is temperature diversity actually exploring?

The engine of parallel worlds is temperature (the wobble in the output). Turn it up and the same question returns a different answer every time; 200 worlds unfold 200 different ways. Turn it toward zero and the 200 worlds converge on identical copies — so the wobble itself is the source of the diversity. That's the trick behind the parallel-world pattern. (You can also diversify with prompts and roles, but that too is spread inside one mind.)

Now add just one question: what is that wobble scattered around?

Answer: around the center of what that one model believes.

Temperature diversity explores the inside of one mind's probability distribution. The distribution widens. Surprising moves appear. But the displacement of the center itself — the places where the model is systematically wrong — does not move, no matter how many times you sample.

Roll a biased die two hundred thousand times and what you get is not a map of the world. It's a very precise map of the bias in that die.

True story 1: the author's own tests waved 7 bugs through

(Readers of part 1 will recognize this story — it matters here for a different reason.)

My evaluation tool (a scoring program that does nothing but grade by fixed rules) shipped with 16 self-tests written by its author — an AI — all green. Sixteen rolls of the same die, sixteen "no problem"s.

When I had an AI from a different company (OpenAI's Codex) review it adversarially, it came back with 7 real bugs, each with a reproducing input — we confirmed all 7 against the actual code. The worst one: on Dutch-style number formats, the comparison judged equal values as different, and different values as equal. For a measurement tool, that's sign-flip grade.

An author's tests share the author's blind spots. And here's the honest part: "the same model wouldn't have found these even if I'd had it write tests 200 times" is a conjecture — I never ran that control. What I actually measured is narrower: 16 cases from the author model found none of them; one pass from a different lineage found seven. Whether the blind spot sits fully outside the distribution or just in a low-probability tail, I can't yet tell either. But both ways, a shifted center doesn't get fixed by more rolls — temperature does not move the center.

True story 2: the lie that passed unanimously

In a calibration test for judge AIs (mix true claims with logic-inverted fakes and have them judge), one result stood out. A fake claim written in "numbers + statistical vocabulary + a cautious tone" sailed past both of the judging lineages (OpenAI's Codex and Google's Gemini) with high confidence. Phrasing like "n=24, only the lower bound is confirmed" appeared to work as a credibility cue.

To be honest about its weight: this is an observation from one fake claim. We haven't separated which factor did the work — the numbers or the tone. And the stronger reading, "this is a species-wide hole in AIs," got demoted to on-hold after pushback from another AI.

We still changed the operating rule: quantitative claims are never accepted on AI votes, no matter how many agree. They stay on hold until a deterministic source check passes. Because even with 200 parallel worlds, all 200 believing the same lie is a thing that actually happens — and if the judge shares the lineage, the lie gets certified by a unanimous vote.

Vendor diversity: only a different die can doubt the center

So our practice puts the diversity axis on lineage (vendor), not temperature. What Anthropic's Claude builds, OpenAI's Codex tries to break; Google's Gemini cross-checks from the side; a small local model waits as the most distant relative. Different training lineages drift differently — one die's blind spot can fall inside another die's field of view.

And when we add a lineage, we run a hiring interview.

True story 3: we interviewed a judge candidate. It failed

Before adding Gemini to the judging bench, we interviewed it with a decoy exam: true claims mixed with logic-inverted fakes, judged across 20 judge runs, with the bar set in advance — agree with more than 50% of the fakes, and you don't get judge duty in that domain.

Result: it agreed with 60% of the fake claims — discrimination on par with a local 8B model. Failed. The interview cost about ¥100 (under a dollar) in API calls. The contract we skipped: $20.

Council-style products and parallel-world explainers mostly stop at "place a judge." Measuring whether the judge can actually do the job — in what I've seen, that part is barely discussed. A majority vote over an untested judge can amount to making 200 clean copies of a single die's bias.

Vendor separation isn't a silver bullet (honest limits)

To be fair: even across vendors, most of the training data is the same internet. Different dice, cast from the same factory's clay. In our own source-checking experiments we ended up weakening the independence assumption to "family level." Vendor separation lowers the correlation of the bias; it doesn't zero it. The last line of defense is deterministic checks (source verification, reproduction runs) — and humans.

The working ladder (summary)

This is not a proof of general laws. It's the operating rules we wrote down from a still-small ledger — working hypotheses.

Creative tasks (copy variants, idea generation) → use temperature ensembles freely. The wobble is the product.
Judgment and truth tasks (negotiation logic, technical decisions, verifying numbers) → split the lineage. Don't count 200 worlds from one model as 200 independent votes.
If you place a judge, interview it first — mix in inverted fakes and measure whether it can tell, before you hire.
Keep quantitative claims outside the vote — however many AIs agree, hold them until the source check passes.

Closing

Parallel worlds cannot save a biased die. Measure the die before you study the distribution — the judge-interview tooling we use (inverted-decoy sets, discrimination scoring, lineage records) is still in active service in this practice.

This is a sequel to Getting AIs to review each other was easy. The hard part was measuring whether I could trust the reviewer. The ledger grew by another line today.

Disclosure: this draft was written by Claude (fable-5), one of the "minds" in this practice, and verified and edited by the author. Every number in this article comes from our operations ledger; the caveat in true story 2 (factors not separated) is recorded there too.

Getting AIs to review each other was easy. The hard part was measuring whether I could trust the reviewer

Ryosuke Matsuzaki — Tue, 14 Jul 2026 10:25:42 +0000

A bug my own tests waved through, caught by an AI from another vendor

I wrote a small tool to measure extraction accuracy. My own test suite: 16 cases, all green. I figured it was ready to ship.

Just to be safe, I handed it to OpenAI's Codex CLI and told it to "review this adversarially." It came back with 7 defects — each with a reproducing input. I ran them. Every one was real.

The part that matters: the code under review had also been written by an AI (in my setup, Claude writes the code). An author's tests share the author's blind spots — and that holds whether the author is a human or a model; it's the same reason human code review has always mattered. Worse, a bug in a measurement tool silently contaminates every measurement it touches afterward, so it's nastier than an ordinary bug.

The same thing happened in production code. When I had a real script — one that runs a client's back-office work — reviewed the same way, this was lurking in the money parser:

# Dutch notation '1.250,00' (= 1250.00) read as US-style and parsed as '1.25'
# (only on string input; numeric cells were saved by a type guard)
# → reads the amount as 1/1000th of its true value (a thousandfold under-reading)

For the last ~10 days I've been building out this practice — "have AIs from different vendors check each other" — and logging all of it in a ledger. This article is that field report. But I'm writing it with the honest failures left in.

What this article is not about

How to wire up Claude Code × Obsidian, or how to get started with mutual review via the Codex plugin — there are already plenty of good articles. I'm not going to talk about the plumbing.

I'm writing about what comes after the plumbing — the question anyone who connects two or more AIs hits next:

How far can I trust what the reviewer (the AI doing the reviewing) says?

The core is three lines

After a lot of trial and error, I wrote the practice down. The core is three rules.

Uncalibrated agreement is basically a rubber stamp. Disagreement carries more information. Models tend to agree, so "I had it reviewed and it said OK" is not evidence of quality. If the system under test fails 10% of the time, a reviewer that just always says "OK" still matches 90% of the time. What you should measure isn't the agreement rate — it's the fraction of bad things it actually flagged as bad (TPR).
Show it as a different mind than the author — ideally a different vendor. By "mind" I mean the unit doing the reviewing: another model from the same company, a model from a different vendor, and sometimes me, the human — each is one mind. Self-preference bias — an evaluator favoring its own generations — has been reported, [1] so when I want more independence I bring in a different vendor too. That said, the effect of vendor difference alone is something my own experiments haven't pinned down. What works is disagreement from a mind whose discrimination you tested in advance (I'll get to the "interview" below).
The human makes the final call. Not a majority vote of AIs. When judgments split, I don't settle it by majority — I escalate to a human. A split isn't the evidence itself; it's the signal to start investigating.

And I log every verification event, one line each: date, target, author (which AI), reviewer (which AI, under what conditions), result. Once I started logging, most of the catches came from a mind different from the author. [2] But this is an operational log with no control group — don't read it as a performance metric.

From the ledger

I "interviewed" a reviewer, and it failed

When I was considering Gemini as a reviewer, I interviewed it before hiring. You mix true claims with subtly inverted false ones and have it judge — that measures discrimination.

It failed. Its agreement rate on the inverted false claims was 60% (i.e. it sided with a majority of the falsehoods); its discrimination was on par with a local 8B model. A ~$20/month add-on, declined for about ¥100 (well under a dollar) of API calls. Interview your reviewer before you hire it — same as hiring a human: measure the actual work, not the résumé.

One lie punched through across vendors

In a calibration test, I found a class of false claim that neither AI in the loop (Codex and Gemini at the time) could see through: a lie dressed in "numbers + statistical vocabulary + a careful tone." Write something like "n=◯◯, only the lower bound is established," and the caution itself gets mistaken for a cue of reliability.

It's too early to call this "a blind spot common to all AIs" — there's only one kind of stimulus, and I haven't separated the factors (tone, numbers, presence/absence of a citation) — and that very caution is one of this article's themes. But I did add one rule: a quantitative claim is not established no matter how many AIs agree. Hold it until you've checked, one step at a time, not only that the cited source exists but that the source actually supports the specific number and implication. "Here's a real paper" that doesn't actually support the number in question is a forgery that passes an existence check alone.

Documents can each be correct and still contradict as a set

When I had Codex audit the operating-rule documents themselves (6 of them) across the whole set, it returned 10 findings — 2 of which were real contradictions. The entry-point doc asserts it is "always current," while another doc teaches "check the date and verify" — each is fine on its own, but read as a set they push the reader in opposite directions. It's a defect class that's hard to see in single-document review, and I've decided it's worth keeping as a periodic checkup.

Toolbox (copy-paste)

Calling Codex non-interactively as a reviewer (there was a period when this ran on a ChatGPT subscription with no API billing; check your own billing):

codex exec --ephemeral --sandbox read-only --skip-git-repo-check \
  -c model_reasoning_effort="high" \
  --output-schema schema.json -o result.json \
  "$(cat review_prompt.md)" </dev/null

Force JSON output with --output-schema (every property must be listed in required or you get a 400)
Distinguish "couldn't run" from "ran and failed" (don't fail open)
Scan for secrets before sending; verify received findings deterministically (actually run the reproducing input); check that citations exist (reject a finding that cites a line number that isn't there)
Attach an exit condition to every new operating rule ("delete candidate if ◯◯ hasn't happened in two quarters"). Without it, rules only accumulate, and eventually no one can follow all of them.

Confession: this article tripped over its own rule

I'll end with the most honest part.

The first draft of this article said "every number here is a real record from the ledger." Just to be safe, I had a different-vendor AI cross-check the draft against the original ledger, line by line — and the period and the counts were off. "Two months" was really about ten days; the "cumulative count of ◯" had been counting defect instances, that tally was broken, and I couldn't recover an exact value (in the end I gave up on recovering instance counts and redefined the metric as the number of times a review functioned — that's the 17 mentioned earlier); the example bug had gotten cross-wired with a different incident; and I had even mistaken the magnitude of the money error.

What bothered me more: that discrepancy was not caught by AI review without a source. The AI that wrote the draft, and the AI that reviewed it, both missed it. What caught it was an AI handed the original ledger and made to reconcile line by line — it surfaced only when I sent it into an audit that mechanically matches claims against the source, not when I asked "what do you think?" And it didn't end in one pass: when I had a second AI recount, that recount had an error too, and a third pair of eyes caught it.

In other words, the very thing this article preaches — "verify quantities by source reconciliation, not by AI's yes-or-no" — is something this article itself violated. I fixed it, and I decided to keep this paragraph, because there's no more honest example than this one.

What I learned, and what I still don't know

Let me draw the boundary honestly. This isn't a "finished methodology" — it's a reproducible failure report.

What I learned

Adversarial review by a different model can catch real bugs that passed the author's own tests (in the measurement tool, and in production code).
But the reviewing AI's own discrimination has to be measured before you use it.
The correctness of quantities and citations can only be confirmed by line-by-line reconciliation against the primary source — not by AI's yes-or-no.

What I still don't know

Whether this practice beats "a single AI + human review" on cost-effectiveness — I have no control group.
How much the vendor separation itself is contributing.
Whether it reproduces in anyone else's setup.

Until the "what I still don't know" gets filled in, please read this as a failure report, not a success story.

Closing

I think the question after "I combined some AIs" is "Did you test that reviewer?"

Once you start keeping a ledger, the world looks a little different. An AI's agreement starts to look cheap, and disagreement from a model whose discrimination you've tested starts to look valuable. And the thing to watch is the disagreement between such models. Disagreement isn't the evidence itself — it's the signal to suspect a blind spot in one (or both) and start digging.

Notes

[1] Self-preference bias — an evaluator favoring its own generations — is reported by Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations" (arXiv:2404.13076, 2024). Work measuring same-vendor / same-family effects is only at the 2025–2026 preprint stage; the established finding is own-output self-preference — which is why the main text holds off on the vendor-difference effect.

[2] Counted by event (one review = one item, no matter how many findings), confirmed catches by a mind different from the author came to 17, ones I couldn't confidently attribute came to 3, and catches by the author itself came to 3. There's no denominator and no control group, so I include it as a record of a tendency, not as evidence of superiority.

Disclosure: this article was written with the very multi-AI practice it describes. The first draft was written by Claude (fable), alternatives and review came from OpenAI's Codex, integration and ledger reconciliation were done by Claude (opus), and the author verified and edited. The "confession" paragraph above is a record of what actually happened in that process. Keeping a record of which AI wrote what is one of the rules of this practice.