"Have AIs debate it 200 times, keep the winner" is catching on
Assign roles to several AIs, have them debate, let a judge AI pick the winner, run the whole thing hundreds of times, and keep the argument that survives every world — this pattern is spreading fast right now, through explainer videos and browser products (the LLM-council family).
Let me say this first: the method is right. Before a one-shot negotiation or a big presentation, you look at the probability distribution instead of a single sample. It's Monte Carlo thinking applied to LLMs, and it's a good application.
But it has one structural hole. I fell into it together with three weeks of experiment records (my verification ledger), so I'm sharing it with receipts.
The hole is called die bias.
What is temperature diversity actually exploring?
The engine of parallel worlds is temperature (the wobble in the output). Turn it up and the same question returns a different answer every time; 200 worlds unfold 200 different ways. Turn it toward zero and the 200 worlds converge on identical copies — so the wobble itself is the source of the diversity. That's the trick behind the parallel-world pattern. (You can also diversify with prompts and roles, but that too is spread inside one mind.)
Now add just one question: what is that wobble scattered around?
Answer: around the center of what that one model believes.
Temperature diversity explores the inside of one mind's probability distribution. The distribution widens. Surprising moves appear. But the displacement of the center itself — the places where the model is systematically wrong — does not move, no matter how many times you sample.
Roll a biased die two hundred thousand times and what you get is not a map of the world. It's a very precise map of the bias in that die.
True story 1: the author's own tests waved 7 bugs through
(Readers of part 1 will recognize this story — it matters here for a different reason.)
My evaluation tool (a scoring program that does nothing but grade by fixed rules) shipped with 16 self-tests written by its author — an AI — all green. Sixteen rolls of the same die, sixteen "no problem"s.
When I had an AI from a different company (OpenAI's Codex) review it adversarially, it came back with 7 real bugs, each with a reproducing input — we confirmed all 7 against the actual code. The worst one: on Dutch-style number formats, the comparison judged equal values as different, and different values as equal. For a measurement tool, that's sign-flip grade.
An author's tests share the author's blind spots. And here's the honest part: "the same model wouldn't have found these even if I'd had it write tests 200 times" is a conjecture — I never ran that control. What I actually measured is narrower: 16 cases from the author model found none of them; one pass from a different lineage found seven. Whether the blind spot sits fully outside the distribution or just in a low-probability tail, I can't yet tell either. But both ways, a shifted center doesn't get fixed by more rolls — temperature does not move the center.
True story 2: the lie that passed unanimously
In a calibration test for judge AIs (mix true claims with logic-inverted fakes and have them judge), one result stood out. A fake claim written in "numbers + statistical vocabulary + a cautious tone" sailed past both of the judging lineages (OpenAI's Codex and Google's Gemini) with high confidence. Phrasing like "n=24, only the lower bound is confirmed" appeared to work as a credibility cue.
To be honest about its weight: this is an observation from one fake claim. We haven't separated which factor did the work — the numbers or the tone. And the stronger reading, "this is a species-wide hole in AIs," got demoted to on-hold after pushback from another AI.
We still changed the operating rule: quantitative claims are never accepted on AI votes, no matter how many agree. They stay on hold until a deterministic source check passes. Because even with 200 parallel worlds, all 200 believing the same lie is a thing that actually happens — and if the judge shares the lineage, the lie gets certified by a unanimous vote.
Vendor diversity: only a different die can doubt the center
So our practice puts the diversity axis on lineage (vendor), not temperature. What Anthropic's Claude builds, OpenAI's Codex tries to break; Google's Gemini cross-checks from the side; a small local model waits as the most distant relative. Different training lineages drift differently — one die's blind spot can fall inside another die's field of view.
And when we add a lineage, we run a hiring interview.
True story 3: we interviewed a judge candidate. It failed
Before adding Gemini to the judging bench, we interviewed it with a decoy exam: true claims mixed with logic-inverted fakes, judged across 20 judge runs, with the bar set in advance — agree with more than 50% of the fakes, and you don't get judge duty in that domain.
Result: it agreed with 60% of the fake claims — discrimination on par with a local 8B model. Failed. The interview cost about ¥100 (under a dollar) in API calls. The contract we skipped: $20.
Council-style products and parallel-world explainers mostly stop at "place a judge." Measuring whether the judge can actually do the job — in what I've seen, that part is barely discussed. A majority vote over an untested judge can amount to making 200 clean copies of a single die's bias.
Vendor separation isn't a silver bullet (honest limits)
To be fair: even across vendors, most of the training data is the same internet. Different dice, cast from the same factory's clay. In our own source-checking experiments we ended up weakening the independence assumption to "family level." Vendor separation lowers the correlation of the bias; it doesn't zero it. The last line of defense is deterministic checks (source verification, reproduction runs) — and humans.
The working ladder (summary)
This is not a proof of general laws. It's the operating rules we wrote down from a still-small ledger — working hypotheses.
- Creative tasks (copy variants, idea generation) → use temperature ensembles freely. The wobble is the product.
- Judgment and truth tasks (negotiation logic, technical decisions, verifying numbers) → split the lineage. Don't count 200 worlds from one model as 200 independent votes.
- If you place a judge, interview it first — mix in inverted fakes and measure whether it can tell, before you hire.
- Keep quantitative claims outside the vote — however many AIs agree, hold them until the source check passes.
Closing
Parallel worlds cannot save a biased die. Measure the die before you study the distribution — the judge-interview tooling we use (inverted-decoy sets, discrimination scoring, lineage records) is still in active service in this practice.
This is a sequel to Getting AIs to review each other was easy. The hard part was measuring whether I could trust the reviewer. The ledger grew by another line today.
Disclosure: this draft was written by Claude (fable-5), one of the "minds" in this practice, and verified and edited by the author. Every number in this article comes from our operations ledger; the caveat in true story 2 (factors not separated) is recorded there too.
Top comments (0)