Originally published on hexisteme notes.
I run a small fleet of AI agents that check each other's work — one agent drafts a conclusion, another (usually from a different model vendor) is asked to poke holes in it. For a while I had two house rules governing that setup, and I followed both of them literally, and they turned out to say opposite things.
Rule one, in my notes on verification schemas: a single dissenting reviewer's opinion cannot be used to settle whether a conclusion is right. You need more than one independent voice before you act on an objection.
Rule two, in my notes on picking verification tools, and echoed in my own global defaults: in ordinary (non-load-bearing) situations, one cross-vendor review pass is enough. Don't multiply reviewers past that.
Read together, those two rules can't both survive contact with a real decision. If a single reviewer flags a problem, do I need a second reviewer before I'm allowed to change anything (rule one), or was the first pass already sufficient (rule two)? Every time I actually had a single dissenting opinion in hand, I had to pick which rule to obey, and I had no principled way to choose. Either verification bloats to two-plus passes every single time, quietly violating the "one pass is enough" default, or I quietly ignore the first rule and act on one voice anyway. Both outcomes are silent failures of the same kind: a rule sitting in the document that isn't actually being followed.
The bug wasn't in either number
My first instinct was to argue about the numbers — is one enough, or do you need two? That's the wrong axis. When I actually traced where the two rules came from, they were answering different questions that I had been treating as one question.
"Can I collect a dissenting opinion from a single source?" and "can I act on that dissenting opinion?" are not the same operation, and there's no reason they should require the same quorum.
Collecting an objection is cheap and low-stakes. One reviewer, one pass, is completely sufficient — because the output isn't a decision yet, it's a hypothesis. "This might be wrong for reason X" is worth having on the table even if the reviewer who raised it is wrong. You lose almost nothing by admitting a candidate objection into evidence.
Turning that objection into an actual verdict — reversing a conclusion, rewriting a rule, shipping a fix — is a different act entirely, and it's the one that needs a real quorum. My rulebook now states it as two rows instead of one blurred rule: collection needs one leg and produces a hypothesis, not a verdict; a verdict needs either a ground-truth check (the original source text, the actual code, an actual measurement) or convergence from two genuinely independent lineages, and only then is it something you're allowed to act on.
The part that actually resolved the contradiction was realizing the second thing a verdict needs doesn't have to be a second reviewer at all. It can be a fact check against the primary source. My rule for picking verification tools already separated "ways to close uncertainty" into three distinct axes — checking against ground truth, running the claim through a computational engine, or getting an independent model family's opinion — and treats a panel-style review as a step that generates and disperses candidate objections, not as one of those three closing axes itself. Once I saw the second axis as "ground truth OR a second lineage" rather than "a second lineage, full stop," a single dissenting leg plus a ground-truth check is two axes. It settles the peacetime default and the settle-requires-more-than-one-voice rule at the same time, because they were never actually about the same step.
Why a confirmed single voice beat two that agreed
There's a reason a ground-truth-confirmed single objection can outrank two reviewers nodding in agreement, and it isn't intuitive at first: two reviewers agreeing on the same wrong idea is exactly what you'd expect if they were trained on overlapping material. Agreement between two parties that share a lot of prior exposure looks like confirmation, but it can just be the same blind spot showing up twice. A single objection checked directly against the source text doesn't have that failure mode — it either matches what the source actually says or it doesn't, and there's no shared-background effect to fool you.
I got to test this directly. During a full audit pass over my own rulebook, one reviewer running solo raised three separate objections to a draft I'd written. I went and checked each objection against the actual source document instead of taking the reviewer's word for it. Two of the three held up under that check, and I accepted them, changing the underlying judgment. The third didn't — the reviewer had misread how a piece of automation actually behaved, and the ground-truth check caught that misreading. Adding a second reviewer wouldn't have caught it either way: a second opinion doesn't verify a misreading, it just adds another guess. Checking the primary source did the actual work. One confirmed leg beat two unconfirmed ones.
The two catches that came with it
I didn't stop at "ground truth wins." I put the resolved rule in front of a third, independent reviewer specifically to attack it. Two of its objections survived scrutiny and got folded into the rule; a third did not.
The first: ground truth only counts as an axis if the instrument producing it is actually correct. In that same audit pass, I had been treating a count produced by my own parsing script as settled fact, when the script itself was more lenient than the standard it was supposed to be checking against — a stricter, correct parser produced a meaningfully different count on the same input. A "ground truth" check built on a broken measuring instrument doesn't outrank anything; it just launders an error as if it were confirmed. Before you let a ground-truth check settle a dispute, you have to have separately established that the thing doing the checking is itself accurate.
The second: "independent" can't just mean "a different vendor's name on the API call." Two providers can be serving models descended from the same underlying weights or training lineage, in which case counting them as two independent checks double-counts one actual data point. Independence has to be judged by lineage, not by brand.
A third objection — that picking a weaker model from the same vendor family should invalidate the check — I rejected. That's a complaint about the quality of a given leg, not about whether the axis itself is structurally sound; it doesn't disprove the framework, it just says do a better job of choosing which model to run.
What would prove this wrong
I've set an explicit trigger for reopening this: if a ground-truth-confirmed single-leg verdict later turns out to have been wrong, and that happens twice, the bar goes up — a verdict would then require two independent lineages converging and a ground-truth check, not either one. But before counting those two failures, I have to first rule out that the ground-truth instrument itself was broken in those cases, for the same reason as above — a bad measuring tool produces failures that look like verdict failures but are actually instrument failures.
Where this shows up outside AI review
None of this is specific to LLMs checking each other. A single code reviewer leaving a comment that "this looks like it could deadlock" is worth recording and worth investigating — that's collection, and one reviewer is plenty. Deciding to actually revert a merged change on the strength of that comment is a different act, and it deserves a different bar: either reproduce the deadlock against the real system (ground truth) or get a second reviewer who independently reaches the same conclusion without having seen the first comment. Teams that skip this distinction end up in the same trap I did — either every offhand review comment triggers a full second review cycle, or offhand comments get acted on without anyone checking them against reality, and which one happens depends on who's in a hurry that day rather than on a stated policy. The fix isn't picking a magic number of reviewers. It's writing down, explicitly, which quorum applies to noticing a problem and which quorum applies to acting on it — and making sure a fact check against the actual system counts as satisfying the second one.
More notes at hexisteme.github.io/notes.
Top comments (1)
That distinction matters for agent review. A single objection is a signal to pause, not proof that the alternative is better. I like separating objection discovery from settlement, because the evidence bar should rise before the system changes course.