3-LLM 交叉验证的共识机制

3-LLM Cross-Validation: A Consensus Mechanism

Introduction

The most counterintuitive audit principle in the ATS-008 architecture is this: the more fluent a model's reasoning chain, the more you should suspect it was fabricated.

A single LLM's verdict is, at best, an unverified witness statement. It can tell a perfectly coherent story about whether a piece of code is safe — and still be completely wrong. That is the confirmation-bias failure mode that single-model pipelines inherit silently.

This post is part of the "Detective Reasoning + Behavior Logs + Technical Puzzles" series. We'll tear down the mechanism this architecture uses instead: 3-LLM cross-validation — three independent models, isolated from each other, whose conclusions are accepted only when they reach consensus.

Core Thesis

• A single model is a single witness: no matter how capable, it cannot testify to its own reliability.
• 3-LLM cross-validation runs three independent models against the same question and adopts only conclusions that reach consensus.
• Consensus is not "vote counting." It is the output of three stacked filters: semantic equivalence → weighted voting → Byzantine threshold.
• Rejected evidence is logged too. In a behavior log, the dissenting vote matters as much as the approving one.

The Mechanism

1. Open a Case: Proposal Creation

Every verification starts with create_proposal. The question under test is registered as a proposal — who participates, whether a reference answer exists. From this moment on, every judgment is recorded. This is the detective's case file.

self.consensus.create_proposal(
    proposal_id=proposal_id,
    content={"agent_count": len(outputs), "has_reference": ...},
    proposer_id="cross_validator",
)

2. Independent Testimony

Three models answer the same question in isolation. The isolation matters: they don't know the others exist, so they can't collude or contaminate each other. If a reference answer exists (e.g., a human-labeled security verdict), it becomes the baseline; otherwise the first model's output is the baseline for comparison.

3. Semantic Equivalence, Not Literal Matching

This is the most misunderstood part. The system does not count two models as agreeing because they used the same words — that would just be repetition. It runs semantic equivalence checking: both outputs are normalized (AST normalization) and structurally compared to decide whether they express the same conclusion.

Model A says "this function has an integer overflow risk"; Model B says "a buffer overrun may occur here." Different words, same meaning → consistent. Listen to the facts, not the phrasing.

equiv_result = self.equivalence.check_equivalence(
    comparison_base, output, threshold=similarity_threshold
)

4. Weighted Voting

Each verifier is not a single equal vote. The engine maintains per-validator weights — models with a stronger track record weigh more. Approvals and rejections are both recorded, each carrying its semantic similarity as a justification, written into the behavior log.

5. Consensus and the Byzantine Threshold

The final gate is 2/3:

# Consensus is only declared when strength exceeds 2/3
if consensus_strength <= 2 / 3:
    # do not flag Byzantine behavior

Why 2/3? With three nodes, the system tolerates exactly one "traitor" — a compromised model, a down node, or one in full hallucination mode. As long as the other two are honest and agree, consensus holds. This is the simplest form of Byzantine fault tolerance: tolerate one bad witness, never trust two.

If consensus is not reached, the proposal is rejected — and the rejection itself is appended to consensus_history. The dissenting vote, the failure, the weight adjustment: all logged. In auditing, the question isn't "who was right" but "why did we rule this way."

Detective's Conclusion

• All three agree → confidence far higher than any single model's output.
• Two agree, one dissents → conclusion stands, but the dissenter is flagged as a candidate Byzantine node and its weight is reduced.
• No agreement at all → consensus fails, the verdict is bounced back for re-verification; nothing flows downstream.

The point of this mechanism is not to pick the smartest model. It is to make sure no single model ever has the authority to convict on its own. In governance and audit pipelines, that matters more than raw accuracy — because auditing isn't about being usually right. It's about never allowing a single point of false testimony.

About the Author

Written by detective-noir — detective reasoning, behavior logs, and technical puzzles: the audit philosophy of the ATS-008 architecture.