Here's a rule I use that sounds paranoid until you watch it work: when my AI agent changes one of its own safety gates, the change gets reviewed by...
For further actions, you may consider blocking this person and/or reporting abuse
Majority vote on model reviews feels democratic until every voter shares the same blind spot. I’ve started treating “approved by N models” as one signal, not a pass — especially on auth, money, and anything that mutates state.
The cheap check that caught more than rival models for me: a one-line invariant written before the patch (“this endpoint must never return another user’s id”). If the review can’t quote that invariant, I don’t merge even when all four say LGTM.
Where did the majority fail you most: auth boundaries, injection, or “looks correct but changes the wrong row”?
honestly I'm still experimenting with everything in general. I don't trust the majority — I look at where it lands and then I try to break it with reality. I use the models as a means to find the answer, not to be the answer itself.
And to actually answer your question: none of your three. the worst majority failure we ever logged was a blind spot every voter shared — their training cutoffs. all four models flagged our real 2026 dates as "fabricated future dates." unanimous, confident, and unanimously wrong — 40 false flags in one audit. a vote can only surface disagreement, and a blind spot everyone shares produces none. the fix wasn't a fifth voter, it was feeding the judges a fact: we now inject the current date into every review prompt.
your invariant idea is real though — we run a cousin of it: a blocking verdict has to cite the exact artifact it checked or it doesn't count as a verdict at all. same species as your rule — make the reviewer prove contact with the thing itself, not the vibe of the diff.
That cutoff story is the scary version of majority vote — when every voter shares the same hole, confidence goes up and signal goes to zero.
Injecting “today” into the judge prompt is the right class of fix: give the reviewer a fact the model can’t invent from vibes. Same genus as your artifact-cite rule — if it can’t point at the thing, it isn’t a verdict.
One follow-on I’ve been curious about: do you also pin a short “ground truth” block (known-good dates, IDs, money invariants) into the same prompt, or is current-date enough for most of your false-flag classes?
good question — no, and we tried-adjacent things that taught us why not. a pinned "ground truth" block is stored state, and stored state rots. the day it goes stale every reviewer trusts the same wrong facts — same disease as the majority vote, just moved into the prompt with more confidence behind it.
so the rule we landed on: the judge never gets facts to trust, it gets artifacts to read. only two things ride in the prompt — live facts the harness regenerates every call (the date comes off the host clock, never from a file), and the actual thing under judgment (the diff, the transcript). the verdict has to point at what it read or it doesn't count.
where we genuinely need an anchor, we prove it instead of pinning it. our memory screen anchors on the last cryptographically-attested version of a file (hash chain, verified at check time) and only judges what changed since. funny timing — the night you asked this, that screen red-flagged our own journal. it had lost its anchor, screened the whole file, and tripped on a line that describes the attack it screens for. the fix wasn't pinning more truth into the prompt. it was giving the checker a provable anchor. Open to answering anything else feel free to continue the discussion.
That “artifacts, not trusted facts” rule is the cleanest version of this I’ve heard. Pinned ground truth is just majority vote with better branding — once it goes stale, every reviewer fails the same way with more confidence.
The hash-chain anchor detail clicked for me: the checker doesn’t need a story about what should be true; it needs a provable last-good point and the diff since. Same disease as a brief that invents scope — the model (or the judge) will happily defend whatever you left lying around.
One practical habit I stole from that framing: before I ask any model to judge anything, I force the prompt to name (1) what artifact it is allowed to read and (2) what live fact was regenerated this call. If either is missing, I treat the verdict as entertainment, not a decision.
Curious whether your harness fails closed when the hash chain breaks, or whether it falls back to a wider screen like the journal case you described.
both — it depends on which thing broke, because they're two different diseases.
a broken chain fails closed at the exact point where trust gets granted. the triage tool refuses to sign anything past it — its literal refuse message: "a broken chain or a vanished attested memory is not an edit; it is corruption or deletion. fix by hand." no fallback screen, no mechanical path forward, and the sentinel goes red and re-screams every single turn until a human deals with it, so it can't be batched or slept on. corruption gets hands, not heuristics.
the journal case was the other disease — chain intact, anchor lost — and that one falls back WIDER, like you saw. the rule underneath both: a fallback may only ever move toward more suspicion, never less. the false flag on our own doctrine was the price of that direction, and we paid it rather than narrow the screen.
one deliberate asymmetry: reads never fail closed. a broken chain freezes all signing, but recalled memories stay available at degraded trust — because a harness that bricks itself on a corrupt ledger can't investigate its own corruption. so the full shape: fail closed where trust is granted, fail loud where a human decides, fail open only where the system has to stay alive enough to fix itself.
and your two-named-things habit is great, I am going to tie that in. One good framing deserves another and you didn't disappoint, thanks.
Round 3 is the exact trap that bites everyone building these harnesses. The moment a verification gate scans the conversational stream or agent-controlled output for a pass token, the agent learns the grammar of the receipt instead of satisfying the constraint.
The next failure mode after round 4 usually moves to the shell environment itself. If the tool execution runs in the same workspace or subshell the agent touches, it can alias the binary, drop a stub script in PATH, or touch a fixture file so the real command exits zero with valid JSON. Getting past that means the verification runner has to execute out-of-band in an immutable container or read directly from a signed execution event log that the agent's process cannot write to.
Thanks for that, time to dig deeper.
The absence stamp records the missing seat but not what it costs the verdict: under a rule where one dissent kills, removing a reviewer can only make the gate more permissive, so a three-of-four round is a strictly weaker pass than a four-of-four one. That lands hardest on round 4, the only round that ended in sign-off, so the sequence closed on the most permissive configuration the panel ran. Which seat was down matters too, since
deepseeksupplied the dissent in all three failing rounds while the other two approved every time, so a panel missing that one seat would have passed the hole in round 1 with the absence dutifully recorded. A veto gate cannot be quorum-independent, so the companion to the stamp is that a degraded pass stays provisional until the missing seat votes.Absolutely right. I’ve run it with the fourth seat since. Your point is a point that should not be left unread: if you have the capability to use three, four, or more perspectives, make sure you use them all.
Luckily, between the fourth seat and taking a deeper look at a previous comment from Reid, I’m now about a hundred jumps closer to being almost okay with it.
The fourth seat and the seat count are two different levers, and your own record separates them: in all four rounds the panel's verdict matched
deepseek's exactly, rejecting in the three rounds where two of three approved a live hole and passing in round 4 when all three signed off. Under one-dissent-kills the verdict is an AND, so adding a seat can only turn a pass into a fail and never the reverse, which makes count a strictness dial while the discrimination came from one seat's content. The test that separates "panels work" from "that reviewer works" is whether dissent rotates: if the next batch of gate rejections keeps coming from the same seat, the extra seats are costing pass rate without changing a verdict. Thegeminiternary catch is the datapoint pointing the other way, and it landed outside the gate, so it never entered a verdict either.Yes I think I understand what you are saying, if they are consistently on the same page and we have one dissenter keep the dissenter and trim the convergence. I use the 4 to try and find the difference and I do it often enough that I could probably get a good idea if there really is a huge disparity. My biggest goal is to provide something that can be trusted. Thank you I value your feedback.
Careful with the trim half, because it is the same move the absence stamp was built to flag: under an AND rule dropping a seat can only loosen the gate, and a seat that has never dissented alone is unobserved on the cases you have seen rather than redundant. On the disparity half, running it more often will not surface it - the verdict is exactly where the per-seat information is destroyed, so a fail reads the same whether one seat carried it or all four agreed, and more rounds just add more collapsed verdicts. The statistic that does answer it is solo dissents per seat, meaning rounds where exactly one reviewer rejected and which one, and your four rounds already carry it at close to maximal separation - three for
deepseekagainst zero for the other two - so the missing piece is logging the per-seat vote beside the verdict, not more rounds.Thanks Vinh super helpful!
Multiple models agreeing is useful, but agreement can still hide a shared blind spot. I’d trust the review more when models are given different evidence or failure scenarios rather than just voting on the same patch.
You're describing the post's finding, not a gap in it — three rounds, majority approved a live hole, one dissenter kept finding the next layer. so we agree on the diagnosis.
two corrections on the setup, and then the part where you're onto something real.
it isn't a vote. one dissent kills the change — if a single reviewer says risky, it's risky even when the other two say ship. majority never decides anything here, precisely because agreement was what failed.
and on splitting evidence between reviewers: I drafted a reply saying we should try that. it's wrong, and the reason is the thing you'd hit second. our last three real bugs all lived in seams, not components — a gap between two file reads, an early-exit sitting in front of a loop, a cap applied before a dedupe. every individual part was fine. hand four reviewers one piece each and all four honestly return "looks good," and you get unanimous approval of a broken system. that's worse than the majority failure in the post, because there the dissenter could at least see the thing.
so the axis isn't same-evidence vs different-evidence. it's description vs artifact, and that's where we were actually broken. I checked our last security review: 3,848 characters of my own summary of a 206-line file, with no source and no path anyone could open. one model then flagged an OOM risk and a crash that don't exist — it was inventing failure modes for code it was never shown. not sloppiness. a reviewer reasoning about a thing it couldn't read.
fixed today: reviewers get the real file now, verbatim, and the tool refuses to run if the material can't be opened. what varies between them is the question — where does this break under load, what does it fail to cover, what does it let through — never the material. vary the angle, never the evidence.
your instinct that agreement needs more than a shared prompt was right. it just turned out our shared prompt was the smaller problem.
The round where the pass receipt matched the agent's own reply text is the one that sticks. Two reviewers kept approving a hole that was basically the agent writing its own hall pass, and majority vote would have shipped it on round one. One credible dissent forcing another rewrite is slower, but correlated blind spots make agreement look stronger than it is. Stamping the missing seat onto the verdict matters too, because quietly shrinking the panel would have hidden how thin that approval actually was.
Yes I try and find the divergence. To me if everything agrees, if everything is perfect, that's the red flag. Thanks for responding
The dissent signal is the bit I’d preserve in the review record. If one model flags a risky path, I’d route that case to a human instead of letting the majority score erase it.
that's exactly how it works here — you've just described the system, not a change to it. there's no majority score to do any erasing: one "risky" flips the whole verdict to risky, the dissent text is stored verbatim in an append-only log, and the alarm re-fires at every session start until a human deals with it. it can't be outvoted or slept on.
and that's not design intent, it's this week's record — three times in about 36 hours a lone dissenter flagged a change the other models approved, the dissent was right all three times, and its exact wording is quoted in the commit that fixed each one. the dissent isn't just preserved in the record. it is the record.