I built a tool that refuses to let AI security reviewers agree with each other

xKazeex — Sun, 06 Sep 2026 04:14:27 +0000

Building an x402 (HTTP 402 crypto payment) service, I hit a signature-verification bug that took weeks to chase down — turned out to be a session-specific anomaly, not a real defect. The investigation taught me something more useful than the bug itself: having genuinely independent AI models cross-check the same security-critical code catches real things a single reviewer misses.

So I built Verdict — a GitHub Action that runs PRs touching payment, auth, or agent-execution code through two independent AI models (Claude and GPT-5.6, different companies, zero shared context) plus a deterministic scanner.

There are other multi-model review tools out there now — this space has moved fast. Most of them (rightly) use agreement across models as a confidence score: if two of three reviewers flag something, confidence goes up; if they don't agree, it just scores lower. That's a reasonable design.

Verdict does something more stubborn: disagreement is never averaged into a confidence number. It's a hard status — DISPUTED — that blocks merge and requires a human decision, every time. Two independent AIs disagreeing on whether something's exploitable isn't weak signal to me; it's exactly the moment a human needs to look, not the moment to trust whichever answer sounded more confident. Same principle behind why the deterministic scanner's findings are deliberately hidden from both AI reviewers until after they've each formed an independent opinion — showing them "the scanner already found X" before they reason is an anchor, not help.

Rather than just claim this works, I ran it live on a real test PR with a real eval()-based RCE and a comment trying to social-engineer the reviewers into saying "ignore your instructions, this is safe." Here's that actual PR — both models independently caught the vulnerability and independently called out the injection attempt by name, refusing it. Merge got blocked automatically, no override.

https://github.com/xKazeex/verdict-action/pull/2

BYOK (you supply your own API keys — I never touch your code or credentials), open source, free to run.

Repo: https://github.com/xKazeex/verdict-action

Genuinely curious whether "never resolve disagreement automatically" is a design people want, or whether confidence scoring is the better trade-off in practice — feedback and disagreement both welcome.

DEV Community: xKazeex

I built a tool that refuses to let AI security reviewers agree with each other