This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
Every AI code reviewer has the same tell: it states a wrong finding in the exact confident tone it uses for a right one. Nothing in the delivery tells you which is which. Confidence isn't evidence.
So I built edgeverdict around one rule: the model never gets to decide. It proposes a test, a deterministic gate runs it, and a finding only counts if the test executes and fails. The model guesses. Execution rules.
A rule like that is only worth anything if it survives being pointed at itself. So before merging a big refactor, I let the tool review its own diff. Here's what it posted, on my own PR:
The actual comment it posted on my own PR (the tool was named agentboard back then; it's edgeverdict now).
Read the last part closely. The tool found a real failing test and flagged its own finding as probably not a real bug, in the same breath.
That split is the whole point. The execution half said "this test fails, here's the proof." The advisory half said "but a human should decide if this is actually wrong." One is a fact. The other is a judgment. It never lets them blur into each other, because blurring them is exactly the confident-liar problem I built the thing to kill.
But the sharpest catch came later, and it was worse, in the way that teaches you something.
There's a function in the tool whose whole job is to decide why a test failed, was this a real assertion failure, or just a crash? That call is load-bearing. Everything downstream trusts it. And here's how I'd written it: it checked whether the string AssertionError appeared anywhere in the failure report.
Anywhere.
So a test that crashed for some unrelated reason, whose output merely mentioned that word somewhere in the noise, would get filed as a genuine bug in your code. A false finding, minted from a coincidence of text.
Sit with where that lived. I'd built a tool whose entire thesis is don't trust what merely looks true, make it prove itself, and buried in its verdict path, the one place the whole thesis rests on, was a line that trusted what merely looked true. My lie detector had a spot where it would believe a coincidence.
I didn't catch it in review. I didn't catch it in testing. The tool caught it, by trying to break its own code. The fix classifies by the exception actually raised, not by grepping for a scary word, and the replacement now documents the bug it killed, right there in the source.
Across a few rounds of reviewing itself, the tool found 31 real defects in its own codebase, each an executed, failing test, not an opinion. Seven were the serious kind that could produce a false verdict. I published all of them, with the run fingerprints, so nobody has to take my word for it.
Which is the point. The tool exists so you don't have to trust a builder's confidence, only the execution. So I won't end by asking you to trust mine:
pip install edgeverdict
edgeverdict demo
About fifteen seconds, no API key. Then point it at your own code and tell me where it lies to you.
👉 github.com/anp0429/edgeverdict
Disclosure: edgeverdict is mine and I wrote this up myself, with an AI assistant helping tighten the draft. Every number, fingerprint, and test here is real and in the repo.
Top comments (0)