The schema defined 2 foreign key pairs. The API reported 4.

Ankita Patel — Mon, 03 Aug 2026 19:38:26 +0000

I hit this in supabase/mcp a few weeks back, it's merged now (PR #317), and the shape of the bug is worth writing down because it's exactly the kind of thing AI agents consume without blinking.

The setup: the MCP server has a list_tables tool. Verbose mode includes foreign key relationships, so an agent asking "how are these tables connected" reads that output as ground truth.

For composite foreign keys it was wrong. The SQL joined source and target columns without pairing them positionally, so a 2-column FK came back as 4 pairings and a 3-column FK as 9. Cartesian product. N columns, N squared relationships, and all but N of them don't exist in the schema. Nothing errors. The output looks completely plausible. A human might squint at it. An agent just uses it, and now it's joining on a relationship that was never there.

I filed it as a question, not a verdict, because I figured I might be missing context: is this intended behavior? The maintainer confirmed it was a bug and suggested going one better while we're in there: group each constraint's columns into ordered arrays, so the pairing is structural instead of implied by row adjacency.

The part I actually want to talk about is proving the fix.

The obvious fix is "order the columns." Ordered by what, though. Alphabetical order works on the happy path. Physical column order also works on the happy path. Both are wrong. The pairing has to follow the constraint's own declared column positions, so the fix uses unnest ... WITH ORDINALITY, and I added a test where the FK's declared order differs from both the alphabetical and the physical order. That test is the invariant. Without it, the bug can come back wearing a green suite.

Once the grouped shape was pushed, I wanted to sanity-check it beyond my own tests before it merged. So I ran an automated review over the branch: an LLM proposes edge-case tests from the intent and the diff, then a deterministic gate executes every one against the real code. No model anywhere in the pass or fail. It proposed 9 cases past what I'd written. Two independent composite FKs between the same table pair staying separate. Self-referential composite FKs. Cross-schema FKs visible from either side. Single-column FKs correctly emitted as one-element arrays. Repeated calls returning identical output. All 9 executed green, 0 gaps found. I posted the list on the PR and offered to commit them.

The maintainer's reply: commit them all, and this merges. So they went in, plus a couple more scenarios that came up in review, and the final suite landed 110/110.

Two things I took from this.

One, the dangerous bugs in agent-facing APIs aren't crashes. They're confidently wrong data. A crash gets caught by the first retry loop. N squared foreign keys gets built on.

Two, "my tests pass" and "the behavior is pinned" are different claims. The nine executed cases didn't find a gap, and that's fine, that's what they were for: turning "I think the fix is right" into "here is what was executed against it and held." What got merged was the evidence, not my confidence.

The gate I used is something I've been building in the open. It's called edgeverdict, it's on PyPI, and there's a no-key demo that runs in about half a minute. If you point it at your own repo, tell me where it lies to you. That's genuinely the most useful thing you can do for it.

The gate: github.com/anp0429/edgeverdict, or pip install edgeverdict for the no-key demo.

I built a lie detector for code. Then it caught itself lying.

Ankita Patel — Wed, 29 Jul 2026 22:54:26 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Every AI code reviewer has the same tell: it states a wrong finding in the exact confident tone it uses for a right one. Nothing in the delivery tells you which is which. Confidence isn't evidence.

So I built edgeverdict around one rule: the model never gets to decide. It proposes a test, a deterministic gate runs it, and a finding only counts if the test executes and fails. The model guesses. Execution rules.

A rule like that is only worth anything if it survives being pointed at itself. So before merging a big refactor, I let the tool review its own diff. Here's what it posted, on my own PR:

The actual comment it posted on my own PR (the tool was named agentboard back then; it's edgeverdict now).

Read the last part closely. The tool found a real failing test and flagged its own finding as probably not a real bug, in the same breath.

That split is the whole point. The execution half said "this test fails, here's the proof." The advisory half said "but a human should decide if this is actually wrong." One is a fact. The other is a judgment. It never lets them blur into each other, because blurring them is exactly the confident-liar problem I built the thing to kill.

But the sharpest catch came later, and it was worse, in the way that teaches you something.

There's a function in the tool whose whole job is to decide why a test failed, was this a real assertion failure, or just a crash? That call is load-bearing. Everything downstream trusts it. And here's how I'd written it: it checked whether the string AssertionError appeared anywhere in the failure report.

Anywhere.

So a test that crashed for some unrelated reason, whose output merely mentioned that word somewhere in the noise, would get filed as a genuine bug in your code. A false finding, minted from a coincidence of text.

Sit with where that lived. I'd built a tool whose entire thesis is don't trust what merely looks true, make it prove itself, and buried in its verdict path, the one place the whole thesis rests on, was a line that trusted what merely looked true. My lie detector had a spot where it would believe a coincidence.

I didn't catch it in review. I didn't catch it in testing. The tool caught it, by trying to break its own code. The fix classifies by the exception actually raised, not by grepping for a scary word, and the replacement now documents the bug it killed, right there in the source.

Across a few rounds of reviewing itself, the tool found 31 real defects in its own codebase, each an executed, failing test, not an opinion. Seven were the serious kind that could produce a false verdict. I published all of them, with the run fingerprints, so nobody has to take my word for it.

Which is the point. The tool exists so you don't have to trust a builder's confidence, only the execution. So I won't end by asking you to trust mine:

pip install edgeverdict
edgeverdict demo

About fifteen seconds, no API key. Then point it at your own code and tell me where it lies to you.

👉 github.com/anp0429/edgeverdict

Disclosure: edgeverdict is mine and I wrote this up myself, with an AI assistant helping tighten the draft. Every number, fingerprint, and test here is real and in the repo.

DEV Community: Ankita Patel

The schema defined 2 foreign key pairs. The API reported 4.

I built a lie detector for code. Then it caught itself lying.