I pointed my code reviewer at its own verifier. It found two ways to lie.

seamstressdev — Mon, 06 Jul 2026 18:15:09 +0000

I built SeamStress. It's a code reviewer with one rule: it only reports what it can prove against your actual code, quoting the exact lines. If it can't prove it, the finding gets demoted to a judgment call. Not presented as fact.

That rule is enforced by one small piece of code: the verification gate. It decides whether a finding may be shown as verified_real. Every other part of the tool can be wrong and the damage is bounded. If the gate is wrong, the tool shows you a confident claim it never earned, with a proof label on it, and it renders as success. Silently.

So before making the repo public, I ran the tool on the gate. Same pipeline it runs on anyone's code: three blind critics, then synthesis, then per finding verification. Eight model calls. It found two critical defects in its own foundation.

Defect one: verified with no evidence behind it

The status authority looked like this:

const result = verifications.find((v) => v.findingId === finding.id);
return result ? result.status : "unverified";

It trusted the verdict on a finding ID match. It never looked at the evidence. And the schema allowed an empty evidence array and an empty quoted code string. So a result shaped like {status: "verified_real", evidence: []} validated cleanly and certified a finding as proven. The report renderer would put that finding in the headline, under copy promising the exact lines quoted as proof, with nothing attached. The evidence block suppressed the display of the missing proof. It did not remove the finding from the verified set.

The fix lives at the authority, not just the schema:

if (!result) return "unverified";
const hasRealEvidence = result.evidence.some((e) => e.quotedCode.trim().length > 0);
return hasRealEvidence ? result.status : "unverified";

A verdict is honored only when at least one non empty quote backs it. Checking at the authority also catches the whitespace quote variant that a naive schema minimum would miss. Fixed in 5fdd680.

Defect two: proof attached to the wrong finding

Finding IDs were namespaced by a slug of the file path. The slug lowercases and collapses every non alphanumeric run to a dash:

const slug = path.replace(/[^a-zA-Z0-9]+/g, "-").replace(/^-+|-+$/g, "").toLowerCase();

Two different paths can slug to the same value. a/Check.ts and a-check.ts. user_check and user-check. When that happens, both seams share a namespace prefix, their finding IDs alias, and the first match lookup binds one seam's verified status and quoted evidence to the other seam's finding. Fabricated proof, on a finding that was never verified. It needs a colliding path pair to trigger, which is rare. The outcome when it triggers is the worst one available.

The fix keys the namespace on the seam's position in the map, which is unique by construction. No collision can produce a duplicate ID. Fixed in bb9c838.

Trust the tests, not the story

Both fixes are pinned by regression tests, and the tests are reversion proven. That is not a figure of speech. We reverted each fix in a scratch worktree, with the tests held at HEAD, and watched the guards go red on the pre fix code while the positive control stayed green. Revert the evidence gate and two tests fail. Revert the collision fix and the binding test fails. Restore either fix and the suite is green again. You can do the same thing yourself. The tests are in src/types/types.test.ts and src/engine/detector.test.ts.

One related gap was found and knowingly left open. An orphaned verification, one whose finding ID matches nothing, silently degrades its finding to unverified. That is under reporting. It fails in the safe direction, toward silence instead of false assurance. So it was documented rather than fixed. Not every finding deserves a commit.

What it misses

The tool has recorded failures and they are public. The benchmark scores real bugs reconstructed from documented incidents on an append only ledger, and the ledger includes the misses. One fixture never even reached review: the keyword pre filter scored it at zero and the run cost nothing because no model was called. That is a detection stage miss and it is the concrete instance of a limit we track. It's on the ledger. Not edited out.

The broader limits are stated in the repo. It's early, bring your own key, validated on a small number of repos. It reasons about the code in front of it, so it can't see design intent that lives in your issue tracker. A finding can be code accurate and still be something you already decided on purpose.

Why publish this

Because the audit is the argument. A code reviewer that claims a proof standard should survive its own methodology, and the honest version of that story includes the part where it didn't, twice, until the gate was fixed. Confidence in the tool should be earned the same way the tool earns its findings. Checked against the real code, exact lines quoted.

The full engineering record, including the verifier's quoted evidence for each finding, is in the repo: docs/seamstress-trust-gate-trio.md. The fastest way to see real output is examples/. The benchmark, misses included, is benchmark/.

Implementation is AI assisted. Architecture, validation, and every irreversible call are mine.

SeamStress is open source: https://github.com/SeamStressDev/seamstress

DEV Community: seamstressdev

I pointed my code reviewer at its own verifier. It found two ways to lie.