I pointed my code reviewer at its own verifier. It found two ways to lie.

I built SeamStress. It's a code reviewer with one rule: it only reports what it can prove against your actual code, quoting the exact lines. If it can't prove it, the finding gets demoted to a judgment call. Not presented as fact.

That rule is enforced by one small piece of code: the verification gate. It decides whether a finding may be shown as verified_real. Every other part of the tool can be wrong and the damage is bounded. If the gate is wrong, the tool shows you a confident claim it never earned, with a proof label on it, and it renders as success. Silently.

So before making the repo public, I ran the tool on the gate. Same pipeline it runs on anyone's code: three blind critics, then synthesis, then per finding verification. Eight model calls. It found two critical defects in its own foundation.

Defect one: verified with no evidence behind it

The status authority looked like this:

const result = verifications.find((v) => v.findingId === finding.id);
return result ? result.status : "unverified";

It trusted the verdict on a finding ID match. It never looked at the evidence. And the schema allowed an empty evidence array and an empty quoted code string. So a result shaped like {status: "verified_real", evidence: []} validated cleanly and certified a finding as proven. The report renderer would put that finding in the headline, under copy promising the exact lines quoted as proof, with nothing attached. The evidence block suppressed the display of the missing proof. It did not remove the finding from the verified set.

The fix lives at the authority, not just the schema:

if (!result) return "unverified";
const hasRealEvidence = result.evidence.some((e) => e.quotedCode.trim().length > 0);
return hasRealEvidence ? result.status : "unverified";

A verdict is honored only when at least one non empty quote backs it. Checking at the authority also catches the whitespace quote variant that a naive schema minimum would miss. Fixed in 5fdd680.

Defect two: proof attached to the wrong finding

Finding IDs were namespaced by a slug of the file path. The slug lowercases and collapses every non alphanumeric run to a dash:

const slug = path.replace(/[^a-zA-Z0-9]+/g, "-").replace(/^-+|-+$/g, "").toLowerCase();

Two different paths can slug to the same value. a/Check.ts and a-check.ts. user_check and user-check. When that happens, both seams share a namespace prefix, their finding IDs alias, and the first match lookup binds one seam's verified status and quoted evidence to the other seam's finding. Fabricated proof, on a finding that was never verified. It needs a colliding path pair to trigger, which is rare. The outcome when it triggers is the worst one available.

The fix keys the namespace on the seam's position in the map, which is unique by construction. No collision can produce a duplicate ID. Fixed in bb9c838.

Trust the tests, not the story

Both fixes are pinned by regression tests, and the tests are reversion proven. That is not a figure of speech. We reverted each fix in a scratch worktree, with the tests held at HEAD, and watched the guards go red on the pre fix code while the positive control stayed green. Revert the evidence gate and two tests fail. Revert the collision fix and the binding test fails. Restore either fix and the suite is green again. You can do the same thing yourself. The tests are in src/types/types.test.ts and src/engine/detector.test.ts.

One related gap was found and knowingly left open. An orphaned verification, one whose finding ID matches nothing, silently degrades its finding to unverified. That is under reporting. It fails in the safe direction, toward silence instead of false assurance. So it was documented rather than fixed. Not every finding deserves a commit.

What it misses

The tool has recorded failures and they are public. The benchmark scores real bugs reconstructed from documented incidents on an append only ledger, and the ledger includes the misses. One fixture never even reached review: the keyword pre filter scored it at zero and the run cost nothing because no model was called. That is a detection stage miss and it is the concrete instance of a limit we track. It's on the ledger. Not edited out.

The broader limits are stated in the repo. It's early, bring your own key, validated on a small number of repos. It reasons about the code in front of it, so it can't see design intent that lives in your issue tracker. A finding can be code accurate and still be something you already decided on purpose.

Why publish this

Because the audit is the argument. A code reviewer that claims a proof standard should survive its own methodology, and the honest version of that story includes the part where it didn't, twice, until the gate was fixed. Confidence in the tool should be earned the same way the tool earns its findings. Checked against the real code, exact lines quoted.

The full engineering record, including the verifier's quoted evidence for each finding, is in the repo: docs/seamstress-trust-gate-trio.md. The fastest way to see real output is examples/. The benchmark, misses included, is benchmark/.

Implementation is AI assisted. Architecture, validation, and every irreversible call are mine.

SeamStress is open source: https://github.com/SeamStressDev/seamstress

Comments

[Luis Cruz]

Fascinating experiment. Using a code reviewer to audit its own verification logic highlights an important point in AI-assisted development: the quality of the guardrails matters as much as the intelligence of the model. Self-checking systems, adversarial testing, and transparent evaluation will become increasingly important as AI agents take on more engineering responsibilities. Great exploration of AI reliability!

[seamstressdev]

Thanks Luis,
One sharpening of the point: the guardrails matter as much as the model, agreed, but the guardrails also can't be trusted on sight. The verification gate was the guardrail here, and it had two ways to lie until it was tested like any other code. Untested guardrails are just confidence with extra steps.
Since this post we took it one layer further: probed the discipline itself in live agent sessions and published the transcripts, misses included, in the repo. The pattern seems to hold at every layer. Test the thing that vouches for the thing.
It has been a fantastic journey! I appreciate the comment.
Cheers!

[Luis Cruz]

This is a really valuable lesson for AI-assisted engineering. The idea that "the thing that vouches for the thing" also needs to be tested is something many systems overlook.

I especially like the decision to publish the failures instead of only showing the final polished result. In AI systems, trust doesn't come from claiming reliability — it comes from showing the evaluation process, failure cases, and how the system improves.

The verification gate example is a great reminder that even safety mechanisms can become a single point of failure. A model can hallucinate, but so can the logic around the model if it isn't continuously challenged.

As AI agents become more involved in software development, I think this mindset of adversarial testing, transparent benchmarks, and evidence-based validation will become a core engineering practice.

Great work documenting the journey. Looking forward to seeing how SeamStress evolves! 🚀

[seamstressdev]

Thanks Luis,
One refinement I'd make to "continuously challenged": in practice the challenge has to be scheduled, not aspirational. Nobody continuously challenges anything. What worked here was making it an event with a record: a red team before publish, probes with committed transcripts, a findings ledger. If it does not produce an artifact, it did not happen.
Cheers!