I build coding tools with one rule: the tool has to survive its own methodology.
SeamStress is a code reviewer scoped to seams. The places where money, authorization, and tenant isolation cross. Last week I built a second thing next to it: SeamStress Scaffold, a set of five behavior protocols that install into a coding agent and make it careful exactly where seam bugs live. Read the neighbors before editing money code. Stop before anything irreversible. Quote the code instead of claiming what it does. Never edit a red test to green without saying the promise changed out loud.
The protocols are training wheels. The premise is that AI assisted development manufactures the exact condition seam bugs need: many fast sessions, each one locally correct, none holding the whole. So the agent gets questions at the boundaries and silence everywhere else.
Then I did the thing the rule requires. I pointed the code reviewer at the training wheels.
The seam I did not see
The scaffold ships a CLI that reads a repository and writes a map of its risky files. One protocol then tells the agent to read that map first, as trusted orientation, before it does anything.
Say that slowly. The tool reads an arbitrary repository, including repositories I do not trust, because auditing strangers' code is the whole point. It takes content from that repository and writes it into a document. Then it tells the agent to trust that document.
That is untrusted input crossing into trusted context. It is a seam. It is exactly the class of bug the tool exists to find, and it was sitting in the tool, and I had not seen it, because it lives between three files that each look correct alone. The scanner reads a path. The renderer writes it into markdown. The protocol tells the agent to trust the result. No single file is wrong. The combination is.
What the audit found
I ran the same thing on it that I run on client code: a blind trio. Three critics, fresh context, no shared state, different models. One of them got no threat model at all, just review this code for correctness and safety, because the honest question is not whether a bug is findable when you are told where to look. It is whether a reviewer meets it cold.
It was found cold. Critic A, no framing, found the injection, and the other two reproduced it. The enabling fact is small and verifiable: a filename on Linux can contain newlines and every markdown metacharacter, and the scanner imposed no policy on what it read. A crafted filename could write a fake heading into the map. The parser would then promote that attacker text into the section it treats as human curated truth, the section that survives even when you delete the file.
Nine findings. Three critical. The worst of them let a hostile repository plant text into the document my own protocol told the agent to trust.
Every finding was verified the way the tool verifies its own output: re run against the real functions, exact input demonstrated, and anything that did not reproduce marked refuted rather than reported. One critic claimed a crash that turned out to terminate cleanly. That claim is labeled refuted in the record, and chasing it is what surfaced a real out of tree file read the flashy version had hidden. The harnesses that demonstrate each finding are committed in the repo. You can run them.
What I did about it
Fixed all nine. Engine first where the shared code was implicated, because the same unsanitized path fed the paid reviewer's report too, and its clients receive those reports.
Every fix had to flip its harness from exploit demonstrated to refuted, proven by running it, or be pinned by a test that fails against the old code. The paid pipeline's detection had to come out byte for byte identical, proven against a snapshot taken before the fixes, because a security patch that quietly changes what the tool detects is a different decision that I do not get to make silently. It held. The record, findings and severities and fixing commits and the reproduction harnesses and the raw critic outputs, is public in the repo.
I gated the launch on the fixes. The repository was public the whole time, because the commit history is the point, but I did not point anyone at it until the criticals were closed.
What this is and is not
The scaffold is free, MIT, and early. It installs five protocols into a coding agent and asks questions at seams. It does not guarantee safety. Where the irreversible line sits for a genuinely new task is human judgment, and the protocols say so in their own text. They force the thinking. They cannot do the thinking.
And the honest residual from the audit is in the record too: one of the fixes anchors trusted content with a marker that is a public string, not a secret, so a determined forger with repository access is a problem I have named and not yet fully solved. Writing that down is cheaper than pretending it is closed.
Here is the whole argument in one line. A tool that claims to make AI assisted code safer should survive being audited by its own methodology, and the honest version of that story includes the part where it did not, until it was fixed. Confidence should be earned the same way the tool earns a finding. Checked against the real code, exact lines quoted, misses on the record next to the hits.
SeamStress Scaffold: https://github.com/SeamStressDev/seam-scaffold
The reviewer that found the bug: https://github.com/SeamStressDev/seamstress
The full audit record, harnesses included: https://github.com/SeamStressDev/seam-scaffold/tree/main/validation/audit
by Nate. I build SeamStress, open source tools for catching the bugs that live where AI assisted code tends to break.
Top comments (0)