I have seven interdependent tool repos — some public, some private — each with its own audit checks, but nobody had ever checked what happens when they're wired together. This post is about handing all seven to Fable5 for a cross-repo audit, and the fact that not one of the real problems it found could have been caught by auditing any single repo alone.
Read on: Previous: the Confrontation-Point Law · Don't ask a model to narrate itself · 繁體中文版
The audit prompt itself went through three drafts
I didn't just paste in "please check these seven repos." The prompt started as a skeleton, then got execution steps, then got filled in with the real seven repos — 5 public tool-chain repos plus 2 private ones (an internal connective framework and a cross-project reuse library). The goal was to write context and boundaries, not a checklist — tell Fable what the terrain is and where the hard lines are, and make verification about mechanism, not about handing over an answer.
Each repo got its own scanning subagent, plus one independent verifier subagent that never saw the scanners' conclusions. That independent verifier is the part that mattered — it overturned four of the scanning agents' conclusions. The topology itself turned out fine; the two public/private seams both worked correctly. The real problems weren't in any single repo. They were three composition-level failures that only become visible once you look at all seven together.
The tool that fixes drift can silently overwrite your work
The most interesting one: one tool's job is to detect and repair drift between a local skill copy and its canonical source — if your local copy diverges from the original, it syncs you back up. The problem: in one environment, that "local copy" is actually a Windows junction (a filesystem shortcut) pointing straight at the canonical repo — and the tool's scan logic only checked "is this a directory," never "is this actually a shortcut."
The result: a tool built to repair drift will write straight through the shortcut into the canonical repo, potentially overwriting uncommitted work sitting on someone's feature branch. The repair tool itself becomes the risk. No single-repo audit could have caught this — the bug doesn't live in any code, it lives in how directories are wired together on a specific machine. That shape is only visible across repos.
The other two composite failures: an orphaned commit left a cross-repo contract stranded with nothing consuming it; and three places documented something as "prompt-level, optional" that was actually being treated as "code-level, mandatory for everyone" — a documentation/enforcement mismatch that leaves different maintainers with completely different mental models of the same rule.
One rule worth stealing: content-level probes, not existence checks
The audit converged on a principle I think is more valuable than any single finding: "the commit exists" or "the file exists" doesn't count as verified. You have to walk the entire propagation chain — local branch, remote default branch, the published artifact or install source, the installed copy — and grep an actual signature string at every single stage, and it has to actually hit, every time.
That principle is what caught the most important issues in this audit. Downgrade it to an existence check, and the audit still runs, still produces a report, and catches nothing — which is worse than not running it at all, because it hands you a false green light.
A real case: how a spec makes frontend and backend asymmetric from birth
The day after the cross-repo checklist converged, a real client project handed us a case that got folded directly into that checklist as a new criterion.
The setup: a real order-management admin feature with a visible frontend/backend asymmetry. The backend RPC defined an error code called BOOKING_NOT_FOUND that the frontend's error-message map never learned about. A refund action returned HTTP 501 in production with zero UI indication. A loading ref was declared and returned but never consumed by any of six admin tabs. An admin-readable transaction table existed and was never queried by the frontend at all.
We gathered the hard evidence first, then asked for the root cause, not just a symptom list. The diagnosis: the asymmetry wasn't born in any one layer — it was born upstream, in the spec's own output grammar. The original spec gave the backend a model (field-level data structures) and gave the frontend only a verb ("view orders and bookings"). Every downstream layer — decomposition, execution, scoring — is deliberately faithful by design, each explicitly instructed not to invent or over-check. So the upstream asymmetry got preserved intact, all the way to the final output. No layer dropped the ball. The pipeline worked exactly as designed — the design itself just had a blind seam nobody had seen.
Even the fix this diagnosis proposed doubted itself out loud: the suggestion was to add a mandatory "Surface Contract" section (a state row plus a seam row) to the five-element goal format. But the same document flagged its own biggest risk: "once a template becomes mandatory, the most likely failure is that people fill it with boilerplate that formally passes the check while saying nothing real — that's the step I'm most likely to have gotten wrong." A model willing to say, in writing, "my own suggestion might just get rubber-stamped" is more useful to me than the suggestion itself.
That proposal is, as of this writing, only designed — not yet wired into the actual goal template. Worth saying plainly: designed is not the same as done.
Takeaway
A single-repo audit can't see the shape that only exists once tools are wired together — and the most expensive lesson here is that the tool meant to repair drift can be the single riskiest thing in the whole system. "Content-level probes, not existence checks" is a rule worth copying into any multi-repo system you maintain, whether or not you have a Fable5 to run it for you.
本文原載於我的部落格：AI Auditing an AI's Own Toolchain: The Repair Tool That Overwrote Its Own Fix
Top comments (0)