Two thirds of my architecture document wasn't enforceable

Kaleem Noorani — Fri, 14 Aug 2026 02:21:30 +0000

I keep an architecture document for a platform I've been building for months. Not diagrams. Just decisions in plain English. "This system fulfills orders, it never creates them." "External integrations log and continue, they never block the core workflow."

I keep it because I build with a coding agent, and the agent's failure mode isn't writing bad code. It's writing locally reasonable code that violates a decision I made three weeks ago. The diff looks fine. Review passes. Nothing catches it.

So I wanted to know how much of that document a tool could actually check. I built one to find out. Then I ran it against the real codebase and it caught me lying about my own numbers four times, which turned out to be the most useful thing it did.

Sorting the document

Before writing any code I went through the document statement by statement. 24 of them. They fell into three groups that need completely different handling.

Eight were about process, not code. Don't claim something works without running it. Pause before destructive operations. Ask instead of guessing when you can't verify a physical fact. These leave no trace. You cannot look at a repo and determine whether the agent paused and asked before dropping a table. It either did or it didn't and the evidence was in a session nobody saved. Not checkable by anything.

Nine were invariants about the codebase. Provable from the file tree. These are the ones a tool can do something with.

Seven were declared knowledge. order_number is a display value, lookups use the internal id. The carton catalog is owned upstream, we mirror it and never write it. "Cart" means a physical picking cart in the warehouse and a pre-transaction container in commerce and the two are unrelated. You can't violate these with a file. But you forget them, and forgetting them is exactly how the bugs happen.

Of the nine invariants, six translated into deterministic checks. Three didn't, and the three that failed were more interesting than the six that worked, because each failed differently.

The first was "only the ingest boundary may create order records." Can't check it. The ingest logic sits inside one enormous service class, so there's no file path to scope the exclusion to. My architecture claims there's a boundary there. My code doesn't have one. The tool didn't find drift, it found that the invariant was unenforceable because the structure it describes doesn't exist yet.

The second was "permission strings must match between backend and frontend." That needs cross-file set comparison — pull strings from A, pull from B, diff them. Not a parsing problem. Just a primitive I hadn't built.

The third was "entities must render as clickable links." That needs to understand what the code means. Out of reach, probably permanently, for anything pattern-based.

Add the two knowledge entries that had a checkable shadow and I ended up with eight deterministic checks out of 24 statements.

Running it

Six checker types. Forbidden call patterns, forbidden dependencies, config value allowlists, non-empty method bodies, error handler presence, and a proximity heuristic for identifier misuse. No AST parser, on purpose. Every check documents in the config how it can be evaded.

First run: 24 violations, 2 advisories.

Thirteen were real. Frontend components calling the HTTP client directly instead of going through the service layer, accumulated over months, nobody noticed.

Six were migrations flagged for having no rollback. I opened all six. Every one of them was deliberately irreversible with a comment explaining why, and I had written all six comments myself and forgotten every one. Which means the rule was wrong, not the code. I don't have "every migration is reversible." I have "reversibility is the default and exceptions are declared." I changed the check to accept a marker or a rollback and six findings went to zero without touching any code.

Four were legitimate exclusions across two files. Blob downloads, a different payload class than the rule targets.

One was a commented-out line. Lexical checks can't tell code from comments. Left it in, documented it.

Second run after the fixes: 14 violations. The thirteen real ones plus the false positive.

The one that fired correctly and was still wrong

An advisory flags the display identifier appearing near lookup contexts. It hit a partner webhook handler and I classified it as the real hazard case. The exact bug the rule exists to prevent.

Then I read the code around it. The partner's documented contract uses that display value as the shared key, because they don't have access to our internal ids. The docblock says so. A sibling method does the same match for the same reason. The check fired correctly on the symbol and the usage was correct.

An AST wouldn't have helped. Parsing tells you where a symbol sits syntactically. It doesn't tell you whether an external contract sanctions it. The information I needed was in a comment and in another method's behavior.

That's the ceiling, and it's not a parser problem. Across 26 findings, nothing pushed me toward building an AST layer. One thing pushed me toward a new primitive. I fully expected to be writing a parser by now.

The unenforceable rules caught more than the checkable ones

This is the part I didn't see coming.

The agent told me the tool's scaffold was working. Two documented interfaces didn't exist. One entrypoint had no top-level invocation and silently did nothing when run. The JSON output flag had never been implemented at all, despite being the thing the whole integration was supposed to hang off.

Both got caught by "don't claim something works without running it the documented way." No check could have found either, because a tool that doesn't run produces no findings, and no findings looks exactly like clean.

Same bucket also stopped it from writing a log entry it couldn't source, stopped it from inventing a config file it hadn't been shown, and got it to volunteer a force-push it had done without being asked.

None of that is verifiable afterward. All of it held because the rule was in front of it at the time.

Four wrong numbers

While writing this up I stated four counts from memory and all four were wrong.

I said 25 findings. The file said 24. Correcting that, I derived 23 arithmetically from remembered subtotals, in a paragraph about the dangers of doing that. I predicted 15 real drift sites before a run. There were 13. I said four exclusions across three files. Two files — and I said it in the middle of explaining that conflating site counts with file counts is the error under discussion.

Every correction came from opening the output file. None came from thinking harder about it.

So the failure mode survives knowing about it. Being actively vigilant about a specific error does not prevent that error. That's a worse result than I wanted and it's the strongest argument in the whole project for writing things down somewhere a machine can check them. Applies to the prose as much as the code.

Where I've landed

There are tools that infer your architecture from the repo, tools that compare code against a formal model, tools that score erosion statistically. Some of them are considerably more mature than mine.

I started somewhere else: some architectural decisions exist only because a human made them, and an agent shouldn't have to rediscover them from the code. So the document is human-authored, the checks enforce the part that's mechanizable, and the rest gets carried as context the agent needs but can't prove.

The checker is the enforcement half of a three-part contract. It turned out to be the smaller half, and I'm still not sure whether that means I should build more checks or write better rules.

Tool and the full drift log, including all four of my wrong numbers: github.com/KaleemNoorani/arch-drift