I audited my own AI agent guardrails. Four of them were walkable.

Most AI agent development advice is about what to write down. This post is
about which of those written rules a machine is actually holding, in the two repositories where I run agents every day, and it is not a tour - it is an audit with the failures left in.

In May I closed a post by promising this one. Architecture before the AI build argued that a dependency graph fixed before any agent started was the load-bearing decision, and deferred the rest: "the skill files, the briefing pattern, the verifier role, and the instruction set that let eight roles coordinate without stepping on each other's work."

I sat down to write that and audited the harness first. Four of my own guards turned out to be walkable, one rule I had filed under "prose nobody checks" turned out to be gated by eleven fail-closed counters, and a skill file I open most days has carried a 30%-wrong number since June.

One claim, and I have tried to break it: every rule I wrote by hand and enforce here is decided by comparing two strings or two integers. The rules that rot are the ones needing judgment. Find a rule in my .claude/ config that a machine holds and that needs more than a comparison to decide, and the claim fails.

That excludes the compilers, deliberately, and the exclusion is the interesting part. tsc and Biome decide genuinely hard questions and hold them perfectly. I did not write them. Everything I authored myself falls on the comparison side, and what I could not reduce to a comparison I wrote down and hoped.

Two AI agent development repos, opposite policies on the same rule

Start with the rule I was most confident about: agents do not touch git.

// flare-engine/.claude/settings.json - permissions.deny (15 entries)
"Bash(git push:*)", "Bash(git commit:*)", "Bash(git add:*)",
"Bash(git reset:*)", "Bash(git rebase:*)", "Bash(git merge:*)",
"Bash(git checkout:*)"

// Pan-Tvardowski/.claude/settings.json - permissions.allow (25 entries)
"Bash(git add:*)", "Bash(git commit:*)"

Same author, opposite policy - and the engine did not start where it ended. Its first agent-config commit, caaca69 on 2026-04-17, had no deny block at all and explicitly allow-listed both git add and git commit. The denial landed a week later in 9fabff1. Pan Tvardowski's allowance landed 2026-06-10, seven weeks after that. Allow, then deny, then allow again in the next repo. Both end states are live.

That is not an inconsistency. The two repos have different units of work: in the engine an agent's output is a paste-ready commit block that I run myself, so an agent commit is a contract violation; in the game repo work advances one roadmap row at a time and the commit is the unit. Two risk models, deliberately chosen.

What I cannot defend is the third repo. This blog's own repository states the rule in eight places and enforces it in none:

• docs/blog/RULES.md:17 - "No git operations. No commit, no push, no branch creation."
• CLAUDE.md:71 - "Propose only - never run git add/git commit/git push. The human pastes."
• Three of eight agent files and three of thirteen slash-command files repeat it.

Its .claude/settings.json carries 33 allow entries and 16 deny entries.
Filter all of them for git commit or git add and you get an empty list, in both directions. Eight read-only-looking verbs are explicitly allow-listed, which is what makes the omission read as an oversight rather than a decision. All eight agents hold Bash.

One of those eight allow entries is worse than an omission. Bash(git branch*) is a prefix match, so it silently auto-approves git branch -D main. The rule it serves says "no commit, no push, no branch creation." A third of that rule is not merely unenforced; it is pre-approved. My own patched guard in the other repo already classifies branch as conditional and blocks that exact command - drift between two files nobody diffed.

The honest limit: unlisted is not unguarded. An unmatched command falls through to an interactive permission prompt, so most of the portfolio's rule is human-gated rather than machine-refused. That is a real control. It is just not the one the documents claim, and it does not cover the verb that is listed.

Here is the whole engine loop, with each step marked by whether it can refuse anything:

SessionStart hook (session-start.sh)
    |  injects next queue row
    v
agent reads CLAUDE.md + AGENTS.md + skill
    |  prose: asks
    v
agent proposes a Bash call
    |
    v
[ PreToolUse: pre-bash-guard.sh ]  --REFUSES (exit 2)--> back to the agent
    |  allows
    v
Edit / Write
    |
    v
PostToolUse: biome, silent rewrite
    |  prose: asks
    v
scope partition ("You never touch:")
    |
    v
/close runs verify.sh
    |
    v
[ 8 gates ]  --REFUSES (exit 1)--> back to /close
    |  green
    v
session block appended + paste-ready commit block
    |
    v
human runs the commit

Two of those steps can refuse. The rest ask.

What it looks like when a machine is holding a rule

Here is the shape of the other half. When I close a session in the engine,
/close runs verify.sh, which runs eight gates in order and refuses to write the session block if any of them fails:

# flare-engine/.claude/hooks/verify.sh:58-84, condensed (scope conditionals removed)
run_gate "lint"      bun run lint
run_gate "typecheck" bun run typecheck
run_gate "test"      bun run test
run_gate "build"     bun run build          # engine + mixed scope only
run_gate "deps"      bun run check:deps
run_gate "changeset" bun run check:changeset
run_gate "perf"           bun run check:perf
run_gate "perf-freshness" bun run check:perf-freshness

Every one of those is a process exiting non-zero. None is an instruction. The test gate alone runs the [deterministic suite (https://grzegorzotto.dev/blog/deterministic-game-testing) that post covers in depth.

Pan Tvardowski goes harder. It has a file of the same name - a different script, 81 lines against the engine's 197, running six gates rather than eight - wired as a Stop hook with a 300-second timeout, so the agent is not supposed to end its turn while the gate is red. Its line 6: "exit 2 -> block the stop; stderr is fed back to Claude to fix."

I have read that wiring and the exit path. I have never watched it fire, so I am not going to tell you what it feels like when it does. And reading it properly, for this post, turned up three exit-0 fast paths I had forgotten, one of which is a plain bypass: if the stop was already blocked once, stop_hook_active is set and the hook returns 0 rather than re-running the gate. Stop once, get blocked. Stop again, and the turn ends with the gate still red.

Something like that has to exist or a red gate traps the agent forever. But my strongest mechanism is two attempts deep, not absolute - and the repo's own /milestone skill already says "Run the gate locally - do NOT rely on the Stop hook to find failures," which is me declining to trust it in writing without having found the reason.

One of those eight gates deserves a correction rather than a description.
check:deps runs scripts/check-deps-tiers.mjs, 264 lines that fail on three things: dependencies declared but never imported, dependencies imported but never declared, and any package depending on a strictly higher tier. It is what holds the layer graph today.

It is not what I said held it, and it did not exist when I said it. In
May I wrote that "pnpm refuses to resolve the import," and a fortnight later that "a violation fails at pnpm install." Three things are wrong with that, in increasing order of how much they matter.

flare-engine has never used pnpm - its first commit declares "packageManager": "bun@1.2.0" and no pnpm lockfile has ever existed.
The monorepo post corrected that tool name in July.

Worse, renaming it does not fix the sentence. Bun installs hoisted, so packages resolve from the root node_modules and there is no per-package boundary for any resolver to refuse. The mechanism is not misnamed; it is absent.

Worst, check-deps-tiers.mjs landed 2026-07-01. For the entire period both posts describe, the layer rule was held by nothing mechanical at all - by agent briefings, review, and the rule being easy to follow. It held. I had the reason wrong, and the reason I gave was the one that made it sound automatic.

That error is the thesis in miniature. It carries no number, so no fact check could see it. It describes a mechanism, so only re-running it catches it. It survived three months and five published artifacts. The dated corrections on those two posts are owed and this is not the place for the full accounting - that belongs in the retrospective at the end of this series.

The blocklist held exactly the verbs it was told to name

The guard I trusted most was pre-bash-guard.sh, a PreToolUse hook that runs before every Bash call an agent makes. Forty-one lines. A case statement over start-anchored globs, catching env-prefixed and chained variants that the deny list would miss.

It blocks git commit -m x. It blocks FOO=1 git push origin main. It blocks cd packages && git add ..

Here is what it did not block, each verified by piping real hook input to the real hook on 2026-08-09:

Command	Old guard
git commit -m x	blocked, exit 2
bash -c "git commit -m x"	allowed, exit 0
eval "git commit -m x"	allowed, exit 0
git -C . commit -m x	allowed, exit 0
git revert / cherry-pick / rm / restore / stash	allowed, exit 0

Three different failures. A wrapper hides the verb from a start-anchored glob. A global option displaces it past the anchor - git -C . commit does not start with git commit. And five state-changing verbs were never written down, because a blocklist can only refuse what somebody remembered to enumerate.

The guard had existed since 2026-04-24 with no test. Three and a half months. It looked mechanical, so nobody re-examined it, which is precisely the failure this post is about, turned on its author.

I rewrote it before publishing. It is an allowlist now: 32 git subcommands
classified read-only, plus 12 more allowed only in specific read-only shapes (git branch lists, git branch -D does not). Anything outside both sets is refused, including subcommands git has not shipped yet.

# flare-engine/.claude/hooks/pre-bash-guard.sh - the inversion (excerpt)
GIT_READONLY = {
    "status", "diff", "log", "show", "blame", "annotate", "rev-parse",
    "rev-list", "ls-files", "ls-tree", "ls-remote", "cat-file", "describe",
    # ... 32 entries. Everything outside this and GIT_CONDITIONAL is
    # treated as state-changing.
}

Four other things changed with it. Segments are now split quote-aware and
interpreter payloads are inspected, because the old hook matched substrings and never looked inside one: python3 -c "import os; os.system('git commit -m x')" matched no pattern it carried and exited 0. Wrappers are unwrapped and re-analysed. Git global options are folded away before the subcommand is read. And it fails closed: if it cannot parse its input, it refuses.

Then I wrote the thing that should have existed in April: a contract test.

And the contract test immediately found that my rewrite was also wrong. I had classified git fetch as read-only. It is not - git fetch origin main:side creates a local ref and git fetch --prune deletes remote-tracking ones. Same for git symbolic-ref, which reads HEAD with one argument and repoints it with two. Two more holes, in the fix, written by someone who had just spent a day thinking about exactly this failure.

Both are now conditional, and both have cases. The file stands at 72 - 46 that must block, 26 that must still be allowed - all passing. That is the difference between a rule and a gate: the gate has a contract you can run, and running it is how I found out the gate was wrong twice.

This is also where the verifier role went, and the answer is the least
glamorous one available. verifier.md was a 107-line agent whose constraints included "No source edits. Ever." It was deleted on 2026-05-13 with five other roles. Its successor is not an agent. It is verify.sh. A role became a shell script, and the shell script is stricter.

The rule I wrote in six files is gated in eleven counters

I went into this audit expecting a clean story: the rules I repeat most are the ones no machine can check. "No allocation in update() or render()" is written into AGENTS.md, docs/conventions/code.md, both agent briefings, CONTRIBUTING.md and the PR template. Six files. No linter can prove the absence of an allocation on a hot path, so I filed it under prose.

That was wrong, and finding out cost one grep.

// flare-engine/.perf-baseline.json - the "counters" object
"alloc.render  1000 sprites / 1 atlas"
"alloc.render  1000 sprites / 100% rotated"
"alloc.render  1000 sprites / 8 atlas interleaved"
"alloc.render  1000 sprites / 8 atlas clustered"
... 11 scenarios total

scripts/check-perf.mjs reconciles those counters against a live run of the allocation suite against a counting Skia stub, and it is wired into verify.sh and into the release script. Its own header, which I wrote and then forgot, is stricter than anything I would have claimed for it:

// flare-engine/scripts/check-perf.mjs:13-17 (em-dashes normalized to hyphens)
//   1. Draw-path counters - allocations, draw calls per flush, batch runs,
//      transform writes - measured live by the benchmarks' alloc suite against
//      a counting Skia stub. Fail-closed in both directions: an empty baseline
//      fails, a scenario with no key fails, a key with no scenario fails, and
//      an IMPROVEMENT fails too (it is still a change to a published claim).

An improvement fails the gate. A scenario cannot be retired by deleting it -
that path demands an explicit flag and a dated review entry.

So the rule is enforced, but look at which part. Not "do not allocate on
hot paths." What is gated is: eleven named scenarios, each producing an
integer, each compared for exact equality against a committed number. The
semantic rule is unenforceable and remains unenforced. The slice of it that reduces to an integer comparison is gated harder than I remembered.

That is the whole finding, and it is smaller and more honest than the one I set out to write.

What rots instead

The prose half is not idle. It is where the drift accumulates, silently,
because prose has no failure mode.

The start skill is what I invoke to open an engine session. Line 39 warns that the localization cascade is "the single most repeated undeclared drift in 188 logged sessions." That was exactly right the day it was written: git show 0c49191:SESSION_LOG.md counts 188 session blocks, and 0c49191 is the commit that added the line, on 2026-06-10. Today the same log holds 270 blocks carrying 231 distinct numbers. Sixty days, and the file understates by 82 blocks - or by 43 distinct session numbers, depending which unit you pick, and the file does not say which one it meant.

The log's own header is worse. Two of its opening lines carry four references to things that were all deleted on 2026-05-13: a /session-close command, the verifier agent, a .github/instructions/session-close.instructions.md file, and a docs/plans/sessions.md named as the source of truth. Three months later the header still describes them, and still calls the file "append-only," which stopped being true when verify.sh began upserting same-day blocks in July.

The numbering in that log is broken too - 270 blocks carry only 231 distinct numbers, because verify.sh:128 mints the next one by counting headings rather than taking the maximum - but that is a bug with a fix, not drift, so it belongs in a different post.

None of the rest was sloppiness. Every one of those lines was correct when it was typed. They are correct-then-stale, and nothing in the stack could have flagged a single one, because none of them is a string a process compares to anything.

The scope partition works the same way. Both agent briefings carry a hard
boundary near the top of their scope section:

<!-- flare-engine/.claude/agents/engine-builder.md:22 -->
**You never touch:** `apps/**`, `.github/**`, `.claude/**`,
`docs/plans/**` (except reading), `SESSION_LOG.md` (verify.sh appends at `/close`).

That line is the anti-collision mechanism the May post promised to explain, and it is prose. Nothing prevents a violation; what exists is a record after one. The session log's frictions ledger counts drift events, and across the 50 sessions carrying numeric counters there are 29. The partition is a convention with an incident log, not a boundary.

And since I promised the skill files: there are six today, 571 lines. In May there were three - growth, not a correction.

The promise was to publish the skills that ran the April sprint, and I can only half keep it. In April the text lived in .github/skills/ - 38 files - and .claude/skills/ held 38 eight-line stubs pointing at them. The May refactor deleted all 38 originals and moved three into .claude/: create-flare-app, flare-game-dev, flare-tier2-stubs, three of today's six. Those three are still the April text almost verbatim - 264, 81 and 29 lines, with 5, 2 and 1 lines changed in the three months since. The other 35 are gone from the working tree and readable only out of history.

Where this breaks

Every gate here measures a filter against itself, including mine. 926 of flare-engine's 928 commits satisfy its commitlint contract, which is not evidence the hook works - a filter cannot report a pass rate other than about 100% except where it was bypassed. Same for my own 72 passing cases: I wrote most of them from the bypass list I had just found, so "all passing" is partly a restatement of what I already knew. The two that count are fetch and symbolic-ref, which I did not know about, and which surfaced only because writing a test forced me to classify every verb rather than the ones I had thought of. Coverage is still unknown - nothing in the file tests an interpreter payload that reaches git without spelling it.

A gate enforces its own config, not your documentation. This repo's CLAUDE.md states a 72-character commit-subject cap. commitlint.config.cjs never sets header-max-length, so the real cap is the inherited 100, and 15 of 348 subjects exceed 72. All passed.

n is two repos, one author, one month, one model vendor. No control repo, no counterfactual, and I am the only variable in every comparison. I also picked the enforced-versus-semantic split after looking at the data.

The counter-explanation is at least as good as mine. Maybe machines do not select checkable rules; maybe I only ever bothered to automate the cheap ones, and the pattern is my laziness rather than a property of enforcement. I cannot separate those with the evidence I have.

There is no outcome data here, and there will not be. No task times, no defect rates, no before-and-after. The external evidence closes both escape routes. METR's 2025 trial measured experienced developers taking about 19% more time with AI; METR's own February 2026 follow-up estimates the same cohort roughly 18% faster, with a confidence interval running from 38% faster to 9% slower - it crosses zero, and METR calls it "an unreliable signal." Their May 2026 survey reports a self-reported 2x using the instrument their own trial showed to overestimate by 40 percentage points. Nobody has settled this in either direction.

The best peer-reviewed evidence cuts against the prose half specifically.
ETH Zurich evaluated repository context files and found they do "not generally improve task success rates, while increasing inference cost by over 20% on average" - while scoping their own null result to exclude my exact use case: context files remain "useful for specifying non-standard coding practices." Adoption is thinner than the conversation suggests, too. An AIware 2026 study sampled 37,249 actively maintained repositories and found 2,853 with any AI configuration at all; within that already adopting group, 158 use skills and 131 use subagents. And the vendor is arguing for less: Anthropic published, seventeen days before this post, that it removed over 80% of Claude Code's own system prompt "with no measurable loss on our coding evaluations."

This does not contradict what I wrote in May. That post concluded the tooling was scaffolding and the architecture was the load-bearing piece, and this audit agrees: nothing here holds the dependency graph up except a 264-line script. Scaffolding is not nothing. It is just not structure.

The rule I would give someone else

Do not count the places a rule is written. Count the processes that exit
non-zero when it is broken.

Then write the test for that process, because naming a check is not enough and this post is the proof. My git guard was a nameable check for three and a half months and did not hold, four ways. The rewrite did not hold either, twice, and the only reason I know is the contract test I wrote afterwards. A check with no test is a rule wearing a machine's clothes.

That is not an argument for deleting the prose. Some rules genuinely cannot be reduced to a comparison, and "no allocation on a hot path" is one. It is an argument for knowing which half a rule is in, so you stop being surprised when the unenforced half drifts - a preference degrades at exactly the rate nobody is looking, which here was 30% in sixty days, in the file I open most days.

I can only show you two repos, but mine divided cleanly: the import boundary a lint rule refuses, and the import boundary a paragraph requests. Only one of those is a boundary. Whether yours divides the same way is worth twenty minutes and a grep.

---

The engine relicensed to Apache-2.0 in July and opens publicly in October. Every figure here was re-derived on 2026-08-09 against flare engine df1e3de, except the guard rewrite and its 72 test cases, which were written the same day and land in a commit after it - if you check df1e3de you will find the old 41-line blocklist and all four bypasses still working. The .claude/ files move weekly, so treat every count as a dated snapshot rather than a spec.

---

Originally posted on grzegorzotto.dev.

Comments

[Alex Shev]

The useful audit question is not "did I have a guardrail?" but "could a normal helpful path walk around it?" Agents rarely break rules like an attacker in a demo; they break them while trying to complete a reasonable task. Testing those ordinary paths is where weak guardrails show up.