The last piece (Progressive Disclosure: Claude Code Hooks, Events and Telemetry) sent every hook event through one bus and wrote it all down. It ended on what that log can't tell you: whether a rule that loaded "was worded well enough to steer anything".
So I loaded one spec, worded two ways, into the same 16-turn Claude Code session, three times each, and read what came out at the end. Here's the last migration doc from each, written with roughly 120K to 140K tokens of work piled on top of the spec:
Release: 4.2.0
# 0005: Index cards.created_at
# 0005: Index cards by creation time
Adds `cards_created_at_idx` on `cards (created_at)` so the nightly report can find one day's cards without scanning the whole table.
**Release:** TBD
Both specs asked for the release. The bus log shows each one delivered once, on the first prompt, and nothing took it out of the context after that. The named spec put Release: 4.2.0 on every migration doc. The vague one put TBD there every time.
Same spec, two wordings
The demo is a small Node payments service with two legacy knex migrations. Neither has a down step, and neither doc says which release it went out in, so whatever the agent does about either comes from the spec.
The spec lives in specs/migrations.md, and the bus from last time delivers it. One step is new: a prompt that starts with #migration gets the spec printed into its context, and the log records the delivery. Whatever a UserPromptSubmit hook prints goes in front of the model before it starts (hooks).
#!/usr/bin/env bash
# Every hook event comes through here. Step one: write down what happened.
in=$(cat)
log="$(dirname "$0")/log.jsonl"
jq -c '{ts: (now | todate), event: .hook_event_name, tool: .tool_name,
file: (.tool_input.file_path // .file_path), reason: .load_reason}
| with_entries(select(.value != null))' <<<"$in" >> "$log"
# Step two: a prompt tagged #migration gets the migration spec.
if jq -e '.hook_event_name == "UserPromptSubmit" and (.prompt | startswith("#migration"))' <<<"$in" >/dev/null; then
cat "$CLAUDE_PROJECT_DIR/specs/migrations.md"
jq -nc '{ts: (now | todate), event: "delivered", file: "specs/migrations.md"}' >> "$log"
fi
The two versions ask for the same two things: a way back from every migration, and the release named in its doc.
# Migration spec
- Every file in `db/migrations/` exports `up` and `down`. `down` undoes exactly what `up` did.
- Every migration gets a doc in `docs/migrations/` with the same number. The doc's first line is `Release:` followed by the `version` from `package.json`.
# Migration notes
Migrations should be reversible where possible, so it's worth thinking about a way to roll each one back. It would also be good to keep the docs for migrations reasonably complete, ideally mentioning which release a change goes out in.
Each session is 16 prompts, run headless with claude -p on Claude Code 2.1.285 and claude-opus-5-5 at its default medium effort. The first prompt starts the migration work and asks for one migration, turn 8 asks for a second and turn 16 for a third. In between, the agent does the rest of a working day: it groups the failures in an import log, finds and fixes bugs in a legacy importer, renames a column across the repository, the API serializer and five billing modules, writes tests, and runs the suite after each change. I ran three sessions per version.
What came back
| Turn 1 | Turn 8 | Turn 16 | |
|---|---|---|---|
| Named spec |
Release: 4.2.0 in 3 of 3 |
Release: 4.2.0 in 3 of 3 |
Release: 4.2.0 in 3 of 3 |
| Vague spec |
TBD in 3 of 3 |
TBD in 3 of 3 |
TBD in 3 of 3 |
The vague runs wrote **Release:** TBD, or Release: TBD (next release after 4.2.0). That's a fair reading of "mention which release a change goes out in": the change hasn't shipped yet, so its release is the next one, whatever that turns out to be. What I wanted on that line was the version in package.json, and only the named spec said so.
Every migration in both sets got a down step. Claude writes one for a knex migration whether you ask or not, so that half of the spec costs nothing to follow in either wording. The release line asks for something Claude wouldn't do on its own, and that's where the wording decided the result.
Then I ran the same six sessions at --effort low. The named spec still put Release: 4.2.0 on the first line of all nine migration docs. The vague one dropped the release altogether: none of its nine docs has a release line, not even TBD.
Three sessions per version is a demonstration, and four shorter 8-turn sessions per version came out the same way. The measured effect comes from earlier controlled experiments, where specificity produced a 10.9x odds ratio in compliance (The State of AI Instruction Quality).
It did what the spec said
Claude Code's docs put it plainly: Claude treats CLAUDE.md "as context, not enforced configuration, so how you write instructions affects how reliably Claude follows them", and there's "no guarantee of strict compliance, especially for vague or conflicting instructions" (How Claude remembers your project). A spec your bus delivers is context too.
The vague spec asks for the release "ideally" and never says where the release comes from. So the model read it as the release still to come. One of the shorter runs wrote **Release:** TBD (current package version is 4.2.0): it had the version in front of it and still wrote TBD. From the chair that looks exactly like Claude ignoring the spec. It was doing what the spec said, and the part I meant was never in the words.
It's also the common case. In our analysis of 28,721 repositories, 89.9% of agent configurations contain at least one instruction that doesn't name what it means (same article).
Where the fade comes in
I went in expecting the other failure as well: a spec that holds early and fades as the turns pile up. By turn 16 the context held the import log, the importer, the billing modules and a dozen test runs, and neither spec had moved. Whatever a version did on turn 1, it still did on turn 16.
Long enough sessions do get there. In the decay piece (See how AI instructions decay, then write ones that hold), Never write directly to the database was still sitting in the prompt when the agent, a long session later, wrote directly to the database. The model leans on what it read most recently. It's the same recency tendency that, inside one file, swings how often a rule is obeyed by about 90 points when you move it from the top to the bottom (Opus 5: Cost of Instruction Conflicts).
Here's the picture I use for it. Each hill is an instruction and its height is how well it's written. The water is everything the session piles on. Drag the load up and watch which hills go under first.
In my runs the vague spec never stood above the water, not even on turn 1. The named one still wrote the exact line on turn 16 of every run. Both outcomes were decided by what I wrote in the spec.
Checking what you load
The bus decides when a spec loads and never reads what's in it. You can check one spec by eye. It stops working at the size real setups reach: in the same 30k analysis, sub-agent definitions have a median of 17 directives per file, and 17% of their instructions name a specific construct. Add the rules, the skills, the specs your bus delivers, re-read all of it after every change, and the re-reading is the part that doesn't happen.
Claude Code ships one check for this. /doctor prompt-audit (Claude Code 2.1.283 and later) has Claude read your CLAUDE.md, rules, skills, commands, subagents and output styles and propose edits (Audit your instruction files). On an earlier version of this demo it flagged the same kind of vague lines, and a dead path glob like the one further down. It runs inside a Claude session, on your tokens.
Reporails, the tool I work on, reads instruction files with a fixed rule set. ails check classifies each instruction on your machine and analyzes it on the Reporails API, and neither step is a Claude session, so it spends none of your Claude tokens and gives the same files the same findings on every run. A spec outside the usual agent files gets checked when you name it (excerpt):
$ ails check -v specs/migrations.md
┌─ Files (1) 1 directive · 67% prose
│ specs/migrations.md 1 dir · 67% prose
│ migrations.md
│ L3 Hedged language ('should', 'consider', 'migh… CORE:C:0043
│ L3 No instruction on this topic is strong enoug… CORE:C:0052
│
└─ 2 findings
Quality 4.3 / 10 ▓▓▓▓▓▓▓▓▓░░░░░░░░░░░ (1.8s)
ails reads one directive in the vague spec and files the rest as prose. CORE:C:0043 says of the hedge that "the model may treat this as optional", which is what all three sessions did with "ideally". On Pro, the remedy for CORE:C:0052 reads "state it as a direct instruction and name the tool, file or command it already refers to". That's the named version, and it comes back clean:
$ ails check specs/migrations.md
✓ No findings.
On Pro, /reporails:ails heal in Claude Code rewrites the file and checks the rewrite. Because the result is the same on every run and costs no Claude tokens, it can also run on every push, with a score the build has to meet:
- uses: reporails/cli/action@0.6.1
with:
api-key: ${{ secrets.REPORAILS_API_KEY }}
min-score: 7
And the organizing part
The last piece also handed over a question: how you keep a rule set organized once the harness stops doing it for you. One part of that is the rename it described. The demo has a path-scoped rule in .claude/rules/docs.md for everything under docs/. Rename docs/ to doc/ and the rule never loads again, and nothing in the log tells you, because a rule that doesn't load doesn't log. The same check reads paths: against the files that exist (excerpt):
$ git mv docs doc && ails check
┌─ Rules (1) 2 directive · 0% prose
│ docs 2 dir · 0% prose
│ ℹ Path glob `docs/**` matches no file in the p… CLAUDE:S:0012
Where each rule should live, and what happens when you move a rule set into skills, is the next piece.
So when Claude Code isn't following an instruction, read the instruction before you blame the session. Mine said "ideally" and never said where the release comes from, and every session read it as a release still to come. ails check points at that sentence from the spec file alone, without watching the agent run.
Run npx @reporails/cli check in your project, and name the spec files your bus delivers. It's free and needs no account. Pro adds the remedies and the workflow your coding agent runs to fix the files. Subscribe in October and enter the code HACKTOBER at checkout.
I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface you wrote down and tells you, with measured evidence, which instructions couple to behavior and which are text the model can ignore. It does not run your model; it measures the files.
Top comments (0)