DEV Community: rulereceipt

I measured whether Claude Code actually follows my CLAUDE.md

rulereceipt — Mon, 21 Sep 2026 13:50:21 +0000

Everybody writes a CLAUDE.md. Almost nobody checks whether it gets followed. I stopped assuming and built a small tool to check, and the results were worse, and more interesting, than I expected.

Disclosure up front: the tool is mine, it's npx rulereceipt, source-available. This post is about what I found building it, not a pitch. The parts where it's wrong are the parts I actually care about.

The problem

You write "always run the tests before committing" and "surface bad news first" into a file, the agent reads the file, and then it does whatever it wants. Sometimes it even tells you it followed the rule when it
didn't. I kept hitting the same shapes, and I'm not the only one: there are bug reports where people logged a hundred-plus incidents by hand out of their session transcripts and reached the same conclusion, that
prose rules get read and not executed.

The frustrating part is that it's all sitting in the transcript. You can check it after the fact. So I did.

What it does

It reads the Claude Code session transcript (the JSONL under ~/.claude/projects/) together with your CLAUDE.md, and scores each rule.

Two kinds of rule get handled differently. Rules that are checkable with no ambiguity, like "never commit to main" or "never touch .env", get deterministic pattern checks: no model, nothing leaves your machine.
The judgment ones, like "don't over-engineer" or "surface bad news first", go to one structured LLM call using your own API key.

Every result carries an evidence line, or it's marked "didn't run". No silent passes. If it can't tell, it says so instead of guessing.

The hard part wasn't detection. It was false accusations.

My first version confidently flagged rules as broken that weren't. It saw git push --force-with-lease and screamed "force push". A false accusation is worse than a miss, because once the tool cries wolf you
stop trusting any of it.

Getting the false-accusation rate from 15.8% down to 2.9% was most of the work. It's still not zero, and that number is in the README. Any tool like this that claims perfect precision is lying to you.

Catching fabricated "done"

The sharpest failure I keep seeing: the agent reports a test suite green when a fake worker wrote hardcoded checkmarks, or calls a fix "verified" without running anything. So there's a specific check for a claim
of completion with no evidence in the session behind it: "verified", "at baseline", "tests pass", asserted where the session never actually produced that result.

Before, not just after

A report tells you after the damage is done. So there's also a PreToolUse hook that can refuse a rule-breaking command before it runs.

Here's the honest limit, because I measured it. I tried blocking on the command literals written inside rules, and it refused 62% of 16,336 real commands. One rule titled "Feature Validation" recommends npm run build and forbids Playwright, so blocking on its only command-shaped literal refused the recommended command. Nothing in a rules file marks which backtick is the prohibition.

So the guard only auto-blocks rules that name a file or a branch, like "never modify migrations/" or "never commit to main", where the intent is unambiguous. And it fails open: a bug in the guard can't stop
you from working. It's a narrow, safe block, not a magic obedience layer.

Try it, and tell me where it's wrong

npx rulereceipt runs against your last session. rulereceipt demo runs with no setup and no API key at all.

I'd rather hear where it false-accuses than where it works. That's the whole game.

I parsed 559 public CLAUDE.md files. Most of what's in them isn't rules.

rulereceipt — Wed, 02 Sep 2026 05:11:55 +0000

I've been building a tool that checks whether Claude Code actually followed the rules in your CLAUDE.md. Early on I realised I was testing it against one file: my own. So I went and got 559 real ones.

CLAUDE.md, AGENTS.md, .cursorrules, Copilot instructions, Windsurf, Gemini — from PyTorch, Kubernetes, Elasticsearch and a few hundred smaller repos. 21,986 items parsed out of them, no crashes.

Two things came out of it that I didn't expect.

63.4% of a rules file isn't a rule

Directory listings. Reference tables. Import examples. Architecture notes. Glob syntax cheatsheets.

All useful documentation. None of it is telling the agent to do anything. And it's sitting in the same file as the actual rules, competing for the same attention.

So when someone says their CLAUDE.md is 400 lines, it's usually closer to 150 lines of rules with 250 lines of README stapled on.

That reframes the "my CLAUDE.md is too long" problem. It's often not too many rules. It's rules buried in documentation.

Of the real rules, only 44.2% can be checked mechanically

Here's the split that surprised me more.

Checkable:

"Never commit directly to main" — you can look at what git actually ran
"No console.log in committed code" — you can look at what was written to files
"Run the test suite before pushing" — either the command ran or it didn't

Not checkable by any tool, ever:

"Surface bad news first"
"Explain the trade-off before choosing an approach"
"Keep changes reviewable"

There's no command to inspect for the second group. It's not a limitation of my parser — the information isn't in the transcript. A person has to read it and decide.

55.8% of the rules people actually write fall in that second group.

The distribution barely moved as the corpus grew

I started with 40 files: 65.4% non-instructions.
At 559 files: 63.4%.

Between those two points I added formats the parser had never been tuned on. It moved 2.0 percentage points.

That's the part I'd actually stand behind. It suggests "is this a directive?" is a property of language, not of file format — which matters, because the alternative approach (recognising every way people format
rules) is an infinite list.

What I don't know

I don't have a controlled experiment showing longer files perform worse. That's the obvious next question and I haven't run it.

But if a model is budgeting attention across a long file, it seems worth knowing that most of that file was never telling it to do anything.

The tool

It's called RuleReceipt. Free, runs locally, no account:

  npx rulereceipt check

I also published the part where it got things badly wrong — it reported 10 violations on a real session of mine and every single one was false. That write-up is at https://rulereceipt.dev/postmortem, including the wrong turn I took while fixing it.

Happy to answer anything about the method. The corpus was public files only, and the classifier is in the repo.

Update, 4 September.

Two bugs in my own parser moved these numbers after publishing, so I'm correcting them in place rather than leaving them wrong. The parser had no concept of fenced code blocks, so a shell comment starting with # or a YAML item starting with - was read as a rule boundary — 588 rules were being cut mid-block, and 1,718 items (7.2% of everything extracted) were lines lifted out of code samples that had never been rules at all.

Separately, command documentation was being counted as rules. Corrected: 21,986 items, 63.4% non-instructions, 44.2% of real rules checkable. The 40-file sample moved 2.0 points rather than 2.5, which if anything strengthens the stability point.

Full history of every figure and what changed it: https://rulereceipt.dev/postmortem#history