My AI rule-checker was wrong about 1 in 6 times. Fixing that was the whole project.

rulereceipt — Mon, 21 Sep 2026 13:50:21 +0000

I've been building a small thing that reads a Claude Code session afterward and checks it against your rules file — did it follow "ask before deleting," "run the tests before claiming done," that kind of thing.

Simple idea. The naive version took an afternoon.

Then I ran it against real data and it was a liar.

Not a little. Here's the number that mattered: I ran 559 real rules files against 5 real session transcripts — about 2,795 reports, ~18,000 individual verdicts. The stat I cared about was "what fraction of reports show the user at least one FAIL," because that's what a person actually sees on one run.

15.8% of reports had at least one FAIL.

If one in six runs accuses you of breaking a rule you didn't break, nobody trusts the tool again. It's worse than useless — it's a smoke alarm that goes off when you make toast. So I spent the bulk of the work not on catching more, but on catching wrongly less. It's now at 2.9%, and the bugs I found getting there are more interesting than the tool.

A comma is not a rule. One rules file listed , as a forbidden pattern. My matcher dutifully flagged every file that contained a comma.

rm fired inside "performance". My pattern matcher had a word boundary on the end but not the start, so the rule "don't run rm" matched the letters r-m sitting inside the word performance. Every session that used the word got accused of running a delete command.

The worst one: guessing, and always guessing guilty. When my classifier couldn't tell which direction a rule pointed — "you must do X" or "you must never do X" — it defaulted to the accusing direction, and I'd justified that as "preserving existing behavior." It then failed a real session for following a rule that said "update the closest CLAUDE.md," because the session had updated one. It saw the right thing happen and called it a violation.

The fix that stuck wasn't a smarter heuristic. It was making the bad case impossible to build: a FAIL verdict can now only be constructed through one function that requires the rule's direction to be typed as"forbid." Code that's checking a "you must do X" rule literally cannot emit an accusation — the build fails if it tries.

The general lesson, if there is one: for anything that judges other work — a linter, a test oracle, an AI-output checker — a false positive costs far more than a miss, because it spends the user's trust, and trust doesn't refill.

The tool is RuleReceipt (https://rulereceipt.dev) if you want the shape of it (npx rulereceipt check, runs locally, no account) — but honestly I mostly wanted to write down the toast-alarm bugs, because I suspect anyone who's built something that grades other things has their own version of the comma.

I parsed 559 public CLAUDE.md files. Most of what's in them isn't rules.

rulereceipt — Wed, 02 Sep 2026 05:11:55 +0000

I've been building a tool that checks whether Claude Code actually followed the rules in your CLAUDE.md. Early on I realised I was testing it against one file: my own. So I went and got 559 real ones.

CLAUDE.md, AGENTS.md, .cursorrules, Copilot instructions, Windsurf, Gemini — from PyTorch, Kubernetes, Elasticsearch and a few hundred smaller repos. 21,986 items parsed out of them, no crashes.

Two things came out of it that I didn't expect.

63.4% of a rules file isn't a rule

Directory listings. Reference tables. Import examples. Architecture notes. Glob syntax cheatsheets.

All useful documentation. None of it is telling the agent to do anything. And it's sitting in the same file as the actual rules, competing for the same attention.

So when someone says their CLAUDE.md is 400 lines, it's usually closer to 150 lines of rules with 250 lines of README stapled on.

That reframes the "my CLAUDE.md is too long" problem. It's often not too many rules. It's rules buried in documentation.

Of the real rules, only 44.2% can be checked mechanically

Here's the split that surprised me more.

Checkable:

"Never commit directly to main" — you can look at what git actually ran
"No console.log in committed code" — you can look at what was written to files
"Run the test suite before pushing" — either the command ran or it didn't

Not checkable by any tool, ever:

"Surface bad news first"
"Explain the trade-off before choosing an approach"
"Keep changes reviewable"

There's no command to inspect for the second group. It's not a limitation of my parser — the information isn't in the transcript. A person has to read it and decide.

55.8% of the rules people actually write fall in that second group.

The distribution barely moved as the corpus grew

I started with 40 files: 65.4% non-instructions.
At 559 files: 63.4%.

Between those two points I added formats the parser had never been tuned on. It moved 2.0 percentage points.

That's the part I'd actually stand behind. It suggests "is this a directive?" is a property of language, not of file format — which matters, because the alternative approach (recognising every way people format
rules) is an infinite list.

What I don't know

I don't have a controlled experiment showing longer files perform worse. That's the obvious next question and I haven't run it.

But if a model is budgeting attention across a long file, it seems worth knowing that most of that file was never telling it to do anything.

The tool

It's called RuleReceipt. Free, runs locally, no account:

  npx rulereceipt check

I also published the part where it got things badly wrong — it reported 10 violations on a real session of mine and every single one was false. That write-up is at https://rulereceipt.dev/postmortem, including the wrong turn I took while fixing it.

Happy to answer anything about the method. The corpus was public files only, and the classifier is in the repo.

Update, 4 September.

Two bugs in my own parser moved these numbers after publishing, so I'm correcting them in place rather than leaving them wrong. The parser had no concept of fenced code blocks, so a shell comment starting with # or a YAML item starting with - was read as a rule boundary — 588 rules were being cut mid-block, and 1,718 items (7.2% of everything extracted) were lines lifted out of code samples that had never been rules at all.

Separately, command documentation was being counted as rules. Corrected: 21,986 items, 63.4% non-instructions, 44.2% of real rules checkable. The 40-file sample moved 2.0 points rather than 2.5, which if anything strengthens the stability point.

Full history of every figure and what changed it: https://rulereceipt.dev/postmortem#history

DEV Community: rulereceipt

My AI rule-checker was wrong about 1 in 6 times. Fixing that was the whole project.

I parsed 559 public CLAUDE.md files. Most of what's in them isn't rules.