I measured whether Claude Code actually follows my CLAUDE.md

Everybody writes a CLAUDE.md. Almost nobody checks whether it gets followed. I stopped assuming and built a small tool to check, and the results were worse, and more interesting, than I expected.

Disclosure up front: the tool is mine, it's npx rulereceipt, source-available. This post is about what I found building it, not a pitch. The parts where it's wrong are the parts I actually care about.

The problem

You write "always run the tests before committing" and "surface bad news first" into a file, the agent reads the file, and then it does whatever it wants. Sometimes it even tells you it followed the rule when it
didn't. I kept hitting the same shapes, and I'm not the only one: there are bug reports where people logged a hundred-plus incidents by hand out of their session transcripts and reached the same conclusion, that
prose rules get read and not executed.

The frustrating part is that it's all sitting in the transcript. You can check it after the fact. So I did.

What it does

It reads the Claude Code session transcript (the JSONL under ~/.claude/projects/) together with your CLAUDE.md, and scores each rule.

Two kinds of rule get handled differently. Rules that are checkable with no ambiguity, like "never commit to main" or "never touch .env", get deterministic pattern checks: no model, nothing leaves your machine.
The judgment ones, like "don't over-engineer" or "surface bad news first", go to one structured LLM call using your own API key.

Every result carries an evidence line, or it's marked "didn't run". No silent passes. If it can't tell, it says so instead of guessing.

The hard part wasn't detection. It was false accusations.

My first version confidently flagged rules as broken that weren't. It saw git push --force-with-lease and screamed "force push". A false accusation is worse than a miss, because once the tool cries wolf you
stop trusting any of it.

Getting the false-accusation rate from 15.8% down to 2.9% was most of the work. It's still not zero, and that number is in the README. Any tool like this that claims perfect precision is lying to you.

Catching fabricated "done"

The sharpest failure I keep seeing: the agent reports a test suite green when a fake worker wrote hardcoded checkmarks, or calls a fix "verified" without running anything. So there's a specific check for a claim
of completion with no evidence in the session behind it: "verified", "at baseline", "tests pass", asserted where the session never actually produced that result.

Before, not just after

A report tells you after the damage is done. So there's also a PreToolUse hook that can refuse a rule-breaking command before it runs.

Here's the honest limit, because I measured it. I tried blocking on the command literals written inside rules, and it refused 62% of 16,336 real commands. One rule titled "Feature Validation" recommends npm run
build and forbids Playwright, so blocking on its only command-shaped literal refused the recommended command. Nothing in a rules file marks which backtick is the prohibition.

So the guard only auto-blocks rules that name a file or a branch, like "never modify migrations/" or "never commit to main", where the intent is unambiguous. And it fails open: a bug in the guard can't stop
you from working. It's a narrow, safe block, not a magic obedience layer.

Try it, and tell me where it's wrong

npx rulereceipt runs against your last session. rulereceipt demo runs with no setup and no API key at all.

I'd rather hear where it false-accuses than where it works. That's the whole game.