The ablation framing is the useful one. Most of a CLAUDE.md used to be coaching for behaviors the model should already know, like prefer small commits or do not invent APIs, and those lines mostly stop earning their keep once the model improves. What still deserves a permanent seat is the project stuff no model can invent: protected paths, naming that only exists here, the one test pattern that must not get rewritten green. I treat a model upgrade as a cue to re-read every instruction against a real agent session, not as a cue to keep the whole file out of habit.
Backend developer turned builder. 20 years in software, now building Reporails - instruction governance for coding agents. Writing about context engineering and patterns that make AI agents reliable.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the 27% number is the part that stings — tbh matches what you see when you trace which lines actually change model behavior vs which are just there because some older version needed them.
the retrieval strength change is the gnarly one. vague rules used to sit quietly because the model could not reach them for most tasks. now they fire, the model has nothing concrete to bind to, fills with its own judgment. that's not CLAUDE.md working too well, it's working too wide.
did the ablation surface any instruction categories that consistently survived the cut vs got yeeted first?
Backend developer turned builder. 20 years in software, now building Reporails - instruction governance for coding agents. Writing about context engineering and patterns that make AI agents reliable.
Not on my corpus, however the validation mechanisms had to be amended. And since I'm building my instructions based on Reporails (github.com/reporails/cli) rules, the instruction surface is tidy and continuously governed. I also have an addition github action that runs alongside with builds that would catch any instruction defects.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
validation as a CI step is the right call. the edge i'd watch: if the GH action evaluates on the same model family you wrote the rules for, you might be checking the bias rather than through it — the model fills vague rules 'correctly' because it still carries your authoring intent.
running against a different model family, or a minimal context window, is what actually surfaces instruction drift.
does the action test semantic behavior or schema compliance?
Backend developer turned builder. 20 years in software, now building Reporails - instruction governance for coding agents. Writing about context engineering and patterns that make AI agents reliable.
That's why github.com/reporails/cli and it's rules are model agnostic. What happens is that there are 100+ generic rules that are true no matter the model and some additional ones that are specific for certain models.
Note: The cli was created having adaptability in mind, currently working out the fine details around users being able to define their own rules there too.
Spot on! Fully agree that "delete-and-observe" is way too blunt for production—watching your own work break just to see what was load-bearing is a costly way to learn.
Your distinction between steering and enforcement is key: a smarter model doesn't mean you drop your hooks or deterministic gates. Also, that 3-line constraint structure (Imperative → Context → Category Restriction) is gold for prompt hygiene.
Backend developer turned builder. 20 years in software, now building Reporails - instruction governance for coding agents. Writing about context engineering and patterns that make AI agents reliable.
It feels like AI needs some housekeeping now too. As models evolve, cleaning up old instructions seems to become part of the maintenance. I guess useful tools always need a bit of upkeep.
Backend developer turned builder. 20 years in software, now building Reporails - instruction governance for coding agents. Writing about context engineering and patterns that make AI agents reliable.
The ablation framing is the useful one. Most of a CLAUDE.md used to be coaching for behaviors the model should already know, like prefer small commits or do not invent APIs, and those lines mostly stop earning their keep once the model improves. What still deserves a permanent seat is the project stuff no model can invent: protected paths, naming that only exists here, the one test pattern that must not get rewritten green. I treat a model upgrade as a cue to re-read every instruction against a real agent session, not as a cue to keep the whole file out of habit.
Good approach and good habit!
the 27% number is the part that stings — tbh matches what you see when you trace which lines actually change model behavior vs which are just there because some older version needed them.
the retrieval strength change is the gnarly one. vague rules used to sit quietly because the model could not reach them for most tasks. now they fire, the model has nothing concrete to bind to, fills with its own judgment. that's not CLAUDE.md working too well, it's working too wide.
did the ablation surface any instruction categories that consistently survived the cut vs got yeeted first?
Not on my corpus, however the validation mechanisms had to be amended. And since I'm building my instructions based on Reporails (github.com/reporails/cli) rules, the instruction surface is tidy and continuously governed. I also have an addition github action that runs alongside with builds that would catch any instruction defects.
validation as a CI step is the right call. the edge i'd watch: if the GH action evaluates on the same model family you wrote the rules for, you might be checking the bias rather than through it — the model fills vague rules 'correctly' because it still carries your authoring intent.
running against a different model family, or a minimal context window, is what actually surfaces instruction drift.
does the action test semantic behavior or schema compliance?
That's why github.com/reporails/cli and it's rules are model agnostic. What happens is that there are 100+ generic rules that are true no matter the model and some additional ones that are specific for certain models.
Rules: reporails.com/rules
Note: The cli was created having adaptability in mind, currently working out the fine details around users being able to define their own rules there too.
Great point! Here is a short and punchy version:
Spot on! Fully agree that "delete-and-observe" is way too blunt for production—watching your own work break just to see what was load-bearing is a costly way to learn.
Your distinction between steering and enforcement is key: a smarter model doesn't mean you drop your hooks or deterministic gates. Also, that 3-line constraint structure (Imperative → Context → Category Restriction) is gold for prompt hygiene.
Awesome write-up! 🚀🔥
thank you ^^
It feels like AI needs some housekeeping now too. As models evolve, cleaning up old instructions seems to become part of the maintenance. I guess useful tools always need a bit of upkeep.
that kind of maintenance is one of the core idea of Reporails (github.com/reporails/cli).