DEV Community: Mayank Kaul

I deleted a 1,350-line CLAUDE.md. Two rules broke. post

Mayank Kaul — Sun, 16 Aug 2026 19:51:01 +0000

Three things get said about project rules files for AI coding agents, and they get said everywhere. They are advisory rather than enforcement. Long ones get ignored. Rules buried deep in the file get skipped, so keep it short and move anything that matters into a hook.

I have never seen any of it measured, so I measured it. Twenty checkable rules in a 1,350-line CLAUDE.md, one ticket-sized task, thirty runs across six configurations: the full file preloaded, the file removed entirely, the file on disk with auto-loading suppressed, nineteen lines plus a Skill, and two more on a stripped-down brief. A deterministic checker reads the resulting file tree.

With the file present, 399 of 400 rule checks passed. Whatever is going wrong with these files, it is not that the model ignores them. The rest of this is about what is going wrong instead.

What removing it cost

Two rules out of twenty.

Seventeen of the twenty rules scored five out of five in every configuration, including both arms with no file at all. Only three rules ever failed anywhere across six hundred checks, and two of them account for the entire gap.

Why you should believe the arms without the file

The first version of this experiment found no effect at all. Every configuration scored full marks including the ones with no rules file, and I had two articles drafted off it before I found out it was wrong.

Deleting the file from the working tree leaves it in git, and git status announces it by printing D CLAUDE.md in the first command an agent runs to orient itself. Ten of ten runs recovered it, and six said outright that they had read the contract out of HEAD. Two other channels were leaking too: a memory plugin I had installed months earlier and forgotten about, and Claude Code's own per-project memory directory, which I did not know was there. The base repository was rebuilt so the file was never committed at any depth, verified by scanning every blob in the object store.

Then there is the evidence I did not build. The test harness carries a comment pointing at CLAUDE.md. In the arms without the file the model reads that comment, goes looking, and reports that the file does not exist. Four of the ten said so, one noting the harness cites a CLAUDE.md rule pinning something but no such file exists in the repo or its history, so it followed the conventions the committed modules demonstrate instead. That is confirmation from the subject rather than the auditor.

It is also a confound worth naming. Those arms are not a model that never heard of a contract. They are a model told a contract exists and unable to reach it.

Which raises the obvious objection to the experiment. The brief told the model to follow the way the existing utilities are wired in, so of course it copied them. Two of the six configurations ran with that clause removed. The score moved from 94 to 93. One check in a hundred.

Checking turns out to be hard enough that the first careful attempt still came out backwards.

Rules 12 and 13, and why those two

The changelog bullet and the version bump.

Rule 12 says put a bullet under ## Unreleased. The repository says the opposite twice over. That heading has held the placeholder - Nothing yet. for the project's entire history while all eight released sections hold the actual bullets, and the changelog's own header instructs you to cut a new version section and move the patch digit on every merged change.

Every run without the file read that header and followed it. One wrote this about its own work:

CHANGELOG.md + version → 0.4.3, per the changelog's stated rule that the patch digit moves on every merged change.

That is not a model ignoring instructions. It is a model following the only instructions it could see, which were in the repository rather than in the file.

Rule 13 broke the same way, more subtly. The file says bump the patch digit. The repository's history says 0.4.0 was a minor bump because it added new API surface. Three runs added a new public module and picked 0.5.0, one citing that precedent outright. A fourth left the version alone.

Rules 12 and 13 are the only two rules in the set where the file contradicts what the repository demonstrates.

Depth in the file predicted nothing

Rules eleven to twenty all sit below line 314, under a heading where the file says of them, in its own words, that in practice they are the ones most often skipped.

Seven of those ten never failed anywhere. Rules 12 and 13 failed only where the file was absent, which cannot be a burial effect, because burial only exists when the file exists.

That leaves one buried-rule failure in a run that had the file: a single missing manifest line. One failure in the fifteen runs that carried the full file. The other fourteen had that rule at exactly the same depth.

Depth explains nothing here. Disagreement with the surrounding code explains all of it.

The rule I would have bet on

Rule 19 forbids dependency changes. The task makes one tempting: npm run typecheck fails on a pre-existing error that only @types/node fixes. Of the twenty runs that had rule 19 loaded, seventeen cited it by number and declined.

No run in any arm shipped a dependency change, including the ten that never saw rule 19. Several installed the package with --no-save to run the suites and reverted it, in arms with the rule and arms without. The repository has no dependencies and never has, and every run read that the same way.

The rule did not change the outcome. It supplied language for a decision the repository had already forced.

What I would do with this

Go through your file rule by rule and ask one question: would a careful reader of this repo already do this? If yes, the rule is probably inert. Delete it, run a real task twice, and see whether anything changes.

The rules worth keeping are the ones your codebase contradicts, which is uncomfortable, because those are also the ones most likely to be wrong. Rule 12 is arguably the worse convention, and the file never reconciled itself with the eight examples sitting next to it.

One number worth having: the same twenty rules delivered as nineteen lines plus a Skill scored a hundred out of a hundred at $0.696 a run. The full file scored ninety-nine at $1.094, and was dearer on all twenty-five pairwise comparisons with no overlap.

Where I could be wrong

This repository was the best possible case for inference: four worked examples of every convention, matching tests, real history. Eighteen of the twenty rules redundant is a floor, not a ratio. On a greenfield repo the file carries most of the load, and I have not tested that. One task, one repo, one model, five runs an arm, and no statistical testing on the compliance results. The rule 12 split is large enough to be obvious; the rule 13 split is not.

Calling those two a violation also overstates it. The models followed a defensible convention documented in the repository and said which one. A rule set that agreed with the repo would probably have produced the null result honestly.

If you have one of these files, how many lines is it? And has anyone on your team ever checked whether a rule in it contradicts something the codebase already demonstrates? I found mine by accident and I would like a better method than that.

I write about architecture and AI engineering for companies that aren’t Google. The newletter lives at mayankkaul.com

The model was not the variable. I think the records were.

Mayank Kaul — Thu, 06 Aug 2026 20:56:48 +0000

Point a model at your repositories, ask why something broke, and you get an answer. Coherent, names a mechanism, shows you how it got there. The working is the problem rather than the reassurance. It looks the same whether it ends at a value in a record or at what usually happens, and only one of those is evidence.

I ran the same question at four levels of evidence, on two systems, across three models. The question was never find me a bug. It was always this already happened, work out how. The pass that worries me is the one just short of enough evidence. There is enough there to build a real hypothesis, and not enough to tell a real one from a plausible one. Two of its guesses sent me to look at things I had not checked. One was a dead end. The other was a real problem I had not known about, and nothing in either answer told me which was which.

A good guess costs the same to chase as a real one. You find out which it was at the end.

The four rungs, and what each one removes

I stopped thinking about this as adding context. Each rung takes away something the model would otherwise have to guess at, and that is the more useful way to look at it.

One. Repository access, broad. Point it at everything and ask. It removes nothing. The answer was structurally sensible, named components that were not involved, and arrived with no more hesitation than the correct one did three rungs later. If your team tried this once, got something confident and wrong, and decided the tooling is not there yet, this is probably where you stopped.

Two. Three bounded repositories, plus a written map. Which service talks to which, over what protocol, with what delivery and ordering guarantees. This removes rediscovery. What made it work was not the map. It was telling the model to treat the map as true and not go and check. That bought focus and gave up verification. Maps go stale quietly, and I have removed the step where it might have noticed.

Three. Plus traces and logs covering real executions. This removes guesswork about sequence and about what crossed a boundary. It is the closest it got without evidence of state, and it is the rung the top of this piece is about.

Four. Plus a small set of records showing the behaviour, customer fields removed and identifiers replaced, structure intact. This is the one that tells you which path the code actually took, and that turns out to be the whole game.

That last pass ran on Sonnet, which had been the weakest of the three lower down: vaguer, and closer to optimistic than the evidence supported. Opus and Fable pointed nearer the right area on thinner evidence. None of them got there. Then the one that had been worst at guessing was enough, once there was nothing left to guess.

I am not arguing that model choice is irrelevant. It changed how good the guessing was. It did not change whether there was guessing.

Why the records worked

The first defect I had already diagnosed, so I may have picked, at every rung, exactly the evidence that leads there. The second was a live bug on a project of my own, unsolved when I started. Old code, the kind that acquires a condition per edge case over years, so one entry point can take a large number of paths through it.

At rung three it reasoned forward. Here is what the code can do, here is what probably happened. At rung four it reasoned backward. Given these values in this record, this branch ran and these did not. Then it wrote a test for that path, ran it, confirmed the behaviour, and handed back a reproduction.

The reason is mundane. A branch you did not log is a branch you cannot rule out. Logging every branch in a service that has been accumulating conditions for years is not a decision anyone makes, because it costs storage, latency and review attention on every change, and the value is hypothetical until the day it is not. A record from an affected run settles it afterwards, because the values in it determine which conditions evaluated true. Not always. Two branches can converge on the same stored state, a later write can erase the evidence, and the deciding value may never have been persisted at all. Where none of those apply, the record tells you which branch ran.

Which is the job a log line would have done, if I had written one.

That is also the real difference between my two runs, and it is not technical. On the project I own, getting records was a query. Anywhere else it is a legal question somebody answered years ago without knowing they were answering it. That is a separate piece and I owe you one.

Where I could be wrong

Two runs, three models from one vendor. Not a benchmark.

Someone will say rung four hands it the diagnosis. I do not think so. Those are the same records a person opens, and opening them is not the same as knowing what they mean. The argument stands or falls on that: whether the records give you the answer, or give you what you need to find it.

What would falsify it: a defect where the branch taken is knowable from code and traces alone, with no conditional ambiguity. Rung four should add nothing.

What I keep ready now

A brief per service and a communication map, written for a model, regenerated on merge rather than maintained by hand. Delivery and ordering guarantees included, since no codebase states them.
A last-verified date on that map, because I am telling a model to trust it.
One field-aware extraction script that strips sensitive values and leaves field names, timestamps and joins intact. Written while nothing is on fire.
The habit of asking which branch it thinks ran, and what in the data says so. That question is the difference between rung three and rung four, and you can ask it from any rung.

Have you chased down a plausible side finding from a model and had it turn out to be real? Or turn out to be nothing, which costs exactly the same.

And if you are pointing a model at documentation you have told it to trust, how do you know the documentation is still true?

I write about architecture and AI engineering for companies that aren’t Google. The newsletter lives at mayankkaul.com

Onboarding was doing two jobs. I think AI removed only one.

Mayank Kaul — Mon, 03 Aug 2026 20:12:13 +0000

The expensive part of working in another team’s service was never writing the code. Writing fifty lines has never taken a week. What took the week was building enough of a model of the service that the fifty lines were safe.

That was the toll on the boundary. You paid it or you did not cross. I think we stopped collecting it, and nothing else changed.

You used to clone the repo, follow one request end to end, and take a small ticket, ideally a bug, because a bug makes you work backwards through code you did not write. Ask why a field is nullable and get a story about a migration nobody finished.

Now you point a model at the repository and get module structure, request lifecycle, dependencies and a defensible architecture summary. An afternoon.

I am not arguing for the old way. The toll cost real money in gatekeeping, knowledge hoarding and three-week waits. It also forced questions, surfaced unknowns and kept blast radius small. My worry is that we removed the delay and the questions together, and only meant to remove one.

What changed and what did not

Changed: the cost of comprehension, the speed of contributing, and how quickly confidence arrives, whether or not it is justified.

Did not change: review bandwidth, on-call, deployment rights, runbooks, and the person who remembers why that field is nullable.

Conway’s Law gets quoted as systems ending up shaped like the org chart. The mechanism underneath is that crossing a team boundary cost a week of comprehension, so people mostly did not. The inverse Conway manoeuvre, reshaping teams to get the architecture you want, leaned on that the entire time. Code can now appear anywhere. It can still only be operated in one place.

What the brief cannot tell you

A reservation service publishes a state change event. Three services consume it. Two deserialise permissively and persist whatever arrives. The third validates strictly and dead-letters anything it does not recognise.

That fact is not in the reservation service’s repository. It lives in three other codebases and in the memory of someone who moved teams. No brief drawn from that repo produces it, because the information is not there to find. A schema registry, registered consumers or contract tests would put it within reach. That is the actual fix, and it is a great deal more work than the change that exposed the need for it.

The brief is not wrong. The problem is that it feels complete. The slow route left you visibly ignorant, which is what made people ask.

What it costs

Someone on the fulfilment team needs a field on that event. The reservation team’s sprint is full, so they read the brief, write the change, raise the merge request. Twenty lines, tests pass, reviewer has four others open. It goes through.

That reviewer is not a bottleneck anyone planned for. I have watched review take forty to fifty percent of the week for particular tech leads and senior developers, because team velocity climbed and the queue never emptied. Nobody wrote that into a role description and nobody staffed for it.

Two consumers store the new field without complaint. The third dead-letters every message. Nightly job, so nothing happens for a day.

At 03:12 the on-call for the replenishment job is paged. Nothing they own has changed. They trace it to the reservation service, which is not theirs either, and from there to a merge request raised by a third team. By the time the author is awake and on the call, three teams are involved and none of them holds the whole picture. The person paged owns neither the service nor the change. The person who owns the service did not write the change. The person who wrote it has never seen this service fail.

The author opens the repository and it looks exactly like the brief said. That is the problem. The brief was right. It described a service that publishes an event and said nothing about who was listening.

They ask the model, get a reasonable answer, make the third consumer tolerant, and the page clears. The visible failure is gone. What nobody has looked at is the other two consumers, which have spent a day persisting a field no one validated. Someone finds that in a fortnight and files it as an unrelated bug.

Nobody there did anything indefensible. The estate had one control doing several jobs and it was withdrawn without replacement.

Where I could be wrong

I have no data showing cross-boundary contribution has actually risen, so this is a hypothesis with a test attached. Measure the share of merged changes whose author sits outside the owning team, defined by CODEOWNERS or the on-call rotation rather than the org chart.

Authorship alone will miss boundaries still enforced by permissions or policy, so pair it with review rounds, lead time and post-merge incidents for owner versus non-owner changes. If none of that moves, I am wrong about the mechanism and I would like to know.

DORA’s 2025 report is the nearest supporting evidence I trust, though it does not test this mechanism directly. It found higher AI adoption associated with increases in both delivery throughput and delivery instability, and time saved during creation being re-allocated to auditing and verification.

Four questions before merging across a boundary

Does someone accountable for each affected consumer know this is coming, and agree with what it does?
Does a test run it through a real consumer rather than a mock?
If this alters permissions, credentials, dependency scope or data access paths, has that been reviewed on its own terms?
If it needs reverting at 3am with the author unreachable, does the runbook cover it, and does the revert deal with what got written downstream?

Any no means it is not ready to merge on its own. Narrow it, pair with an owner, or go and find the missing evidence. Handing the whole thing over should be the last option rather than the first, or you have rebuilt the gate you were trying to route around.

We treated the cost of comprehension as waste. Some of it was. Some of it was signalling, coordination and control, and those need explicit replacements now that the cost has gone.

Which control did your organisation remove, and did you find out before or after the phone went off?

And if your team has written down what a service brief must contain before an outsider can safely change that service, I would like to read it.

Source: DORA, 2025 State of AI-assisted Software Development.

I write about architecture and AI engineering for companies that aren’t Google. The newletter lives at mayankkaul.com