I looked at 558 AGENTS.md files: here's a 5-minute check for yours

Short version: I labeled 558 public AGENTS.md files against a 9-category taxonomy. The measured
base rates say something boring and useful — almost every file prohibits things (85.7%) and lists
build/test commands (82.8%), while almost none of them record a gotcha (13.6%). Two of the nine
slots are nearly empty across the whole corpus: gotchas and agent_meta (rules about the agent itself,
25.8%).

Then I ran the same ruler over two big, well-maintained files. Both missed gotchas. So here is a
five-minute check you can run on your own file, and the exact numbers behind it.

The base rates

Measured on 516 substantive files (558 collected, the rest were one-line pointers):

boundaries     85.7%   what must never be done
build_test     82.8%   the commands CI runs
workflow       67.1%   commit format, branches, release steps
structure      59.1%   layout, where new code belongs
style          54.5%   naming, formatting — or a pointer to the config that enforces it
environment    45.0%   toolchain versions, required env vars
overview       32.2%   one paragraph: what this is, what it deliberately is not
agent_meta     25.8%   rules about the agent: tone, when to ask first
gotchas        13.6%   pitfalls that are NOT derivable from the code

The shape is not surprising once you see it as a genre: an AGENTS.md is usually written defensively,
as a list of things not to break. The file that would actually save you time is the one almost nobody
writes.

Two receipts

compare prints your file's coverage next to the corpus baseline. Two real examples from the corpus:

file	size	sections	coverage	missing
langchain-ai/deepagents	10 KB	22	7/9	overview, gotchas
openai/openai-agents-python	34 KB	27	6/9	style, agent_meta, gotchas

Both are good files. The 34 KB one is one of the more thorough agent-instruction files in the corpus —
27 sections, 19 separate boundary markers. It still has nothing in it that you could only learn by
running the thing.

Why gotchas are rare (and why that is not laziness)

You can only write a gotcha after being bitten by it — and by the time you have been bitten, the
temptation is to fix the thing rather than write the sentence down. The fix is visible in the code;
the sentence is a liability nobody wants to maintain.

There is a second, worse failure mode. I sampled 347 entries from the Gotchas / Common Pitfalls /
Troubleshooting sections in the corpus (an earlier snapshot, 507 files) and hand-labeled 120 of them:

• 58% are readable from the repo itself (interface contracts, platform limits, build requirements)
• 34% are not pitfalls at all — they are generic advice ("remember to install dependencies", "don't commit .env"), the same sentence you would write for any project
• 8% are genuinely experience-only: upstream/third-party behaviour, past incidents, and the places where the docs disagree with the code

So the section is rare, and a third of what does live there is filler. The 8% is the part worth
handing to an agent, and it cannot be generated from a reading of the repository. It has to come from
a person who was there.

The five-minute check

No tool needed. Ask these five questions about your own file:

1. Are the commands copy-pasteable? Not "run the tests" — the actual command CI runs, with the working directory. If your README says one port and production uses another, say so (that mistake is in the corpus, in a file that otherwise looks complete).
2. Does it name what must never be committed or never touched? This is the one thing the corpus does well (85.7%) — check that yours names the tempting case, not the obvious one. "Don't commit secrets" is obvious; "don't hand-edit the production database to fix a row, use the backfill script" is a boundary that will actually stop someone.
3. Does it say anything about the agent's own behaviour? Only 25.8% do. Tone, when to stop and ask, which actions need explicit approval, what must not leave the machine.
4. Is there at least one sentence that is not derivable from the code? If every line in your file could have been written by reading the repo, the file is documentation, not a charter. This is the gotcha test.
5. Do the paths it points at exist? Measured: 49% of files route to another file, and 15% point at a knowledge store or rules directory. A pointer to a file that moved is worse than no pointer — an agent will go looking, and will read whatever it finds there as authoritative.

If you want the baseline instead of the feeling

git clone https://github.com/janzong/agent-charters   # CN mirror: gitee.com/janzong/agent-charters
cd agent-charters
python -m venv .venv && .venv/bin/pip install .
.venv/bin/agent-charters compare path/to/AGENTS.md   # coverage vs the 558-file baseline, plus gaps
.venv/bin/agent-charters brief                       # the checklist + a paste-ready prompt
.venv/bin/agent-charters refs path/to/AGENTS.md      # external pointers and dangling references

(Not on PyPI — the install is a clone. I verified the sequence above in a clean virtualenv on a
machine that had never seen the repo.)

compare is the one that answers question 4 in aggregate. It also does something I did not expect:
when I used brief's prompt to write a charter for a real project, compare flagged coverage I had
skipped — and one of the nine slots it missed was the name of the slot itself, which is a bug in my
taxonomy, not in the file. That is the kind of thing a rule-based labeler gives you: you can point at
the pattern that fired and argue with it.

There is no LLM in the labeling loop. Every label is recomputable and arguable, which is the point —
if you disagree with a label, you can find the rule that produced it and overrule it.

What this is not

I do not want to oversell the numbers, so:

• The classifier scores 92% precision / 70% recall on a 55-file held-out English set, and 88% / 73% on 50 held-out Chinese files. The recall number is the honest one: it misses roughly three in ten of the labels it should have produced. gotchas and agent_meta are the weakest slots in both languages.
• The held-out sets were labeled by one person (me). No second annotator, no inter-annotator agreement.
• Coverage is a process metric, not a quality metric. In a 3-repo test, a checklist that names all nine slots pushed a generator from 4–5 categories to 9/9 — and filling all nine slots is not the same as writing a good file. It is a prompt for the questions, not a grade.
• The labels and the rates come from public files and a rule-based classifier, not from a language model. The one LLM in this story is the generator in the 3-repo test, which is why that number is reported as n=3.

The ask

The weakest part of this project is that the only person who has ever tested it is its author. If you
have an AGENTS.md (or a CLAUDE.md, or a .cursorrules) on a real project, run compare on it and
tell me what it gets wrong — the file, the label, or the baseline rate. A wrong label on your file is
worth more to me than a star.

Repo: https://github.com/janzong/agent-charters

Comments

[Alex Shev]

The useful test is whether each instruction changes a decision at the moment it matters. A compact “known failure modes” section with trigger, consequence, and recovery path is more actionable than a long list of generic cautions—and it gives future agents a reasoned boundary rather than a vague prohibition.

[Janz]

Agreed, and that is the test my own labeling keeps pointing at. I hand-labelled 120 entries from the Gotchas / Common Pitfalls sections in the corpus: 58% were readable from the repo (contracts, platform limits, build requirements), 34% were generic cautions that would fit any project, and 8% were experience-only. "Changes a decision at the moment it matters" draws that 34% line more cleanly than anything I wrote — a caution that changes no decision is decoration, a failure mode with trigger / consequence / recovery is an instruction.
The awkward half is that I cannot measure it. A rule-based classifier can count sections; it cannot tell whether the paragraph inside one would change a decision. Which is why the number the tool prints is a floor and not a score: the corpus has 8 files that cover all nine categories, and the shortest is 7.7 KB. It names all nine slots — that is not the same as each of them changing a decision.

[Raknaos]

The 58% figure is the part that stuck with me. I keep a gotchas section in my own agent files and I just re-read it against your ruler: most of the lines are things a careful person would learn from the repo in an afternoon, and the genuinely cheap ones are the ones where the wrong choice is invisible in code — why a dependency is pinned, which directory is generated and must never be edited by hand.

Two questions since you ask for outside labels. Did the 3-repo checklist test survive being run on a file that already scored 9/9, or only on the failing ones? And do you have a read on whether the gotcha gap is a writing habit or a review habit — i.e. do teams that treat postmortems as a required artifact score higher on that slot than teams that don't?

[Janz]

Two good questions, and the honest answer to the first is no.
The three-repo test only ran on files that were already falling short: the human versions scored 6, 5 and 5 of nine categories, and the checklist-generated ones came out at 9/9. I had never pointed it at a complete file until just now. Result: it prints "All nine categories covered — nothing to add" and stops. So the checklist is a floor, not a reviewer. It can name the empty slot; once none are empty it has nothing to say. Your framing — whether an instruction changes a decision — is the reviewer I do not have yet.
One related wart, since you re-read your own gotchas against my ruler: write a file using the checklist's own slot names as headings and my classifier misses agent_meta. Five variants tested, the other eight slot names are recognized, that one never is — a file written from my prompt can be reported as 8/9 by my own tool.
On writing habit versus review habit: I cannot see postmortem culture in this data, but there is one proxy I can measure — whether the file points at an external knowledge store or rules directory. Files that do write a gotchas section 24.7% of the time (19/77) against 11.6% (51/439). That is 2.1×, Fisher p ≈ 0.004, and it holds inside every size quartile, so it is not just "bigger files cover more" (the smallest quartile has 5 files with the pointer, so treat that cell as noise). It is a pointer-style proxy, not a postmortem measure, and it is cross-sectional, so I cannot tell you the direction: the pointer and the gotchas section may both come from the same habit of keeping notes outside the code. But of the two hypotheses you offered, it leans process rather than writing style.
Since you offered an outside label: if any of your agent files is public, agent-charters compare AGENTS.md on it would tell me something I cannot get from my own machine — the only person who has ever run this classifier is its author. Current held-out numbers are precision 92% / recall 70%, and both kinds of mistakes are useful to hear about.