DEV Community: Janz

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

Janz — Mon, 14 Sep 2026 13:25:06 +0000

Short version: I labeled 558 public AGENTS.md files against a 9-category taxonomy. The measured
base rates say something boring and useful — almost every file prohibits things (85.7%) and lists
build/test commands (82.8%), while almost none of them record a gotcha (13.6%). Two of the nine
slots are nearly empty across the whole corpus: gotchas and agent_meta (rules about the agent itself,
25.8%).

Then I ran the same ruler over two big, well-maintained files. Both missed gotchas. So here is a
five-minute check you can run on your own file, and the exact numbers behind it.

The base rates

Measured on 516 substantive files (558 collected, the rest were one-line pointers):

boundaries     85.7%   what must never be done
build_test     82.8%   the commands CI runs
workflow       67.1%   commit format, branches, release steps
structure      59.1%   layout, where new code belongs
style          54.5%   naming, formatting — or a pointer to the config that enforces it
environment    45.0%   toolchain versions, required env vars
overview       32.2%   one paragraph: what this is, what it deliberately is not
agent_meta     25.8%   rules about the agent: tone, when to ask first
gotchas        13.6%   pitfalls that are NOT derivable from the code

The shape is not surprising once you see it as a genre: an AGENTS.md is usually written defensively,
as a list of things not to break. The file that would actually save you time is the one almost nobody
writes.

Two receipts

compare prints your file's coverage next to the corpus baseline. Two real examples from the corpus:

file	size	sections	coverage	missing
`langchain-ai/deepagents`	10 KB	22	7/9	`overview`, `gotchas`
`openai/openai-agents-python`	34 KB	27	6/9	`style`, `agent_meta`, `gotchas`

Both are good files. The 34 KB one is one of the more thorough agent-instruction files in the corpus —
27 sections, 19 separate boundary markers. It still has nothing in it that you could only learn by
running the thing.

Why gotchas are rare (and why that is not laziness)

You can only write a gotcha after being bitten by it — and by the time you have been bitten, the
temptation is to fix the thing rather than write the sentence down. The fix is visible in the code;
the sentence is a liability nobody wants to maintain.

There is a second, worse failure mode. I sampled 347 entries from the Gotchas / Common Pitfalls /
Troubleshooting sections in the corpus (an earlier snapshot, 507 files) and hand-labeled 120 of them:

58% are readable from the repo itself (interface contracts, platform limits, build requirements)
34% are not pitfalls at all — they are generic advice ("remember to install dependencies", "don't commit .env"), the same sentence you would write for any project
8% are genuinely experience-only: upstream/third-party behaviour, past incidents, and the places where the docs disagree with the code

So the section is rare, and a third of what does live there is filler. The 8% is the part worth
handing to an agent, and it cannot be generated from a reading of the repository. It has to come from
a person who was there.

The five-minute check

No tool needed. Ask these five questions about your own file:

Are the commands copy-pasteable? Not "run the tests" — the actual command CI runs, with the working directory. If your README says one port and production uses another, say so (that mistake is in the corpus, in a file that otherwise looks complete).
Does it name what must never be committed or never touched? This is the one thing the corpus does well (85.7%) — check that yours names the tempting case, not the obvious one. "Don't commit secrets" is obvious; "don't hand-edit the production database to fix a row, use the backfill script" is a boundary that will actually stop someone.
Does it say anything about the agent's own behaviour? Only 25.8% do. Tone, when to stop and ask, which actions need explicit approval, what must not leave the machine.
Is there at least one sentence that is not derivable from the code? If every line in your file could have been written by reading the repo, the file is documentation, not a charter. This is the gotcha test.
Do the paths it points at exist? Measured: 49% of files route to another file, and 15% point at a knowledge store or rules directory. A pointer to a file that moved is worse than no pointer — an agent will go looking, and will read whatever it finds there as authoritative.

If you want the baseline instead of the feeling

git clone https://github.com/janzong/agent-charters   # CN mirror: gitee.com/janzong/agent-charters
cd agent-charters
python -m venv .venv && .venv/bin/pip install .
.venv/bin/agent-charters compare path/to/AGENTS.md   # coverage vs the 558-file baseline, plus gaps
.venv/bin/agent-charters brief                       # the checklist + a paste-ready prompt
.venv/bin/agent-charters refs path/to/AGENTS.md      # external pointers and dangling references

(Not on PyPI — the install is a clone. I verified the sequence above in a clean virtualenv on a
machine that had never seen the repo.)

compare is the one that answers question 4 in aggregate. It also does something I did not expect:
when I used brief's prompt to write a charter for a real project, compare flagged coverage I had
skipped — and one of the nine slots it missed was the name of the slot itself, which is a bug in my
taxonomy, not in the file. That is the kind of thing a rule-based labeler gives you: you can point at
the pattern that fired and argue with it.

There is no LLM in the labeling loop. Every label is recomputable and arguable, which is the point —
if you disagree with a label, you can find the rule that produced it and overrule it.

What this is not

I do not want to oversell the numbers, so:

The classifier scores 92% precision / 70% recall on a 55-file held-out English set, and 88% / 73% on 50 held-out Chinese files. The recall number is the honest one: it misses roughly three in ten of the labels it should have produced. gotchas and agent_meta are the weakest slots in both languages.
The held-out sets were labeled by one person (me). No second annotator, no inter-annotator agreement.
Coverage is a process metric, not a quality metric. In a 3-repo test, a checklist that names all nine slots pushed a generator from 4–5 categories to 9/9 — and filling all nine slots is not the same as writing a good file. It is a prompt for the questions, not a grade.
The labels and the rates come from public files and a rule-based classifier, not from a language model. The one LLM in this story is the generator in the 3-repo test, which is why that number is reported as n=3.

The ask

The weakest part of this project is that the only person who has ever tested it is its author. If you
have an AGENTS.md (or a CLAUDE.md, or a .cursorrules) on a real project, run compare on it and
tell me what it gets wrong — the file, the label, or the baseline rate. A wrong label on your file is
worth more to me than a star.

Repo: https://github.com/janzong/agent-charters

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Janz — Mon, 14 Sep 2026 12:23:13 +0000

TL;DR — I collected 558 AGENTS.md files from public repos and labeled each one against a 9-category
taxonomy with a rule-based classifier (no LLM in the loop, so it is auditable and recomputable). Then I
blind-labeled held-out samples and compared: 92% precision / 70% recall on 55 English files,
88% / 73% on 50 Chinese files. The most common categories are prohibitions (85.7%) and build/test
commands (82.8%). The rarest: gotchas (13.6%) and instructions about how the agent itself should behave
(25.8%).

Why bother

Almost every discussion about AGENTS.md is anecdote-led: my repo's file works, my agent ignores it,
a good one is a model upgrade, a bad one is worse than nothing. All of that may be true — but nobody
seems to have the distribution. So I built it: snapshot of 558 files from 558 public repos
(2026-09-10, 5.3 MB, 516 usable for statistics), labeled, versioned, and published with the tooling.

Method, in one paragraph

Nine categories: boundaries, build_test, workflow, structure, style, environment, overview,
agent_meta (rules about the AI itself), gotchas. Labeling is done by pattern rules over headings and
body text — deliberately, because a rule set can be read, argued with, and re-run, and every number below
can be recomputed from the released dataset. I then measured how well the rules match a human reading:
100 files in-sample (upper bound, 90%/75%) and two held-out sets I had never tuned against —
55 English (92%/70%) and 50 Chinese (88%/73%). Held-out numbers use the conservative reading
(items I was unsure about count as classifier errors).

Five things the numbers say

1. Two categories dominate — and they are tied

category	share of 516 files
`boundaries` (what you must never do)	85.7%
`build_test` (install/build/test/CI commands)	82.8%
`workflow` (branching, commits, review, release)	67.1%
`structure`	59.1%
`style`	54.5%
`environment`	45.0%
`overview`	32.2%
`agent_meta`	25.8%
`gotchas`	13.6%

The 2.9 pp gap between the top two is smaller than the known false-positive rate (~3%) of the
prohibition pattern — so the honest statement is tied for first, not "prohibitions beat build commands".

2. Nobody writes down their scars

gotchas is dead last at 13.6%. Worse: when people do open a "known issues" section, a third of it
isn't a gotcha. I hand-read 120 items from those sections:

58% were readable straight from the repo (config, code, README mismatch),
34% were not gotchas at all (generic advice: "remember to install dependencies"),
8% needed experience or the outside world (OS behavior, an upstream outage, yesterday's incident).

That 8% is the part an agent can never derive from the code — and it is exactly the part that is
almost never written down.

3. The slot language models skip is `workflow`

In a controlled experiment (11 repos × 3 prompt styles), prompts that listed topics explicitly produced
9/9 categories, while prompts that left the slots implicit skipped workflow in 11 out of 11 files.
Point at workflow by name and it appears 3/3 times, with real content. The gap is not knowledge,
it is questions — which is why I turned the corpus distribution into a checklist tool.

4. Your weakest category depends on the language

English files fail differently from Chinese ones. English: gotchas recall 32–38% — the classifier
misses casual "watch out" prose. Chinese: agent_meta recall 26% — Chinese files express agent rules
in the second person ("you are the dispatcher, not the executor"), and the body-pattern rules for that
category are entirely English, so the whole style is invisible to them. File-level exact agreement
(9/9 categories identical) is 12% in both languages.

5. Half of these files are entry points, not documentation

49% point to some other file; 15% route to a knowledge or rules directory. That's a structural
fact about the format, and it means "does this repo have an AGENTS.md?" is a much weaker question than
"what is actually in it".

The tool

pip install agent-charters

agent-charters brief      # checklist of the 9 slots + a paste-ready prompt
agent-charters compare your-AGENTS.md   # your coverage vs the 558-file baseline
agent-charters refs your-AGENTS.md      # does your file point at paths that exist

Honest note: compare is a checklist, not an oracle. It warned me that one of the nine categories was
missing from a file I wrote myself — it was actually present, but the heading used the tool's own slot name
instead of natural language. That is documented in the repo (along with the exact experiment) rather than
quietly patched, because a tool that tells you "you're missing X" should be checked by a human.

What this is not

Rule-based labels, not per-file human labels. Precision/recall above are the honest measures; the per-category numbers in the dataset carry that error.
Not a representative sample of GitHub. Repos were found through AI/agent topics and Chinese keyword search; the Chinese set came out 97% Chinese by construction, which says nothing about GitHub's language mix.
Single annotator. The blind labeling was done by one model-driven annotator, not multiple raters.
A snapshot, not a trend. A baseline of file hashes is stored so that a future re-crawl can measure how these files change — that measurement doesn't exist yet.

The ask

I'm looking for 2–3 people who are not me to run compare on an AGENTS.md they actually maintain and
tell me where it's wrong — missing a category you clearly have, or claiming one you don't. That is the one
piece of evidence this project doesn't have yet: an external user. Issues and comments are both fine.

DEV Community: Janz

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

The base rates

Two receipts

Why gotchas are rare (and why that is not laziness)

The five-minute check

If you want the baseline instead of the feeling

What this is not

The ask

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Why bother

Method, in one paragraph

Five things the numbers say

1. Two categories dominate — and they are tied

2. Nobody writes down their scars

3. The slot language models skip is workflow

4. Your weakest category depends on the language

5. Half of these files are entry points, not documentation

The tool

What this is not

Links

The ask

3. The slot language models skip is `workflow`