DEV Community: Janz

Your agent run passed. Can you prove it was allowed?

Janz — Wed, 23 Sep 2026 03:03:42 +0000

Short version: Most agent teams can show a run. Few can show, in one re-runnable command, whether that run was allowed under a policy. I added a policy-as-code audit to agent-lab-trust: it checks cost and call caps, the declared artifact contract, and forbidden markers, then emits a canonical audit_hash. On 13 archived runs, the default contract passed 0/13 and a declared GenMentor contract passed 13/13. The data did not change. The contract did.

The gap

Dashboards answer "what happened". Governance needs "what was allowed, and by which rule". A run can succeed and still violate a budget, write an unexpected artifact, or carry a marker that should never ship.

The audit is deliberately boring:

agent-lab-trust audit <run-root> --policy policy.json

A policy can declare:

max_cost_usd and max_calls per run;
required_artifacts plus required_artifacts_mode: all|any;
forbidden_markers.

The output is a finding list and a canonical audit_hash. deletion-proof --output <path> writes the deletion evidence used in the same policy flow.

The demonstration

I ran the audit over 13 archived GenMentor runs.

Trust-layer default contract (output/structured.json or results/structured.json): 0/13 passed, missing_artifact ×13, cost_exceeded ×1.
Declared GenMentor contract (archive.json and summary.json, mode all, cap 5.00): 13/13 passed, no findings.

Audit hashes:

default structured contract: 2c122c4ff2c0d5eed11a0fc23b4ff717002864dbd941402e8fe18d6d17c96ce6
GenMentor contract: 368b75a06c0f5c205436d287881cfddcbabd94b7ad34a860e852fba59dad7ab5

The one cost finding is real: replay-8of8-20260920-a recorded 3.9529266 under the trust-layer cap 0.60. Under the GenMentor policy cap 5.00, it passes. Caps are part of the contract too.

The lesson is not "the audit is noisy". It is that an audit without an explicit run-family contract silently embeds one family's format. The policy has to name the contract, the caps, and the markers.

Reproduce

docker run --rm ghcr.io/janzong/agent-lab-trust:rc2

or bash scripts/reproduce.sh. Expected: 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, deletion audit_hash 4f0193abbd49a0f9.

The :rc2 tag points at the latest published build; SERIES.md pins the verification digest sha256:2e178e63fff30e70ac68501cb17e5316097875932d1d7085e669ce4f8d5a105f.

What this does not prove

no real governance deployment;
no external reproduction yet;
no real decision changed yet;
the 13 runs are a private synthetic family, not published data.

The series

This is piece 3 of a three-part line: write it down → test it → govern it.

agent-charters: what people actually tell agents.
agent-lab-trust: whether an agent run is valid and reproducible.
this audit: whether a run was allowed under a declared policy.

Index: https://github.com/janzong/agent-lab-trust/blob/main/SERIES.md

Call

Run the audit on a run family you own. Tell me where the policy and the artifacts disagree. I am also looking for 3 independent reproductions of the trust layer: https://github.com/janzong/agent-lab-trust/issues/1

We ran 2 vs 4 agents six times. Four agents cost 2.1 and did not improve success

Janz — Tue, 22 Sep 2026 21:51:04 +0000

Short version: I ran a preregistered 2-agent versus 4-agent comparison on a deterministic task.
Six live runs completed under hard caps. Both groups succeeded in exactly one of three repeats.
The 4-agent group cost 2.115× more per run and per complete success. More agents were operationally
viable; they were not better on this task.

The task

The scenario is a deterministic public-repair contribution task:

eight named participants;
each participant chooses how many repair units to contribute;
the task succeeds only if the total reaches a fixed threshold;
participant coverage is forced, so every selected agent acts;
no LLM reasoning is used for the choice; the model returns a strict JSON choice.

The point of forcing coverage was to avoid the earlier failure mode where one agent dominated every
turn and the other participants never acted.

Protocol

Item	Value
Groups	2 agents / 3 steps; 4 agents / 5 steps
Repeats	3 per group
Caps per run	150 calls / USD 0.50 / 600s
Choice model	`deepseek-flash` on a Responses API contract
Outcome	machine-decidable `outcome.json`
Validity	provenance, coverage, integer choices, arithmetic, caps, port closure, archive hashes

All six runs passed every validity gate. No parser failure, no budget breach, no port leak.

Results

Run	Agents	Total / threshold	Success	Zero contributors	Calls	Total tokens	Cost USD
2-agent r1	2	2 / 3	no	1	17	10,060	0.0269116
2-agent r2	2	3 / 3	yes	0	17	11,070	0.0313716
2-agent r3	2	0 / 3	no	2	17	10,255	0.0281116
4-agent r1	4	3 / 5	no	2	33	21,493	0.0597772
4-agent r2	4	2 / 5	no	2	33	22,679	0.0641372
4-agent r3	4	5 / 5	yes	0	33	21,459	0.0588042

Group aggregates:

Metric	2-agent	4-agent	Ratio
complete successes	1 / 3	1 / 3	1.000
mean calls	17.00	33.00	1.941
mean total tokens	10,461.67	21,877.00	2.091
mean cost USD	0.028798	0.060906	2.115
cost per complete success USD	0.0863948	0.1827186	2.115
mean zero contributors	1.000	1.333	1.333

What I take from it

More agents are not automatically better. In this task they doubled cost and left more participants contributing nothing on average.
The result is a valid negative result, not an apparatus failure. Every run completed and every artifact passed its gates.
The 4-agent group did produce one complete success, so the mechanism is not broken. It is simply not worth 2.115× the cost on this task.

What this does not show

It does not show that four agents are worse in general.
It does not transfer to another task or to a real user.
It says nothing about agent quality under human review, because no human scored these runs.
The sample is three repeats per group on one synthetic task.

Reproduce it

Two pieces are now public:

Method package: https://github.com/janzong/agent-lab-method (MIT, commit 6cc70156facb37baf23fe5fe57dad93d43502b91) — schema, synthetic GenMentor adapter, fixtures, tests;
Trust layer: https://github.com/janzong/agent-lab-trust (MIT, release v0.1.0-rc2) — local-first run validation, hashed reports, and a synthetic deletion proof.

The full protocol and the archived runs stay private. The result is intentionally boring: a negative result with caps, hashes, and every failed run preserved.

I am also looking for three independent reproductions by non-authors. The trust layer guide expects 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, and deletion audit_hash 4f0193abbd49a0f9. If you run it and any hash differs, that is the most useful reply I can get. If you have a task where you believe more agents should win, that is the experiment I want to run next.

How common is AGENTS.md, really? I sampled GitHub: 6.2% of active repos, 1.0% of all repos

Janz — Sat, 19 Sep 2026 08:30:25 +0000

Short version: Every rate in my previous two posts had a denominator I picked myself — the 558 repos
that already had an AGENTS.md. That is a fine way to describe a corpus and a terrible way to answer
"how common is this?". So I sampled GitHub two different ways:

6.2% of active repos (pushed in the last 90 days, not a fork, not archived) contain an AGENTS.md — 51 of 817, 95% CI [4.8, 8.1]
1.0% of all public repos do — 9 of 924, 95% CI [0.5, 1.8]

Same file, same counting rule, two numbers that differ by 6×. Which one you quote depends entirely on
the question you are asking, and I had been quietly dodging that choice.

Three things I did not expect, in order of how much they changed my mind:

CLAUDE.md is at 5.4% of active repos. Statistically indistinguishable from AGENTS.md. If you assumed one format won, it has not.
93% of public repos have not been pushed in 90 days, 29% are forks, and 8.3% are completely empty. "GitHub" as a population is mostly a graveyard, which is why the stock rate is so low.
Most AGENTS.md files in the wild are tombstones. Of the 9 files my population sample found, 7 were in repos that have not been pushed in three months.

Why the denominator was missing

Here is the trap I was in. To build the corpus I searched GitHub for repos containing an AGENTS.md, then
labeled what I found. Every percentage since — "85.7% of files prohibit things", "13.6% record a
gotcha" — has a denominator of files that already exist. Those numbers are real and I stand by them,
but they cannot answer the question every reader actually has: should I write one of these?

For that you need a sample of repos drawn independently of whether they have the file. That is a
different sampling problem, and it needs two different frames, which I kept conflating:

quantity	question it answers	frame
stock rate	"is this mainstream?"	every public repo
active rate	"is this what working projects do?"	repos pushed in the last 90 days
trend	"is it spreading?"	rates by repo creation year

Mixing them produces confident nonsense, because the stock population is dominated by abandoned
one-off repos and the active population is not.

I wrote the decision rule down before running anything, so I could not move the goalposts after
seeing the number: <1% ⇒ describe it as an early-adopter curiosity; 1–5% ⇒ "early but measurable",
and every rate must be labeled as in-corpus or ecosystem-wide; >10% ⇒ "standard practice". It landed
in the middle band, slightly high — so: not a curiosity, not a standard either.

Method, briefly

Both frames use one tree call per repo —
GET /repos/{owner}/{repo}/git/trees/HEAD?recursive=1 — and check every path with a case-insensitive
match on the filename. Recursive matters: a root-only check would miss docs/AGENTS.md and friends and
under-count. Roughly 1,900 repos, all responses cached, seed 20260918.

Frame A — active rate, by creation cohort. For each year 2010–2026 I picked one random slice of
creation time, queried
created:<slice> pushed:>2026-06-20 fork:false archived:false, pulled every result (rather than
taking the top page, which is ranked by GitHub's relevance and would bias toward popular repos), then
randomly sampled 50 repos from the slice. A 2026 week contains ~62,000 active repos, which exceeds the
1,000-result search cap, so recent cohorts narrowed to a random day and then a random hour.

Frame B — stock rate, unweighted. GitHub search cannot give you a random sample of the population —
there is no random sort, and ranking favors stars and activity. So I enumerated the ID space instead:
binary-searched the current maximum repo ID (1,375,203,308), then sampled IDs uniformly and asked
GET /repositories/{id}. Only 35% of IDs correspond to an existing public repo (the rest are
deleted, private, or never existed), so 1,000 usable repos cost 2,920 probes.

Results

Active repos (Frame A, n=817):

AGENTS.md                        6.2%   [4.8, 8.1]
CLAUDE.md                        5.4%   [4.0, 7.2]
.github/copilot-instructions.md  1.1%   [0.6, 2.1]
.cursorrules / .cursor/rules     0.7%   [0.3, 1.6]

All public repos (Frame B, n=924):

all public repos                 1.0%   [0.5, 1.8]
non-fork, non-archived           0.5%   [0.2, 1.4]
AGENTS.md ∩ CLAUDE.md             18 repos
CLAUDE.md only, no AGENTS.md      26 repos

The number I got wrong twice

In my corpus, 59.1% of repos that have an AGENTS.md also have a CLAUDE.md. In the wild it is
35.3%. Both are correct; they are answered by different populations, and only one of them is
"typical".

The reason for the gap is a selection effect I should have predicted: a repo that has one agent
instruction file is already a repo whose author cares about agent tooling, so it is much more likely to
have several. My corpus is a sample of the enthusiastic end, and it over-represents multi-tool setups
by about 1.7×. If I had quoted 59.1% as a base rate, I would have been describing my sample, not the
world.

The trend measurement failed, and I am not going to dress it up

I wanted to show adoption rising by cohort and I could not measure it. In Frame A,
p_2026 / p_≤2022 = 0.67× — if anything, older active repos are more likely to have the file.

That number is not evidence that adoption is flat, because creation year and repo age are perfectly
confounded. A repo created in 2010 that is still receiving pushes in 2026 is a survivor — a project
that lived long enough to accumulate conventions. A repo created in 2026 is mostly somebody's first
weekend project. The cohort axis is really an age axis, and age predicts having-writers and having-time.

Separating those would require reading commit history to find when each file was added. I did not do
that, so the honest deliverable here is "not measured", not "not spreading".

A GitHub API gotcha worth knowing

Half a day of this project went into a bug that was not a bug. 403 from the REST API has three
distinct meanings, and you have to read the body to tell them apart:

Primary rate limit — X-RateLimit-Remaining: 0.
Secondary rate limit — burst/concurrency. GitHub's docs say it plainly: make requests for a single user serially. Three threads was enough to get me permanently throttled, while serial requests with connection reuse ran at ~8/s without complaint.
A single repo blocked by GitHub — {"message":"Repository access blocked","block":{"reason":"tos"}}.

I had classified (3) as (2), so my code kept sleeping and retrying the same blocked repo, forever. If
you write a bulk GitHub crawler, put that string in your error handling; it is stable, it is not a rate
limit, and it will silently eat your retry budget.

What this changes about my own claims

I have to soften things I said earlier:

"AGENTS.md is the emerging standard" — no. 6.2% of active repos is a real practice, not a standard. It is roughly 6–9× more common than the vendor-specific alternatives, which is a measurable reason to keep the word "format" instead of a vendor name.
"CLAUDE.md is just a companion file" — that was a conditional rate described as if it were a base rate. Unconditionally, the two are neck and neck.
"adoption is spreading" — unmeasured, see above.

What survives: it is a habit of active projects (6.2%) rather than something the population does
(1.0%). Those two sentences imply completely different advice, and I could not tell them apart until I
sampled for it.

The experiment I have not run

There is still a hole underneath all of this, and it is the one that matters: does any of it work?
Every number in this project — mine and everyone else's — is descriptive. Nobody has shown that a repo
with an AGENTS.md produces better outcomes than the same repo without one, because that requires a
controlled task, a blind judge, and an effect size, not a sample.

I wrote the design for that experiment (three arms, including a placebo arm that gets an equal-length
unrelated document, so "more context" and "this document" can be told apart) but I deliberately did not
run it yet. The sample sizes are brutal and the honest outcome is probably "we could not detect it".

If you have actually noticed a charter changing a decision — a rule that stopped you from doing
something you would otherwise have done — that is the data I cannot generate myself, and it is worth
more to me than a star.

Reproduce it

pip install agent-charters              # the CLI
# the sampling code is in the repo, not the package:
git clone https://github.com/janzong/agent-charters
cd agent-charters
.venv/bin/python work/prevalence.py active --per-gen 50   # ~10 min, search-rate-limited
.venv/bin/python work/prevalence.py report
.venv/bin/python work/prevalence.py stock  --n 1000       # ~40 min
.venv/bin/python work/prevalence.py report-stock

Seed 20260918, every response cached, so report is instant and costs nothing after the first run.
The full write-up with every caveat I could think of — including the two frames disagreeing (6.2% vs
1.8% on a 57-repo active sub-sample; the honest answer is a range of roughly 2–6%) — is in
work/audit/prevalence.md in the repo.

Repo: https://github.com/janzong/agent-charters

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

Janz — Mon, 14 Sep 2026 13:25:06 +0000

Short version: I labeled 558 public AGENTS.md files against a 9-category taxonomy. The measured
base rates say something boring and useful — almost every file prohibits things (85.7%) and lists
build/test commands (82.8%), while almost none of them record a gotcha (13.6%). Two of the nine
slots are nearly empty across the whole corpus: gotchas and agent_meta (rules about the agent itself,
25.8%).

Then I ran the same ruler over two big, well-maintained files. Both missed gotchas. So here is a
five-minute check you can run on your own file, and the exact numbers behind it.

The base rates

Measured on 516 substantive files (558 collected, the rest were one-line pointers):

boundaries     85.7%   what must never be done
build_test     82.8%   the commands CI runs
workflow       67.1%   commit format, branches, release steps
structure      59.1%   layout, where new code belongs
style          54.5%   naming, formatting — or a pointer to the config that enforces it
environment    45.0%   toolchain versions, required env vars
overview       32.2%   one paragraph: what this is, what it deliberately is not
agent_meta     25.8%   rules about the agent: tone, when to ask first
gotchas        13.6%   pitfalls that are NOT derivable from the code

The shape is not surprising once you see it as a genre: an AGENTS.md is usually written defensively,
as a list of things not to break. The file that would actually save you time is the one almost nobody
writes.

Two receipts

compare prints your file's coverage next to the corpus baseline. Two real examples from the corpus:

file	size	sections	coverage	missing
`langchain-ai/deepagents`	10 KB	22	7/9	`overview`, `gotchas`
`openai/openai-agents-python`	34 KB	27	6/9	`style`, `agent_meta`, `gotchas`

Both are good files. The 34 KB one is one of the more thorough agent-instruction files in the corpus —
27 sections, 19 separate boundary markers. It still has nothing in it that you could only learn by
running the thing.

Why gotchas are rare (and why that is not laziness)

You can only write a gotcha after being bitten by it — and by the time you have been bitten, the
temptation is to fix the thing rather than write the sentence down. The fix is visible in the code;
the sentence is a liability nobody wants to maintain.

There is a second, worse failure mode. I sampled 347 entries from the Gotchas / Common Pitfalls /
Troubleshooting sections in the corpus (an earlier snapshot, 507 files) and hand-labeled 120 of them:

58% are readable from the repo itself (interface contracts, platform limits, build requirements)
34% are not pitfalls at all — they are generic advice ("remember to install dependencies", "don't commit .env"), the same sentence you would write for any project
8% are genuinely experience-only: upstream/third-party behaviour, past incidents, and the places where the docs disagree with the code

So the section is rare, and a third of what does live there is filler. The 8% is the part worth
handing to an agent, and it cannot be generated from a reading of the repository. It has to come from
a person who was there.

The five-minute check

No tool needed. Ask these five questions about your own file:

Are the commands copy-pasteable? Not "run the tests" — the actual command CI runs, with the working directory. If your README says one port and production uses another, say so (that mistake is in the corpus, in a file that otherwise looks complete).
Does it name what must never be committed or never touched? This is the one thing the corpus does well (85.7%) — check that yours names the tempting case, not the obvious one. "Don't commit secrets" is obvious; "don't hand-edit the production database to fix a row, use the backfill script" is a boundary that will actually stop someone.
Does it say anything about the agent's own behaviour? Only 25.8% do. Tone, when to stop and ask, which actions need explicit approval, what must not leave the machine.
Is there at least one sentence that is not derivable from the code? If every line in your file could have been written by reading the repo, the file is documentation, not a charter. This is the gotcha test.
Do the paths it points at exist? Measured: 49% of files route to another file, and 15% point at a knowledge store or rules directory. A pointer to a file that moved is worse than no pointer — an agent will go looking, and will read whatever it finds there as authoritative.

If you want the baseline instead of the feeling

pip install agent-charters                 # 0.4.1, on PyPI — no runtime deps, ~110 KB wheel
agent-charters compare path/to/AGENTS.md   # coverage vs the 558-file baseline, plus gaps
agent-charters brief                       # the checklist + a paste-ready prompt
agent-charters refs path/to/AGENTS.md      # external pointers and dangling references

(Updated 2026-09-19 — when this was first posted the only way in was a clone, and this block said so.
It is on PyPI now: verified in a clean virtualenv on a default index, install takes a couple of
seconds, because the corpus ships as gzipped JSONL and pandas / pyarrow are optional extras.
Source, if you would rather read or pin it: github.com/janzong/agent-charters — CN mirror
gitee.com/janzong/agent-charters.)

compare is the one that answers question 4 in aggregate. It also does something I did not expect:
when I used brief's prompt to write a charter for a real project, compare flagged coverage I had
skipped — and one of the nine slots it missed was the name of the slot itself, which is a bug in my
taxonomy, not in the file. That is the kind of thing a rule-based labeler gives you: you can point at
the pattern that fired and argue with it.

There is no LLM in the labeling loop. Every label is recomputable and arguable, which is the point —
if you disagree with a label, you can find the rule that produced it and overrule it.

What this is not

I do not want to oversell the numbers, so:

The classifier scores 92% precision / 70% recall on a 55-file held-out English set, and 88% / 73% on 50 held-out Chinese files. The recall number is the honest one: it misses roughly three in ten of the labels it should have produced. gotchas and agent_meta are the weakest slots in both languages.
The held-out sets were labeled by one person (me). No second annotator, no inter-annotator agreement.
Coverage is a process metric, not a quality metric. In a 3-repo test, a checklist that names all nine slots pushed a generator from 4–5 categories to 9/9 — and filling all nine slots is not the same as writing a good file. It is a prompt for the questions, not a grade.
The labels and the rates come from public files and a rule-based classifier, not from a language model. The one LLM in this story is the generator in the 3-repo test, which is why that number is reported as n=3.

The ask

The weakest part of this project is that the only person who has ever tested it is its author. If you
have an AGENTS.md (or a CLAUDE.md, or a .cursorrules) on a real project, run compare on it and
tell me what it gets wrong — the file, the label, or the baseline rate. A wrong label on your file is
worth more to me than a star.

Repo: https://github.com/janzong/agent-charters

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Janz — Mon, 14 Sep 2026 12:23:13 +0000

TL;DR — I collected 558 AGENTS.md files from public repos and labeled each one against a 9-category
taxonomy with a rule-based classifier (no LLM in the loop, so it is auditable and recomputable). Then I
blind-labeled held-out samples and compared: 92% precision / 70% recall on 55 English files,
88% / 73% on 50 Chinese files. The most common categories are prohibitions (85.7%) and build/test
commands (82.8%). The rarest: gotchas (13.6%) and instructions about how the agent itself should behave
(25.8%).

Why bother

Almost every discussion about AGENTS.md is anecdote-led: my repo's file works, my agent ignores it,
a good one is a model upgrade, a bad one is worse than nothing. All of that may be true — but nobody
seems to have the distribution. So I built it: snapshot of 558 files from 558 public repos
(2026-09-10, 5.3 MB, 516 usable for statistics), labeled, versioned, and published with the tooling.

Method, in one paragraph

Nine categories: boundaries, build_test, workflow, structure, style, environment, overview,
agent_meta (rules about the AI itself), gotchas. Labeling is done by pattern rules over headings and
body text — deliberately, because a rule set can be read, argued with, and re-run, and every number below
can be recomputed from the released dataset. I then measured how well the rules match a human reading:
100 files in-sample (upper bound, 90%/75%) and two held-out sets I had never tuned against —
55 English (92%/70%) and 50 Chinese (88%/73%). Held-out numbers use the conservative reading
(items I was unsure about count as classifier errors).

Five things the numbers say

1. Two categories dominate — and they are tied

category	share of 516 files
`boundaries` (what you must never do)	85.7%
`build_test` (install/build/test/CI commands)	82.8%
`workflow` (branching, commits, review, release)	67.1%
`structure`	59.1%
`style`	54.5%
`environment`	45.0%
`overview`	32.2%
`agent_meta`	25.8%
`gotchas`	13.6%

The 2.9 pp gap between the top two is smaller than the known false-positive rate (~3%) of the
prohibition pattern — so the honest statement is tied for first, not "prohibitions beat build commands".

2. Nobody writes down their scars

gotchas is dead last at 13.6%. Worse: when people do open a "known issues" section, a third of it
isn't a gotcha. I hand-read 120 items from those sections:

58% were readable straight from the repo (config, code, README mismatch),
34% were not gotchas at all (generic advice: "remember to install dependencies"),
8% needed experience or the outside world (OS behavior, an upstream outage, yesterday's incident).

That 8% is the part an agent can never derive from the code — and it is exactly the part that is
almost never written down.

3. The slot language models skip is `workflow`

In a controlled experiment (11 repos × 3 prompt styles), prompts that listed topics explicitly produced
9/9 categories, while prompts that left the slots implicit skipped workflow in 11 out of 11 files.
Point at workflow by name and it appears 3/3 times, with real content. The gap is not knowledge,
it is questions — which is why I turned the corpus distribution into a checklist tool.

4. Your weakest category depends on the language

English files fail differently from Chinese ones. English: gotchas recall 32–38% — the classifier
misses casual "watch out" prose. Chinese: agent_meta recall 26% — Chinese files express agent rules
in the second person ("you are the dispatcher, not the executor"), and the body-pattern rules for that
category are entirely English, so the whole style is invisible to them. File-level exact agreement
(9/9 categories identical) is 12% in both languages.

5. Half of these files are entry points, not documentation

49% point to some other file; 15% route to a knowledge or rules directory. That's a structural
fact about the format, and it means "does this repo have an AGENTS.md?" is a much weaker question than
"what is actually in it".

The tool

pip install agent-charters

agent-charters brief      # checklist of the 9 slots + a paste-ready prompt
agent-charters compare your-AGENTS.md   # your coverage vs the 558-file baseline
agent-charters refs your-AGENTS.md      # does your file point at paths that exist

Honest note: compare is a checklist, not an oracle. It warned me that one of the nine categories was
missing from a file I wrote myself — it was actually present, but the heading used the tool's own slot name
instead of natural language. That is documented in the repo (along with the exact experiment) rather than
quietly patched, because a tool that tells you "you're missing X" should be checked by a human.

What this is not

Rule-based labels, not per-file human labels. Precision/recall above are the honest measures; the per-category numbers in the dataset carry that error.
Not a representative sample of GitHub. Repos were found through AI/agent topics and Chinese keyword search; the Chinese set came out 97% Chinese by construction, which says nothing about GitHub's language mix.
Single annotator. The blind labeling was done by one model-driven annotator, not multiple raters.
A snapshot, not a trend. A baseline of file hashes is stored so that a future re-crawl can measure how these files change — that measurement doesn't exist yet.

The ask

I'm looking for 2–3 people who are not me to run compare on an AGENTS.md they actually maintain and
tell me where it's wrong — missing a category you clearly have, or claiming one you don't. That is the one
piece of evidence this project doesn't have yet: an external user. Issues and comments are both fine.

DEV Community: Janz

Your agent run passed. Can you prove it was allowed?

The gap

The demonstration

Reproduce

What this does not prove

The series

Call

We ran 2 vs 4 agents six times. Four agents cost 2.1 and did not improve success

The task

Protocol

Results

What I take from it

What this does not show

Reproduce it

How common is AGENTS.md, really? I sampled GitHub: 6.2% of active repos, 1.0% of all repos

Why the denominator was missing

Method, briefly

Results

The number I got wrong twice

The trend measurement failed, and I am not going to dress it up

A GitHub API gotcha worth knowing

What this changes about my own claims

The experiment I have not run

Reproduce it

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

The base rates

Two receipts

Why gotchas are rare (and why that is not laziness)

The five-minute check

If you want the baseline instead of the feeling

What this is not

The ask

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Why bother

Method, in one paragraph

Five things the numbers say

1. Two categories dominate — and they are tied

2. Nobody writes down their scars

3. The slot language models skip is workflow

4. Your weakest category depends on the language

5. Half of these files are entry points, not documentation

The tool

What this is not

Links

The ask

3. The slot language models skip is `workflow`