DEV Community: Janz

Audit your UI before you redesign it: baseline first, then compare

Janz — Tue, 13 Oct 2026 05:53:06 +0000

We just spent two weeks improving six dashboard pages in our research system. The tempting move was to open each page, feel what was wrong, and start fixing. That approach produces exactly what you expect: pages that each look individually fine but feel like they were designed by six different people.
Instead, we ran an audit first. Here is the method and what it caught.

The method: baseline, compare, classify

Step 1: Extract a baseline. Pick two or three pages in your system that already feel right — not because they are beautiful, but because they are consistent. Write down every convention they share: segmented controls, button styles, icon usage, font sizes, color semantics, spacing.
In our case, three pages shared a set of conventions we hadn't written down anywhere:

Segmented controls: gray track, pill-shaped, white floating active state
Back buttons: shared icon component, gray hover state
Icons: one shared icon library, no emoji
Text buttons: link-style gray-to-accent on hover
Loading: shared spinner component

Step 2: Compare every target page against the baseline. Not "does this look good?" but "does this match?" Each deviation gets written down as a specific, nameable difference.

Step 3: Classify each deviation. There are only two kinds:

The page is wrong — it should adopt the baseline convention
The baseline is wrong — the deviation reveals a better pattern that should be promoted to the system This classification prevents the two failure modes: inconsistency (everything is special) and rigidity (nothing can improve).

What the audit caught: 8 deviations in 6 pages

Deviation Fix
Square segmented controls Changed to gray-track pill with floating white active
Back button with red hover Changed to shared icon + gray hover
Three-line page headers Aligned to single-line pattern from reference page
Star icon in accent red Changed to neutral gray
Inline SVG chevron Changed to shared icon component
Outlined box button Changed to link-style text button
No loading indicator Added shared spinner
Inconsistent filter-clear hover Unified to same color token

None of these are dramatic individually. Together, they made six pages feel like six different products. After the audit, they feel like one system.

The color rule that fell out

One deviation taught us a principle worth keeping: a back button had red on hover. Red in UI means destructive — delete, danger, irreversible. A back button is none of those. Using red there trains users to hesitate on every red hover.
The rule we wrote down:
Accent red is reserved for numeric values and selected states. Icons and button hovers use neutral gray.

This is not a aesthetic preference. It is semantic hygiene: red should mean "pay attention to this number" or "you are here", never "this is clickable decoration".

The discipline part

After fixing, three things make it stick:

Shared components only. Icons go through one file. Buttons go through one component. No page-local SVGs or ad-hoc button styles.
Full verification after changes. TypeScript zero errors, all frontend tests pass, build succeeds, deployed chunk hash verified on both production ends.
When a new pattern is better, promote it to the baseline. The point is not stasis — it is that changes happen at the system level, not scattered across pages.

The connection to agent governance

If you have been following this series, you will see the pattern: this is the same method as the shadow-to-gate pipeline from the previous post. In both cases:

Measure before you change (shadow audit / UI baseline)
Fix the mechanism, not individual instances (citation rebuild / shared components)
Add a gate to prevent regression (citation gate / component-only rule) Whether you are governing agent output or UI consistency, the sequence is the same: baseline → compare → fix mechanism → gate.

Takeaway

If your multi-page application feels inconsistent, resist the urge to redesign page by page. Instead:

Write down the conventions your best pages already share
List every deviation in every other page
Fix deviations — or promote better patterns to the baseline
Enforce with shared components and full verification The audit itself took 30 minutes. The fixes took one afternoon. The result is six pages that feel like one system — something no amount of per-page tweaking had achieved.

Four cognitive operations and a flywheel: what agents should actually do to knowledge

Janz — Mon, 12 Oct 2026 19:21:23 +0000

Most agent systems answer one question: how do I get the model to write something for me? That is the wrong first question. The right one is: what kind of cognitive work should the agent do at all?
After building and running a research system that has processed 7,243 documents into 138,000 knowledge nodes, 588,000 links, and 966 reports, I can tell you the framework that survived contact with production. It has two parts: four cognitive operations and a flywheel.

The four cognitive operations

When I ask an agent to help with knowledge work, I want it to do exactly one of four things:

Systematize — take scattered material and organize it into a coherent structure. Not create new claims; just make what already exists navigable and comparable.
Theorize — connect specific observations to general principles. This is the hardest one and the one where agents hallucinate most, because it requires restraint: only link what the evidence supports.
Academicize — bring scholarly rigor to raw arguments: citations, methods, evidence chains, counter-arguments. The agent's job is not to be smart; it is to be checkable.
Systematize (again, but as architecture) — take working pieces and turn them into a reproducible pipeline. This is where "agent writes code" becomes "agent maintains a system." Every feature request gets tested against these four. If a proposed capability does not clearly strengthen one of them, it does not get built. This single filter has killed more features than any budget constraint.

The flywheel

The four operations describe what agents do. The flywheel describes how knowledge compounds:

Documents → Extract → Research → Discover → Challenge → Feed back → New Research

Each step feeds the next:

Extract: turn raw documents into structured knowledge nodes with verifiable source links.
Research: use those nodes to generate new analyses on topics a human chooses.
Discover: identify patterns, collisions, and gaps the research pipeline surfaces automatically.
Challenge: have a separate pipeline question those discoveries — not just "is this correct?" but "is this new?"
Feed back: unresolved challenges become new research inputs. The wheel turns. The flywheel is not a metaphor. Each turn measurably increases the knowledge base: more nodes, more links, more reports, more challenges. After several months of continuous operation, the numbers compound.

The gate question

Before any batch of work starts, the agent must answer four questions — and if the answers do not align with the design, the batch does not run:

Which cognitive operation does this strengthen?
Which flywheel stage does it feed?
Which evolution line does it advance — knowledge breadth or method depth?
What are the three expected outcomes: more truthful, less human effort, more consumption? This is not bureaucracy. It is the cheapest form of strategic alignment I have found: four lines, written before the work starts, checked against the master design. It prevents the most common failure mode of agent systems — drifting into producing impressive-looking output that nobody asked for.

Why "agent as editor, not author" matters

The deepest architectural lesson came from a failure. Early versions of our research pipeline let the writing agent see the full source text. It would then paraphrase everything — technically correct, but the output was just the source material with different words.
The fix was to change the agent's role: editor, not author. The agent receives curated material nodes and is told: "The material below is the only evidence source for this section. Understand it, digest the core arguments, and organize them into a coherent narrative in your own reasoning language." The agent organizes; the material provides the substance.
This distinction — editor vs. author — is the difference between a system that generates text and a system that generates knowledge.

The measurement connection

Here is where this connects to my previous posts on trust and validity states. The flywheel only compounds if each turn produces output you can trust. And trust is not a feeling — it is measurable:

Are citations reconciled? (citation reconciliation rate)
Are method claims backed by data? (method honesty rate)
Are quality scores computed on the final artifact, not a prefix? (measurement-window validity)
Are absent events reported as unknown, not zero? (the absent-zero rule) The flywheel tells you what to produce. The validity rules tell you whether you can believe it. A system without both is either productive but untrustworthy, or trustworthy but sterile.

What to take

If you are building an agent system for knowledge work, take three things:

Classify cognitive operations before writing prompts. "Write me an article" is not a cognitive operation. "Systematize these 50 sources into a framework" is.
Make the agent an editor, not an author. Give it material, not a blank page. The output will be less impressive and more useful.
Put a gate question before every batch. Four lines about which operation, which flywheel stage, which evolution line, what expected outcome. If the answers are fuzzy, the work will be fuzzy too. The full system that these principles came from processes tens of thousands of documents on real research topics. It is not a demo. The cognitive operations and the flywheel are what survived.

From shadow to gate: how we fixed agent citation fraud in one production batch

Janz — Mon, 12 Oct 2026 09:05:44 +0000

The previous post ended with a question: which of your zeros are just absences? This one is about what happened after we found ours.
Our research pipeline had a citation problem. A blind review of 150 generated reports found what three independent analyses converged on: references that did not match the text, method claims with no data behind them, and quality scores computed on a truncated prefix of each report.
We fixed it in one day, but not by making the model smarter. The fix was a pipeline: shadow audit, body-driven rebuild, then a gate. This is the recipe.
Step 0: Blind review before you fix anything
Two agents analyzed the same corpus independently — one implementation agent, one audit agent from a different harness, without seeing each other's conclusions. Both found the same three priority failures:

Citation mismatch: the reference table was built from the source material list, not from what the text actually cited.
Method dishonesty: reports claimed "survey research" when no survey had been conducted.
Scoring on a prefix: the quality grader read only the first 6000 characters; references and conclusions were appended after that. Independent convergence is the cheapest signal you will ever get that the problem is real. Step 1: Shadow audit — record, do not block The first change was a shadow audit. It ran before every report was saved and computed two numbers:
inbound rate: what fraction of in-text citations appear in the reference table?
dirty-entry count: how many entries contain internal markers, section fragments, or pipe residue? It wrote to a JSONL file and changed nothing. On the existing 148-report batch, the inbound rate median was 0.32 — roughly one in three citations was broken. The point of shadow mode is calibration without risk: you learn the distribution before you set any threshold. Step 2: Fix the source, not the symptom The root cause of the mismatch was structural: the reference table was reconstructed from the input material list, not from the body text. The fix inverted the direction — at finalize time, the system now rebuilds the reference table from the citations that actually appear in the body, resolving each through the knowledge base for a canonical title. Replaying the same 148 reports: inbound rate median 0.32 → 1.00 reports rebuilt 142/148 (6 correctly fell back) dirty entries 120/326 → 8/326 The remaining 8 were legitimate long titles, not defects. Step 3: Gate — block ingestion, not delivery Only after shadow numbers were calibrated did we turn on a gate. The design choice matters: A failed citation check blocks the report from entering the knowledge base. It does not block delivery.

Rationale: a report with broken citations may still be useful to the requester; poisoned knowledge is not. The cost asymmetry runs in one direction. We set the threshold at 0.5 — below the 10th percentile of the repaired batch (0.857), high enough to catch structural failure without blocking honest edge cases.
Step 4: A dashboard that runs itself
The final step was a batch quality contract: five mechanical metrics computed after every batch, stored in a governance database, with thresholds calibrated from the pre-fix distribution.
Metric Before After
Citation reconciliation (median) 0.45 1.00
Dirty entries (P50) — 0
Fixed closing-sentence rate 97.3% 0%
Method honesty — 100%
Deep-source share 31.6% 58%
Blank-source share 74% 20%

No LLM in the metrics. No human in the loop for the counting. If the pipeline regresses, the dashboard catches it on the next batch.
What made it work
Three things, in order:

Shadow before gate. We could not have chosen 0.5 without knowing the distribution. Blocking on a guess would have either frozen production or let everything through.
Fix the mechanism, not the prompt. The reference table was wrong because of how it was built, not because the model was careless. Rebuilding from body citations is deterministic.
Block ingestion, not delivery. This is the absent-zero principle applied to failure handling: a bad report is visible and correctable; a poisoned knowledge base is silent and compounding. The part that is reusable None of this is specific to our stack. Any agent pipeline that produces claims with citations can run the same sequence: blind review → shadow audit → source fix → calibrated gate → batch dashboard The three-state validity rule from the previous post does the same job at the measurement layer: absent evidence is unknown, not zero. This recipe does it at the production layer: a broken citation blocks knowledge, not the user. The trust layer that made the audit reproducible is at janzong/agent-lab-trust, and the validity-state specification is docs/validity-states.md. If your agents write anything that gets cited later, run the shadow audit first. The first number is usually worse than you expect.

I preregistered 30 real agent tasks. Three rules didn't survive contact.

Janz — Sun, 11 Oct 2026 19:46:23 +0000

We just closed a preregistered cohort of 30 real development tasks executed by a paired-agent workflow: one agent implements, a second agent from a different harness independently recomputes the tests and records process events before anything ships. The cohort was small, descriptive, and single-operator by design. No causal claims. The point was to learn what actually happens when you write governance rules down and then do real work.
The headline numbers look excellent:
Recorded slots 30/30
Complete required chains 30/30
Gate coverage 30/30
Verification events 111
Export gate ready
Three of the most interesting numbers are the ones that look good but should not.

Reservation timing collapsed under real work The preregistration said: reserve a slot before work begins, never backfill completed tasks. Real life disagreed three times. In three later slots, implementation commits landed before the slot was reserved. On one dense day, 17 commits of real work shipped with no slot at all — completed tasks that, per our own rule, can never be backfilled. The mechanism is boring: pressure arrives, work starts, and "open a slot first" is an extra step that nobody remembers at 2 a.m. Writing the rule down did not change the behavior. The honest recording is that reservation discipline degraded exactly when task density went up.
Rework = 0 is a blind spot, not a virtue The final report says Rework incidence: 0/30. That is false in spirit. During the cohort we redid a slot closure twice after an infrastructure incident wiped local commits, and plenty of fixes went red-test-first before going green. The schema records rework only when someone voluntarily writes a rework event. Nobody volunteered. This is the absent-zero error wearing a governance costume: an absent record rendered as a confident zero. We now write that rule down explicitly — an absent event must be reported as unknown, never as zero — in docs/validity-states.md.
Duration coverage drifted mid-cohort Only 9 of 30 slots carry explicit durations. The early slots predate the duration rule; the later ones record it. So duration coverage improves over time because the recorder changed, not because the process got better. Any "trend" in that column is a measurement artifact. What v2 must do differently
Machine-check reservation timing. If the first commit predates the slot reservation, that is an automatic violation event — not a paragraph in a handover note.
Make rework mandatory. A failing verification that is followed by a correction must emit a rework event. Voluntary rework reporting produced zero.
Freeze the recorder before slot 1, including duration and rework fields.
Add at least one more operator. A single-operator cohort cannot separate discipline from personality. The gate said "ready". We chose not to export. The export gate returned ready: manifest valid, safety scan clean, source registry verified. We are still not exporting the event data. Thirty production-adjacent tasks from one operator is a small, non-random sample where cross-field reidentification risk is highest and learning value for outsiders is lowest. That is the gate working correctly. A gate that can only say "no" is theater. A gate that can say "ready" — while a human still says "not worth the risk" — is an actual decision. The durable output is the negative result and the rules it produced, not the dataset. As before: descriptive only, no causal claims, and the full trust layer remains reproducible at janzong/agent-lab-trust. If you run agents on real work, the uncomfortable question is not "did my rules pass?" It is "which of my zeros are just absences?"

Your agent run passed. Can you prove it was allowed?

Janz — Wed, 23 Sep 2026 03:03:42 +0000

Short version: Most agent teams can show a run. Few can show, in one re-runnable command, whether that run was allowed under a policy. I added a policy-as-code audit to agent-lab-trust: it checks cost and call caps, the declared artifact contract, and forbidden markers, then emits a canonical audit_hash. On 13 archived runs, the default contract passed 0/13 and a declared GenMentor contract passed 13/13. The data did not change. The contract did.

The gap

Dashboards answer "what happened". Governance needs "what was allowed, and by which rule". A run can succeed and still violate a budget, write an unexpected artifact, or carry a marker that should never ship.

The audit is deliberately boring:

agent-lab-trust audit <run-root> --policy policy.json

A policy can declare:

max_cost_usd and max_calls per run;
required_artifacts plus required_artifacts_mode: all|any;
forbidden_markers.

The output is a finding list and a canonical audit_hash. deletion-proof --output <path> writes the deletion evidence used in the same policy flow.

The demonstration

I ran the audit over 13 archived GenMentor runs.

Trust-layer default contract (output/structured.json or results/structured.json): 0/13 passed, missing_artifact ×13, cost_exceeded ×1.
Declared GenMentor contract (archive.json and summary.json, mode all, cap 5.00): 13/13 passed, no findings.

Audit hashes:

default structured contract: 2c122c4ff2c0d5eed11a0fc23b4ff717002864dbd941402e8fe18d6d17c96ce6
GenMentor contract: 368b75a06c0f5c205436d287881cfddcbabd94b7ad34a860e852fba59dad7ab5

The one cost finding is real: replay-8of8-20260920-a recorded 3.9529266 under the trust-layer cap 0.60. Under the GenMentor policy cap 5.00, it passes. Caps are part of the contract too.

The lesson is not "the audit is noisy". It is that an audit without an explicit run-family contract silently embeds one family's format. The policy has to name the contract, the caps, and the markers.

Reproduce

docker run --rm ghcr.io/janzong/agent-lab-trust:rc2

or bash scripts/reproduce.sh. Expected: 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, deletion audit_hash 4f0193abbd49a0f9.

The :rc2 tag points at the latest published build; SERIES.md pins the verification digest sha256:2e178e63fff30e70ac68501cb17e5316097875932d1d7085e669ce4f8d5a105f.

What this does not prove

no real governance deployment;
no external reproduction yet;
no real decision changed yet;
the 13 runs are a private synthetic family, not published data.

The series

This is piece 3 of a three-part line: write it down → test it → govern it.

agent-charters: what people actually tell agents.
agent-lab-trust: whether an agent run is valid and reproducible.
this audit: whether a run was allowed under a declared policy.

Index: https://github.com/janzong/agent-lab-trust/blob/main/SERIES.md

Call

Run the audit on a run family you own. Tell me where the policy and the artifacts disagree. I am also looking for 3 independent reproductions of the trust layer: https://github.com/janzong/agent-lab-trust/issues/1

We ran 2 vs 4 agents six times. Four agents cost 2.1 and did not improve success

Janz — Tue, 22 Sep 2026 21:51:04 +0000

Short version: I ran a preregistered 2-agent versus 4-agent comparison on a deterministic task.
Six live runs completed under hard caps. Both groups succeeded in exactly one of three repeats.
The 4-agent group cost 2.115× more per run and per complete success. More agents were operationally
viable; they were not better on this task.

The task

The scenario is a deterministic public-repair contribution task:

eight named participants;
each participant chooses how many repair units to contribute;
the task succeeds only if the total reaches a fixed threshold;
participant coverage is forced, so every selected agent acts;
no LLM reasoning is used for the choice; the model returns a strict JSON choice.

The point of forcing coverage was to avoid the earlier failure mode where one agent dominated every
turn and the other participants never acted.

Protocol

Item	Value
Groups	2 agents / 3 steps; 4 agents / 5 steps
Repeats	3 per group
Caps per run	150 calls / USD 0.50 / 600s
Choice model	`deepseek-flash` on a Responses API contract
Outcome	machine-decidable `outcome.json`
Validity	provenance, coverage, integer choices, arithmetic, caps, port closure, archive hashes

All six runs passed every validity gate. No parser failure, no budget breach, no port leak.

Results

Run	Agents	Total / threshold	Success	Zero contributors	Calls	Total tokens	Cost USD
2-agent r1	2	2 / 3	no	1	17	10,060	0.0269116
2-agent r2	2	3 / 3	yes	0	17	11,070	0.0313716
2-agent r3	2	0 / 3	no	2	17	10,255	0.0281116
4-agent r1	4	3 / 5	no	2	33	21,493	0.0597772
4-agent r2	4	2 / 5	no	2	33	22,679	0.0641372
4-agent r3	4	5 / 5	yes	0	33	21,459	0.0588042

Group aggregates:

Metric	2-agent	4-agent	Ratio
complete successes	1 / 3	1 / 3	1.000
mean calls	17.00	33.00	1.941
mean total tokens	10,461.67	21,877.00	2.091
mean cost USD	0.028798	0.060906	2.115
cost per complete success USD	0.0863948	0.1827186	2.115
mean zero contributors	1.000	1.333	1.333

What I take from it

More agents are not automatically better. In this task they doubled cost and left more participants contributing nothing on average.
The result is a valid negative result, not an apparatus failure. Every run completed and every artifact passed its gates.
The 4-agent group did produce one complete success, so the mechanism is not broken. It is simply not worth 2.115× the cost on this task.

What this does not show

It does not show that four agents are worse in general.
It does not transfer to another task or to a real user.
It says nothing about agent quality under human review, because no human scored these runs.
The sample is three repeats per group on one synthetic task.

Reproduce it

Two pieces are now public:

Method package: https://github.com/janzong/agent-lab-method (MIT, commit 6cc70156facb37baf23fe5fe57dad93d43502b91) — schema, synthetic GenMentor adapter, fixtures, tests;
Trust layer: https://github.com/janzong/agent-lab-trust (MIT, release v0.1.0-rc2) — local-first run validation, hashed reports, and a synthetic deletion proof.

The full protocol and the archived runs stay private. The result is intentionally boring: a negative result with caps, hashes, and every failed run preserved.

I am also looking for three independent reproductions by non-authors. The trust layer guide expects 13 passed under both TZ=UTC and TZ=Asia/Shanghai, report_hash a841b192981fd7e7, and deletion audit_hash 4f0193abbd49a0f9. If you run it and any hash differs, that is the most useful reply I can get. If you have a task where you believe more agents should win, that is the experiment I want to run next.

How common is AGENTS.md, really? I sampled GitHub: 6.2% of active repos, 1.0% of all repos

Janz — Sat, 19 Sep 2026 08:30:25 +0000

Short version: Every rate in my previous two posts had a denominator I picked myself — the 558 repos
that already had an AGENTS.md. That is a fine way to describe a corpus and a terrible way to answer
"how common is this?". So I sampled GitHub two different ways:

6.2% of active repos (pushed in the last 90 days, not a fork, not archived) contain an AGENTS.md — 51 of 817, 95% CI [4.8, 8.1]
1.0% of all public repos do — 9 of 924, 95% CI [0.5, 1.8]

Same file, same counting rule, two numbers that differ by 6×. Which one you quote depends entirely on
the question you are asking, and I had been quietly dodging that choice.

Three things I did not expect, in order of how much they changed my mind:

CLAUDE.md is at 5.4% of active repos. Statistically indistinguishable from AGENTS.md. If you assumed one format won, it has not.
93% of public repos have not been pushed in 90 days, 29% are forks, and 8.3% are completely empty. "GitHub" as a population is mostly a graveyard, which is why the stock rate is so low.
Most AGENTS.md files in the wild are tombstones. Of the 9 files my population sample found, 7 were in repos that have not been pushed in three months.

Why the denominator was missing

Here is the trap I was in. To build the corpus I searched GitHub for repos containing an AGENTS.md, then
labeled what I found. Every percentage since — "85.7% of files prohibit things", "13.6% record a
gotcha" — has a denominator of files that already exist. Those numbers are real and I stand by them,
but they cannot answer the question every reader actually has: should I write one of these?

For that you need a sample of repos drawn independently of whether they have the file. That is a
different sampling problem, and it needs two different frames, which I kept conflating:

quantity	question it answers	frame
stock rate	"is this mainstream?"	every public repo
active rate	"is this what working projects do?"	repos pushed in the last 90 days
trend	"is it spreading?"	rates by repo creation year

Mixing them produces confident nonsense, because the stock population is dominated by abandoned
one-off repos and the active population is not.

I wrote the decision rule down before running anything, so I could not move the goalposts after
seeing the number: <1% ⇒ describe it as an early-adopter curiosity; 1–5% ⇒ "early but measurable",
and every rate must be labeled as in-corpus or ecosystem-wide; >10% ⇒ "standard practice". It landed
in the middle band, slightly high — so: not a curiosity, not a standard either.

Method, briefly

Both frames use one tree call per repo —
GET /repos/{owner}/{repo}/git/trees/HEAD?recursive=1 — and check every path with a case-insensitive
match on the filename. Recursive matters: a root-only check would miss docs/AGENTS.md and friends and
under-count. Roughly 1,900 repos, all responses cached, seed 20260918.

Frame A — active rate, by creation cohort. For each year 2010–2026 I picked one random slice of
creation time, queried
created:<slice> pushed:>2026-06-20 fork:false archived:false, pulled every result (rather than
taking the top page, which is ranked by GitHub's relevance and would bias toward popular repos), then
randomly sampled 50 repos from the slice. A 2026 week contains ~62,000 active repos, which exceeds the
1,000-result search cap, so recent cohorts narrowed to a random day and then a random hour.

Frame B — stock rate, unweighted. GitHub search cannot give you a random sample of the population —
there is no random sort, and ranking favors stars and activity. So I enumerated the ID space instead:
binary-searched the current maximum repo ID (1,375,203,308), then sampled IDs uniformly and asked
GET /repositories/{id}. Only 35% of IDs correspond to an existing public repo (the rest are
deleted, private, or never existed), so 1,000 usable repos cost 2,920 probes.

Results

Active repos (Frame A, n=817):

AGENTS.md                        6.2%   [4.8, 8.1]
CLAUDE.md                        5.4%   [4.0, 7.2]
.github/copilot-instructions.md  1.1%   [0.6, 2.1]
.cursorrules / .cursor/rules     0.7%   [0.3, 1.6]

All public repos (Frame B, n=924):

all public repos                 1.0%   [0.5, 1.8]
non-fork, non-archived           0.5%   [0.2, 1.4]
AGENTS.md ∩ CLAUDE.md             18 repos
CLAUDE.md only, no AGENTS.md      26 repos

The number I got wrong twice

In my corpus, 59.1% of repos that have an AGENTS.md also have a CLAUDE.md. In the wild it is
35.3%. Both are correct; they are answered by different populations, and only one of them is
"typical".

The reason for the gap is a selection effect I should have predicted: a repo that has one agent
instruction file is already a repo whose author cares about agent tooling, so it is much more likely to
have several. My corpus is a sample of the enthusiastic end, and it over-represents multi-tool setups
by about 1.7×. If I had quoted 59.1% as a base rate, I would have been describing my sample, not the
world.

The trend measurement failed, and I am not going to dress it up

I wanted to show adoption rising by cohort and I could not measure it. In Frame A,
p_2026 / p_≤2022 = 0.67× — if anything, older active repos are more likely to have the file.

That number is not evidence that adoption is flat, because creation year and repo age are perfectly
confounded. A repo created in 2010 that is still receiving pushes in 2026 is a survivor — a project
that lived long enough to accumulate conventions. A repo created in 2026 is mostly somebody's first
weekend project. The cohort axis is really an age axis, and age predicts having-writers and having-time.

Separating those would require reading commit history to find when each file was added. I did not do
that, so the honest deliverable here is "not measured", not "not spreading".

A GitHub API gotcha worth knowing

Half a day of this project went into a bug that was not a bug. 403 from the REST API has three
distinct meanings, and you have to read the body to tell them apart:

Primary rate limit — X-RateLimit-Remaining: 0.
Secondary rate limit — burst/concurrency. GitHub's docs say it plainly: make requests for a single user serially. Three threads was enough to get me permanently throttled, while serial requests with connection reuse ran at ~8/s without complaint.
A single repo blocked by GitHub — {"message":"Repository access blocked","block":{"reason":"tos"}}.

I had classified (3) as (2), so my code kept sleeping and retrying the same blocked repo, forever. If
you write a bulk GitHub crawler, put that string in your error handling; it is stable, it is not a rate
limit, and it will silently eat your retry budget.

What this changes about my own claims

I have to soften things I said earlier:

"AGENTS.md is the emerging standard" — no. 6.2% of active repos is a real practice, not a standard. It is roughly 6–9× more common than the vendor-specific alternatives, which is a measurable reason to keep the word "format" instead of a vendor name.
"CLAUDE.md is just a companion file" — that was a conditional rate described as if it were a base rate. Unconditionally, the two are neck and neck.
"adoption is spreading" — unmeasured, see above.

What survives: it is a habit of active projects (6.2%) rather than something the population does
(1.0%). Those two sentences imply completely different advice, and I could not tell them apart until I
sampled for it.

The experiment I have not run

There is still a hole underneath all of this, and it is the one that matters: does any of it work?
Every number in this project — mine and everyone else's — is descriptive. Nobody has shown that a repo
with an AGENTS.md produces better outcomes than the same repo without one, because that requires a
controlled task, a blind judge, and an effect size, not a sample.

I wrote the design for that experiment (three arms, including a placebo arm that gets an equal-length
unrelated document, so "more context" and "this document" can be told apart) but I deliberately did not
run it yet. The sample sizes are brutal and the honest outcome is probably "we could not detect it".

If you have actually noticed a charter changing a decision — a rule that stopped you from doing
something you would otherwise have done — that is the data I cannot generate myself, and it is worth
more to me than a star.

Reproduce it

pip install agent-charters              # the CLI
# the sampling code is in the repo, not the package:
git clone https://github.com/janzong/agent-charters
cd agent-charters
.venv/bin/python work/prevalence.py active --per-gen 50   # ~10 min, search-rate-limited
.venv/bin/python work/prevalence.py report
.venv/bin/python work/prevalence.py stock  --n 1000       # ~40 min
.venv/bin/python work/prevalence.py report-stock

Seed 20260918, every response cached, so report is instant and costs nothing after the first run.
The full write-up with every caveat I could think of — including the two frames disagreeing (6.2% vs
1.8% on a 57-repo active sub-sample; the honest answer is a range of roughly 2–6%) — is in
work/audit/prevalence.md in the repo.

Repo: https://github.com/janzong/agent-charters

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

Janz — Mon, 14 Sep 2026 13:25:06 +0000

Short version: I labeled 558 public AGENTS.md files against a 9-category taxonomy. The measured
base rates say something boring and useful — almost every file prohibits things (85.7%) and lists
build/test commands (82.8%), while almost none of them record a gotcha (13.6%). Two of the nine
slots are nearly empty across the whole corpus: gotchas and agent_meta (rules about the agent itself,
25.8%).

Then I ran the same ruler over two big, well-maintained files. Both missed gotchas. So here is a
five-minute check you can run on your own file, and the exact numbers behind it.

The base rates

Measured on 516 substantive files (558 collected, the rest were one-line pointers):

boundaries     85.7%   what must never be done
build_test     82.8%   the commands CI runs
workflow       67.1%   commit format, branches, release steps
structure      59.1%   layout, where new code belongs
style          54.5%   naming, formatting — or a pointer to the config that enforces it
environment    45.0%   toolchain versions, required env vars
overview       32.2%   one paragraph: what this is, what it deliberately is not
agent_meta     25.8%   rules about the agent: tone, when to ask first
gotchas        13.6%   pitfalls that are NOT derivable from the code

The shape is not surprising once you see it as a genre: an AGENTS.md is usually written defensively,
as a list of things not to break. The file that would actually save you time is the one almost nobody
writes.

Two receipts

compare prints your file's coverage next to the corpus baseline. Two real examples from the corpus:

file	size	sections	coverage	missing
`langchain-ai/deepagents`	10 KB	22	7/9	`overview`, `gotchas`
`openai/openai-agents-python`	34 KB	27	6/9	`style`, `agent_meta`, `gotchas`

Both are good files. The 34 KB one is one of the more thorough agent-instruction files in the corpus —
27 sections, 19 separate boundary markers. It still has nothing in it that you could only learn by
running the thing.

Why gotchas are rare (and why that is not laziness)

You can only write a gotcha after being bitten by it — and by the time you have been bitten, the
temptation is to fix the thing rather than write the sentence down. The fix is visible in the code;
the sentence is a liability nobody wants to maintain.

There is a second, worse failure mode. I sampled 347 entries from the Gotchas / Common Pitfalls /
Troubleshooting sections in the corpus (an earlier snapshot, 507 files) and hand-labeled 120 of them:

58% are readable from the repo itself (interface contracts, platform limits, build requirements)
34% are not pitfalls at all — they are generic advice ("remember to install dependencies", "don't commit .env"), the same sentence you would write for any project
8% are genuinely experience-only: upstream/third-party behaviour, past incidents, and the places where the docs disagree with the code

So the section is rare, and a third of what does live there is filler. The 8% is the part worth
handing to an agent, and it cannot be generated from a reading of the repository. It has to come from
a person who was there.

The five-minute check

No tool needed. Ask these five questions about your own file:

Are the commands copy-pasteable? Not "run the tests" — the actual command CI runs, with the working directory. If your README says one port and production uses another, say so (that mistake is in the corpus, in a file that otherwise looks complete).
Does it name what must never be committed or never touched? This is the one thing the corpus does well (85.7%) — check that yours names the tempting case, not the obvious one. "Don't commit secrets" is obvious; "don't hand-edit the production database to fix a row, use the backfill script" is a boundary that will actually stop someone.
Does it say anything about the agent's own behaviour? Only 25.8% do. Tone, when to stop and ask, which actions need explicit approval, what must not leave the machine.
Is there at least one sentence that is not derivable from the code? If every line in your file could have been written by reading the repo, the file is documentation, not a charter. This is the gotcha test.
Do the paths it points at exist? Measured: 49% of files route to another file, and 15% point at a knowledge store or rules directory. A pointer to a file that moved is worse than no pointer — an agent will go looking, and will read whatever it finds there as authoritative.

If you want the baseline instead of the feeling

pip install agent-charters                 # 0.4.1, on PyPI — no runtime deps, ~110 KB wheel
agent-charters compare path/to/AGENTS.md   # coverage vs the 558-file baseline, plus gaps
agent-charters brief                       # the checklist + a paste-ready prompt
agent-charters refs path/to/AGENTS.md      # external pointers and dangling references

(Updated 2026-09-19 — when this was first posted the only way in was a clone, and this block said so.
It is on PyPI now: verified in a clean virtualenv on a default index, install takes a couple of
seconds, because the corpus ships as gzipped JSONL and pandas / pyarrow are optional extras.
Source, if you would rather read or pin it: github.com/janzong/agent-charters — CN mirror
gitee.com/janzong/agent-charters.)

compare is the one that answers question 4 in aggregate. It also does something I did not expect:
when I used brief's prompt to write a charter for a real project, compare flagged coverage I had
skipped — and one of the nine slots it missed was the name of the slot itself, which is a bug in my
taxonomy, not in the file. That is the kind of thing a rule-based labeler gives you: you can point at
the pattern that fired and argue with it.

There is no LLM in the labeling loop. Every label is recomputable and arguable, which is the point —
if you disagree with a label, you can find the rule that produced it and overrule it.

What this is not

I do not want to oversell the numbers, so:

The classifier scores 92% precision / 70% recall on a 55-file held-out English set, and 88% / 73% on 50 held-out Chinese files. The recall number is the honest one: it misses roughly three in ten of the labels it should have produced. gotchas and agent_meta are the weakest slots in both languages.
The held-out sets were labeled by one person (me). No second annotator, no inter-annotator agreement.
Coverage is a process metric, not a quality metric. In a 3-repo test, a checklist that names all nine slots pushed a generator from 4–5 categories to 9/9 — and filling all nine slots is not the same as writing a good file. It is a prompt for the questions, not a grade.
The labels and the rates come from public files and a rule-based classifier, not from a language model. The one LLM in this story is the generator in the 3-repo test, which is why that number is reported as n=3.

The ask

The weakest part of this project is that the only person who has ever tested it is its author. If you
have an AGENTS.md (or a CLAUDE.md, or a .cursorrules) on a real project, run compare on it and
tell me what it gets wrong — the file, the label, or the baseline rate. A wrong label on your file is
worth more to me than a star.

Repo: https://github.com/janzong/agent-charters

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Janz — Mon, 14 Sep 2026 12:23:13 +0000

TL;DR — I collected 558 AGENTS.md files from public repos and labeled each one against a 9-category
taxonomy with a rule-based classifier (no LLM in the loop, so it is auditable and recomputable). Then I
blind-labeled held-out samples and compared: 92% precision / 70% recall on 55 English files,
88% / 73% on 50 Chinese files. The most common categories are prohibitions (85.7%) and build/test
commands (82.8%). The rarest: gotchas (13.6%) and instructions about how the agent itself should behave
(25.8%).

Why bother

Almost every discussion about AGENTS.md is anecdote-led: my repo's file works, my agent ignores it,
a good one is a model upgrade, a bad one is worse than nothing. All of that may be true — but nobody
seems to have the distribution. So I built it: snapshot of 558 files from 558 public repos
(2026-09-10, 5.3 MB, 516 usable for statistics), labeled, versioned, and published with the tooling.

Method, in one paragraph

Nine categories: boundaries, build_test, workflow, structure, style, environment, overview,
agent_meta (rules about the AI itself), gotchas. Labeling is done by pattern rules over headings and
body text — deliberately, because a rule set can be read, argued with, and re-run, and every number below
can be recomputed from the released dataset. I then measured how well the rules match a human reading:
100 files in-sample (upper bound, 90%/75%) and two held-out sets I had never tuned against —
55 English (92%/70%) and 50 Chinese (88%/73%). Held-out numbers use the conservative reading
(items I was unsure about count as classifier errors).

Five things the numbers say

1. Two categories dominate — and they are tied

category	share of 516 files
`boundaries` (what you must never do)	85.7%
`build_test` (install/build/test/CI commands)	82.8%
`workflow` (branching, commits, review, release)	67.1%
`structure`	59.1%
`style`	54.5%
`environment`	45.0%
`overview`	32.2%
`agent_meta`	25.8%
`gotchas`	13.6%

The 2.9 pp gap between the top two is smaller than the known false-positive rate (~3%) of the
prohibition pattern — so the honest statement is tied for first, not "prohibitions beat build commands".

2. Nobody writes down their scars

gotchas is dead last at 13.6%. Worse: when people do open a "known issues" section, a third of it
isn't a gotcha. I hand-read 120 items from those sections:

58% were readable straight from the repo (config, code, README mismatch),
34% were not gotchas at all (generic advice: "remember to install dependencies"),
8% needed experience or the outside world (OS behavior, an upstream outage, yesterday's incident).

That 8% is the part an agent can never derive from the code — and it is exactly the part that is
almost never written down.

3. The slot language models skip is `workflow`

In a controlled experiment (11 repos × 3 prompt styles), prompts that listed topics explicitly produced
9/9 categories, while prompts that left the slots implicit skipped workflow in 11 out of 11 files.
Point at workflow by name and it appears 3/3 times, with real content. The gap is not knowledge,
it is questions — which is why I turned the corpus distribution into a checklist tool.

4. Your weakest category depends on the language

English files fail differently from Chinese ones. English: gotchas recall 32–38% — the classifier
misses casual "watch out" prose. Chinese: agent_meta recall 26% — Chinese files express agent rules
in the second person ("you are the dispatcher, not the executor"), and the body-pattern rules for that
category are entirely English, so the whole style is invisible to them. File-level exact agreement
(9/9 categories identical) is 12% in both languages.

5. Half of these files are entry points, not documentation

49% point to some other file; 15% route to a knowledge or rules directory. That's a structural
fact about the format, and it means "does this repo have an AGENTS.md?" is a much weaker question than
"what is actually in it".

The tool

pip install agent-charters

agent-charters brief      # checklist of the 9 slots + a paste-ready prompt
agent-charters compare your-AGENTS.md   # your coverage vs the 558-file baseline
agent-charters refs your-AGENTS.md      # does your file point at paths that exist

Honest note: compare is a checklist, not an oracle. It warned me that one of the nine categories was
missing from a file I wrote myself — it was actually present, but the heading used the tool's own slot name
instead of natural language. That is documented in the repo (along with the exact experiment) rather than
quietly patched, because a tool that tells you "you're missing X" should be checked by a human.

What this is not

Rule-based labels, not per-file human labels. Precision/recall above are the honest measures; the per-category numbers in the dataset carry that error.
Not a representative sample of GitHub. Repos were found through AI/agent topics and Chinese keyword search; the Chinese set came out 97% Chinese by construction, which says nothing about GitHub's language mix.
Single annotator. The blind labeling was done by one model-driven annotator, not multiple raters.
A snapshot, not a trend. A baseline of file hashes is stored so that a future re-crawl can measure how these files change — that measurement doesn't exist yet.

The ask

I'm looking for 2–3 people who are not me to run compare on an AGENTS.md they actually maintain and
tell me where it's wrong — missing a category you clearly have, or claiming one you don't. That is the one
piece of evidence this project doesn't have yet: an external user. Issues and comments are both fine.

DEV Community: Janz

Audit your UI before you redesign it: baseline first, then compare

The method: baseline, compare, classify

What the audit caught: 8 deviations in 6 pages

The color rule that fell out

The discipline part

The connection to agent governance

Takeaway

Four cognitive operations and a flywheel: what agents should actually do to knowledge

The four cognitive operations

The flywheel

Each step feeds the next:

The gate question

Why "agent as editor, not author" matters

The measurement connection

What to take

From shadow to gate: how we fixed agent citation fraud in one production batch

I preregistered 30 real agent tasks. Three rules didn't survive contact.

Your agent run passed. Can you prove it was allowed?

The gap

The demonstration

Reproduce

What this does not prove

The series

Call

We ran 2 vs 4 agents six times. Four agents cost 2.1 and did not improve success

The task

Protocol

Results

What I take from it

What this does not show

Reproduce it

How common is AGENTS.md, really? I sampled GitHub: 6.2% of active repos, 1.0% of all repos

Why the denominator was missing

Method, briefly

Results

The number I got wrong twice

The trend measurement failed, and I am not going to dress it up

A GitHub API gotcha worth knowing

What this changes about my own claims

The experiment I have not run

Reproduce it

I looked at 558 AGENTS.md files: here's a 5-minute check for yours

The base rates

Two receipts

Why gotchas are rare (and why that is not laziness)

The five-minute check

If you want the baseline instead of the feeling

What this is not

The ask

I labeled 558 AGENTS.md files. Here's what they say — and what almost nobody writes down

Why bother

Method, in one paragraph

Five things the numbers say

1. Two categories dominate — and they are tied

2. Nobody writes down their scars

3. The slot language models skip is workflow

4. Your weakest category depends on the language

5. Half of these files are entry points, not documentation

The tool

What this is not

Links

The ask

3. The slot language models skip is `workflow`