DEV Community: Reporails

The State of AI Instruction Quality

Gábor Mészáros — Tue, 21 Apr 2026 12:41:52 +0000

Everybody has opinions about AGENTS.md/CLAUDE.md files.

Best practices get shared. Templates get copied, and this folk-type knowledge dominates the industry. Last year, GitHub analyzed 2,500 repos and published best-practice advice. We wanted to go further: measure at scale, publish the data, and let anyone verify.

When the agent doesn't follow instructions and does something contradictory, the usual suspects are: the model is inconsistent, LLMs are not deterministic, you need better guardrails, you need retries.

The failures almost always get attributed to the model.

So we decided to measure. We built a diagnostic tool that treats instruction files as structured objects with measurable properties. Deterministic. Reproducible. No LLM-as-judge. Then we pointed it at GitHub repositories with instruction files for five agents - Claude, Codex, Copilot, Cursor, and Gemini.

28,721 repositories. 165,063 files. 3.3 million instructions.

... and one question:

What if the instructions are the problem?

The dataset

28,721 projects. Sourced from GitHub via API search, cloned, and deterministically analyzed. Each project was scanned for instruction files across five coding agents — then deduplicated to remove false positives from agent detection overlap.

Agent	Projects	% of corpus
Claude	12,356	43.0%
Codex	11,206	39.0%
Copilot	7,755	27.0%
Cursor	7,291	25.4%
Gemini	5,942	20.7%

The percentages add up to more than 100% because 37% of projects configure multiple agents. More on that later.

Key distributions stabilized early. A 9,582-repo sub-sample produced identical tier shares (±0.2pp) and the same mean scores as the full 12,076-repo intermediate sample. The final 28,721-repo corpus moved nothing. The patterns reported below are not small-sample artifacts.

All classifications are deterministic — the same file produces the same result every time. No LLM-as-judge. Sample classifications are published for inspection (methodology below). The tool is source-available.

How we measured

The analyzer parses each instruction file into atoms — the smallest semantically distinct units of content. A heading is one atom. A bullet point is one atom. A paragraph is one atom. Each atom gets classified along a few dimensions, all deterministic, no LLM involved:

Charge classification. A three-phase pipeline determines whether an atom is a directive ("use X"), a constraint ("do not use Y"), neutral content (context, explanation, structure), or ambiguous (could be read either way). Phase 1 detects negation and prohibition patterns. Phase 2 detects modal auxiliaries and direct commands. Phase 3 uses syntactic dependency parsing to catch imperatives that the first two phases missed. First definitive match wins. Atoms that partially match but don't clear any phase are marked ambiguous. Everything else is neutral.

Specificity. Binary: does the instruction name a specific construct — a tool, file, command, flag, function, or config key — or does it stay at the category level? "Use consistent formatting" is abstract. "Format with ruff format" is named. This is a text property, not a judgment call.

File categorization. Each file is classified as base config (your main CLAUDE.md or .cursorrules), a rule file, a skill definition, or a sub-agent definition — based on file path conventions for each agent.

Content type. Charge classification separates behavioral content (directives and constraints) from structural content (headings, context paragraphs, examples). That's how we know what fraction of your file is actually doing work.

The full tool is source-available (BUSL-1.1). You can run npx @reporails/cli check on your own project and inspect every finding. More on that at the end.

Finding 1: Most of your instruction file isn't instructions

Here's what the median instruction file actually contains:

50 content items total
12 of those are actual directives
The rest is headings, context paragraphs, examples, structure

Only 27% of your instruction file is doing what you think it does.

The other 73% is scaffolding. Headings that organize but don't instruct. Explanation paragraphs that compete for the model's attention without adding behavioral weight. Example blocks. Context-setting prose.

That's not inherently bad. Structure matters. But if you're writing a 200-line CLAUDE.md and only 54 lines are actual instructions, you should probably know that.

The average instruction is 8.9 words long. That's a sentence fragment.

Finding 2: 90% of instructions don't name what they're talking about

This is the big one.

We measured whether each instruction references specific tools, files, commands, or constructs by name — or whether it stays at the category level.

Two-thirds of all instructions are abstract.

Agent	Names specific constructs	Uses category language
Gemini	39.3%	60.7%
Codex	38.3%	61.7%
Copilot	33.3%	66.7%
Cursor	30.8%	69.2%
Claude	30.6%	69.4%

What does this look like in practice?

Abstract: "Use consistent code formatting"
Specific: "Format with ruff format before committing"

Abstract: "Avoid using mocks in tests"
Specific: "Do not use unittest.mock — use the real database via test_db fixture"

In previous controlled experiments, specificity produced a 10.9x odds ratio in compliance (N=1000, p<10⁻³⁰). The instruction that names the exact construct gets followed. The one that describes it abstractly... mostly doesn't. This is consistent with independent findings from RuleArena (Zhou et al., ACL 2025), where LLMs struggled systematically with complex rule-following tasks — even strong models fail when the rules themselves are ambiguous or underspecified.

89.9% of all agent configurations contain at least one instruction that doesn't name what it means. It's not a few projects. It's nearly everyone.

Finding 3: `agents.md` is the most common instruction file

Before we get into quality, let's look at what people are actually naming their files:

#	File	Count
1	`agents.md` / `AGENTS.md`	20,654
2	`claude.md` / `CLAUDE.md`	14,014
3	`gemini.md` / `GEMINI.md`	5,703
4	`.github/copilot-instructions.md`	5,647
5	`.cursorrules`	2,415

49,071 unique file paths across the corpus. That's not a typo. The format fragmentation is real.

A few things jumped out:

claude.md (lowercase, 10,642) is 3x more common than CLAUDE.md (3,372). Both work. The community clearly prefers lowercase.
agents.md dominates — the Codex/generic format is the single most popular instruction file name.
Skills and rules are already showing up in meaningful numbers: .claude/rules/testing.md (422), .agents/skills/tailwindcss-development/skill.md (334).

Finding 4: Different agents, completely different config philosophies

Not all agents are configured the same way. Not even close.

We categorized every file into four types: base config (your main CLAUDE.md, .cursorrules, etc.), rules (scoped rule files), skills (task-specific skill definitions), and sub-agents (role-based agent definitions).

Agent	Base	Rules	Skills	Sub-agents	Total files
Claude	18,733	4,638	10,692	10,538	44,601
Cursor	5,903	19,843	6,237	1,716	33,699
Copilot	16,026	4,486	10,352	3,012	33,876
Codex	19,001	81	8,911	165	28,158
Gemini	10,253	74	3,039	53	13,419

Cursor is 60% rules files. The .cursor/rules/ system dominates its configuration surface. One agent's config looks nothing like another's.

Claude is the only agent with a roughly balanced architecture across all four config types. Codex and Gemini are almost entirely base config — single-file setups.

The median Cursor project has 3 instruction files. The median Codex project has 1. These aren't just different tools. They're different configuration philosophies.

Finding 5: 37% of projects configure multiple agents

10,620 projects in the corpus target two or more agents. That's not a niche pattern — it's over a third of all projects.

Agents	Projects
1	18,101
2	6,776
3	2,687
4	949
5	208

The dominant pair is Claude + Codex (5,038 projects). Makes sense — CLAUDE.md + AGENTS.md is the most natural multi-agent starting point.

Here's what's interesting about multi-agent repos: the same developer, writing instructions at the same time, for the same project, produces measurably different instruction quality across agents. The person didn't change. The project didn't change. The instruction format did.

Some of that is structural. Cursor's .mdc rules enforce a different format than Claude's markdown. Codex's AGENTS.md invites a different writing style than Copilot's copilot-instructions.md. The format shapes the content.

Finding 6: The most-copied skills are the vaguest

This is where it gets interesting.

13,309 unique skills across the corpus. Some of them appear in hundreds of repos — clearly copied from shared templates or community sources. So we measured them.

Named% = what fraction of a skill's instructions name a specific tool, file, or command (instead of using category language).

Skill	Repos	Named%	What it means
`frontend-design`	271	2.8%	Almost entirely abstract advice
`web-design-guidelines`	197	10.2%	Generic design principles
`vercel-react-best-practices`	315	30.7%	Mix of specific and vague
`pest-testing`	216	55.1%	Names actual test constructs
`livewire-development`	87	75.5%	Names specific Livewire components
`next-best-practices`	76	92.6%	Names almost everything

frontend-design is in 271 repos with 2.8% specificity. It's a wall of "follow responsive design principles" and "ensure accessibility compliance." That reads well. It sounds professional. It gives the model almost nothing concrete to act on.

next-best-practices is in 76 repos with 92.6% specificity. It says things like "use next/image for all images" and "prefer server components over client." It reads like a checklist. It tells the model exactly what to do.

One is shared 3.5x more than the other.

The most popular skills are the most decorative. The well-written ones barely spread.

The best and worst skills (>50 repos)

Most specific:

Skill	Repos	Named%
`next-best-practices`	76	92.6%
`shadcn`	74	82.6%
`livewire-development`	87	75.5%
`pest-testing`	216	55.1%
`laravel-best-practices`	94	49.7%

Most vague:

Skill	Repos	Named%
`openspec-explore`	110	2.5%
`frontend-design`	271	2.8%
`web-design-guidelines`	197	10.2%
`vercel-composition-patterns`	131	10.7%
`find-skills`	113	18.9%

Notice a pattern? The Laravel/Livewire ecosystem produces specific skills. The generic frontend/design ones stay abstract. Domain-specific communities write better instructions than cross-cutting ones.

Finding 7: Sub-agents are almost entirely persona prompts

5,526 unique sub-agent roles in the corpus. Developers are building agent teams: code reviewers, architects, debuggers, testers, security auditors.

The problem? Sub-agents are the most abstract config type in the entire corpus. Only 17% of sub-agent instructions name specific constructs.

Role	Repos	Named%
`code-reviewer.md`	236	14.4%
`architect.md`	89	18.2%
`debugger.md`	66	9.4%
`security-auditor.md`	57	14.8%
`test-runner.md`	54	10.5%
`frontend-developer.md`	47	9.0%

Most of these are persona prompts. "You are a senior code reviewer. You care about code quality, security, and maintainability." That's a role description, not an instruction set. It tells the model who to be, not what to do.

Compare this to a base config that says "run uv run pytest tests/ -v before suggesting any commit" — that's 100% named, and the model knows exactly what action to take.

The anatomy chart: more directives, worse quality

Here's where it all comes together.

We measured three things for each config type: how big the files are, how many directives they contain, and what fraction of those directives actually name something specific.

Sub-agents have the largest files (61 items median), the most directives (17), and the worst specificity (17%). They're the wordiest config type in the corpus and the least effective.

Base configs are the opposite. Fewer directives (11), but 40% of them name specific constructs. The developer writing their own CLAUDE.md by hand, for their own project, produces the most actionable instructions.

Config type	Files	Median size	Median directives	Specificity
Base configs	69,916	50 items	11	39.8%
Rules files	29,122	34 items	9	31.2%
Skills	39,231	59 items	14	30.8%
Sub-agents	15,484	61 items	17	17.0%

The pattern is clear: what developers write by hand is the most specific. What gets templated and shared gets progressively vaguer. And what tries hardest to sound authoritative — sub-agent persona prompts — is the most hollow.

More instructions is not better instructions.

Independent research supports the structural angle: FlowBench (Xiao et al., 2024) found that presenting workflow knowledge in structured formats (flowcharts, numbered steps) improved LLM agent planning by 5-6 percentage points over prose — across GPT-4o, GPT-4-Turbo, and GPT-3.5-Turbo. Structure is not decoration. It changes what the model retrieves.

Limitations

Five things to know about these numbers.

Sampling bias. GitHub API search, public repos only, English-skewed. Enterprise configurations, private repos, and non-English projects are not represented. This is not a random sample of all instruction files in production.

Classification accuracy. The charge classifier is deterministic but not perfect. Edge cases exist: mixed-charge sentences, implicit constructs, domain jargon that looks like a category term but is actually a named tool. Specificity detection (named vs abstract) is simpler and more robust. Sample classifications are published for inspection.

Association, not causation. "More directives correlate with lower specificity" is an observed pattern. We do not claim that adding directives causes quality to drop.

Snapshot. Collected March–April 2026. Instruction practices are changing fast — agents.md didn't exist six months ago. These numbers describe the ecosystem at collection time.

No popularity weighting. A 10-star hobby project counts the same as a 50K-star production repo. The distribution of instruction quality in production agent work may differ.

What this means

This isn't an article about AI models being bad at following instructions. The models are fine.

This is an article about what we actually give them to work with.

Most instruction files are three-quarters scaffolding. Two-thirds of the actual instructions don't name what they're talking about. The most popular community skills are the most decorative. Sub-agent definitions are the wordiest files in the corpus and the least specific.

None of that is obvious from reading your own files. It wasn't obvious to us before we measured it. A well-structured CLAUDE.md feels thorough. A shared skill with 271 repos feels battle-tested. A sub-agent with 17 directives feels comprehensive.

Measurement shows something different.

In The Undiagnosed Input Problem, I argued that the industry is great at inspecting outputs and weak at inspecting inputs. This corpus analysis is the evidence for that claim.

The instruction files are there. The developers wrote them. They just have no way to know which parts are working and which parts are wallpaper.

Try it yourself

The analyzer we used for this corpus analysis is available as a CLI you can run against your own instruction files.

Reporails — instruction diagnostics for coding agents. Deterministic. No LLM-as-judge. 97 rules across structure, content, efficiency, maintenance, and governance.

npx @reporails/cli check

That scans your project, detects which agents are configured, and reports findings with specific line numbers and rule IDs. Here's what the output looks like:

Reporails — Diagnostics

  ┌─ Main (1)
  │ CLAUDE.md
  │   ⚠       Missing directory layout             CORE:C:0035
  │   ⚠ L9    7 of 7 instruction(s) lack reinfor…  CORE:C:0053
  │     ... and 16 more
  │
  └─ 21 findings

  Score: 7.9 / 10  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░

  21 findings · 4 warnings · 1 info
  Compliance: HIGH

The corpus analysis used the same classification pipeline at scale. Fix the findings, run again, watch your score improve.

The dataset

The full corpus is published at reporails/30k-corpus. Three files:

File	Records	What it contains
`repos.jsonl`	28,721	Per-project record: agents configured, stars, language, license, topics
`stats_public.json`	1	Every aggregate statistic in this article
`validation_key.csv`	2,814	Sample classifications with source text for inspection

Verify any claim:

# "28,721 repositories"
cat repos.jsonl | wc -l

# "43% Claude"
cat repos.jsonl | python3 -c "
import sys, json
repos = [json.loads(l) for l in sys.stdin]
claude = sum(1 for r in repos if 'claude' in r['canonical_agents'])
print(f'{claude}/{len(repos)} = {claude/len(repos)*100:.1f}%')
"

Every number in every table traces to that dataset. If you disagree with a finding, count the rows.

This is part of the Instruction Quality series. Previous: The Undiagnosed Input Problem. Related: Precision Beats Clarity · Do Not Think of a Pink Elephant · 7 Formatting Rules for the Machine.

The Undiagnosed Input Problem

Gábor Mészáros — Wed, 08 Apr 2026 11:51:12 +0000

The AI agent ecosystem has built a serious industry around controlling outputs. Guardrails. Safety classifiers. Output validation. Monitoring. Retry systems. Human review.

All of that matters, but there is simpler upstream question that still goes mostly unmeasured:

Are the instructions any good?

That sounds obvious, yet it is not how the industry behaves.

When an agent fails to follow instructions, the usual explanations come fast:

Models are probabilistic
Agents are inconsistent
You need stronger guardrails
You need better monitoring
You need retries
You need humans in the loop

… and while those explanations are right to a certain degree, they also have a side effect: they turn instruction quality into a blind spot.

The ecosystem has become extremely good at inspecting what comes out of the model, and surprisingly weak at inspecting what goes in.

The symptom

Consider τ-bench.

It gives agents policy instructions and measures whether they follow them in realistic customer-service tasks. Airline and retail workflows. Real constraints. Real multi-step behavior.

The benchmark result that gets repeated is the model result: even strong systems still fail a large share of tasks, and consistency across repeated attempts remains weak.

The conclusion most people draw is straightforward: we need better models, better agents, better orchestration.

My take: Maybe.

But there is another question sitting underneath the benchmark:

Were the instructions themselves well-formed and well structured?

Not just present. Not just long enough. Not just sincere.

Well-formed. Well-structured. Well-organized.

Specific enough to anchor behavior. Structured enough to survive context mixing. Non-conflicting across files. Positioned where the model can actually use them.

Those questions usually never gets asked.

The industry response

I had a conversation recently where a lead solutions architect put the standard view plainly:

“The instruction merely influences the probability distribution over outputs. It doesn’t override it.”

That is right about the mechanism but it is wrong about what follows from it.

Yes, instructions operate probabilistically. But that does not mean all instructions are weak in the same way.

The shape of the distribution is not fixed. It changes with the properties of the instruction itself. Specificity sharpens it. Structure sharpens it. Conflict flattens it. Vague abstractions flatten it. Bad formatting can suppress it almost entirely.

Across my earlier controlled experiments, small changes in wording and placement produced large changes in compliance:

Instruction ordering moved compliance by 25 percentage points with the same model and the same directive.
Specificity produced roughly a 10x compliance effect when the instruction named the exact construct instead of describing it abstractly.
Formatting changed whether the model reliably registered the instruction at all.

The problem is that most instruction systems are built without diagnostics.

That is not an AI limitation. That is an engineering failure.

The folk system

Right now, instruction practice spreads mostly through imitation.

A popular repository posts “best practices” for Claude Code. Shared Cursor rules circulate as templates. People copy AGENTS.md files between projects. Teams accumulate CLAUDE.md, .cursorrules, copilot-instructions.md, etc and project-specific rule files across multiple tools.

Copy, paste, hope, repeat.

Some of that advice is useful. Almost none of it is tested in any controlled, reproducible way. That would be fine if instruction quality were self-evident. It is not.

A long instruction file can feel thorough while being internally contradictory. A highly opinionated ruleset can feel disciplined while producing almost no behavioral influence on the model.

A sprawling multi-file setup can look sophisticated while making the system worse.

Without diagnostics, developers do not know which instructions are binding, which are noise, and which are actively interfering with each other.

The gap

The tooling split is now pretty clear.

Output tooling is mature. Guardrails AI validates structure. Lakera focuses on prompt injection and security. NeMo Guardrails enforces safety and conversational rails. Llama Guard classifies risky content. The output edge is crowded.

Prompt testing is real. Promptfoo, Braintrust, and LangSmith can all help evaluate behavior. But they are primarily black-box systems: did the prompt produce the output you wanted?

That is useful.

It is not the same as measuring the instruction artifact itself.

Instruction-quality tooling exists only in fragments. Some tools use LLM-as-judge. Some use deterministic local rules. But the category is still early, inconsistent, and mostly disconnected from measured behavioral outcomes.

What is still largely missing is a deterministic way to inspect instruction files as engineered objects:

how specific they are
how directly they state intent
whether they conflict across files
whether they overuse headings
whether they provide alternatives instead of bare prohibitions
whether the system is getting denser while getting weaker

Code gets static analysis.

Instruction systems usually get vibes.

What we measured

We built an analyzer that treats instruction files as structured objects with measurable properties. Deterministic. Reproducible. No LLM-as-judge.

I am running it across a large live corpus of real repositories. The full run completes this week; what follows is what the partial sample already shows - stable enough to publish, not yet the full picture.

Quality is reported on a 0-to-100 scale: 0 means the file produces no measurable influence on model behavior, 100 is the ceiling the framework can score.

A fresh aggregation over 12,076 completed instruction-file scans is virtually identical to an earlier 9,582-repo sample:

bottom tier: 40.3% vs 40.1%
top tier: 12.1% vs 12.2%
mean quality score: 27 vs 27
directive content ratio: 27.9% vs 27.9% - the share of instruction sentences that directly tell the model what to do

That matters because it means the pattern is stable.

This does not look like a small-sample artifact.

And the strongest finding is not what I expected.

More rules, lower quality

The common response to bad agent behavior is to add more rules.

More files. More guidance. More scoping. More edge-case coverage.

The corpus says that strategy tends to backfire.

Across 12,076 repositories, instruction quality falls as instruction-file count rises:

Files per repo     N      Mean score   Bottom tier %   Top tier %
1                  4681   28           46.3%           16.9%
2-5                4796   26           37.3%            9.5%
6-20               1972   26           36.0%            8.8%
21-50               438   25           31.3%            5.7%
51-500              186   25           33.3%            5.4%

The key number is the top-tier share.

It collapses from 16.9% in single-file setups to 5.4% in repositories with 51 to 500 instruction files.

That is a roughly 3x drop.

The article version of that finding is simple:

Developers respond to bad agent behavior by adding more rules. In the corpus, that strategy correlates with a 3x collapse in the probability of landing in the top tier.

That does not prove file count causes low quality by itself.

But it does show that rule proliferation is not rescuing these systems. At scale, it is associated with weaker instruction quality, not stronger.

The sweet spot

There is also a more subtle result in the partial sample. Instruction quality appears to be non-monotonic in directive density: more directives help at first, then stop helping, and past a point start to hurt.

The full curve is in next week’s piece. The short version is that there is an optimal density range, after which additional directives stop strengthening the system.

Enough force to bind behavior. Not so much that the system turns into an overpacked rules document.

A real example

Here is the kind of instruction block the corpus is full of:

# Code should be clear, well documented, clear PHPDocs.

# Code must meet SOLID DRY KISS principles.

# Should be compatible with PSR standards when it need.

# Take care about performance

It is not malicious. It is not absurd.

It is just weak.

Everything is abstract. Nothing is anchored. Headings are doing the work prose should do. The agent can read it, represent it, and still walk past most of it.

Now compare:

Never use `var_dump()` or `dd()` in committed code. Use `Log::debug()` instead.
Run `./vendor/bin/phpstan analyse src/` before every commit. Level 6 minimum.

Same general intent. Completely different binding strength.

The second version names the construct, names the alternative, names the command, and names the threshold. It gives the model something concrete to hold onto.

That is what diagnostics should make visible.

What this means

Output guardrails still matter.

Prompt evaluation still matters.

Safety systems still matter.

But they do not answer the upstream question: Are the instructions themselves well-formed?

If the answer is no, then a large class of downstream failures will keep showing up as mysterious agent unreliability when the real problem is earlier and simpler.

The agent loaded the instruction and walked past it.

That is often not a model problem.

It is an input problem.

And input quality is measurable.

What’s next

These are corpus-level findings from a partial sample, not universal laws.

The sample is still in flight. The strongest claims here are about association, not proof of causality. Specific conflict-count case studies need source verification before publication. Popularity weighting is not yet applied, so “40% of repositories score in the bottom tier” is not the same claim as “40% of production agent work scores in the bottom tier.”

The full corpus run completes this week. Next week I publish the end-of-run analysis across the full sample — the complete distribution, the cross-cuts the partial sample cannot yet support, and the specific case studies this article deliberately held back. If you want to know where your stack lands, that is the piece to come back for.

For now, the central pattern is already stable enough to matter:

The ecosystem keeps responding to weak agent behavior by adding more instructions, while the corpus shows that more instruction files are usually associated with lower measured quality.

That is the undiagnosed input problem.

Not that instructions do not matter.
That they matter, measurably, and most teams still have no way to see whether theirs are helping or hurting.

This is part of the Instruction Best Practices series. Previous: Do NOT Think of a Pink Elephant, Precision Beats Clarity, 7 Formatting Rules for the Machine. I’m building instruction diagnostics for coding agents. Follow for the full corpus analysis.

DEV Community: Reporails

The State of AI Instruction Quality

The dataset

How we measured

Finding 1: Most of your instruction file isn't instructions

Finding 2: 90% of instructions don't name what they're talking about

Finding 3: agents.md is the most common instruction file

Finding 4: Different agents, completely different config philosophies

Finding 5: 37% of projects configure multiple agents

Finding 6: The most-copied skills are the vaguest

The best and worst skills (>50 repos)

Finding 7: Sub-agents are almost entirely persona prompts

The anatomy chart: more directives, worse quality

Limitations

What this means

Try it yourself

The dataset

The Undiagnosed Input Problem

The symptom

The industry response

The folk system

The gap

What we measured

More rules, lower quality

The sweet spot

A real example

What this means

What’s next

Finding 3: `agents.md` is the most common instruction file