DEV Community

Cover image for State of AI Instruction Quality: Coding Agent Comparison
Gábor Mészáros Subscriber for Reporails

Posted on Originally published at reporails.com

State of AI Instruction Quality: Coding Agent Comparison

Claude Code, Codex, Copilot, Cursor and Gemini are the five coding agents in the Q3 report. We compared them across the 10,000 most-starred repos that ship an instruction file, and put the companies that make them through the same check, using their own repos.

Claude Code is everywhere. It's set up in 69 of the 100 most-starred repos and in 6 of the top 10. Copilot is in 26 of the top 100, Gemini in 11, Cursor in 7 and Codex in 3.

The file Claude Code reads is losing anyway. One CLAUDE.md in five is a line or two telling the agent to go read AGENTS.md.

Which coding agents the top repos use

Agent Top 10 Top 100 Top 1,000 All 10,000
Claude Code 6 69 646 6,030
AGENTS.md (shared) 6 72 678 6,125
Copilot 2 26 225 1,756
Gemini 2 11 51 430
Cursor 1 7 96 734
Codex 0 3 32 162

Share of repos that set up each agent, in the 100 most-starred repos against the 9,784 of the 10,000 with a setup: Claude Code 69% against 61.6%, AGENTS.md 72% against 62.6%, Copilot 26% against 17.9%, Gemini 11% against 4.4%, Cursor 7% against 7.5%, Codex 3% against 1.7%.

The most-starred repo in the set, freeCodeCamp, uses Copilot. React uses Claude Code. Gemini punches above its size at the top: 430 of the 10,000 set it up, and 11 of the top 100 do.

Many repos don't pick one. 4,213 of the 9,784 with a setup (43.1%) use two or more coding agents, and the most common pairing is Claude Code plus AGENTS.md, in 2,399 repos.

AGENTS.md is winning

998 of the 5,464 CLAUDE.md files in the 10,000 (18.3%) are 25 words or fewer and point at AGENTS.md. Going the other way, 39 of 6,128 AGENTS.md files point at CLAUDE.md. That's 26 to 1. 11.3% of GEMINI.md files point to AGENTS.md too, and 4.0% of Copilot's instructions files do.

The CLAUDE.md files people do write are the least instruction-like main files of any agent. About a quarter of what's in them tells the agent what to do. The rest describes the project.

Median share of items in each agent's main file that are directives or constraints: CLAUDE.md 26.1%, GEMINI.md 35.2%, AGENTS.md 36.4%, Codex's AGENTS.md 38.1%, copilot-instructions.md 38.9%, .cursorrules 46.0%. Pointer files left out.

Claude Code setups hedge less. The "should" and "try to" kind runs at a median 3.3 per 100 instructions, against 4.7 for the other agents.

The instructions moved into skills

While the debate is still ongoing between AGENTS.md and other provider-specific files, most of the instructions moved to a different location. Skills hold 42.3% of every directive and constraint in the set. All the main files together hold 31.3%.

Share of all directives and constraints in the 10,000: skills 42.3% (13,190 SKILL.md files plus their reference files), main files 31.3% (13,375 CLAUDE.md, AGENTS.md and other main files), everything else 26.4% (rules, sub-agents, commands).

Peculiar oddity, but only a quarter of repos (2,348) have any skills, and 249 repos hold half of all skill instructions between them. There are almost as many SKILL.md files as main files: 13,190 against 13,375.

They spread like packages, without a package manager. 977 SKILL.md files are byte-for-byte copies of one in another repo. The most-copied, frontend-design, sits unchanged in 14 repos. A fix upstream reaches none of the copies.

Two names, one file, and copies drifting apart

3,562 repos carry main files for two or more agents. A third of them (1,246) keep the same text under two names, mostly a CLAUDE.md identical to the AGENTS.md beside it. Another third use a pointer.

How the 3,562 repos with main files for two or more agents relate the second file to the first: the same text under two names 1,246 repos (35.0%), a pointer to the other file 1,158 (32.5%), different files 937 (26.3%), copies drifting apart 221 (6.2%).

221 repos keep two copies that have already drifted. The typical pair shares 32 lines, and each side has two or three lines the other lacks.

In 92 of 430 Gemini setups (21.4%), GEMINI.md and AGENTS.md hold the same text, and Gemini reads both. For the other agents, the same shape turns up in 1.3% of setups. Gemini setups are also the heaviest at startup: 8.1% hand the agent more than 5,000 words before the first prompt, against 2.3% across the field.

Cursor goes the other way. 66.3% of Cursor setups carry rule files, and a third (33.5%) have no main file at all. The other agents' setups almost always have one.

The makers, by their own repos

Company Agent Repos in the 10,000 Median score Share of repos with custom sub-agents
Anthropic Claude Code 11 7.0 27%
OpenAI Codex 13 6.8 0%
Google Gemini 28 7.2 4%
GitHub Copilot 11 7.9 45%
Cursor Cursor 2 8.6 0%
All repos 10,000 7.5 5.7%

The scores cluster around the field's. What each company builds is where they split.

Which agents each company sets up in its own repos: Anthropic, 11 repos: Claude Code 11, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 0. OpenAI, 13 repos: Claude Code 2, AGENTS.md 12, Copilot 1, Gemini 0, Cursor 0. Google with its Gemini orgs, 28 repos: Claude Code 5, AGENTS.md 15, Copilot 1, Gemini 18, Cursor 1. GitHub, 11 repos: Claude Code 1, AGENTS.md 5, Copilot 10, Gemini 0, Cursor 0. Cursor, 2 repos: Claude Code 0, AGENTS.md 1, Copilot 0, Gemini 0, Cursor 2.

GitHub builds its own agents. 5 of its 11 repos ship custom sub-agents, eight times the field's rate of 5.7%. One of them is spreading: an agentic-workflows sub-agent for GitHub's own workflow tool sits in 44 repos across the 10,000, the second most common sub-agent role after code-reviewer.

The agents' own repos shout too. A third of the 10,000 write rules in capitals. Anthropic's prompting guide tells users of its newest models to dial back that kind of language: "Where you might have said 'CRITICAL: You MUST use this tool when...', you can use more normal prompting like 'Use this tool when...'." Claude Code's own repo ships a triage command that says "IMPORTANT: Don't post any comments or messages to the issue." Codex's own AGENTS.md says "Features that change the agent logic MUST add an integration test", and a Gemini CLI command tells the agent "you MUST first consult the getting started documentation".

OpenAI's repos leave the agent guessing. Of every writing habit we compared across the makers, only one difference held up statistically. OpenAI's median repo leaves 48.7 of every 100 instructions without a file, command or tool to act on, against 30.8 for the field. One of its SDKs opens a section with:

Solve the narrow problem with the simplest implementation. Avoid unrelated refactors, reformatting, dependency or lockfile churn, generated-file edits, speculative abstractions, compatibility shims, and capabilities not requested.

A senior engineer would agree with every word. An agent gets two judgment calls, "the narrow problem" and "the simplest implementation", and nothing to check its work against.

Cursor's two repos score highest, 8.6, with the fewest instructions left vague. One still carries the hedged kind, a skill that says "subtask_prompt should read like a standalone request".

Who checks the makers

These companies write the agents, the documentation and the prompting guides. Their engineers know their codebases far better than any tool does, and that's how the gaps get in. To the person who wrote it, "the narrow problem" has an obvious answer. The agent reading the file has the file, and nothing else.

The agent can't flag the gap either. It does what the line says, and when the line names nothing, it guesses. A company reviewing its own files reads them with the same knowledge that left the gap there.

ails check reads every instruction file with the same deterministic rules, whoever wrote it. The 10,000 most-starred repos and the companies that build the agents get the same treatment, and the findings above are what came back.

Lines 26 to 41 of the AGENTS.md in OpenAI's openai-node repo, the section

Check one yourself

Every repo in this post is public. Clone one and run the same check:

npx @reporails/cli check
Enter fullscreen mode Exit fullscreen mode

It lists each finding by file, so you can read the instructions and decide whether we counted fairly.

This is the last post from the Q3 report. The data behind all of them is in the 2026-Q3 folder of reporails/reports.


From The State of AI Instruction Quality: Q3 2026: 10,000 of the most-starred GitHub repos that ship an instruction file, September 2026 snapshot; 9,784 have a setup we could map. "Top 100" and "top 1,000" rank the 10,000 by stars at the snapshot. A setup is one repo and one agent; repos often have several, so agent counts add up to more than the repos. Codex counts repos with a Codex marker in .codex/ (a config.toml, rules/, agents/ or hooks.json) or an AGENTS.override.md; a plain AGENTS.md setup sits in the shared row. A pointer is a main file of 25 words or fewer that names or imports another agent's main file. Items are what the instrument maps from a file; directives and constraints are the items that tell the agent what to do. "Skills" include the reference files beside SKILL.md; setups above the batch's 90-file cap are outside the run, so the skills share is a floor. "The same text" cannot tell a symlink from a copy; only the drifted pairs are certainly two files. "Gemini reads both" is the instrument's model of Gemini, which concatenates AGENTS.md and GEMINI.md. Companies are the GitHub organizations that ship each agent: anthropics, openai, google with google-gemini and gemini-cli-extensions, github and cursor; Cursor's two repos carry no test. Group differences are two-sided Mann-Whitney or Fisher's exact tests, read against a Bonferroni threshold for all 209 comparisons in the run; the top-100 shares are counts. Scores are each company's median repo score on ails check's 0 to 10 scale; for Anthropic, OpenAI, Google and GitHub the 95% bootstrap interval includes the field's 7.5. Vague instructions name no file, command or tool (CORE:C:0042), per 100 directives and constraints, in repos with at least 5.

Repos measured. anthropics: anthropic-sdk-typescript, buffa, claude-agent-sdk-python, claude-code, claude-code-action, claude-code-base-action, claude-for-legal, commerce-agents, defending-code-reference-harness, financial-services, launch-your-agent. openai: apps-sdk-ui, codex, codex-security, harmony, openai-agents-js, openai-cli, openai-cookbook, openai-dotnet, openai-go, openai-java, openai-node, openai-python, realtime-voice-component. google: adk-go, adk-java, adk-python, benchmark, clusterfuzz, comprehensive-rust, docsy, dotprompt, filament, go-github, gvisor, heir, mantis, nomulus, oss-fuzz, osv-scanner, osv.dev, perfetto, play-services-plugins, sam, site-kit-wp, syzkaller, xls. google-gemini: gemini-cli. gemini-cli-extensions: code-review, nanobanana, security, workspace. github: branch-deploy, codeql, codeql-action, copilot-cli-for-beginners, copilot-sdk, docs, gh-aw, gh-skyline, gh-stack, github-mcp-server, spec-kit. cursor: community-plugins, cookbook.


I work on Reporails, deterministic diagnostics and governance for the instruction files, rules, and prompts that steer coding agents. It reads the instruction files your coding agent loads, from CLAUDE.md to Copilot and Cursor rules, measures which instructions couple to behavior, and points to the lines worth tightening.

Top comments (0)