Start with the artifact
Yesterday's Claude Code changelog shipped something small that says a lot about where agent harnesses are heading:
2.1.277: "Added AGENTS.md support: in a project with no CLAUDE.md, Claude Code reads AGENTS.md instead; change it under 'Project instructions' in /config (not yet on Bedrock, Vertex or Foundry"
Read that once and it sounds like a compatibility hook. Read it twice and it's confirmation that agent instruction context is becoming a cross-tool portability layer. AGENTS.md is described as an "open format for guiding coding agents, used by over 60k open-source projects," stewarded by the Agentic AI Foundation under the Linux Foundation, on agents.md — a README for agents, separate from the human-facing README, adopted by Codex, Factory, Cursor, Zed, Windsurf and more. Now the biggest names in the space are converging on the same file as the default place to tell an agent who it is, how to behave, and what it must not touch. That trend is good for portability and bad for anyone trying to reproduce an agent benchmark.
The same file, resolved differently
The problem is not that tools now share AGENTS.md. It's that they still resolve it with their own rules on top, and those rules are part of the harness. Claude Code, per this release, prefers CLAUDE.md and only falls back to AGENTS.md when there is no project-level CLAUDE.md.
Codex has a whole different discovery chain. In its own documentation, Codex checks each directory for AGENTS.override.md, then AGENTS.md, then any fallback names you define in project_doc_fallback_filenames. It layers guidance from a global file in ~/.codex down to the deepest nested directory, merging root-first with nearer files overriding earlier ones, and stops adding when the combined prompt hits a 32 KiB cap (project_doc_max_bytes). It even has a section telling you to put ## Code Review Rules in the AGENTS.md closest to the code they govern.
So the exact same repository, with the same AGENTS.md sitting in the root, produces a different resolved instruction context depending on which agent you run. One tool falls back to it only when its own vendor file is absent. Another builds a root-down chain with override files, fallback filenames, and a byte budget. "The agent's instructions" is not a single thing you can hold constant across two tools, because the harness decides how those instructions get in.
This is the model-plus-harness problem wearing a different jacket. A benchmark score is a property of the model and the harness together — the scaffold can move a score more than reasoning effort does. Instruction context is exactly that lever. Two runs of the same model on the same task will differ if one load has an AGENTS.md telling it to prefer a code style and one doesn't. And you can't cleanly attribute which agent did better, because the better one had better instructions injected by its own resolution order.
What this does to comparison
To compare two coding agents on the same repo, you want the environment constant: same task, same timeout, same starting state. Instruction context has to be part of that constant. But you cannot freeze it to "the AGENTS.md that exists" and get a clean test, because each tool reads a different effective instruction set from that same file.
Concretely, the reproducible setup becomes: clone the repo, drop the same AGENTS.md in, and run agent A and agent B. The result is not "which agent is better at the task." It's "which agent is better at the task given this instruction file, as interpreted by each tool's precedence rule." The precedence rule is not yours to compare — it's baked into the harness. Every agent benchmark run this way is really reporting a model-plus-harness-plus-instruction-resolution composite.
This is also why a flat number from someone else's harness tells you very little about whether it will work on your repo. Your team's AGENTS.md (or the one you have not yet written) is not part of their score. Their number was produced under their resolution order and their conventions.
Why this is actually good news for teams
The portability side is the upside, and it's real. Teams that have been maintaining CLAUDE.md, Codex.md, and whatever else now have a shot at one file that travels across tools. If a standard AGENTS.md becomes the common denominator, a new agent can come in and honor instructions that were written before it existed, with minimal migration.
The practical reading for an engineering team: write AGENTS.md as the portable baseline, not as a vendor artifact. Keep vendor-specific bits (CLAUDE.md extras, tool-specific hooks, AGENTS.override.md files) in places you expect to differ. That way the portable layer you test against is the one that holds across agents, and the harness-specific stuff is visible and separable. Treat AGENTS.md as a contract you can reproduce, not a blob you hope each tool reads the same way.
What it means for review
This also surfaces in code review, which is the other half of my beat. When the code you're reviewing came from an agent, the review has to account for the fact that the decisions behind the code are artifacts of that agent's instruction context. The same task run under two different AGENTS.md-resolving tools can produce visibly different style, structure, and even correctness trade-offs. A review harness that treats every agent's output the same, without knowing what instruction set shaped it, is judging two different things side by side.
When you run a review tool across agents (which is the direction a lot of teams are heading, running one reviewer against whatever model their devs happened to invoke), the instruction context becomes part of what you verify. A review tool that does not fold in "which agent produced this and under what instructions" is doing a partial job. Tooling in this space — review harnesses like Kodus and the agent-wrapping reviewers — needs to surface which agent and what instruction context produced a change, not just score the diff.
The one number you can trust: the empty-AGENTS.md baseline
If you want a reproducible comparison despite all this, flatten the variables. Run both agents on a repo with no AGENTS.md and no vendor instruction file at all. That gives you the closest thing to a model-plus-default-harness number, with the instruction layer removed. It is the honest baseline, because it is the only configuration where "the instructions" is genuinely constant (namely, empty) across tools.
Then run your real AGENTS.md on top and measure the delta. That delta is what your instruction file actually buys you per tool, which is more useful than an absolute score anyway. You learn which agent honors your instructions as written and which one does its own thing with them. That is an eval you can set up today and one that does not get invalidated the day a vendor ships another file-convention tweak.
The change in this changelog reads like a compatibility win. It is, mostly. But it is also a reminder that every new convention tools adopt becomes a variable you have to control before you can compare anything. Control it, and the comparison means something. Ignore it, and you are comparing instruction interpreters, not coding agents.
Top comments (0)