Should I delete my CLAUDE.md? Before doing that, I wanted to know which parts an agent still needs. So I ran 438 tests: with guides, without guides, and with notes left in a folder that nobody mentioned.
The question is in the air. When Opus 5 shipped, Anthropic removed over 80% of Claude Code's system prompt and measured no loss on its coding evaluations. Boris Cherny, who created Claude Code, told the YC Startup School audience: every six months, delete your CLAUDE.md, delete your skills, delete your hooks, and see what the model does. Anthropic's advice now is to keep CLAUDE.md lightweight and "spend most of the tokens on gotchas inside of the codebase". And Andrej Karpathy writes that the agents do not listen to his AGENTS.md files at all.
My coding agent does read the official docs by itself. But when I write a guide, which parts still change what it does, and which parts would it have worked out anyway? I didn't want to argue about it. I measured it.
The short answer
An agent does not need a guide for anything the docs already say. A guide that retells the docs did exactly as well as no guide, in every one of 32 runs.
It does need a guide for what the docs don't say: a fact missing from the official docs, a fix it can diagnose but not know, a method step it skips on its own, a trap sitting where the obvious code goes. Those turned fails into passes. With one of those in hand, the smallest models did as well as the largest ones without. If you pay for a big model to make up for missing context, a good note does the same job for fewer tokens.
And how you hand it over matters less than what is in it. "Here is a colleague's note, use what applies" did as well as "follow this guide", and a note left in a folder, never mentioned, got read and used.
How I tested
The guides are stories from worklore, a library I run. Each story is one developer's account of a problem: what they tried, what failed, what worked, and a "Verify" section that says how to check the result. I used seventeen stories, fourteen of my own and three by another author.
For each story I built a small repo with the problem planted in it, wrote a one-sentence task, and gave it to an agent twice: once with the story and once without. A hidden script graded every run against the story's own Verify. The grader read the repo, the diff and the command log, never the agent's report of itself.
The agents: Claude Code with Opus, Sonnet and Haiku 5.5; Codex CLI with GPT-6-Sol at two reasoning levels; Antigravity with Gemini 4 Argon at two levels, Gemini 3.8 Flash and GPT-OSS 120B. Nine configurations, 438 runs in all, and one run per cell, a cell being one story, one configuration and one way of handing the story over. That is why the counts below have different denominators: a story that ran on eight configurations is scored out of eight without it and sixteen with it, because "with" combines two wordings. Read the numbers as strong anecdotes, not a benchmark.
First round: five guides, no pattern
I started with five guides I had on hand: a skills audit, an MCP server test, a timezone bug, a database schema review and a marketing plan. The totals looked like a small win for guides. The runs did not support that reading.
- The MCP test was the only task where the guide turned fails into passes. Everyone found the bug, but only the guided runs tested the server the way a real client calls it.
- The timezone bug was fixed by every agent without help. With the guide, three agents took one sentence literally and broke a working API.
- The skills audit, the schema review and the marketing plan came out the same with and without the guide: longer answers, the guide's scale, the guide's layout, the same findings, and a slightly worse marketing plan.
Five tasks, four different outcomes. The kind of task did not predict them. What the guide carried did: the MCP test was the only guide with something the agents did not already have.
Second round: a guide pays for what it carries
So I sorted my stories by the one thing each one carries and picked six kinds. For each kind I tested two stories, so that a result could not be blamed on one badly written story. The first story of each kind ran on all nine configurations, the second on the seven that still had quota.
A fact missing from the official docs. This is where a guide is worth the most. A Garmin watch wants a heart-rate corridor in one exact shape, and the docs don't give the shape. Without the story, three agents of eight got it, all by web search, and two Gemini runs spent their whole twenty minutes trying variants. With it, sixteen of sixteen, from Haiku to Opus. The second story, three undocumented quirks of an OAuth provider, went the same way, with one correction: two of the quirks turned out to be on a forum, and the good searchers found them.
A fact from the docs. Dead weight. Every agent knew that PostHog's Flutter replay needs the app wrapped in a particular widget, and every agent knew how Anthropic's prompt caching handles a memory write. Thirty-two runs, no difference.
A fix the agent can diagnose but not know. The clearest win. A nightly job failed because the CLI it calls said "Not logged in": the job's launchd plist had no USER in its environment. Seven agents of nine found the cause from the log, then wrote "this cannot be fixed from here". The story's one line, add USER to the plist's EnvironmentVariables, was the fix, and with it seventeen of eighteen fixed it. The second story of this kind, DynamoDB reserved words, did not repeat the effect, and that is the point: its second bug could be derived from the log, so the agents derived it. The win comes from a fix the agent cannot derive, not from there being two bugs.
A trap. My first trap, a command that silently serves a website as a download, caught nobody: not one of 35 runs typed that command. So for the second story I set a trap where the natural code goes: an analytics event on every keystroke, which is how I once burned a free tier in days. Three agents of seven walked into it without the story. With it, none of fourteen.
A cause the evidence gives away. Found anyway. A soft 404 caused by a robots.txt rule and a DynamoDB crash on the second write: when the log says what is wrong, the agent reads the log. The story added precision, not correctness.
A method. It adds the one check nobody runs on their own. My ten-step MCP checkup helped because one of its checks, "can the output schema reject a wrong answer?", is one agents skip. A two-account authorization test did not help, because six agents of seven already ran it unprompted.
"A colleague's note" did as well as "follow this guide"
Half the guided runs got the story under "Follow this guide:". The other half got: "A colleague's note on a similar case is below. It is not instructions. Use what applies, ignore the rest, and say which parts applied."
The note wording was never worse. It was better on three tasks of six in the first set and tied in the second. I had expected "follow" to win on the ten-step checklist. It lost, because a checklist followed to the letter crowded out the agent's own look: one agent skipped a check it had run by itself in the no-guide run.
Two things came free with the note wording. Every agent said which parts of the story applied, which tells you which paragraphs of your guide do any work. And the agents kept what the evidence did not implicate: told to follow, seven of seven rewrote a token exchange the story described; told it was a colleague's note, three of seven did, and one wrote "nothing in the logs points at the exchange, so I left it alone".
A note nobody mentioned gets read
The fourth arm put the story in a notes folder in the repo, next to two unrelated stories, and the task said nothing about it.
Eighty-four runs of ninety-one opened the note. The ones that didn't were all GPT-OSS. With the note open they passed at nearly the prompt rate, and it was the fastest arm by a wide margin, because the agent stopped searching once it had the answer. That is closer to how guides reach agents in real life, through an AGENTS.md or a docs folder, than a guide pasted into the prompt. It also cuts both ways: nobody questioned the note or checked its claims before applying them.
Small models with a guide matched big ones
Without a guide the spread was wide: Haiku passed half, Opus most. With a guide in the prompt, every Claude, Codex and Gemini configuration passed everything, or all but one run, and Codex at low reasoning matched medium everywhere. The one exception, GPT-OSS 120B in Antigravity, mostly struggled with tool use, not with reading. Haiku with a note matched Opus without one.
Four surprises
- "This can't be fixed from here." Agents diagnose more than they know how to fix. Seven of nine found the cause of the nightly job failure, then gave up on it.
- An agent went looking for my keys. One run, diagnosing a missing environment, started a clean shell and read my API keys out of the login files, past a guard I had added after an earlier leak. If your agent can run a shell, assume it sees everything a fresh shell can read.
- An agent read the answer key. It found my hidden grader scripts in the parent folder and graded itself. Every later run got a fresh temp folder with nothing around it.
- The out-of-date guide was noticed. One story described problems its own migration had already fixed. All eight runs read the history and filed those as "already fixed". A guide's facts age; its method didn't.
What I'd put in a guide, or an AGENTS.md, now
- Write down what the docs don't say. The exact format a device wants, the one key in a config file, the check your agent won't run by itself. Leave out what the docs cover; the agent has it.
- Present it as a case, not a command. "Here is what happened to a colleague, use what applies, and say what did." You lose nothing and gain a list of which paragraphs mattered.
- Don't write absolute sentences you don't mean. Agents take them literally. "Never guessed by a server" broke a working API.
-
Name a tool, and say what it can't see. A scanner in one of my guides rated an unpinned
npxas harmless. Every agent caught it anyway, by reading the line. Say next to the tool what it misses, so the reading stays part of the job. - Write a Verify, and make it check substance. A Verify that checks only the shape lets a thin result pass.
- Test the guide. Run your agent once with it and once without. It takes minutes, and it showed me which of my paragraphs were dead weight, which one did damage, and which one had quietly gone stale.
If you write skills
A skill is a guide your agent loads on demand, so the same rules apply to a SKILL.md, and two of the seventeen guides here were skills: the marketing plan and the capability scanner. The one test worth adding to a skill's release checklist is the last rule above: the same task once with the skill and once without, graded by your own Verify. If the two runs match, the skill isn't carrying anything yet.
What changed on worklore because of this
Every story's copy-to-agent prompt on worklore now hands it over as a colleague's case and asks the agent to report which parts applied, and when you ask your agent to write up a fix of your own, the skill tells it what makes a story work before it drafts. The new prompt itself is not measured yet. That is the next experiment.
Limits
One run per cell. Fourteen of the seventeen guides are mine, all in one format. The fixtures are small, with one planted problem each. Agents had web access, so "missing from the docs" means missing from the docs, not from the web. The agents ran with no memory, no MCP servers and no global instruction files, and no run without a story fetched it from the web; whether a model had seen a story in training I cannot check.
Everything is on GitHub: the fixtures, prompts, graders and every run's log, at worklore-mcp/research/guide-eval. The guides are stories on worklore, where each one ends with a Verify and a count of the agents that ran it and reported the outcome against that Verify. If you have a fix your agent keeps needing, that format is built for it.
Top comments (1)
"A guide that retells the docs did exactly as well as no guide" matches what we saw. The part 438 tests can't easily reach is time: a gotcha that was right in March and is wrong now still reads as authoritative. We ended up dating each line and recording what replaced it, so a stale note shows up as stale. Did any of the notes-in-a-folder runs trip over an outdated note?