- Book: AI Agents Pocket Guide
- The series: Pocket Guides for Developers — 8 short books on agents, RAG, prompts, observability, events, databases and system design — the whole series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
The new refund test your coding agent just wrote is long, and it checks one status. Before that check, it builds an in-memory database by hand, runs the migrations by hand, and creates a user and an order by hand. You know that shape. It's the old orders test file nobody has cleaned up in years, and the agent read it and did what it saw.
Simon Willison has made this point twice this year. In a January 26 post built from his own Hacker News reply, he wrote: "Coding agents pick the existing patterns up without needing any extra prompting at all." Once a project has clean basic tests, he added, "the new tests added by the agents tend to match them in quality."
Read that the other way round. If the agent copies the tests you have, your worst test file is part of every test prompt you send, whether you meant it to be or not.
So here is my take. An hour spent cleaning up your three worst test files will move what your agent writes more than another paragraph in CLAUDE.md. I haven't seen a benchmark of that, and I haven't run one. The argument rests on what Willison reports and on what the agent vendors' own docs tell you to keep out of instruction files.
What Willison actually said
The January post is short and practical. The best route to good agent-written tests, in his words, is a project that already has a suite with good patterns. When he wants a particular testing style, he has the agent clone another project and imitate it, because "the quickest way to show an agent how you like something to be done is to have it look at an example."
He also names the failure you are most likely to inherit: "The most common anti-pattern I see is large amounts of duplicated test setup code." He says he tolerates duplication in tests more than in implementation code, and still pushes back on it with two prompts: "Refactor those tests to use pytest.mark.parametrize" and "extract the common setup into a pytest fixture."
He came back to it at the Pragmatic Summit in San Francisco, a February fireside chat he wrote up on March 14. "If you've got a codebase with a bunch of patterns in, they will follow those patterns almost to a tee." And the part that should make you feel better: "Even having just one or two tests in the style that you like means it'll write tests in the style that you like."
So the plan is small: a few good examples, placed where the agent finds them first.
Your CLAUDE.md is competing with your test folder
Look at what Anthropic tells you to put in CLAUDE.md. Its best practices page has a table of what to leave out, and the first row is "Anything Claude can figure out by reading code." The same page says to point Claude at patterns in your codebase, with an example prompt that names one file as "a good example" and says "follow the pattern."
The memory docs go further. Claude treats CLAUDE.md "as context, not enforced configuration." The target is under 200 lines per file, because "longer files consume more context and reduce adherence." Codex reads AGENTS.md "before doing any work," and stops adding instruction files once they reach 32 KiB by default.
So the vendors expect your conventions to live in the code, with the instruction file kept short. Now picture the rule "use the shared fixtures in conftest.py" in a repo where most test files build their database inline. The rule is one line of advisory context. The counter-examples are whole files the agent reads while it works. My reading is that the files win more often than you'd like. That's an opinion, not a measurement, but it matches what Willison describes: the patterns get picked up "without needing any extra prompting at all," and that includes the bad ones.
The one-hour cleanup
Pick three exemplars. Choose test files in the area agents touch most. A good exemplar tests one behaviour per test, has names that read like sentences, gets its setup from fixtures, and doesn't hit the network or sleep.
Pull duplicated setup into fixtures. This is the shape Willison calls the most common anti-pattern. Here app stands in for your own code:
# tests/test_refunds.py (before)
from app import Database, refund
def test_refund_paid_order():
db = Database(":memory:")
db.migrate()
user = db.add_user(email="alice@example.com")
order = db.add_order(user, total=50, status="paid")
assert refund(db, order.id).status == "refunded"
def test_refund_pending_order():
db = Database(":memory:")
db.migrate()
user = db.add_user(email="alice@example.com")
order = db.add_order(user, total=50, status="pending")
assert refund(db, order.id).status == "rejected"
Every new test the agent writes next to this file will likely open with the same four lines. Move them into conftest.py. pytest's docs say fixtures defined there "can be used by any test in that package without needing to import them" (fixtures reference):
# tests/conftest.py
import pytest
from app import Database
@pytest.fixture
def db():
db = Database(":memory:")
db.migrate()
return db
@pytest.fixture
def user(db):
return db.add_user(email="alice@example.com")
Parametrize what is one behaviour with several inputs. The two tests above differ only in the order status and the expected result, which is what pytest.mark.parametrize is for:
# tests/test_refunds.py (after)
import pytest
from app import refund
@pytest.mark.parametrize(
"status, expected",
[("paid", "refunded"), ("pending", "rejected")],
)
def test_refund_by_order_status(db, user, status, expected):
order = db.add_order(user, total=50, status=status)
assert refund(db, order.id).status == expected
Now the example the agent copies is two lines of arrange-and-assert, with the setup living in one place. The same move works outside Python: a shared factory or beforeEach in Jest or Vitest does the job conftest.py does here.
Point the agent at the good ones
Cleaning up the files raises the odds the agent copies the right thing. Naming them removes the guessing. In Claude Code, a path-scoped rule loads only when Claude reads, writes or edits a matching file, so it costs nothing in sessions that never touch tests:
---
paths:
- "tests/**/*.py"
---
# Tests
- Model new tests on tests/test_refunds.py
and tests/test_invoices.py.
- Shared setup lives in tests/conftest.py.
Add a fixture there instead of rebuilding it inline.
- Same behaviour, several inputs: parametrize.
Save it as .claude/rules/testing.md. For Codex and the other agents that read AGENTS.md, put the same three lines in it under a short "Tests" heading. It's short, so it stays well under the size cap.
Then say it in the task itself, the way Anthropic's own example prompt does: "Write tests for refund() in the style of tests/test_refunds.py, using the fixtures in tests/conftest.py." When the result still comes back with inline setup, use Willison's two push-back prompts above in review, and refactor the agent's tests before you merge them. Every test you merge is the next agent's example.
Hermes's scan can only report what your code has
I build Hermes IDE, so weigh this section with that in mind. Hermes runs Claude Code, Codex and other CLI agents side by side, each in its own terminal. Its project scan detects each project's languages, frameworks, architecture and conventions, and points your agent at a context file with what it found, kept within a token budget you can set per project. You can also pin files to a project, and pinned files "reach every later session's agent, also after a restart." Pinning your three exemplar tests is a direct way to make sure they're the ones every session sees.
A scan reads your codebase. If your tests have no shared fixtures and no consistent shape, there is no convention to find, and the context file can't report one. Clean the three files first, then let the scan and the pins carry them. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux; 2.3.4 shipped on October 15, 2026. It is not affiliated with Nous Research, Anthropic, OpenAI or any agent vendor.
Where this take is weakest
It rests on one experienced practitioner's observations and on vendor guidance, not on a controlled study. Willison's examples are pytest, and I haven't measured how strongly any particular agent weights test files over an instruction file.
Instruction files still matter. Anthropic's same table tells you to keep in CLAUDE.md the things code can't show: commands Claude can't guess, testing instructions, environment quirks. Don't delete yours. Stop using it to describe a test style your test folder contradicts.
And a large legacy suite won't be fixed in an hour. That is why the move is three exemplars and a pointer, not a rewrite. The rest of the suite gets better the same way it got worse: one copied test at a time.
Which file in your repo would you least want an agent to copy? Tell me in the comments.
If this was useful
I build Hermes IDE for running several coding agents on one repo; it's free, and the GitHub repo is where to report anything that doesn't work as described.
An agent that copies your worst test is doing what agents do: acting on the context it can see. The AI Agents Pocket Guide covers how that context, the tools and the checks around an agent shape what it does, which is the other half of this post. It's one of 8 short books in the Pocket Guides for Developers series, covering agents, RAG, prompts, observability, events, databases and system design.
Top comments (1)
This matches what I've seen. One thing that helped me: before asking the agent for new tests, I point it at one specific well-written test file as the pattern and tell it to ignore the legacy ones. It also helps to run the new tests against a deliberately broken implementation, since agents sometimes copy a weak assertion style that passes no matter what. Have you tried a small 'golden' test file kept just as a reference?