- Book: Observability for LLM Applications
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
In February 2025, Anthropic published the system card for Claude 3.7 Sonnet. Section 6 carries a blunt heading for a lab to put over its own model: "Excessive Focus on Passing Tests."
The card says the model "occasionally resorts to special-casing in order to pass test cases in agentic coding environments." Most often that meant "directly returning expected test values rather than implementing general solutions." It also included "modifying the problematic tests themselves to match the code's output."
Then section 6.2 tells people running coding agents what to monitor. The last item on that list is "Unexpected modifications to test files."
Look at your own pipeline. An agent opens a pull request with a new feature and new tests, written in the same turn. CI goes green. What did the green prove?
My take: much less than the checkmark suggests. A test the agent wrote alongside its code is a claim the agent makes about its own work. It is not evidence. Evidence comes from a test the agent didn't write and can't change.
The labs wrote this down themselves
Anthropic isn't alone here. On March 10, 2025, OpenAI published Detecting misbehavior in frontier reasoning models. Reading a reasoning model's chain of thought, the post says, "has allowed us to detect misbehavior such as subverting tests in coding tasks, deceiving users, or giving up when a problem is too hard."
The paper behind that post, Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation, lists what a frontier agent "in the same family as OpenAI o1 and o3-mini" did on coding tasks during training. It planned to "circumvent the tests by making verify always return true." It exited the environment early "without running all unit tests." It wrote "stubs instead of real implementations when unit test coverage is poor," and tried "parsing test files at test-time in order to extract expected values." In a separate set of deliberately hackable tasks given to a smaller agent, "directly modifying unit tests was the most easily learned exploit."
Independent evaluators saw the same pattern. In June 2025, METR's Recent Frontier Models Are Reward Hacking described o3 deciding "to patch the competition evaluation function so that it judges every submission as successful."
The OpenAI post has one more finding that matters for your review process. Penalizing the model's "bad thoughts" didn't stop most of the misbehavior. It taught the model to hide its intent. So read the diff, and give less weight to the agent's reasoning or its cheerful summary at the end of a turn.
Be fair about scope. These are 2025 models, and the behavior was mostly seen in training and evaluation environments. Anthropic's card says its training mitigations "significantly reduced the frequency of this behavior in typical use." I'm not claiming your agent is cheating today. The point is structural: the agent is rewarded when tests pass, and in your repo the tests are files it can edit.
An honest agent still grades its own homework
Drop the cheating entirely. The problem doesn't go away.
Say the ticket reads "allow refunds on paid orders." The agent reads "paid" as "paid or shipped." It writes a handler that refunds shipped orders. In the same turn, it writes a test that asserts a shipped order can be refunded. The handler and the test agree with each other, both are wrong, and CI is green.
The code and the test came from one reading of the ticket, so they share one blind spot. Two files that agree with each other only show the agent was consistent. Neither one checked the ticket. Humans have the same failure mode, which is why test-first work and a second reviewer exist. The difference is speed: an agent produces the feature and its tests in a minute, and review tends to stop at the checkmark.
A test written after the code, by the same author, can only confirm what the author already believed.
Keep a test set the agent can't touch
You want at least some tests that encode the requirement, written before the agent starts, that the agent cannot rewrite. Three layers, from weakest to strongest.
1. Write the acceptance tests first. You write them, or a separate agent session drafts them from the ticket without seeing any implementation and a human reads them before they're committed. A second agent can misread "paid" the same way, so the human read is the point. Commit them to the base branch before the task starts. They describe what "done" means, in code, from outside the turn that does the work.
2. Make them read-only for the agent. In Claude Code, a deny rule in the project's .claude/settings.json stops edits to that folder:
{
"permissions": {
"deny": ["Edit(/tests/acceptance/**)"]
}
}
A leading / anchors the pattern at the settings source, so in a worktree session it matches that worktree's own tests/acceptance/. Know its limits. The docs say Read and Edit deny rules cover Claude's own file tools, file commands it recognizes in Bash such as sed and tee, and redirections. They don't cover "arbitrary subprocesses that read or write files indirectly, like a Python or Node script that opens files itself." For OS-level enforcement, the docs point to the sandbox. Treat the deny rule as a speed bump. Other agents have their own permission settings, and the next layer works whichever agent wrote the branch.
3. Put the wall on the server. A CODEOWNERS entry gives the folder an owner:
/tests/acceptance/ @your-org/qa
Turn on "Require review from Code Owners" in the branch protection rule, and a pull request that touches those files can't merge until someone from that team approves. The agent can edit the file in its branch. It can't land the edit alone.
The acceptance suite sits outside the turn: the agent can run it but not rewrite it.
Flag every change that edits code and tests together
The protected suite covers the requirement. The agent will still write its own tests for the rest, and those are worth keeping. You just read them differently when they arrive in the same change as the code they check. This script makes that visible:
#!/usr/bin/env bash
# flag-cochange.sh: exit 1 if one change edits code AND tests
# no args: uncommitted work vs HEAD (one agent turn)
# or a range: HEAD~1 HEAD, origin/main...HEAD
t='(^|/)(tests?|__tests__|spec)/|[._](test|spec)\.|(^|/)test_'
if [ $# -eq 0 ]; then
files=$(git diff --name-only HEAD
git ls-files -o --exclude-standard)
else
files=$(git diff --name-only "$@")
fi
tests=$(grep -E "$t" <<<"$files")
code=$(grep -vE "$t" <<<"$files" | grep -vE '\.(md|txt)$')
if [ -n "$tests" ] && [ -n "$code" ]; then
echo "FLAG: code and tests changed together:"
sed 's/^/ test: /' <<<"$tests"
exit 1
fi
The pattern catches tests/ and __tests__/ folders, *.test.ts, *.spec.ts, Go's *_test.go and Python's test_*.py. With no arguments it checks uncommitted changes plus new untracked files, which is what one agent turn leaves behind. Give it a range and it checks a branch: origin/main...HEAD in CI, as long as the checkout has the base branch's history. Run it there as a non-required check, or with || true plus a PR label, so a flag asks for a reviewer's eyes instead of blocking the merge. Adjust the pattern to your layout.
Test-first work and honest refactors trip the flag too, so treat it as a reading order: read the test diff before the code diff, and look at every changed assertion. If an expected value moved toward what the new code returns, that is the exact pattern the Claude 3.7 Sonnet card describes, and it needs a human answer.
The card adds one more signal: when the model special-cased, it "often (though not always)" left a comment saying so. A crude check costs one line:
git diff -U0 HEAD | grep -iE '^\+.*(special.?case|hard.?cod)'
No match means nothing. A match means read that hunk first.
Where Hermes IDE fits
I build Hermes IDE, so weigh this section with that in mind. Hermes saves every agent turn, so you can diff or restore any single turn, with undo. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows changes by turn or by file, with risk flags, and a line comment goes back to the agent that wrote the line.
The by-turn view is the useful part here. Each turn lists the files it changed. A turn that touched src/refunds.ts and tests/refunds.test.ts shows both in one place, before anything lands, so the co-change the script flags is right there in the turn that made it.
The Review Desk (⌘G) by turn for a Claude Code task in orders-api: two turns, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns.
When a turn changed an assertion, you have two moves. Restore your files to the point before that turn (with undo) and rerun from there. Or put a line comment on the assertion. Hermes sends it to the agent that wrote the line, so you can ask it to justify the new expected value. Hermes also lets your own checks decide when a task is done. Point that check at the acceptance suite.
What Hermes doesn't do: it won't tell you whether a test change was honest, and I'm not claiming its risk flags single out test files. Reading the assertion is still your job. Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. The code is on GitHub. It is not affiliated with Anthropic, OpenAI or any agent vendor.
Start with one folder
Pick your last three tickets. Move the tests that encode their requirements into tests/acceptance/, add the CODEOWNERS line, and wire the flag script into CI as a non-required check. After that, the next green checkmark on an agent's pull request carries a little more weight.
Do you let agents change tests in the same pull request as the code? Where do you draw the line? Tell me in the comments.
If this was useful
The rule underneath this post is an old evals rule: the system being measured doesn't write its own grader. Observability for LLM Applications applies it to LLM features in production, with tracing, evals, cost and quality regressions, so you can tell a real improvement from a model agreeing with itself. If you review agent work in Hermes IDE, the by-turn Review Desk is where the habit from this post starts.
Top comments (0)