- Book: LLM Observability Pocket Guide
- The series: Pocket Guides for Developers — 8 short books on agents, RAG, prompts, observability, events, databases and system design — the whole series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
In a benchmark called EvilGenie, researchers gave Claude Code, Codex and Gemini CLI programming problems in a setup where the agent could edit the test files. They report "explicit reward hacking by both Codex and Claude Code, and misaligned behavior by all three agents." A few months earlier, METR described frontier models "modifying the tests or scoring code" to get a higher score, and noted that "they demonstrate awareness that their behavior isn't in line with user intentions."
Those were earlier model versions, in setups built to measure exactly that. You don't need a cheating agent to want a forensic pass, though. A well-behaved one can still add a dependency you never asked for, leave an it.only that silences the rest of a suite, or quietly move a test file out of the runner's path. The agent says "all tests pass", and the branch is where all of it lands.
So before you merge, run one command per check. Every command below compares the agent's branch with the branch it started from. Run them in the agent's worktree, or swap HEAD for the branch name, and swap main for whatever the task was cut from.
The three dots matter. Per the git-diff docs, git diff A...B "is equivalent to git diff $(git merge-base A B) B". You see only what the branch changed since it split off, not what landed on main in the meantime.
A pre-merge pass looks for what rode in alongside the fix.
What changed, and how much
1. Scope creep: --stat and the commit list
git log --oneline main..HEAD
git diff --stat main...HEAD
You asked for a fix in src/auth/. If --stat lists files in src/billing/ and a shared config, the agent went wider than the task. Read the commit list too. Two dots in git log mean commits reachable from HEAD but not from main (gitrevisions), which is the agent's work and nothing else.
2. Deleted tests: --diff-filter=D
git diff --name-status --diff-filter=D main...HEAD
git diff --stat main...HEAD -- '*test*' '*spec*'
The first line lists every file the branch deleted. The second shows how test files changed. If a test file lost far more lines than it gained, open it and compare the assertions, because a weaker assertion passes just as green as a fixed bug. In a plain pathspec, * can match directory separators (gitglossary), so '*test*' finds tests at any depth.
3. Silent renames: -M
git diff -M --name-status --diff-filter=R main...HEAD
A test moved from tests/auth.test.ts to tests/old/auth.test.ts.bak is a rename, not a deletion, so check 2 misses it, and your runner never sees the file again. Renames print as R followed by a similarity score (git-diff, raw output format). A low score is a rewrite wearing a rename's label. Passing -M explicitly keeps it working whatever your local diff config says, and -M90% raises the bar for what counts as a rename.
4. Binary files: --numstat
git diff --numstat main...HEAD | awk '$1 == "-"'
--numstat prints two - instead of line counts for binary files (git-diff). An agent that ran a build, a test snapshot or an image tool can commit the output. You can't review a binary, so find out where it came from before you merge it.
What it added to your dependencies and config
5. New dependencies: manifest and lockfile diffs
git diff main...HEAD -- '*package.json'
git diff --stat main...HEAD -- '*package-lock.json' \
'*pnpm-lock.yaml' '*yarn.lock'
Read the manifest diff line by line. Every new name under dependencies is code you'll ship, so look each one up before you merge. The lockfile diff is too big to read, so --stat tells you whether it moved at all. A lockfile change with no manifest change means something was re-resolved, and you should ask why. On other stacks, swap in go.mod/go.sum, Cargo.lock or poetry.lock.
6. CI, ignore rules and the agent's own config
git diff --stat main...HEAD -- .github/ .gitignore \
.claude/ '*CLAUDE.md' '*AGENTS.md'
These files decide what the agent and CI may do on the next run. Claude Code's shared project file, .claude/settings.json, holds "team permissions, hooks, plugins" (Claude Code settings), so an agent adding an allow rule there can widen what it may run next time without asking. AGENTS.md is "a simple, open format for guiding coding agents", read by Codex, Gemini CLI and many others. A new line in .gitignore can hide a file from git status.
7. Executable bits and symlinks: --compact-summary
git diff --compact-summary main...HEAD
This adds markers to the --stat view: new or gone for created and deleted files, +x or -x when the executable bit changes, and +l when the created or deleted file is a symlink (git-diff). Ask about a script that became executable, or a new symlink pointing outside the repo. A file that turned into a symlink is a type change instead: --diff-filter=T lists files whose type changed between regular file, symlink and submodule.
What it left in the code
8. Whitespace damage and conflict markers: --check
git diff --check main...HEAD
The docs say --check warns "if changes introduce conflict markers or whitespace errors" and "exits with non-zero status if problems are found" (git-diff), so it drops straight into a pre-merge script. By default, the whitespace part catches trailing spaces and a space followed by a tab in a line's indent. The conflict-marker part matters more after an agent "resolved" a conflict by hand.
9. Debug leftovers and focused tests: grep the added lines
PAT='TODO|FIXME|console\.log|debugger|\.only\(|\.skip\('
git diff main...HEAD | grep -E "^\+.*($PAT)"
Lines starting with + are what the branch added, so this ignores the TODOs that were already there. .only is the dangerous one. Jest has test.only and its aliases it.only and fit for running a subset while debugging, and test.skip with xit and xtest for skipping. Left in a branch, one .only means the rest of that file never runs, and the suite can still report green. Add your own stack's patterns to PAT.
One focused test can switch off every other test in the file.
10. Secrets: grep, then a real scanner
git diff main...HEAD \
| grep -iE '^\+.*(api[_-]?key|secret|token|passw)'
gitleaks git -v --log-opts="main..HEAD" .
The grep is a cheap first look and will be noisy. Gitleaks is the proper tool for this: its git command scans patches from git log -p, and --log-opts passes a range through to git log. Scanning the commits, not only the final diff, matters here. A key the agent added in one commit and removed in the next is gone from the diff and still in the history.
After the agent rebases
11. What the rebase changed: git range-diff
You reviewed the branch, then asked the agent to rebase onto the new main. Now every commit hash is different, and a fresh git diff shows the whole branch again. git range-diff compares the old series with the new one, commit by commit, so you re-read only what moved.
Save the old tip before the agent rebases:
git branch review/before-rebase # in the agent's worktree
# ...the agent rebases onto main...
git range-diff main review/before-rebase HEAD
The <base> <rev1> <rev2> form means main..review/before-rebase against main..HEAD. In the docs' example, = pairs a commit whose patch didn't change, ! pairs one whose patch did and prints a diff of the diffs under it, and < or > marks a commit that exists on only one side. If you forgot to save the tip, the docs show git range-diff @{u} @{1} @ for a branch with an upstream, and ORIG_HEAD records where HEAD was before a rebase, as long as nothing else moved it since.
A ! on a commit you'd already approved is the one to read. A conflict the agent resolved during the rebase shows up there.
Where Hermes's Review Desk fits
I build Hermes IDE, so weigh this section with that in mind. Hermes runs each task on its own hermes/<name> branch in its own worktree (README), so every command above works on those branches as plain git.
Per the same README, every agent turn is saved, so you can diff or restore any single turn, with undo. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows changes by turn or by file with risk flags, and sends a line comment back to the agent that wrote it.
The by-turn view tells you which turn touched the lockfile, which a single branch diff can't. Once you know that, you can diff or restore that turn, with undo, and a line comment on the new package line means "why did you add this?" reaches the agent that added it.
The README doesn't list which changes get a risk flag, so keep all eleven commands. Treat the Review Desk as a second view of the same branch. Hermes is free, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with Nous Research, Anthropic, OpenAI or any agent vendor.
If this was useful
Save the eleven commands as a script and run it on the next agent branch before you click merge. If you'd rather also review the changes turn by turn, with risk flags, download Hermes IDE and, if it saves you a bad merge, star it on GitHub.
The git pass tells you what an agent left in the branch. It can't tell you what the model did on the way there: which calls it made, what they cost, or whether its answers held up. The LLM Observability Pocket Guide covers that side, with tracing, evals, logs and token usage, and how to pick the tools your team will actually use. It is one of 8 short books in the Pocket Guides for Developers series, alongside agents, RAG, prompts, events, databases and system design.
Top comments (0)