An agent says it's done. The tests are green. The PR description reads like a thoughtful engineer wrote it. You're two clicks away from hitting merge.
I've learned to stop at that point. The green build and the confident summary are the minimum bar, not the signal. Three cheap checks sit between "looks right" and "actually right," and each one catches a different class of failure that the agent's own self-review reliably misses.
1. Ask for the failing log line, not the passing test
When an agent fixes a bug, it usually writes a test that exercises the fix and reports that the test passes. That proves the new code path works. It does not prove the bug is gone.
The habit I picked up: before I accept the fix, I ask the agent to paste the test output from before it changed anything. The diff between the old failing assertion and the new passing one tells me whether the test actually covers the regression or whether the agent quietly changed the assertion to match the new behavior.
Agents sometimes rewrite a test so that the expected value equals what the code now produces. The build goes green. The bug stays. Asking for the pre-fix log line makes that visible in ten seconds.
2. Read the diff, not the summary
The PR description summarizes the intent. The diff tells the truth. I don't mean skimming the file list. I mean reading every added line in the core logic — usually under 40 lines if the agent scoped the task well.
Agents have a predictable blind spot: they add error handling that looks reasonable but catches exceptions they never trigger. They add a retry loop around a call that cannot fail. They rename a variable in one file and miss the string reference in a test fixture or a config lookup. The summary says "refactored X," and the diff quietly changes the behavior of Y.
The trick I use: scan the diff for any line that touches something the task didn't mention. If the agent was asked to add pagination and it also changed the auth middleware, that's not efficiency — that's scope creep I want to interrogate.
3. Re-run the external contract by hand
This is the one unit tests almost never cover. When an agent changes a backend handler, it updates the local tests and maybe the internal README. It does not, on its own, prove that the live endpoint still matches the documented contract.
The drift is subtle. The agent changes a field name, or makes a previously required field optional, or changes an error code from 404 to 400. Internal tests pass because they were updated in the same commit. The OpenAPI doc in the repo still says the old behavior. The frontend three repos over is about to break in production.
Before I merge handler changes, I re-run the described contract against the live endpoint and compare. I used to do this by hand with curl and a notebook. Now I keep a local OpenAPI viewer that re-runs the spec against the running server and flags mismatches — the same thing I built into Powerduck, because I got tired of rebuilding the throwaway script every quarter.
This check is cheap and it catches the failures that show up in prod on Monday morning.
The pattern
None of these checks are sophisticated. They're all "don't trust the agent's self-report on the dimension it is most likely to be wrong about." The agent is the author, the test writer, and the reviewer. In human teams we separate those roles because a single mind has blind spots. With AI, the blind spots are wider and the confidence is higher.
Three checks. A failing log line. The untouched diff lines. The live contract. They add maybe fifteen minutes to a merge and they catch the failures that would otherwise cost you an hour of debugging on the other side.
Top comments (0)