This week on Product Hunt the theme was unmistakable: TestSprite, Appxiom, Currents, tensorViz — every launch was about making coding agents observable. The digest line said it plainly: "agents are now an assumption; the shift is from building agents to making them observable and trustworthy."
I run a small store of agent-ops tooling (honest disclosure: it's early, revenue is still $0, I'm building in public). When I read those launches I realized most teams are buying observability dashboards when what they actually need first is a pre-merge checklist. Dashboards tell you an agent failed after the damage. A five-minute review catches it before.
Here are the five checks I run on every agent-produced diff, in order of how often they catch something.
1. Did the agent touch files outside the task scope?
This is the highest-yield check and the one people skip. An agent asked to "add retry logic to the payment client" will happily refactor the auth middleware, bump three dependency versions, and reformat a file you didn't ask about — all in one diff.
git diff --stat main...HEAD
If the file list contains anything you didn't discuss, treat it as a red flag, not a nuisance. Scope creep in agent diffs is where subtle breakage hides, because the extra changes look plausible in isolation.
2. Are the tests testing the new behavior or the old behavior?
Agents are excellent at writing tests that pass. The failure mode is writing tests that assert the pre-change behavior, or mocking so aggressively that the test would pass even if the feature did nothing.
Concrete check: pick one assertion in the new tests and ask "what input would make this fail?" If you can't construct one in ten seconds, the test is decoration.
3. Did it delete the unhappy path?
Ask any agent to "handle errors" and watch it replace a try/except with a generic except Exception: pass, or collapse five distinct error codes into one. The diff gets shorter, CI stays green, and your failure telemetry dies.
Scan the diff for removed lines containing raise, return None, except, assert, log. Every deletion of an error path deserves a justification you can articulate.
4. Does the diff contradict its own commit message or PR description?
Agents generate confident descriptions. The description describes what was asked; the diff contains what was done. When they diverge — and they do, regularly — the description is the part humans read first and trust most.
I keep a rule: if the PR body mentions a file or function that doesn't appear in the diff, I stop reviewing and re-ask the agent what it actually changed.
5. Revert test: can you undo it cleanly?
Before approving, mentally (or actually) run git revert. Agent diffs that interleave changes across unrelated files make clean reverts painful. If the diff can't be reverted file-by-file, it probably shouldn't have been one diff.
When the checklist isn't enough
The checklist catches bad diffs. It does nothing for the slower problems: an agent that gradually forgets your conventions over a long session, a prompt that works on Tuesday and drifts by Friday, or token spend that triples because the context window is full of stale tool output.
That longer-horizon stuff is what observability tooling is actually for — session logs, eval suites, spend reports. It's also why this category is exploding right now.
If you want to go deeper, I sell a few small tools in this space at my store — a known-answer eval suite for LLM agents ($17), a stdlib-only CLI that scans agent session logs for leaked keys and dangerous calls ($19), and a transcript search tool ($14). They're deliberately boring: single files, no dependencies, MIT-licensed source you can read before you pay. Everything is at https://demand-dedicated-usc-hydrocodone.trycloudflare.com — and there's a free eval-mini sample if you just want to poke at the format first.
The honest pitch: none of this is magic. It's the same discipline SREs apply to human-written code, applied to diffs that happen faster than anyone can read them carefully. The teams winning with coding agents aren't the ones with the best prompts — they're the ones who made "trust but verify" a two-minute habit.
What checks do you run on agent diffs? I'm collecting them — the best one gets folded into the next version of the eval suite.
Top comments (0)