My coding agent said "everything preserved." A method was already gone.

Akash Khanna — Fri, 03 Jul 2026 08:27:08 +0000

I've spent most of two decades in enterprise Java, where "done" meant it is working and you have done what you were asked for. It's in production, it works, and you can prove it. So when I started leaning on a coding agent and it told me "done — everything preserved, tests pass,"my instinct was the one every long-tenured engineer has: show me.

Here's the one that actually stung. I had Claude collapse three classes into one — it had grown too complex — and later split another apart. Copy-paste the whole refactored class, done in the chat window and pasted into my IDE, not in an agent that touches the repo directly. It reported everything preserved. I believed it. Days later, something broke in a path no test covered, and I traced it to a method that wasn't there anymore. And this happened even when I asked three different agents to compare and even with the max usage and the best of models I could use at that point. It hadn't made the trip during the merge. Nothing flagged it, because nothing forced the claim to be checked. I just prayed the refactor was clean, and it wasn't.

The thing that clicked: a claim is not evidence, and a rule in the agent's context is not enforcement. "Everything preserved" is an assertion about a mapping the agent reshaped and never verified. "Tests pass" is a claim about a run that, it turned out, never happened that session. I'd been treating the agent's summary as if it were the ground truth. It's the press release, not the primary source.

So I built a small thing to read the primary source. It's a Claude Code Stop hook. When the agent says it's finished, the hook reads the actual git diff and the session transcript — not the summary — and checks the claim against what really changed, before the turn is allowed to end. It's deterministic: no model sits in the check itself, which matters, because the same confident narration that talks an LLM reviewer into "looks good" has nothing to grab onto in a diff parser.

On the refactoring pain specifically, here's the honest scope, so I too do not sound like the agent I faced an issue with. It doesn't try to detect "a method was dropped" — that needs intent, and guessing intent is where false positives come from. It detects the observable consequence: a call that no longer resolves, in a turn that claimed nothing changed. That one reframe makes the messy cases fall out correctly — a rename with the callers updated stays silent; a rename that missed a caller fires on the real broken call; a merge or a split doesn't matter, because it checks whether the symbol is defined anywhere in the whole tree, not just where it used to live. What it can't catch, and says so plainly: a method dropped with no caller left behind — no dangling reference, no textual signal, invisible to a diff. It trades recall for precision on purpose. It would not have caught every possible silent drop in my refactor — but it would have caught the one that broke, because something still called it.

The broader set of things it flags, in plain terms: "tests pass" when no test ran this session, stubs and TODOs in new code, a file the agent said it changed that isn't in the diff, imports pointing at nothing, hardcoded secrets, and rules from your own project docs that got overridden anyway.

What surprised me is how much company this idea has now. There's a wave of people landing in the same place — that you shouldn't ask the same agent to write code and verify it, because it has no incentive to fail its own work; that the validator needs separate context; that the answer is guardrails that make the bad behaviour hard, not prompts that ask nicely. A year ago this felt like a personal irritation. Now it feels like a category forming, which is a good sign I'm not the only one muttering "show me" at a green checkmark.

It's open source (MIT), it's deterministic, and in-session it's tamper-evident, not tamper-proof , not perfect and I am still fixing issues as I find them— the hook shares a filesystem with the agent it audits, so the real guarantee lives in the pre-commit/CI rung, in a separate trust domain (that's in there too). It checks that the agent did what it said — not that what it said was right.

Link's below. The one thing I actually want back: tell me where it false-flags you. False positives are what kill a tool like this, and I'd rather hear about them than not.
https://github.com/akahkhanna/groundtruth

DEV Community: Akash Khanna

My coding agent said "everything preserved." A method was already gone.