Coding agents are pretty good at finding suspicious code.
Give one a repository and it will happily spend an hour following calls, reading tests, checking configuration and coming back with a list of things that look wrong.
The awkward part starts after that.
Suppose the agent tells me that a particular code path can leave the application in an invalid state.
What do I actually know at this point?
Not much.
I know that the model found something worth checking.
It could be a real bug. It could also have missed a guard somewhere else, misunderstood how the code is called, assumed the wrong configuration, or simply produced a convincing explanation for something that cannot happen.
I've become increasingly uncomfortable with treating those two things — finding a possible problem and proving the problem exists — as the same job.
Asking another model isn't a great answer either.
If Claude finds a bug and another model agrees with Claude, that's useful. I probably want to investigate it sooner.
But agreement is still not evidence.
If possible, I want the claim to survive something that doesn't care what either model thinks.
Run the code. Reproduce the behaviour. Write a test that fails for the reason being claimed. Check the actual configuration. Look at the log, the dependency version, the Git history, whatever can answer the question directly.
Sometimes that is easy.
If an agent says a function throws on a particular input, call it with that input.
Sometimes it isn't. Architecture and concurrency bugs are obvious examples. You may only be able to establish part of the claim, or discover that reproducing it would require assumptions you cannot confirm.
That's fine too.
"Couldn't verify this" is useful information.
I'd much rather keep a suspicious claim unresolved than turn it into a finding because a model sounded confident.
This is also why I don't think static analysis and coding agents are competing approaches.
Static tools are great when we already know what to look for and can describe it precisely.
The interesting thing about an agent is that it can notice something nobody thought to write a rule for.
That is the part I want from it.
Let it dig through an unfamiliar repository and come back with weird questions.
Then make those questions earn their way into the bug list.
The more code agents can write and inspect, the more this distinction matters. Generating another thousand lines is getting very cheap. Carefully checking whether a subtle claim about those thousand lines is true is not.
So when an agent tells me it found a bug, I don't want the next question to be:
"How confident are you?"
I want it to be:
"How can we prove it?"
Top comments (0)