I let a one-shot Sonnet 5 agent QA a live app. Then I ran our QA tool on the same one.

Shahid Helal — Fri, 18 Sep 2026 11:48:24 +0000

Every week, someone tells me our company shouldn't exist.
The line is always some version of this: "QA? I'll just point Claude or Sonnet at my app." And honestly — it's a fair shot. These models are good. If a one-shot agent already does what we do, then we're selling snake oil.

So we stopped arguing and ran the experiment.

We took a real, production EU rail-booking app. We turned a strong Sonnet 5 agent loose on it — from scratch, no hand-holding — to hunt bugs and write tests the way a sharp QA engineer would. Then we ran our own tool's full pipeline on the exact same app. Then we handed both outputs, unlabelled, to an independent judge and asked one question: which is better, and why?

The scoreboard

The agent was genuinely good — a smart pair of eyes for an hour, and it found real bugs. I'll say that plainly, because the interesting part isn't "AI bad." It's where a single clever run quietly runs out of road.

Our tool wrote 415 executable, grounded test cases. The one-shot agent wrote 36. That gap isn't a trick — it's the difference between "someone poked at your app for an afternoon" and "someone mapped it and left you a suite that reruns forever."
The depth held up under a blind judge: feature coverage 9/10, test-case depth 8/10, edge cases 8/10. Every expected result tied to behavior actually observed on the page — not a plausible guess that reads well and dies in CI.

Then the part I didn't expect: our tool caught bugs the agent walked straight past. A checkout total that silently drifted from the price on screen. A station index that mis-sorted — just enough to be wrong, not enough to notice. Boring, expensive, real: the exact bugs that survive a quick look and reach production.

The honest bit

Neither output was a superset of the other. The one-shot agent found a couple of things our tool didn't. A single pass — human or AI — always misses something, in both directions.

Which is the whole argument, really.

QA was never one brilliant run. It's coverage that's broad, grounded, deduplicated and repeatable — that keeps working on the next deploy, and the one after that, without a human re-prompting it at midnight. An agent gives you a run. A real QA system gives you a system.

So next time someone says "I'll just use an AI agent for QA": they're right that they can do a run. They're wrong that it's the same thing.

We measured it. On a real app. Blind. And then we shipped the improvements.

(Numbers from a blind September 2026 audit on real production apps, anonymized. Ask me anything in the comments — happy to show my work.)

DEV Community: Shahid Helal

I let a one-shot Sonnet 5 agent QA a live app. Then I ran our QA tool on the same one.

The scoreboard

The honest bit