Every team I've worked with has shipped something that passed its automated checks but still confused people. The button did its job, but nobody could find it. The form submitted, but only after you'd typed your phone number three different ways. Automated tests only catch what someone thought to write an assertion for. A good human QA tester is the one who notices what nobody wrote down, and says "this is annoying."
So I wanted to know: can an AI agent do that part of the job? Not "does the page load," but "would a real person get confused here?"
And if it can, how much does it depend on how you ask?
A coffee shop with 40 problems hidden in it
To test a tester, you need to know the right answers in advance. A real website doesn't give you that. If the agent reports five issues, you can't tell whether it missed ten or there were only five.
So I made a fake one: Ponto Coffee Roasters, a small online store with a shop page, product pages, a basket, a two-step checkout and a sign-up form. There is no real server behind it. A small piece of code pretends to be one, with random delays and an order that always fails on the first attempt, because how an app behaves while waiting and failing is a big part of user experience.
Then it got filled with problems, from two sources. Claude built the store and planted the first 20 bugs. I then asked Codex, a different AI, to introduce 20 more, and kept that request and the new bug list outside anything Claude could see. Claude was also the model doing the testing, so I didn't want all the bugs to come from the same model that would be hunting them.
The 40 bugs cover a wide range. There's pale grey text you can barely read, an "Add to bag" button that gives no sign it worked, and a review step that says shipping is "Free" while the total quietly includes €6.90. The basket's minus button actually adds coffee, the logo takes you to the basket, and on a phone the checkout spills off the screen. Some are quiet: the search box clears itself when you click away, and the surname field stops accepting letters after eight characters.
Three ways of asking
The experiment changed only one thing: the prompt.
Level 1 was the one-liner most people would type first:
You are a QA engineer. Test the website and look for UX errors.
Level 2 gave the agent a role (a senior QA tester who cares about user experience), some context about the store, a way of working (main journey first, then mistakes and small screens) and a report format.
Level 3 was level 2 plus a full checklist, built on Jakob Nielsen's 10 usability heuristics (a classic list of what makes interfaces easy to use) and the basics of WCAG, the web accessibility guidelines. It also told the agent to test in passes: desktop, mistakes, waiting and failure, phone size, keyboard only.
The checklist was written before any bug existed. Otherwise it might have described the answers without me noticing.
Keeping it fair
Every run used the same agent and model: Claude Code, Anthropic's coding agent, running Claude Sonnet 5.5. It drove a fresh browser through Playwright, a tool that lets an AI click, type and look at web pages. The browser was its only tool. It couldn't read files or use a terminal, so it couldn't peek at the source code or the answer key. It could only use the site the way a visitor does. Each level ran three times, because AI output varies from run to run.
Every report was scored against the answer key: a full point if a bug was clearly described, half a point if it was only touched on, nothing if it was missed. I'll call the result a weighted detection score. To reduce scoring subjectivity, a separate AI agent scored all nine reports against the answer key and scoring rules, without seeing my scores. Its numbers came within half a point of mine on every report.
The results
| Prompt | Runs | Average score | Percentage | Time per run | Cost per run |
|---|---|---|---|---|---|
| Level 1: one-liner | 24, 25, 25 | 24.7 / 40 | 62% | 1.7 min | $0.51 |
| Level 2: role + context | 21, 22, 23.5 | 22.2 / 40 | 55% | 1.9 min | $0.51 |
| Level 3: full checklist | 27.5, 31.5, 32 | 30.3 / 40 | 76% | 3.5 min | $0.81 |
The first surprise is right there in the middle row.
Surprise one: the "better" prompt did worse
I expected each level to beat the one before. Level 2 lost to the one-liner.
Level 2 produced longer, more polished entries, with a title, location, impact and severity for every issue, and it scored lower. The one-liner runs wrote quick, dense lists. The reporting format may have contributed, but these runs don't isolate its effect from differences in how each agent explored the site. It's a pattern worth testing, not a conclusion.
Surprise two: a checklist changes behaviour, not just wording
Level 3 didn't only describe things better. It did different things.
It was the only level that ever pressed the Tab key. That's how you test a site for people who don't use a mouse, and it's how all three level 3 runs noticed there was no visible keyboard focus. None of the six other runs caught it.
Every single run, at every level, shrank the window to phone width at some point. But only level 3 caught the checkout overflowing on a phone, in two of its three runs. None of the level 1 or 2 runs reported it, and one of them stated there was "no horizontal scroll at 375px."
The checklist worked like a test plan. It didn't make the agent smarter. It made it go to places it otherwise skipped.
Surprise three: a checklist also narrows attention
The opposite happened too. The newsletter box in the footer has a spinner that never stops. One of the one-liner runs clearly reported it, and the other two got partial credit. None of the six runs with detailed instructions got any credit. They followed the journey they were given, and the footer wasn't on it.
I've seen this in people before. Give a tester a checklist and they'll cover the checklist. Freedom finds odd things, and structure finds thorough things.
The bugs nobody found
Five bugs were missed by all nine runs. Four came from Codex's set:
- the search box only accepts 12 characters
- the search clears itself when you click away
- the surname field stops at 8 characters
- the Huila Sunrise card on the shop page shows another coffee's picture
The fifth was one of Claude's: double-clicking "Add to bag" adds the item twice.
Most of these have one thing in common. They don't produce an obvious error message or broken layout. Finding them requires trying a boundary value or comparing behaviour across interactions. None of the reports identified the image mismatch between the shop and product pages.
The double-click bug is a messier case, and it taught me something about the experiment itself. Codex's set also made a single click add two items, so a double click added four. Because one click already added two items, the repeated-click issue was harder to distinguish. Planted bugs can hide each other.
The bugs everyone found
On the other hand, every run reported the logo leading to the basket and the password shown in plain text. The same went for the "Free" shipping that wasn't, the basket row that ignored quantity and the price that changed between pages: all nine runs caught these numerical inconsistencies, including the one-liner runs.
The scorer flagged at most one false alarm per run, mostly complaints about slow loading caused by my test environment rather than the app. Every run also found real problems nobody planted, like form fields with no autocomplete hints, error messages that screen readers wouldn't announce, and a header that still said "Create account" after you'd made one.
What I'd be careful about
This is a small experiment. Nine runs with one model is a hint, not proof, and another model could shift the numbers. The tests ran in a cloud sandbox that blocked web fonts, which isn't quite a real visitor's experience. All 40 bugs came from AIs, two different ones, so a human would probably plant different problems. And some planted bugs interacted: the clipped product description partly hid the wall-of-text problem, and the doubled "Add to bag" muddied the double-click bug. That changes how visible some bugs were, and it means the five universal misses don't all point to a single weakness in the agent.
What I'd try next: a checklist that learns
I haven't tested the workflow below yet. It's what I'd try next.
The checklist made the agent thorough, and the one-liner's freedom made it curious. So I'd run two complementary passes before each release: a checklist run, and a free-form run told to explore and report anything odd. The free run's job isn't to find more bugs. Looking back at my saved scores, I combined runs in pairs, giving each bug the best credit either run earned. One free-form run plus one checklist run averaged 31.6 out of 40, and two checklist runs averaged 32.3. That's a retrospective calculation, not a separately tested workflow, but it suggests a free run won't raise the score. Its job is to find the bugs that aren't on the checklist yet.
Those discoveries, along with what human testers find and what real users complain about, feed the checklist. Some become general habits, like "type an unusually long value into every field," and stay there because they need judgment.
Others are specific regressions, like "the newsletter spinner must stop." For those, the checklist is a cheap placeholder. Adding a line takes seconds. An end-to-end test (a script that clicks through the site the same way each time, written with a tool like Cypress) takes real attention to write. So the line goes on the checklist first. Later, when there's time or the list gets long, it graduates into a Cypress test and leaves the list.
Neither half is a guarantee. My own newsletter results show that an instruction in a prompt doesn't ensure the agent follows it, and scripted tests can be flaky too. But the split puts judgment where AI helps and repetition where scripts are cheaper.
So, can an AI be your QA tester?
In this small experiment, a browser-driven AI found many of the planted functional and usability problems, quickly and cheaply. The best setup scored 76% for under a dollar a run. A checklist improved coverage, but several quiet bugs went unnoticed in every run. I'd use it as a first pass before a release, not as a replacement for a person who types a long name, clicks away by accident and thinks "wait, didn't this look different a second ago?"
The thing I'll take away is about prompting. A prompt for an AI tester works a lot like a test plan. What you put in it tends to get covered carefully. What you leave out may never be visited. And like any good test plan, it should never be finished: every bug the agent misses today is a candidate line for tomorrow's checklist.
The whole experiment is open: the store, the 40 bugs, the three prompts, every agent's report and the scores. You're welcome to run your own prompt against it and see how many you can catch. github.com/rodrinobrega/agent-qa-experiment
Have you tried using AI agents for QA or UX reviews? I'd love to hear what they caught for you, and what they walked right past.
Top comments (0)