I have a teammate who never sleeps, never gets bored, and spends every hour poking at our app trying to break it. It is an agent. Every hour it opens the real app, clicks around like a tester would, and files a bug report for anything that looks off.
A second agent then picks up those reports and fixes them. I want to walk you through how it works, what it has actually found, and the parts that do not work at all.
The setup
The app under test is The AI Platform by Zephyr Cloud, a desktop app where teams work alongside AI specialists in channels. The agent drives the real, signed-in desktop app with Playwright over CDP, the Chrome DevTools Protocol. Not a stripped-down test build, the same app a person uses.
That distinction matters. It is not clicking through a mockup or hitting an API. It is looking at the actual product, the way a new user would, and noticing when something feels wrong.
Hunt, reproduce, fix
There are really two agents, running on their own schedules.
The first one hunts. It explores routes, opens dialogs, fills forms, and watches how the app responds. When it finds something, it writes a proper bug report with reproduction steps and a screenshot, and files it as a GitHub issue.
The second one fixes. It picks up an issue, reproduces the bug for itself first, patches it, captures before and after proof, and opens a pull request. A human still reviews and merges. The agents just do the tedious middle bit.
Finding bugs was the easy part
Here is the thing I did not expect. Getting an agent to find bugs is not hard. Getting it to be honest about what it found is the whole game.
An eager agent will report everything as a bug, including things that are working as designed, things caused by test data, and things it simply is not sure about. That noise is worse than silence, because you stop trusting it.
So the hunter has to classify every finding as one of:
- Bug: genuinely broken
- Expected but bad UX: works, but should not
- Environment or data issue: setup, not the product
- Test gap: missing coverage, not a live bug
- Inconclusive: could not confirm it
And it attaches a confidence level. High means it reproduced the issue in that run with clear steps. Medium means probably, but one thing is uncertain. Low means suspicious but not enough to file.
The rule that makes it trustworthy: it only files an issue for a high-confidence, reproducible product bug. Everything else it holds back. I would much rather it say "I could not confirm this" than guess and cry wolf.
What it has actually found
The hunter has filed real issues, and they are the kind of quiet, low-key bug that is easy to miss when you are focused on shipping the next feature. A few real ones:
- A chat channel that just sits on "Fetching message history" forever. No error, no timeout, you are simply stuck.
- A GitHub token field that looks like it saved, but quietly did not because the token format was off. It never told you.
- A settings page that loads the home screen instead, and leaves the home buttons stuck and unclickable.
- A chat pane that shrinks down to a one-pixel sliver the moment you open all the side panels together.
None of these throw an error. None would have failed a normal test. They are just wrong in a small, quiet way, and having something patiently checking for them every hour means they get caught early instead of piling up.
My favourite part: it fixed a bug it caused
Early on, the hunter found that our workflow editor had no unsaved-changes guard. You could edit a workflow, navigate away, and your changes vanished silently with no warning. It filed an issue. We added a guard.
Weeks later, the same hunter came back around and found a new problem: that guard now fired a spurious "Unsaved changes" dialog after every successful save, even when there was nothing unsaved. It filed that too.
Then the fixer picked it up, reproduced it, traced it to a save that navigated before React had re-rendered, patched it, and opened the pull request that closed it.
An agent flagged the absence of a feature, we built it, and the same agent later caught the bug that feature introduced, and another agent fixed it. A full circle, and I barely touched it. That is the moment this went from a fun experiment to something we can actually use.
The rule I care about most: reproduce before you fix
The fixer is not allowed to touch code until it has reproduced the bug itself. No repro, no pull request.
It sounds obvious, but it is the difference between a fix and a guess. Plenty of times the honest outcome is "I could not reproduce this," or "this needs a human," and in those cases it deliberately does not open a PR. A confident-looking patch for a bug you never actually saw is not a fix, it is a liability.
When it does open a PR, it includes a before screenshot showing the real broken state and an after screenshot showing it resolved. And the before shot has to be genuine. An agent will happily produce a convincing "before" from an already-fixed branch if you let it, so that is exactly the thing we lock down.
Now the honest part: what does not work
If I stopped here it would sound like magic. It is not. Here is where it breaks.
The webview wall. CDP only sees inside the app's web view. It cannot see the operating system around it. Native file pickers, the system login window, OS notifications, keychain prompts, none of that is visible to the agent. So a whole category of bugs lives just outside its reach, and it has to be honest about that boundary rather than pretend it checked.
Silent failures make it hallucinate. The hardest problems were never loud errors. They were the quiet ones, where something failed without saying so, and the agent happily narrated a success that did not happen. Most of the engineering went into making failure loud, so the agent notices and admits it instead of inventing a happy ending.
It is not a CI gate. These runs drive a real, running, signed-in app. That is wonderful for realism and useless as the fast check you run on every commit. It is a separate, slower tier, and treating it like a unit test would only make you sad.
What I would tell you to steal
If you want to try something like this, the mechanics are not the hard part. Playwright over CDP, a schedule, a couple of prompts. The hard part, and the part worth your time, is the honesty.
Make the agent classify what it found. Make it attach a confidence level. Make it reproduce before it fixes, and let "I could not" be a perfectly good answer. An agent that files ten real bugs and admits to the three it was unsure about is worth far more than one that files thirteen and makes you check every one.
The bugs were never the impressive bit. Building something I could trust to be honest about them was.
Top comments (5)
The silent-failure class you hit is the one I would build around, because it is invisible by construction to a CDP observer. The GitHub token that "saved" but did not is a UI reporting success while the state underneath never changed. An agent watching the web view sees what the app rendered, so "saved" is reachability, the button did its animation, not exercise, the token actually persisted. The fix that scales better than a louder UI is a second observation point on the far side of the action: read the token back and assert the state it claimed to change actually changed. Reading only the surface keeps missing the bugs that matter most, the ones that fail quietly.
"No repro, no PR" is the sharpest part of the design. A self-assessed confidence label is a vibe on its own, an agent can mark something High and be confidently wrong. The fixer's independent reproduction is the negative control that turns that label into evidence, which is what makes the agent's honesty structural instead of decorative.
"Getting an agent to find bugs is not hard, getting it to be honest about what it found is the whole game" is the line. Your confidence-tiered classification (bug / expected-but-bad-ux / environment issue / test gap / inconclusive) plus the rule that only high-confidence reproducible bugs get filed is doing the actual work, everything else is UI. The bug-that-fixed-a-bug-it-caused loop is a genuinely good story, but the part I'd want more detail on is the "before shot has to be genuine" line, an agent will happily produce a convincing before from an already-fixed branch if you let it. How do you actually lock that down mechanically? Timestamp ordering, git ref pinning, something else? That's the exact kind of confabulation risk that looks fine in the PR and only breaks trust the first time someone catches a doctored screenshot, and it seems like the piece worth writing up in more depth on its own.
Could dig deeper on how this is setup? Are we talking about multiple detailed AGENTS.md's with everything you want or prompts sent to any LLM or a type of copilot SDK running in CI all with their respective MCPs. Just wanted to get a clearer picture. Thank God sharing!
This is where agents become more useful than a normal scheduled test: they can explore weird paths while humans are not staring at the app. The risk is noise. A bug-hunting agent earns trust when each finding includes replay steps, environment state, screenshots or traces, and a clear reason it is not just random poking.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.