We pointed our chaos-QA agent at our own site. It found a shipped bug.

We build an AI QA engineer, so the fair test is the obvious one: point it at
ourselves. On 15 June 2026 we ran Gremlin mode — Prufa's chaos-testing
modality — against our own marketing site, prufa.dev. It found a real,
user-facing bug that our CI had gone green on and shipped that same day. Here is
the whole run, including the parts where the tool was wrong about itself.

What Gremlin mode actually does

A normal Prufa flow checks a path you already know to check. Gremlin is for the
paths you didn't. An LLM-backed agent drives a real browser as a deliberately
difficult user — a confused newbie, an impatient double-clicker, a fat-finger
typist, a back-button masher, a hostile poker — and chooses its own next action
every step. It is the part of QA that needs a model: absorbing an unfamiliar UI
and deciding what a frustrated human would try next.

What the agent never does is decide whether anything broke. That is the same
invariant as the rest of Prufa —
the LLM navigates, plain code verifies —
and it is the whole reason a finding from an LLM-driven tester can be trusted: a
separate layer of deterministic detectors grades the run. A 500 response, an
uncaught exception, a form that accepts invalid input, content wider than the
viewport, two clickable elements overlapping — those are facts, read off the
live page, not opinions.

The bug: a mobile overflow CI had just shipped

Across three personas, every run reported the same verified finding at the
390px mobile viewport: the page was 103 pixels wider than the screen, with
the "Run a free audit" button in the header hanging off the right edge.

Here is the part that makes the case for chaos QA. Earlier that same day, a
commit titled "fix" had added exactly the rule meant to prevent this:

@media (max-width: 520px) { .header-cta { display: none; } }

It never applied. The button is styled by a.btn-primary { display: inline-block },
whose selector specificity (0,1,1) outranks the bare .header-cta (0,1,0), so
the display: none was silently overridden on every phone-width render. The CSS
was valid. The build passed. The linter was happy. CI was green. And the bug
shipped to production, where it sat 103px wide until an agent that had never seen
our codebase resized the viewport and measured the document.

The fix was to out-specify the button:

@media (max-width: 520px) { header a.header-cta { display: none; } }

header a.header-cta is specificity (0,1,2), which beats a.btn-primary
regardless of source order. After the change, a fresh build measured 0px of
horizontal overflow at 390px and the button correctly hidden. The class of bug
matters here: nothing errored. A test that asserts known selectors would have
stayed green forever, because the breakage was in a layout dimension no one had
written an assertion about. You catch that by measuring the rendered page, not by
re-running the path you already trusted.

The safety guarantee, demonstrated on a live site

A chaos tester loose on a real site is only acceptable if it cannot change
anything. In Prufa, mutations are denied by default: the run is dry-run and a
network-layer guard aborts every non-GET request before it leaves the browser. A
destructive click becomes a "would have mutated" finding instead of an action.

We didn't have to take that on faith — the run logged it. Across the three
personas the agent attempted between 0 and 4 mutations each; every one was
blocked, and the run recorded which control it would have submitted. Real
payment instruments are never used at all. To let Gremlin submit forms for real,
you explicitly authorise a domain you own — and even then, hard caps bound how
many submissions it can make.

Where the tool was wrong about itself

The honest part. In an earlier run, two of the gremlin's own detectors fired
on things that were not bugs:

• A "dead-end / error page" detector matched the bare string 500 in ordinary marketing copy (think "save $500"), calling a healthy page an error page.
• A "bad input accepted" detector treated any navigation after a form fill as a successful submission — so clicking a normal link after typing in a field looked like the app had swallowed invalid input.

A verified finding that turns out to be noise costs more trust than a missed bug
costs coverage, so we did not ship around it. We added a detector
false-positive policy: the error-page check now requires a strong error phrase
in the page's prominent text (title or heading) on an error-shaped page, not a
substring match anywhere in the body; the bad-input check now requires a real
form submission — an actual non-GET request — before it fires. Both false
positives are gone, and the genuine findings (the mobile overflow) still land.

We also measured discovery quality directly. On a seeded-bug fixture with five
planted bugs, the agent's first pass found four of five (0.80 coverage); after
we gave it an exploration frontier — a running list of same-site pages it
hasn't visited yet, fed back into each decision — it found all five (1.00),
because it stopped looping one corner and started covering the whole app. That
number is fixture discovery quality, not a claim about your site; the point is
that "does the chaos actually find the planted bugs" is something we test, with
a number, not assert.

Why we publish the misses

A QA product that only tells flattering stories about itself is exactly the
product you shouldn't trust to test you. The mobile bug is a good demo. The
false positives are a better one: they show the failure mode that matters for an
LLM-driven tester — a confident, wrong "this is broken" — and they show the line
we hold against it. The model proposes; plain code disposes; and when plain code
gets it wrong, we fix the plain code, in the open.

Gremlin mode is available on any paid plan — read how it works on
the chaos-testing page, or
run a free audit to see the deterministic side of the same engine on your
own URL first.