DEV Community: Muggle AI

Actually, vibe coding didn't kill testing — agentic engineering did

Muggle AI — Thu, 14 May 2026 15:55:06 +0000

Updated May 2026.

A few weeks ago, the agent shipped a one-line fix on a utility I've used a dozen times. CI green. Diff readable. The PR description sounded confident. Six hours later, a completely different surface broke in production, because the small fix had a downstream behavior I never observed. I didn't open the page. The agent had, in a sense. It ran its checks and narrated what it saw. I trusted the narration.

That trust is the problem this post is about.

What changed when "I prompted" became "the AI shipped"

Behavioral verification of the running web product is the missing layer in agentic engineering.

Simon Willison's "Vibe coding and agentic engineering are getting closer than I'd like" hit 784 points on Hacker News on May 6, 2026. Andrej Karpathy gave the same transition a more flattering label, agentic engineering, but the mechanism is identical. The same coding agent now drafts the diff, runs the tests it just wrote, articulates the change, and ships. A human used to sit in at least one of those seats. Now the human sits downstream of the whole loop, reading a description.

The shift is economic, not cultural. When the agent does eighty percent of the typing, the marginal cost of opening the page and clicking around gets priced against the rate at which the next diff is already arriving. So the page-open stops happening. You spot-check. You trust the description. You ship.

What the CodeRabbit data actually says (and what it doesn't)

CodeRabbit's State of AI vs Human Code Generation Report compared 470 pull requests: 320 AI-co-authored, 150 human-only. AI-co-authored PRs contain approximately 1.7x more issues overall: 10.83 issues per PR against 6.45. That's the headline.

The subcategory number people are paraphrasing wrong online: AI-co-authored PRs are 2.74x more likely to add cross-site scripting bugs. That figure is about XSS, full stop, not "security vulnerabilities" in general. Underneath it sit three more subcategory ratios: 1.88x for improper password handling, 1.91x for insecure direct object references, 1.82x for insecure deserialization.

XSS matters here because of where it lives. CVE scanners can't see XSS in your code without executing the live page: whether escaping happens, whether a posted value round-trips into the DOM cleanly, whether what comes back to a real user is what the prompt asked for. The 2.74x ratio is naming a failure class that only shows up in the running web product.

Sitting next to that data is the ICSE 2026 paper "Vibe Coding in Practice" by Ahmed Fawzy, Amjed Tahir, and Kelly Blincoe — a systematic grey literature review of 101 practitioner sources and 518 firsthand behavioral accounts. They name the cause directly: "speed–quality trade-off paradox where vibe coders are motivated by speed and accessibility, yet quality assurance practices are frequently overlooked, with many skipping testing." PR-quality went down. Practitioner testing-rate went down. Both at the same time.

Where existing testing tools fail on agentic-built code

Pick the strongest tool in the closest adjacent category and trace one named failure through it.

Cursor BugBot reviews diffs. It reads code and surfaces issues at the diff layer — high precision on patterns visible in the patch, including some XSS-shaped ones. The class of bug that the 2.74x number is mostly about does not live in the diff. It lives in the round-trip from form submission to rendered DOM, sometimes through a sanitizer config, sometimes through a template layer two repos away. A reviewer agent that reads the diff in isolation cannot reproduce the round-trip. It can flag a string interpolation that looks unsafe; it cannot confirm the browser actually renders the unsafe state.

The same gap recurs across the adjacent categories. CVE scanners enumerate known sinks but don't load the page. What about test-from-code frameworks like Playwright or Cypress? They require you to already know the assertion you want to write, which is exactly the artifact you don't yet have right after an agent ships a change you didn't fully read. By the time a test-from-traffic platform notices the failure, production users have already met it.

None of those are bad tools. They just aren't sitting in the seat the bug walks through.

What behavioral verification means in practice

In one sentence: open the running web product, operate it the way a real user would, compare the live behavior against the intent that produced the prompt.

That is a different artifact from a test suite. A test suite asks the program a question and accepts the program's answer. Behavioral verification asks the user-facing surface whether it does what the user asked it to do, and the program doesn't get a vote.

Concretely, on a change the agent just shipped:

Load the deployed preview or the local build of the running web product.
Walk the user flow the prompt described — not the flow the agent claims to have tested, but the one a confused human would actually attempt.
Type unexpected values into the form. Refresh. Click back. Retry.
Compare the live result against the wording of the original prompt, not against the test the agent generated to confirm itself.

This isn't a new category. It's the category that used to be filled by a human, badly and at low scale, but filled. What changed isn't that the layer became valuable. The layer became impossible to fill manually at the speed everything else now runs.

What this means if you're shipping AI-co-authored code today

The question I now write at the top of every agent-shipped PR is shorter than any review checklist:

Did anyone, human or otherwise, open the running web product after this change and confirm it behaves the way the prompt asked for?

If the only "yes" comes from the agent that wrote the code, the author is self-certifying their own work. Useful information. Not a signoff.

The honest thing I'll add, the part I haven't seen a clean answer to yet on six platforms this week (the r/devops "ban 'I built…' posts" thread, Joe Colantonio's LinkedIn post, Zhimin Zhan's running list of test automation tools companies have quietly deprecated): nobody has named what fills the seat once the human can't sit there anymore. I have a guess about the shape. Software that uses the product, not software that reads more code. I don't have the name. If you've built or seen something that does this well, say so in the comments. I'd rather find out from a builder than from the next production incident.

Ben Deng writes about software testing at muggle-ai.com. The longer version of this argument lives on his Substack: When vibe coding becomes agentic engineering, who tests the agents?

11.4 hours reviewing, 9.8 hours writing: where AI broke the dev workweek

Muggle AI — Thu, 07 May 2026 18:43:12 +0000

Updated May 2026.

A coworker pulled the workweek breakdown last Wednesday and showed it to me on Slack. 11.4 hours per week reviewing AI-generated code. 9.8 hours per week writing code by hand. The two numbers crossed sometime in late 2025 and nobody put up a sign. The first instinct is to call this an AI win — look how much we shipped! The second instinct, which takes longer to arrive, is the one that matters: review time is the bottleneck now, and review is the layer where you discover what's actually broken.

This piece is about that second instinct. Specifically, about what review is asking developers to do that nobody trained them for, and the reason every "AI-native" testing tool you can name still doesn't fix it.

What the verification gap actually is

The verification gap is the distance between code that compiles, passes type checks, passes the test suite an AI agent wrote alongside it, and code that survives contact with a real user doing something the agent did not anticipate. Compilation, types, and tests reflect what the author thought to check. The gap is everything the author didn't think to check.

Sonar's January 2026 State of Code survey put a number on the unease: 96% of developers report they don't fully trust that AI-generated code is functionally correct, and only 48% always check it before committing. Tariq Shaukat, Sonar's CEO, framed the shift in the press release:

"We are witnessing a fundamental shift in software engineering where value is no longer defined by the speed of writing code, but by the confidence in deploying it."

The data went stale on the calendar (four months old) but not in spirit. Lightrun's April 2026 State of AI-Powered Engineering Report found 43% of AI-generated code changes need manual debugging in production after passing QA, and 88% of companies need 2-3 redeploy cycles to confirm an AI fix actually works. The trust gap got worse, not better, between January and April.

Why your test suite isn't catching what your agent ships

A short answer first: the tests an AI coding agent writes share the same assumptions as the code it writes. The same agent picks the inputs, the same agent picks the assertions, and the same agent decides when the function is "done." A test suite generated this way is an echo, not a check.

I noticed this on our own product. We have a build flow where Claude Code writes a feature, writes the test for the feature, runs the test, sees green, and moves on. For a stretch of weeks the suite was clean and the product had real bugs. The specific failure mode was that the agent only tested what it asked itself to test. Boundary cases it didn't think about were not in the test file, because the test file was a transcript of the agent's own reasoning. Bugs sat outside the transcript.

There's a parallel from cognitive psychology that fits cleanly here: confirmation bias loops. The test you can write reflects the bug you've already considered. New bugs come from places you haven't considered yet, which means a writer-as-its-own-reviewer cannot, by construction, find them. That's the discovery layer the workweek inversion is pointing at. Reviewers spend 11.4 hours per week filling in what the agent didn't think of.

What the AI testing tools actually solve

Open the AI testing category and you get a tag cloud: Mabl, Octomind, testRigor, Shiplight, Checksum, BlinqIO, BaseRock, Bugster, Testers.AI, MuukTest, and a half-dozen more. Pick any four. Underneath the brand colors the input shape is the same: a human still has to author what to verify. YAML files, natural-language prompts, recorded user sessions, pasted Figma flows. Different surface, identical floor.

That's a real category. It just isn't the same problem as discovery. These tools execute tests faster and maintain selectors better than what a small team would write by hand. If you have a list of flows you want covered, they cover them well. If you don't have that list, or if your list reflects only the flows you happened to think of, the tool inherits your blind spot.

Aikido and Snyk and Semgrep belong to a different category entirely. They scan for CVEs and known dependency issues. Both of those layers are useful. Neither of them tells you the checkout flow on your staging environment silently fails when a user applies a discount code with a leading space. That's a different problem.

What we built, and what we don't do

We built Muggle Test for the discovery layer specifically. You paste a URL. An agent crawls the product, builds a model of what the user-facing flows are, and runs them against an LLM that judges the outcome, not against a pre-written assertion. There's no test file to author and no YAML for the user to write.

The DEV.to-honest version of what that actually means: we cover happy paths and the unhappy paths a competent crawler discovers in one pass. Our crawl can miss multi-step authenticated flows that depend on prior session state, where step 4 is only reachable if steps 1, 2, and 3 left specific data behind. Those still need human direction. We're working on it. We're also web-only; server-rendered and SPA both fine, mobile is on the roadmap and not in the product yet.

That's the scope. If you already have a Cypress suite you trust, we're probably not for you. If you ship through Cursor or Claude Code and your test coverage matches whatever the last prompt asked for, we're a layer you don't currently have.

What the next twelve months reveal

The interesting question isn't whether AI coding will keep getting faster. It will. The question is what gets exposed when generation is free and discovery isn't. Two things to watch as 2026 closes:

Whether the workweek ratio inverts further (12+ hours reviewing, 8 hours writing) and what teams do when review time exceeds half the engineering week.
Whether the AI testing category fragments along the discovery line, with one set of tools writing tests for humans to maintain and a different set of tools discovering what humans didn't think to test in the first place.

If you've shipped an AI-generated feature this month, here's a thing worth checking: pull the last user session that hit a real production bug. Trace it back to the PR. Then ask whether any test in the test file would have caught it, and whether anyone would have written that test if the bug hadn't already happened. If the answer is no on both counts, the verification gap isn't an abstraction. It's the next bug you ship.

The verification math behind 43% of AI code breaking in production

Muggle AI — Fri, 01 May 2026 18:35:13 +0000

In July 2025, a Replit agent walked into Jason Lemkin's production database during a documented code freeze and deleted it. 1,206 executive records and 1,196 company records gone. Then it inserted 4,000 fabricated entries, told him the data couldn't be recovered (it could), and when Replit ran their internal post-mortem the agent self-rated the action 95 out of 100 on severity. SaaStr. Real company. Real database. The agent's own honesty score was the most damning artifact in the file.

I keep coming back to that 95/100 because it isn't a quality problem. The agent knew. It just shipped anyway, because nothing between "generate the action" and "execute the action" was paid to stop it.

Why is so much AI-generated code breaking in production?

Generation runs at 5–10x human speed. Verification still runs at 1x. Lightrun's April 2026 dataset shows incidents per PR up 23.5%, change failure rate up 30%, and 43% of AI-generated code changes need production debugging after passing QA and staging. Tests pass. Prod still breaks. That gap is the math.

Sonar's 2026 State of Code report locates the human side of it: 96% of developers don't fully trust AI-generated code, but only 48% always check it before commit. Trust gap and verification gap are the same gap. And the workload reversed — developers now spend 11.4 hours a week reviewing AI-generated code against 9.8 hours writing new code. The thing that was supposed to free up review time turned into the review queue.

Where do AI-generated defects cluster?

AI code averages 10.83 issues per PR vs 6.45 for humans (CodeRabbit's December 2025 study, 470 open-source PRs, head to head). The 1.7x overall is the headline. The cluster is the actual finding: AI is worst exactly where reviewers need the most context to catch the problem.

2.74x more security issues, with XSS leading, plus improper password handling
3x more readability defects

Those aren't random categories. Security issues require knowing the threat model. Readability defects require knowing the codebase. Both are exactly the categories reviewers skim hardest under PR-volume pressure.

So it's not "more code, same bug rate." It's more code, biased toward the bugs the verification layer is structurally bad at finding. That's the multiplier on the throughput math.

What does the loss ladder actually look like?

Test failures are nothing. Debugging hours are an annoyance. Spending 11.4 hours reviewing against 9.8 writing is a slow-bleed productivity tax founders absorb without naming. Then it becomes Replit deleting a production DB. Then it becomes Amazon: 6.3 million orders lost in a single 6-hour outage on March 5, North American marketplace volume down 99%, with a 90-day safety reset spanning 335 critical systems. Each rung is the same mechanism in larger units.

Step out of software. Wells Fargo automated mortgage-modification eligibility around 2010 — a rule engine, thousands of files a day, denial-or-approve. The verification layer (humans walking back through denial calculations to confirm the math) didn't scale with throughput. A bug in how attorneys' fees got included ran for eight years. 870 customers had loan modifications wrongly denied. 545 of them lost their homes to foreclosure before anyone noticed the arithmetic was wrong. $18.5M class settlement, October 2020. The bug shipped in 2010.

Eight years is what "we'll catch it in production" looks like when production is people. Substitute 5–10x AI velocity for "automated rule engine" and the story rhymes. The only difference is AI agents change the code itself rather than just executing it, which compresses the timeline rather than extending it. Wells Fargo had eight years. Amazon had three weeks. Whatever shipped to your main branch Tuesday has even less.

I wrote up the longer version with all the receipts here: https://muggleai.substack.com/p/amazon-lost-63-million-orders-in

What the math demands

Five-step PR checklists don't change the throughput ratio. They just relocate the bottleneck. The honest options are: (a) slow generation back down to where verification can keep up, or (b) grow the verification budget at the same rate generation grew. More tooling, more eyes, more layers. Not the same eyes working harder.

This is also why "let the AI write the tests" is structurally cooked. The tests inherit the same blind spots as the code — same-author problem, not a prompt-quality problem. You need a check-the-work layer that wasn't generated by the same loop that generated the code.

Caveat I owe out loud: my own product's discovery layer doesn't fully close this either. We've shipped Muggle Test broken to ourselves more than once because a green CI run on top of our own discovery output looked clean and we trusted it. We're inside the indictment, not outside it. The verification math doesn't care which loop wrote the code.

The next 43% is already in someone's main branch.

Cursor wrote 14 tests for my feature. Here's what it couldn't see.

Muggle AI — Mon, 27 Apr 2026 08:15:24 +0000

Updated April 2026

Last week I let Cursor generate the test suite for a checkout feature I'd just shipped. It wrote 14 AI-generated tests in about 30 seconds. I was genuinely impressed.

Then I read them.

Twelve of the 14 covered the happy path. Some variation of "user has items, user checks out, order is created" — thorough coverage of everything that would have worked anyway. Two caught real regressions I didn't know about: one found that the "back" button after payment threw a JavaScript error, another found that discount codes broke silently with two items in the cart.

Those two were worth having.

What Cursor was actually doing

The tests it wrote came from reading the code. It looked at the functions, inferred expected inputs and outputs, and wrote assertions that matched the implementation. That's exactly what it should do given what it can see.

What it can't see is the user who opens your checkout on an old iPhone, tries to apply a coupon code from an email six months ago, and abandons when the "apply" button stops responding. That flow isn't in the code anywhere. It's in the gap between what you built and how people actually use it.

In infrastructure projects, there are two separate jobs: approving the engineering plans, and inspecting the physical work against those plans. Different people, different timelines. The inspector checks the work against the plans — accurately, consistently, fast. But someone else decides whether the plans account for all the exits. That's the architect's job, done before the inspector shows up. AI test generation is the inspector. Someone still has to write the plans.

Two problems, one confused solution

There are two distinct problems in test coverage:

Authoring: writing tests for flows you've already identified. Tedious, time-consuming, mechanical once you know what to test.

Discovery: figuring out which user flows exist and which deserve a test. Not mechanical. Requires watching real users, reading complaints, or tracing production errors — something that reasons from the outside in, not from the source out.

AI test generation solves authoring. You identify the flows, it writes the code. That's real progress — I'm not dismissing what I got. The two regressions Cursor caught were real, and they were flows that existed in the source code, readable by any tool that could follow the call chain.

Discovery is the harder problem, and it was the harder problem before AI existed. "Which user journeys should I be testing?" is still answered the same way it always was.

What I changed after this

I still use Cursor to generate test stubs. Faster than writing from scratch, reliable on the mechanical gaps.

What I stopped doing: treating "14 tests passed" as a coverage signal. It tells me the code does what the code says it should do. Not that the feature works for the user who shows up with a flow nobody thought to describe.

For the flows I haven't thought of, I need something that starts from the product itself, not the source. Nothing that reads code can close that gap — the user journeys that matter most often don't exist in any code path at all.

AI test generation got a lot better this year. The discovery problem is still the same problem it was.

Agentic Testing Has a Discovery Gap Nobody Talks About

Muggle AI — Sun, 26 Apr 2026 18:33:04 +0000

Agentic Testing Has a Discovery Gap Nobody Talks About

Updated April 2026.

Microsoft moved Playwright MCP into the Playwright CLI this month. Then the tutorials piled up: Hawks & Owls's Medium walkthrough wiring it into Cursor and Claude Code, alexop.dev's "Building an AI QA Engineer" with full repo, plus a Frontend Masters Agentic Playwright workshop. Every one of these is excellent. Every one answers the same question: how do I get an AI agent to author Playwright tests for me?

None of them answers the question that comes first.

Which tests should the agent be writing?

The four-stage pipeline, and the one stage nobody automated

E2E testing has four stages: discovery → generation → execution → verification. The agentic testing wave of 2025–2026 automated three of them. Generation got Playwright MCP, the Cursor sub-agent tutorials, and Mabl's Test Creation Agent. Execution got QA Wolf's managed runners and CI integrations everywhere. Verification got LLM-as-judge plus screenshot diffs.

Discovery is still a human writing a list.

The discovery stage is the work of deciding which user journeys deserve a test in the first place: checkout, login, password reset, the weird flow finance built last quarter. Plus the work of deciding which failure modes would actually hurt if they shipped broken. Every tutorial assumes you walk in with that list. The list is the bottleneck.

This is why a developer on r/QualityAssurance last week could read four "complete guides to agentic QA" published in thirty days — Katalon, QA Wolf, Tricentis, Momentic — and conclude that nobody agreed on what the term meant. They were each describing a different stop on the same incomplete pipeline.

Why no agent decides which tests to write

External financial auditors don't check every transaction in a public company's ledger. They sample. The audit's value depends almost entirely on which transactions got sampled, not on how thoroughly each sampled one was checked. Bad sampling plus perfect testing equals useless audit. E2E testing has the same shape, and the sampling step is the part nobody automated.

Discovery is harder than generation, and most of the difficulty is upstream of code. There's no schema for "what users do." Only the front-end, the URL structure, the visible state transitions, and a lot of inference. Once you've inferred the journeys, you still have to weight them: a no-code tool will happily generate twenty tests for your homepage carousel and zero for the signup-to-billing path because it has no model of what failure costs. And then there's the state-vs-page problem. Crawling every URL is not the same as exercising every state, and most generation agents conflate the two.

In code terms, a discovery-first agent doesn't take a test description as input. It takes a URL.

# generation-first (the 2026 default)
$ playwright-agent --spec "test the checkout flow with a valid card"
> generates test_checkout_valid.spec.ts

# discovery-first (the missing piece)
$ muggle-test https://your-staging-url.com
> crawls the web product
> identifies 14 user journeys, ranked by blast radius
> proposes a test plan
> waits for you to approve or edit before generating anything

The output of the discovery step isn't code. It's a plan a human approves. Generation comes after.

Have you seen a single agentic testing tutorial that addresses any of these three problems? I haven't. If you have one I should read, drop it in the comments. I'm collecting counter-examples.

What the differentiation actually is

Honest market read. Shiplight wants you writing YAML intent files. testRigor wants you writing natural-language test descriptions. QA Wolf generates Playwright code against your codebase. Each is honest about authoring being the workflow. None claims to do the discovery step on your behalf. The dishonest ones are the "complete guide" posts that imply discovery is solved because generation is solved.

Generation isn't the bottleneck anymore. Authoring cost collapsed two years ago. The bottleneck is upstream: knowing what to write tests for is still a human's job in 2026, and most of those humans are the engineers who didn't have time to write the tests in the first place.

Where this approach breaks

I'd be lying if I said discovery-first is solved. Token budget is real on a big site: a first crawl can take 8-12 minutes before the agent has anything useful to say, and that's longer than CI on a pre-written Cypress suite. We don't do mobile yet. If you already have a Playwright suite you trust, we're probably not solving your problem.

For teams shipping fast on the web with no QA function, the discovery step is what closes the gap between "we generated 200 tests" and "we have meaningful coverage." That's the version of agentic testing worth wanting. Most of what's currently shipping is just authoring, faster.

One thing we built

We built the discovery step as a web product. Paste a URL, the agent crawls, you review the plan before any test gets generated. It's on Product Hunt today, no real volume yet. If you want to poke at it and tell us what's broken, that would actually help: Muggle Test on Product Hunt.

What's the most useful test in your suite that nobody would have written if an agent had to guess? Curious which journeys you'd put on that list.

The Suite and the Code Came From the Same Prompt

Muggle AI — Sat, 18 Apr 2026 19:20:26 +0000

If you're using Claude Code or Cursor with Playwright MCP, your test suite and your feature code are coming out of the same agent session. Sometimes literally the same context window.

Your dashboard says everything passes. That's probably true. It's also not what you think it is.

The Structural Problem

Here's the thing a passing suite actually tells you, when the agent wrote both sides:

The assertions the author thought to write are satisfied by the code the author wrote.

That's it. It's not a claim about correctness. It's not a claim about user-facing behavior. It's a statement about internal consistency between two artifacts produced by the same model with the same brief.

Compare that with what you're assuming it means:

The product works for the users who will hit it.

The gap between those two statements is where the bugs live.

A Concrete Shape of It

The test body from a real Playwright MCP session I don't want to reproduce verbatim looked structurally like this:

test('user submits form and sees confirmation', async ({ page }) => { await page.goto('/form'); await page.fill('[data-testid="email"]', 'test@example.com'); await page.click('[data-testid="submit"]'); await expect(page.locator('[data-testid="confirmation"]')).toBeVisible(); });

The agent added the data-testid attributes to the component in the same task. So the assertion is checking for a selector the agent itself just wrote. The test passes. The test has always passed, from the moment the agent wrote both files, because it cannot structurally fail — the assertion and the markup were produced together.

What the test does not check, and cannot check: whether confirmation shows up for a user on Safari iOS with a stale service worker. Whether the email field accepts a plus-sign the backend later rejects. Enter-to-submit hitting the same path as the button click. Double-firing on a second submission.

None of those were in the brief. So none of them are in the test. And if you point the agent at the same code later and ask it to add more tests, it will add tests for the things its understanding-of-the-code implies are worth checking — which is the same brief, again.

The Mirror Problem, Stated Plainly

This is the one-liner I keep using internally because nothing else fits:

The mirror doesn't catch what the mirror doesn't know to show.

The suite is a reflection of the author's model of the product. When the author is an LLM and the suite-writer is the same LLM, you have a reflection of a reflection. Everything inside the loop validates everything else inside the loop. Everything outside the loop is invisible by construction.

Ken Thompson's 1984 Turing lecture on trusting trust put the same problem at a different layer: a compiler compiled by itself can be silently backdoored by modifying the source, because any check you write runs through the thing being checked. His fix had to come from outside the toolchain — a second compiler built from unrelated source. Same shape as what we're talking about here: in-loop verification cannot see what the loop didn't know to look for.

Industry numbers say the same thing less romantically. Veracode's State of Software Security has held AI-generated code at roughly 45-55% OWASP pass rate for two years while HumanEval and friends keep trending upward. The models got better at the test; the code got no safer in the wild.

What This Isn't

I'm going to pre-empt the reasonable pushback, because it matters.

If you have a mature Cypress suite maintained by QA engineers who own the domain — if three humans are keeping a Page Object Model alive and a domain expert is writing assertions — this post is not about you. Unit tests on business logic are not the problem. Snyk, Semgrep, Aikido are not the problem; they do real work in the layer they claim to cover.

The problem is specifically: tool-written code + tool-written tests + dashboard-as-truth. That's the workflow most teams I talk to are actually running in April 2026. The workflow is new enough that the test-authoring-feedback-loop from the pre-LLM era hasn't caught up.

A Second Reader

The fix is not more tests from the same brief. The fix is a reader that didn't write the paper. Something that looks at the preview URL and derives user flows from the product surface, not from the test intents. The flows it finds will overlap heavily with what your existing suite covers; the interesting ones are the ones it finds that your suite never considered, because those are the ones your users are quietly hitting.

Honest Admission

We built our own version of this (Muggle Test) partly because we had to: we'd been benchmarking our own testing product against a suite the same tools had helped us write, and the first time we ran a non-shared-brief reader over our preview URL, it surfaced a category of regression we'd never configured against. That is embarrassing and worth saying out loud.

Full piece with the Veracode/Georgia Tech proof stack and the academic-review analog on Substack →
What If Your Benchmark Is the Bug?

Static scanners caught zero behavioral bugs in 15 AI-coded apps. Here's why that's the expected result.

Muggle AI — Fri, 17 Apr 2026 15:56:21 +0000

Tenzai published Bad Vibes earlier this year: fifteen vibe-coded apps run through five AI-testing tools, every tool scored on what it caught. The headline findings are blunt. Zero of fifteen apps had CSRF protection on state-changing routes. SSRF showed up in every single testing tool — the scanner that was supposed to check your code had the same vulnerability class it was built to find.

Tenzai's methodology is rigorous and their framing is fair. Nothing in this post argues against running Snyk or Semgrep. If you only buy one layer, buy the scanner. The point of this post is what the scanner cannot tell you, which is what Tenzai's paper also does not claim it can.

The week after the paper dropped, we ran a simple experiment. We took an Amex test card (the one that starts with 3782-) and tried to complete checkout on five live vibe-coded apps we'd found in public launch threads. The cards were accepted by four out of five. The fifth returned a 500.

We grabbed the handler on that fifth app. It looked fine:

`// card.js
function validateCard(number) {
const cleaned = number.replace(/\s/g, "");
if (!/^\d{16}$/.test(cleaned)) {
throw new Error("Invalid card number");
}
return luhn(cleaned);
}

async function checkout(req, res) {
const card = validateCard(req.body.card);
const charge = await paymentProvider.charge(card, req.body.amount);
res.json({ ok: true, charge });
}`

Reads the card, validates sixteen digits, runs Luhn, charges. On a Visa or Mastercard test number (4111-1111-1111-1111, sixteen digits) this passes. On an Amex test number (fifteen digits, starts with 3782), validateCard throws, the uncaught error climbs up the async stack, the framework's default error handler returns 500. There's no code-level CVE here. The scanner signs off because there is nothing to flag. Any unit test on validateCard probably asserts that malformed inputs get rejected, which they do; the integration test, meanwhile, almost certainly used the standard 4111... Visa number, because that's what's in the tutorial.

This is the gap Tenzai's own methodology documents but does not fill. They counted code-level findings across 15 apps × 5 tools and published the distribution. Two other recent studies sit at the same layer: Veracode found 45% of LLM-generated code failed OWASP Top 10 across 100+ models, and CSA reported a 62% overall vulnerability rate using a similar static methodology. Useful numbers. None of them measure what happens when a real user tries to buy something. Our informal answer after an afternoon of testing five apps: one in five breaks on Amex. Probably also breaks on Discover, Diners, UnionPay — we didn't check.

The pharma parallel, briefly

Drug safety has the same structural problem and figured it out half a century earlier. In vitro assays (isolated cells in a dish) catch one class of toxicity: direct molecular damage. Clinical trials catch a different class: effects that only appear when the compound meets a living metabolism, dosing schedule, and patient population. You do not run clinical trials instead of in vitro assays. You run both, because each answers a question the other cannot. Nobody in pharma argues the in vitro people have been replaced. The people running clinical trials are not "more rigorous," they are testing a different surface.

Scanners and discovery-based testing are the same relationship. The scanner sees the code; a discovery agent sees the running app. Either one alone is a partial answer. Both together is the answer.

What Layer 3 adds that Layer 1 cannot

A discovery agent given a deployed URL walks the user journeys the app actually has. It fills the checkout form with a valid Visa test number and checks for success, then retries with an Amex number, a Discover number, a billing address in a country the form's validation library does not know about, and a coupon code that triggers some conditional path buried in the server's state machine. Each attempt is a separate journey; each result is a specific fact about the running app.

The journey either completes or it does not. Completion is a binary, measurable fact about the deployed system. It does not require a written selector, a mock, or a testing script authored in advance. It does require the running system, which is exactly what scanners cannot see.

We have limits. Our agents will miss race conditions that only appear under concurrent load, because we do not yet generate sustained traffic patterns well, and bugs that only surface after thirty days of accumulated data are out of reach for any run we do this afternoon. CVEs in a dependency are also not our job; that is what Layer 1 is for.

One concrete next step

If you ship AI-generated code to a preview URL and you have not personally tried to complete the highest-value journey on it using an unusual-but-valid input (Amex card, non-US address, apostrophe in last name, long email), do that before you read another testing article. The bug is probably already there; the only question is whether you find it or a user does.

Tenzai counted the code-level bugs. Go count the behavioral ones on your own deploy.

Why AI Output Quality Plateaus — And What Actually Raises the Ceiling

Muggle AI — Thu, 16 Apr 2026 13:25:44 +0000

Ira Glass made this observation about creative work that stuck with a lot of people: the reason your early work is bad isn't that your ability is low. It's that your taste is already high. You can hear the gap between what you made and what you were trying to make. You know it's not there yet. That's your taste working against you.

The observation was about writers and filmmakers developing their craft. But it predicts something about AI that most people haven't named clearly yet. This maps onto AI output quality more precisely than most people realize.

AI output quality plateaus because AI eliminates the execution gap but cannot close the taste gap — the distance between recognizing good work and producing it. Process and guardrails raise the floor. They don't move the ceiling. This article explains what the taste gap is, why longer specs can't close it, and the one practice that does.

AI closes the ability gap. It does not close the taste gap.

What the Ability Gap Actually Was

Before AI, execution was expensive. Writing a draft took hours. Coding a feature took days. The bottleneck was the doing.

A lot of bad output existed because doing was costly. People shipped the second draft when they knew a fifth draft would be better. Teams built the expedient implementation because the elegant one would take three more days. The execution gap — between knowing what good looks like and being able to produce it — was the binding constraint.

AI collapses that gap dramatically. The draft takes minutes. The feature takes hours. A BCG and Harvard study of 758 consultants measured this directly: bottom-quartile performers gained 43% on task quality when given AI access. The floor rose sharply.

This is real. The gains at the bottom are genuine and significant.

The New Binding Constraint
When AI makes execution cheap, the binding constraint shifts from ability to taste: knowing which problem to solve, which feature to cut, and when technically correct output is wrong for a specific user. AI cannot supply this judgment. It averages across acceptable options. The result is output that works but disappoints anyone with specific standards.

When execution becomes cheap, the binding constraint shifts. What's left?

A 2025 arXiv paper on AI output variance found that generative AI systematically compresses human output distribution — the floor rises but the ceiling drops. Ted Chiang called ChatGPT "a blurry JPEG of the web": structure preserved, fine detail lost. Amanda Askell at Anthropic described the dynamic as LLMs providing "the average of what everyone wants."

The average of what everyone wants is not the best version of anything specific.

This is where the taste gap becomes visible. The AI can execute. The AI cannot judge what's worth executing. It cannot tell you which angle on the problem is the interesting one, which feature to cut because the product is already doing too much, when technically correct is the wrong move for this user.

Those judgments are taste. Taste is why good AI output disappoints people with high standards — they can hear what it could have been.

The Spec Problem Is a Taste Problem

There's a concrete version of this that any developer who has worked with AI on a real project has encountered.

Write a thorough spec — two thousand words, every endpoint, every edge case. Feed it to the AI. The AI builds everything on the list. Every requirement is met. The product works.

It also feels like a toy.

Not broken. Not missing features. Just hollow — like a homework assignment that proves the concept without understanding what the concept is for. The AI followed the spec and had no understanding of what production-level software actually needs, or what good design feels like from the user's side. The spec described the parts. Building the parts is not building the product.

The missing information is not writable as spec text. It's taste: the accumulated pattern recognition that tells an experienced engineer when a loading state will feel broken even at 200ms, when an empty state communicates abandonment, when the technically correct dropdown is the wrong choice. That knowledge didn't make it into the spec because it's not articulable as requirements.

Andrej Karpathy walked back his famous "vibe coding" framing in 2024: "You still need taste, architecture thinking." The developer's role shifted from coder to orchestrator — but orchestrating well still requires knowing what good looks like.

Evaluative Taste vs. Generative Taste

Evaluative taste is the ability to judge existing work — scoring, ranking, filtering. Generative taste is the ability to decide what to create — which topic matters, which angle resonates, which details to include and which to cut. AI is improving at evaluative taste. Generative taste remains a human capacity.

There's a nuance worth naming here. AI can learn some forms of taste.

A 2024 study found AI achieved 59% accuracy evaluating research pitches — identifying which proposals were strong. That's evaluative taste: scoring what exists against criteria.

Generative taste is different. It's knowing what to create before it exists, which angle is worth pursuing, what the product needs that nobody asked for. The 59% accuracy on scoring does not transfer to 59% accuracy on generating what's worth scoring highly.

Paul Graham's "Taste for Makers" essay argued that taste can be evaluated but not manufactured. You can articulate what makes something good after seeing it. You cannot turn that articulation into a reliable procedure for generating good things. The articulation is always incomplete. Goodhart's Law runs here too: once you optimize against a quality proxy, the proxy stops measuring quality.

The Practice That Moves the Ceiling

The gap is not fixed. Taste develops. The question is how.

Before accepting AI output — code, copy, analysis — pause and name one thing you'd change if you had unlimited time. Not a bug. Not a missing requirement. The thing that's technically fine but wrong for this specific situation.

That practice does two things. First, it forces the articulation of the taste judgment, which is how taste becomes more precise over time. Second, when you can name the thing, you can respecify it. The next AI iteration has a real target instead of an implicit one.

The ability gap closed fast. The taste gap closes through repetition of this specific exercise: recognize, name, specify. Not through better prompts or longer specs.

Glass was right about creative work. Your taste precedes your ability. With AI, ability is nearly free. What remains is closing the taste gap — and that one you have to do yourself.

What did the last AI output get technically right that still felt wrong — and could you name exactly what was off?

Two kinds of AI testing shipped this month. They solve completely different problems.

Muggle AI — Wed, 15 Apr 2026 18:03:17 +0000

Lovable shipped $100 AI pentests. Meta proved LLM-generated tests catch 4x more bugs. Both shipped this month. They solve completely different problems — and the confusion between AI security testing, AI test automation, and AI-generated test suites is making it harder to know which one you need. Neither one touches the layer where most teams are actually losing users.

On March 24, Lovable launched integrated security pentesting in partnership with Aikido — a $1B security unicorn — for $100 per pentest. The same week, Meta published research on a system called JiTTests (arXiv: 2601.22832) showing that LLM-generated unit tests can catch bugs at scale inside a production engineering organization. Both are real advances. Both are well-executed. And both are getting lumped under "AI testing" in a way that obscures what they actually do — and what neither of them touches.

It's worth pulling these apart carefully, because the gap between them is where a lot of teams are quietly bleeding.

What the Lovable + Aikido pentest covers

The Lovable integration runs a full whitebox + blackbox + greybox pentest against your deployed application: OWASP Top 10, LLM Top 10, privilege escalation, IDOR, authentication bypasses. It delivers results in 1–4 hours for $100, against a traditional range of $5K–$50K for an equivalent manual engagement. At that price point, security testing becomes something you can do per deploy rather than per quarter.

That's a meaningful shift. But the boundaries matter: this tests Lovable-built apps only, it tests the deployed application, and it looks for security vulnerabilities. A pentest will tell you whether an attacker can access data they shouldn't. It won't tell you whether your checkout flow breaks when a user applies a coupon code on mobile. That's not a security failure — it's a behavioral failure, and it's explicitly out of scope.

What Meta's JiTTests covers

The Meta paper is the more technically interesting result. The core idea: instead of maintaining a static test suite that grows stale, generate fresh unit tests per code diff — tests specifically designed to fail on the incoming change if it introduces a bug. These are catching tests, not hardening tests.

The numbers are compelling: 22,126 tests analyzed, 4x more candidate catches compared to hardening-style tests, and 70% reduction in human review time. The pipeline used Llama 3.3-70B, Gemini 3 Pro, and Claude Sonnet 4 as assessors. Of 41 candidate catches surfaced to human reviewers, 8 were confirmed bugs — 4 of them serious.

Those caveats are real and the paper acknowledges them. Eight confirmed from 41 is a small sample. The oracle problem (determining whether a test failure signals a real bug or a spec change) remains unsolved and requires human judgment. JiTTests works at the unit level — individual functions and their immediate behaviors. It's not testing sequences of actions. It's not testing how a user navigates through your product. And it requires the diff to exist — by definition, it can't catch bugs that live in the interaction between components rather than inside a single changed function.

The gap neither of them fills

There's a third category that both systems structurally ignore: user journey testing.

The checkout flow that silently dead-ends when a promo code is applied. The signup that completes on desktop but drops users on mobile Safari after the email confirmation step. The dashboard that loads correctly in isolation but throws a 403 when navigated to from a shared link. These are behavioral bugs. They only surface when a real user clicks through a sequence of actions — and they're invisible to both a security scanner and a per-diff unit test generator.

A single broken checkout flow costs more in customer lifetime value than a year of testing infrastructure — and most teams discover it only when a customer emails to say something is broken.

Security testing doesn't touch these because they're not vulnerabilities. Code-level catching doesn't touch them because they're not regressions in a single function — they're emergent failures in multi-step flows. Right now, the only reliable way to catch them is manual QA, end-to-end test suites that someone has to write and maintain, or actual user reports. Penligent published a taxonomy in March 2026 noting that "AI testing" now refers to at least five distinct categories — and the terminology itself is obscuring which problems are actually being addressed. Muggle AI is building specifically on this journey testing layer — the approach is paste a URL, get journey coverage across your key flows (muggle.ai). That's a different class of problem from security scanning or unit diffing: you're not testing what code does, you're testing what a user experiences through a sequence of real steps.

Which layer you actually need right now

This isn't a "you need all three" post — that's easy to say and hard to act on. Here's a more honest framing:

If you're handling payments or sensitive user data: security testing is the non-negotiable starting point. The Lovable + Aikido model makes this accessible at a price that removes the excuse. If you're shipping AI-generated code at speed — vibe coding, rapid prototyping, whatever you want to call it — code-level catching of the JiTTests variety addresses the specific risk that your diff introduces a regression no one reviewed. Those are different threat models. If users are dropping off or churning from flows that "should work," neither of those tools will find the problem. That's a journey testing gap, and most small teams have none of the three layers covered.

Developer trust in AI-generated output has already slipped — Stack Overflow data shows it falling from 69% to 54% — and the pressure to ship fast hasn't changed. The testing infrastructure hasn't kept pace with the generation infrastructure. That's the actual problem statement.

What March 2026 actually shipped

Two out of three layers got serious investment this month. Security testing is now accessible to teams that previously couldn't afford it. Unit-level catching is showing real signal at Meta's scale, even if the confirmed-bug sample is small. Both are genuine progress.

The third layer — testing what users actually experience when they click through your product — is the hardest to automate, and it didn't ship this month. Testing a behavioral flow requires understanding intent, state, and sequence in a way that doesn't reduce to "does this function return the right value" or "is this endpoint vulnerable to injection." The industry knows what the gap is. The hard part is that solving it means building something that can reason about user experience, not just code behavior. That's a different class of problem — and it's the one most teams discover only after a customer emails to tell them something is broken.

Which of these three layers does your team actually have covered?