Resque

Posted on Jul 23

I let an AI test my app 55 times in a week. Here's what it broke (and what I broke).

#testing #devops #webdev #ai

I had a passing test. Green checkmark, banked in my suite, ran clean every time I looked at it. "Pet sprite renders, no broken image."

Then I opened the app in an incognito window and the pet was a broken image icon. On every page. For days, apparently.

The test was green the whole time. This is the story of how that happened and what I did about it — but also the story of the 54 other rounds of letting an AI testing agent use my live app before I trusted my own code.

some context

I was building Nora for a hackathon — a pixel-art study app with spaced repetition, Feynman explanations, and a virtual pet that grows as you learn. About 50 routes, Next.js 16, Supabase, deployed on Vercel.

The hackathon theme was "Build the Loop." So instead of testing once at the end, I put TestSprite (an AI testing agent) between me and every deploy. The rule: write a feature, let the agent open the live app in a cloud browser and use it like a real person, get a verdict. Pass = banked. Fail = I get a failure bundle with screenshots, DOM snapshots, and a root cause guess. Fix it, redeploy, rerun.

I did this 55 times over a week. Here's what I learned.

how the loop actually works

Four commands:

# describe what the feature should do, run against live app
testsprite test create --plan-from ./plans/signup.plan.json \
  --type frontend --run --wait \
  --target-url https://norastudy.vercel.app

# it failed? pull the bundle
testsprite test failure get <test-id> --out ./.testsprite/failure/

# fix. redeploy. then:
testsprite test rerun <test-id> --wait

# green? banked. next feature.

That's the whole thing. Not complicated. The interesting part is what it finds.

the blank page every new user would've seen

I always test logged into my own account. My account has data. So my dashboard looks great.

TestSprite creates a fresh account every run. Fresh account = no data = my redirect logic dumped it on a blank /app shell with nothing on it. No onboarding, no welcome, just white.

Every new signup would've seen that and left. The fix was one redirect line. But I never would've found it because I never test as someone with zero history.

the analytics page that took four rounds to pass

Round 1: 404. I renamed a route during a refactor and forgot.

Round 2: fixed the route, agent couldn't navigate there through my nested sidebar. Got lost clicking.

Round 3: told it to hit the URL directly. Worked! Then it asserted a chart that doesn't render on a low-activity account.

Round 4: fixed the assertion. Still failed.

The problem was the test's name. It was called "Analytics page shows stats and charts." The agent read its own name as context and kept hunting for charts that weren't supposed to be there anymore.

I renamed it. It passed.

The name of your test is part of the prompt. If you update a test plan but leave the old name, the agent reads the name and gets confused. Weird lesson to learn from a testing tool. But it stuck.

the lying test

So — the pet sprite thing.

Nora has companion pets (Pokémon-like sprites). I had a banked test: "pet sprite renders, not a broken image placeholder." It ran green every single time.

The sprites were hotlinked from a third-party CDN. That CDN rate-limits. Sometimes it serves the image, sometimes it 404s. My test was running during "good" windows. The green was real but meaningless — it was measuring someone else's uptime, not my code.

I fixed it by self-hosting all 39 sprites locally. Then banked a new regression test.

But here's the thing I want to be honest about: TestSprite did not catch this bug. I found it myself — I was demoing the app to a friend on their laptop, opened it fresh with no cache, and there it was. Broken images everywhere. The existing test had been masking it with flaky passes. I logged it as a manual find in my iteration log, not a tool catch.

If your testing writeup isn't honest about what the tool finds vs. what you find yourself, none of it means anything.

testing the layer a browser can't see

This was genuinely a surprise to me.

A browser test tells you "the page loads." Cool. But it can't tell you whether the database actually rejects unauthorized access. Those look identical in the UI — the page either loads your data or loads nothing. You can't tell from the outside whether RLS is working or broken.

TestSprite has a --type backend mode that runs Python server-side instead of Playwright:

testsprite test create --type backend \
  --name "RLS rejects unauthorized reward manipulation" \
  --code-file ./test_rls_security.py --run --wait

I wrote 7 of these. They prove anonymous users can't call reward RPCs, can't read other users' profiles, can't insert fake flashcards. Stuff that's invisible from the browser.

If you use Supabase and you don't have something testing your RLS from the outside... you're just trusting it works. Which is brave, I guess.

the test I deleted

I wrote a mobile navigation test. The component worked fine on my phone. But the cloud runner uses a fixed desktop viewport and can't resize.

So the test kept failing. Not because my code was wrong — the runner just couldn't see the mobile nav.

I deleted the test and wrote exactly why in my log. A suite should only contain tests that can give honest verdicts. A test that fails because of infrastructure limitations trains you to ignore red. That's worse than having no test at all.

I also broke the CLI (and fixed it)

I ran the TestSprite CLI so many times that I hit bugs in it. A timeout that swallowed output. A credential edge case. So I fixed them upstream — 10 PRs, all merged. Including a new command called test flaky that replays a test N times and reports stability.

Which is exactly what I needed after the green-test-lying-to-me situation. Irony noted.

MCP vs CLI vs portal — honest comparison

I tried all three. Here's where I actually ended up:

MCP server (IDE integration): used it day one. You tell your AI assistant "test this project" and it figures out what to test. Good for getting started. But for a repeatable loop with specific assertions, the CLI gives you more control.

CLI: where I lived. 90% of my workflow. Create, run, fail, fix, rerun. Scriptable. Works in CI/CD. This is the thing.

Web portal: used for forensics only. When a test fails you can see the exact screenshot, the DOM at that moment, network logs. Useful for debugging. Not where I created or managed tests.

the final count

55 iterations, 7 build days
57 tests banked (50 frontend, 7 backend), all green
100+ total runs
10 real product bugs found and fixed
1 test deleted (runner limitation, documented honestly)
10 PRs merged to the CLI itself

what I'd actually tell someone starting this

Let it test as a stranger. The signup bug existed because I only ever tested with my own data-rich account.

Read blocked runs carefully. blocked means the agent couldn't finish the scenario (usually your assertion is bad). failed means your app broke. Different problems.

Test the invisible layer. If your security is at the database level, a browser test can't prove it works.

Distrust green tests that lean on third parties. They measure someone else's reliability, not yours.

Be honest. I deleted a test, credited a manual find as manual, and documented every "the tool didn't catch this" moment. That honesty is the only reason the other 54 iterations are credible.

The app is open source: github.com/lxcario/Nora. The full iteration log: LOOP.md. TestSprite CLI: github.com/TestSprite/testsprite-cli.

If you've tried AI-powered testing or have thoughts on flaky test detection, I'd genuinely like to hear about it.

Built for TestSprite Hackathon S3. This won first place — but the workflow is the part that actually stuck with me after.