Green tests don't mean signup works. How to catch the bugs AI coding agents ship before your users do, with a free open-source CLI.
You ask your agent for a small refactor. It finishes in 40 seconds. The diff looks clean, TypeScript is quiet, lint passes, every test is green.
You merge.
Two days later someone asks why nobody has signed up since Tuesday.
That's the bug AI coding agents are best at shipping. Not a syntax error. Not a failing test. Code that's correct in the file the agent was looking at, and wrong everywhere else.
How it happens
Here's a common pattern, and you can reproduce it with the demo app in the repo below.
The agent "tidies up" the signup API. The handler now reads emailAddress. The form still sends email.
TypeScript doesn't complain, because the request body is "any" at that boundary. Unit tests pass, because the agent updated them to the new field name. And the agent says "done".
In the browser, the Create account button does nothing. No message on screen. A 500 in the network tab. A TypeError in server logs nobody is reading.
Your tests checked the code. Nobody checked the product.
The fix is boring, and that's the point
Before trusting an AI change, click through the flows your business depends on: signup, login, checkout. In a real browser, the way a user would.
Nobody does that by hand after every change. So I automated it and open-sourced it.
Meet stillworks
npx stillworks init
npx stillworks run --url http://localhost:3000
The init command creates a journeys.config.ts file and installs Chromium. Each journey is a list of named steps in plain Playwright:
{
name: "signup",
steps: [
{ name: "Open the signup page", run: ({ page }) => page.goto("/signup") },
{ name: "Fill in email and password", run: async ({ page }) => {
await page.getByLabel("Email").fill(`test+${Date.now()}@example.com`);
await page.getByLabel("Password").fill("correct-horse-battery-staple");
} },
{ name: "Submit the form", run: ({ page }) => page.getByRole("button", { name: "Create account" }).click() },
{ name: "See the welcome screen", run: ({ page }) => page.getByText("Welcome").waitFor() },
],
}
Don't want to write them? Ask your agent: "Read journeys.config.ts and add a journey for inviting a teammate."
The part that actually saves you time
When a journey breaks, you don't get a stack trace to decode. You get a REPORT.md file written to be pasted straight into your agent:
The signup journey failed at step 4 of 4: "See the welcome screen"
Failed requests:
POST /api/signup → 500 Internal Server Error
Response: {"error":"TypeError: Cannot read properties of undefined (reading 'toLowerCase')"}
Fix this without changing unrelated files. Find the root cause; don't change the
journey or weaken the check to make it pass. These journeys still pass, keep them
passing: login, checkout.
The failed step, console errors, the failed request with the server's response, and a screenshot. Your agent goes straight to the renamed field, instead of "adding more error handling to the signup form".
Or skip the copy-paste entirely
In Claude Code, a Stop hook runs your journeys every time Claude finishes. If one breaks, the report goes back to Claude and it keeps fixing until everything passes. It skips turns where no file changed and never blocks you when the app isn't running.
In Cursor, a rule makes the agent verify after every change. For Codex and Copilot, there's a ready-made AGENTS.md section. And a GitHub Action runs your journeys on every Vercel preview deploy.
Watch it catch the bug in under a minute
git clone https://github.com/valmirjunior2109-spec/stillworks && cd stillworks
npm install && npm run build && node dist/cli.js install
npm run demo:broken # in one terminal
node dist/cli.js run --url http://127.0.0.1:4173 --config templates/journeys.config.ts
MIT licensed. No telemetry. Works on macOS, Linux and Windows.
Where this is going
stillworks runs when you ask. I'm also building Publishub, the version that runs by itself: on every PR, catching broken flows and security holes in AI-written code, and comparing each run with your last working version. It's not out yet, and early access is free: https://getpublishub.com/?utm_source=devto#early-access
If this saves you from even one "nobody signed up since Tuesday" moment:
- Try it: npx stillworks init
- Star the repo so other devs find it: https://github.com/valmirjunior2109-spec/stillworks
- Tell me in the comments which flow you'd check first. I'm adding templates for the most requested stacks.
Top comments (0)