DEV Community

Valmir Júnior
Valmir Júnior

Posted on Edited on

Your AI wrote the code, the tests passed, and signup is still broken

Green tests don't mean signup works. How to catch the bugs AI coding agents ship before your users do, with a free open-source CLI.

You ask your agent for a small refactor. It finishes in 40 seconds. The diff looks clean, TypeScript is quiet, lint passes, every test is green.

You merge.

Two days later someone asks why nobody has signed up since Tuesday.

That's the bug AI coding agents are best at shipping. Not a syntax error. Not a failing test. Code that's correct in the file the agent was looking at, and wrong everywhere else.

How it happens

Here's a common pattern, and you can reproduce it with the demo app in the repo below.

The agent "tidies up" the signup API. The handler now reads emailAddress. The form still sends email.

TypeScript doesn't complain, because the request body is "any" at that boundary. Unit tests pass, because the agent updated them to the new field name. And the agent says "done".

In the browser, the Create account button does nothing. No message on screen. A 500 in the network tab. A TypeError in server logs nobody is reading.

Your tests checked the code. Nobody checked the product.

The fix is boring, and that's the point

Before trusting an AI change, click through the flows your business depends on: signup, login, checkout. In a real browser, the way a user would.

Nobody does that by hand after every change. So I automated it and open-sourced it.

Meet stillworks

npx stillworks init
npx stillworks run --url http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

The init command creates a journeys.config.ts file and installs Chromium. Each journey is a list of named steps in plain Playwright:

{
  name: "signup",
  steps: [
    { name: "Open the signup page", run: ({ page }) => page.goto("/signup") },
    { name: "Fill in email and password", run: async ({ page }) => {
        await page.getByLabel("Email").fill(`test+${Date.now()}@example.com`);
        await page.getByLabel("Password").fill("correct-horse-battery-staple");
    } },
    { name: "Submit the form", run: ({ page }) => page.getByRole("button", { name: "Create account" }).click() },
    { name: "See the welcome screen", run: ({ page }) => page.getByText("Welcome").waitFor() },
  ],
}
Enter fullscreen mode Exit fullscreen mode

Don't want to write them? Ask your agent: "Read journeys.config.ts and add a journey for inviting a teammate."

The part that actually saves you time

When a journey breaks, you don't get a stack trace to decode. You get a REPORT.md file written to be pasted straight into your agent:

The signup journey failed at step 4 of 4: "See the welcome screen"

Failed requests:
POST /api/signup → 500 Internal Server Error
Response: {"error":"TypeError: Cannot read properties of undefined (reading 'toLowerCase')"}

Fix this without changing unrelated files. Find the root cause; don't change the
journey or weaken the check to make it pass. These journeys still pass, keep them
passing: login, checkout.
Enter fullscreen mode Exit fullscreen mode

The failed step, console errors, the failed request with the server's response, and a screenshot. Your agent goes straight to the renamed field, instead of "adding more error handling to the signup form".

Or skip the copy-paste entirely

In Claude Code, a Stop hook runs your journeys every time Claude finishes. If one breaks, the report goes back to Claude and it keeps fixing until everything passes. It skips turns where no file changed and never blocks you when the app isn't running.

In Cursor, a rule makes the agent verify after every change. For Codex and Copilot, there's a ready-made AGENTS.md section. And a GitHub Action runs your journeys on every Vercel preview deploy.

Watch it catch the bug in under a minute

git clone https://github.com/valmirjunior2109-spec/stillworks && cd stillworks
npm install && npm run build && node dist/cli.js install
npm run demo:broken      # in one terminal
node dist/cli.js run --url http://127.0.0.1:4173 --config templates/journeys.config.ts
Enter fullscreen mode Exit fullscreen mode

MIT licensed. No telemetry. Works on macOS, Linux and Windows.

Where this is going

stillworks runs when you ask. I'm also building Publishub, the version that runs by itself: on every PR, catching broken flows and security holes in AI-written code, and comparing each run with your last working version. It's not out yet, and early access is free: https://getpublishub.com/?utm_source=devto#early-access

If this saves you from even one "nobody signed up since Tuesday" moment:

  1. Try it: npx stillworks init
  2. Star the repo so other devs find it: https://github.com/valmirjunior2109-spec/stillworks
  3. Tell me in the comments which flow you'd check first. I'm adding templates for the most requested stacks.

Top comments (0)