I Automated My SaaS Testing With AI Agents (Playwright + LLM Test Generation)

#ai #python #testing #devops

Every indie SaaS founder knows the feeling: the product works on your machine, in your happy path, and you have absolutely no idea what happens when a real user logs in.

There's a thread on r/SaaS that gets this exactly right: "Love building SaaS, but I absolutely hate testing. I ship and pray." Hundreds of upvotes. That's not laziness — it's that manual QA is the most soul-draining, low-leverage task in a solo founder's week.

Here's the workflow I built to kill manual testing entirely. It costs nothing to run, plugs into CI, and the AI part writes the test cases for you.

The three layers of automated QA

Layer 1: A smoke test that catches the obvious stuff

Before any AI, you need a fast "is the site actually working" check. A Playwright script that hits your key routes and verifies the critical elements:

# smoke_test.py — the skeleton, extend per route
import sys
from playwright.sync_api import sync_playwright

CHECKS = [
    ("/",           200, "Welcome"),
    ("/login",      200, "Sign in"),
    ("/pricing",    200, "Pricing"),
    ("/dashboard",  302, None),   # auth redirect expected
]

def run():
    with sync_playwright() as p:
        b = p.chromium.launch()
        page = b.new_page()
        failed = 0
        for path, expect_status, expect_text in CHECKS:
            resp = page.goto(f"https://your-app.com{path}", wait_until="networkidle")
            status = resp.status if resp else -1
            ok = status == expect_status and (expect_text is None or expect_text in page.content())
            print(f"{'PASS' if ok else 'FAIL'} {path} -> {status}")
            failed += 0 if ok else 1
        b.close()
        return failed

if __name__ == "__main__":
    sys.exit(1 if run() else 0)   # exit code for CI

Every deploy, this runs. 20 seconds, zero thought, catches the embarrassing stuff before users do.

Layer 2: AI-generated test cases (the actual superpower)

Here's the part that changes the game. Instead of you writing 200 test scenarios by hand, you write prompts — one per area of your app — and an LLM generates the test matrix.

# ai_test_cases.md — prompt library, one section per area

## Auth & Sessions
Generate 15 test cases for a login flow with email+password.
Include: wrong password, empty fields, unregistered email, rate limiting,
session expiry, password reset token reuse, concurrent sessions,
plus 5 edge cases an experienced pentester would try.

## Payments
Generate 12 test cases for a Stripe checkout.
Include: card decline, insufficient funds, 3DS challenge, webhook replay,
idempotency, refund flow, currency mismatch, plus 4 abuse cases.

## i18n
Generate 8 test cases for a locale switcher.
Include: RTL layout, pluralization, date formatting, missing translation keys.

Why this works: LLMs are great at enumerating test scenarios — it's pattern matching over millions of bug reports. The human skill you're replacing is the "what could break here?" thinking, and honestly, an LLM with a good prompt does that better than a tired founder at 11pm.

Feed each section to your favorite coding model, get a list of concrete test cases, paste the good ones into your Playwright spec. I get 40-60 usable cases per 10-minute prompt session.

Layer 3: CI integration (so it actually runs)

The whole thing is worthless if it only runs when you remember. Wire it into GitHub Actions:

# ci_github_actions.yml — runs smoke + regression on every push
name: QA
on:
  push:
  schedule:
    - cron: '0 6 * * *'   # daily, just in case

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install playwright && playwright install chromium
      - run: python smoke_test.py
      - if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: failure-screenshots
          path: screenshots/

The screenshot artifact on failure is the killer feature — CI tells you what broke, with visual proof, before your users find it.

The payoff

Since I set this up on my own side projects:

Deploys went from "ship and pray" to "ship and watch the green checkmark" — the smoke test is faster than my coffee.
The AI prompt library pays for itself on the first session — it consistently surfaces edge cases I would never have typed out (password reset token reuse, webhook idempotency, 3DS challenge handling).
Daily scheduled runs catch drift — the kind of bug where a third-party script tag or a CDN setting silently breaks something and nobody notices for a week.

Testing doesn't have to be the thing you avoid. The setup is a Playwright skeleton, a prompt library, and a CI file — I packaged the whole thing (skeleton, 10-area prompt library, CI config with failure artifacts, and a README that explains the workflow) over at AgentChip. The blog posts there are free; the template pack is a cheap one-time download for founders who'd rather ship than write test specs.

Your future self — the one debugging a production issue at 2am — will thank you.

Originally published on the AgentChip blog.

DEV Community