AutoQA-Agent: Write Acceptance Tests in Markdown, Run Them with AI + Playwright

NEE — Fri, 19 Dec 2025 02:31:42 +0000

AutoQA-Agent is a Docs-as-Tests CLI: you write acceptance tests in plain Markdown, and it runs them via a lightweight Claude Agent SDK loop + Playwright.

It focuses on:

Reducing script fragility with snapshot-first, ref-first interactions
Letting non-engineers contribute (Markdown specs)
Leaving great artifacts (logs / snapshots / screenshots / traces)
Exporting passing runs into standard @playwright/test specs

TL;DR

Write test steps in Markdown.
Run: autoqa run <spec-or-dir> --url <baseUrl>
Get artifacts under .autoqa/runs/<runId>/...
If the spec passes, AutoQA-Agent can export to tests/autoqa/*.spec.ts

Why this exists

UI automation tends to break for boring reasons:

Locators become unstable after small UI refactors.
Test code is often unreadable for PMs/QAs.
Failures are hard to diagnose without good context.

AutoQA-Agent treats acceptance tests as living documentation, then uses an agent loop to drive a real browser and recover from transient failures.

Quick start

Prerequisites

Node.js >= 20
Claude Code authorized (recommended) or ANTHROPIC_API_KEY

Install & build

git clone https://github.com/terryso/AutoQA-Agent.git
cd AutoQA-Agent
npm install
npm run build
npm link # optional

Initialize

autoqa init

Run a spec

autoqa run specs/saucedemo-01-login.md --url https://www.saucedemo.com/

Debug (headed browser)

autoqa run specs/saucedemo-01-login.md --url https://www.saucedemo.com/ --debug

What a Markdown spec looks like

# Login

## Preconditions
- Test account exists

## Steps
1. Navigate to /login
2. Verify the login form is visible
3. Fill the username field with standard_user
4. Fill the password field with secret_sauce
5. Click the "Login" button
6. Verify the user is redirected to dashboard

Notes:

Base URL is provided via --url (the “Base URL” line in preconditions is for readability).
Steps starting with Verify/Assert (also supports Chinese “验证/断言”) are treated as assertions.

How it works (high level)

Parse Markdown into preconditions + ordered steps.
Run an observe → act → recover loop using Claude Agent SDK.
Use accessibility snapshots (with stable refs) to drive ref-first actions.
When a tool/assertion fails, return structured errors (instead of crashing) so the agent can retry, bounded by guardrails.

Artifacts you get for free

After a run:

.autoqa/runs/<runId>/
├── run.log.jsonl
├── ir.jsonl
├── screenshots/
├── snapshots/
└── traces/

This makes failures much easier to debug locally and in CI.

Export to `@playwright/test`

Successful specs can be exported to:

tests/autoqa/*.spec.ts

So you can keep what worked and run it as classic Playwright tests later.

Roadmap (short)

Richer export (more semantic parsing + more assertion mappings)
More example specs and demo projects
Continuous improvement in docs and diagrams

Try it / contribute

Repo: https://github.com/terryso/AutoQA-Agent
Issues & PRs: welcome

DEV Community: NEE