I built a BYOK AI agent that tests your app while you're building it, then turns the passing run into a real Playwright spec

Sekhar — Tue, 04 Aug 2026 15:58:49 +0000

I kept doing the same manual thing while building features: change some code, reload the page, click through the flow by hand to check it still works, repeat. So I built five46 to do that clicking-around for me.

What it does

You give it a URL and a goal in plain English — "log in and confirm the dashboard loads" — and an LLM (your own OpenAI/Anthropic/Gemini/Groq/Bedrock key) drives a real local Playwright browser toward it, one action at a time.

When it succeeds, that exact run gets written out as a real, standalone Playwright spec you keep — so the manual check you were already going to do becomes a permanent regression test afterward, with no LLM or five46 involved in ever running it again.

npm install -g five46
five46 test http://localhost:3000 --goal "log in and confirm the dashboard loads"

Why I built it this way

Fully local, BYOK

Everything runs on your machine.

The only thing that leaves it is the page's visible text sent to your LLM provider per step — always disclosed, never hidden. No cloud sandbox sees your app's traffic or screenshots, which matters if your organization can't send application data off-machine.

Honest failure reporting

A failed assertion comes back as a real finding:

Screenshot
DOM snapshot
LLM-generated root-cause hypothesis

These are clearly separated from tooling failures such as:

A stuck or looping agent
An unparseable LLM response

I spent a lot of this project's time specifically hunting for ways it could silently claim success when it hadn't:

A compound goal only half-verified
A weak assertion passing before the real check
One scenario's crash silently hiding another scenario's passing result in batch mode

Speed

LLM round-trip latency dominates wall-clock time, not five46's own code.

Structured planning (enabled by default) performs one upfront LLM call to plan the entire goal, then most execution steps happen without additional live decision-making.

I actually measured this instead of guessing. Using the same login flow against a live demo site with the free Gemini tier:

Mode	Time
Default (structured planning)	~7–8 seconds
Fully adaptive fallback	~11 seconds

Other things it does

API/backend testing — No browser required. Drives real HTTP requests and generates a plain node:test script instead of a Playwright spec.
MCP server — five46 mcp exposes five46 as a tool that IDE coding agents (Claude Code, Cursor, etc.) can call directly.
Self-healing selectors — A stale selector gets one bounded, fully disclosed recovery attempt instead of immediately failing the step.
Story mode — Splits a multi-acceptance-criteria user story into independent goals, executes them concurrently, and reports pass/fail for each acceptance criterion.

Status

This is still an early project.

It's been tested end-to-end with real LLM API keys across dozens of live websites and APIs, but I'd genuinely appreciate feedback—especially on places where "the AI decided this passed" still feels too unproven to trust.

Repository

https://github.com/sekharsdet/five46

DEV Community: Sekhar