Most AI-powered browser testing tools ask a language model what to do at every step of every run. That's impressive in a demo and painful in CI: every run costs tokens, takes longer, and can make a different choice than the last one. Flaky tests caused by your test tool are the worst kind.
I wanted the opposite trade-off: let AI do the tedious part once, then get out of the way. That's s1pilot, an open-source (Apache-2.0) web testing platform built on Playwright.
The loop: explore, replay, heal
- Explore. You write a test case in plain language. Each step is an action and an expected result:
steps:
-action: Choose colour {colour} and size {size}, then add to cart
expected: Cart badge shows 1 item
assert:
-{ type: elementText, role: status, contains: "1" }
s1pilot opens a real browser, observes the page (the accessibility tree plus numbered element boxes), and asks a decision model which element to act on.
Replay. When a run passes, the resolved locators and actions are saved as a learned script. The next run replays it like any Playwright script, with zero model calls. It's fast, cheap and deterministic.
Heal. When the page changes and a step breaks, only that step goes back to the model. The fix is proposed, not silently applied: it lands in a review queue for a human to approve.
Why a "System 1" model and not a chatbot
The model that drives the browser isn't asked to write text. It's asked a multiple-choice question: "Which of these 14 elements completes the step?" It answers with a choice and a probability, like a classifier.
That gives you two things a chat agent can't easily give you:
- Inspectability. Every step records the options, their probabilities and the chosen action. When something goes wrong, you can see why.
- A principled "I don't know". If the top option scores below a confidence threshold, s1pilot doesn't guess. An attended run pauses for your approval; an unattended run marks the step blocked. For harder calls it can escalate to a "System 2" chat model or a person.
Thresholds are set per model in capability profiles, and s1pilot calibrate suggests better ones from your human review outcomes. It never applies them automatically.
More than a runner
Testers asked for the boring things, so s1pilot ships a local-first test management app: projects, environments, suites, plans, a live run view with the browser screencast, manual and hybrid runs, a review queue, defects and analytics. Tests sync both ways with YAML files in your repo, so changes go through pull requests. In CI, s1pilot run writes JUnit/JSON/HTML reports and exits 0 passed, 1 failed, 2 needs review, 3 config error.
Secrets are referenced (env:NAME, secret:key), never sent to a model or written to traces. Production environments are read-only by default, and sensitive actions (pay, delete, send) need approval.
Try it
npm install -g s1pilot
npx playwright install chromium
s1pilot demo # [confirm: no API key needed]
Then point it at your own app and your own model. It supports System 1 endpoints, OpenAI-compatible log-probability servers, and chat models for escalation.
How it was built, and what's next
I wrote the design as small specs (they're all in docs/specs/) and built them one by one with heavy help from coding agents. Known gaps are listed in the README: for example, Playwright traces aren't secret-masked yet.
Feedback, bug reports and skeptical takes are all welcome.
Top comments (0)