Most AI-powered browser testing tools ask a language model what to do at every step of every run. That's impressive in a demo and painful in CI: every run costs tokens, takes longer, and can make a different choice than the last one. Flaky tests caused by your test tool are the worst kind.
I wanted the opposite trade-off: let AI do the tedious part once, then get out of the way. That's s1pilot, an open-source (Apache-2.0) web testing platform built on Playwright.
The loop: explore, replay, heal
- Explore. You write a test case in plain language. Each step is an action and an expected result:
steps:
-action: Choose colour {colour} and size {size}, then add to cart
expected: Cart badge shows 1 item
assert:
-{ type: elementText, role: status, contains: "1" }
s1pilot opens a real browser, observes the page (the accessibility tree plus numbered element boxes), and asks a decision model which element to act on.
Replay. When a run passes, the resolved locators and actions are saved as a learned script. The next run replays it like any Playwright script, with zero model calls. It's fast, cheap and deterministic.
Heal. When the page changes and a step breaks, only that step goes back to the model. The fix is proposed, not silently applied: it lands in a review queue for a human to approve.
Why a "System 1" model and not a chatbot
The model that drives the browser isn't asked to write text. It's asked a multiple-choice question: "Which of these 14 elements completes the step?" It answers with a choice and a probability, like a classifier.
That gives you two things a chat agent can't easily give you:
- Inspectability. Every step records the options, their probabilities and the chosen action. When something goes wrong, you can see why.
- A principled "I don't know". If the top option scores below a confidence threshold, s1pilot doesn't guess. An attended run pauses for your approval; an unattended run marks the step blocked. For harder calls it can escalate to a "System 2" chat model or a person.
Thresholds are set per model in capability profiles, and s1pilot calibrate suggests better ones from your human review outcomes. It never applies them automatically.
More than a runner
Testers asked for the boring things, so s1pilot ships a local-first test management app: projects, environments, suites, plans, a live run view with the browser screencast, manual and hybrid runs, a review queue, defects and analytics. Tests sync both ways with YAML files in your repo, so changes go through pull requests. In CI, s1pilot run writes JUnit/JSON/HTML reports and exits 0 passed, 1 failed, 2 needs review, 3 config error.
Secrets are referenced (env:NAME, secret:key), never sent to a model or written to traces. Production environments are read-only by default, and sensitive actions (pay, delete, send) need approval.
Try it
npm install -g s1pilot
npx playwright install chromium
s1pilot demo # [confirm: no API key needed]
Then point it at your own app and your own model. It supports System 1 endpoints, OpenAI-compatible log-probability servers, and chat models for escalation.
How it was built, and what's next
I wrote the design as small specs (they're all in docs/specs/) and built them one by one with heavy help from coding agents. Known gaps are listed in the README: for example, Playwright traces aren't secret-masked yet.
Feedback, bug reports and skeptical takes are all welcome.

Top comments (1)
The review queue for healed steps is the part I'd watch most. A heal can pick a different element that still satisfies the step's assertion, for example the 'Cart badge shows 1 item' check passing because a different badge on the page also says 1. The replay then stays green with the wrong locator and nobody looks at it again. Showing the reviewer the old and new element side by side, plus whether the step's assertion would also have passed on the old element, would make the approval a real check instead of a rubber stamp.
On the confidence thresholds: 'calibrate' from human review outcomes is only trustworthy if the review set includes steps the model got right at low confidence, not just the ones it escalated. Otherwise the threshold is fitted on the hard cases only and drifts toward always asking.
Have you measured how often a heal gets approved unchanged versus edited by the reviewer?