Yesterday I pivoted my shelved testing project into edge-pilot: let an agent drive a browser once, then replay that flow forever at zero LLM cost, with the captured API calls as the oracle. The Day 1 milestone was small on purpose — an MCP skeleton that can record and replay one real checkout end to end.
Today I tried to build it. The demo worked immediately. Replaying it did not.
What worked
I stood up the MCP authoring server. An external agent (Claude Code, in my case) drives the browser through edge-pilot's tools — ep_navigate, ep_snapshot, ep_click, ep_fill, ep_save_scenario — and edge-pilot does the hardening behind each action.
From one prompt, the agent walked our real shop: opened the treatments page, added an Aromatherapy Massage, filled in guest, therapist, date, and time, entered a test card, and completed checkout. A real order came back. I froze the verified run and replayed it with no model calls at all — it created the next orders (I ended the day with test orders #513950001 through #513950006). The "drive once, replay free" loop exists.
Then I tried to make replay reliable, and every weak assumption fell out at once.
What broke
The recording was garbage. Replay kept failing because the recorder simply wasn't capturing every action the user took. There's no clever fix here — if the recording is lossy, nothing downstream can be trusted. My note to myself for the week:
Recording must be perfect. It has to save every single one of the user's browser actions.
The selector logic grabbed the wrong field. My old resolveTarget walked a fixed priority list — css → data-testid → name → aria → role → text — and judged each signal in isolation. On the real payment form (MUI), every signal was non-unique: three inputs shared #outlined-basic, there were no test IDs, everything was type=text. So it fell through to a shared CSS selector, .first() picked the wrong input, and a zip code (12342) got typed into the Address line. The order was rejected.
A test that silently fills the wrong field is worse than no test at all.
So I rebuilt element identity as a multi-signal fingerprint with a weighted scorer. Instead of trusting one signal, weak signals combine: role=textbox + a nearby label "Zip code" + nth-position-in-form can be uniquely identifying even when id, name, and type are all shared. If nothing clears a confidence threshold, edge-pilot refuses to guess and emits a testability warning ("add a data-testid here") instead of quietly doing the wrong thing. It's framework-agnostic — no data-testid required.
The live site drifted under me. A time slot I kept booking went SOLD OUT from my own repeated test orders, so a recorded step stopped matching. A cart-page dropdown came back with a different id mid-run and broke its selector.
Annoying — but also the entire reason edge-pilot exists. Selectable dates move, totals change, options disappear. This is the drift that makes me re-verify flows by hand every time. Watching it break my own replay on day two was oddly validating: this is the problem, reproduced.
The oracle, wired in
I also made the captured network calls into assertions on save. Replay no longer just checks that a button was clicked — it checks that POST /order returned 200 with an orderId in the body. "It didn't crash" is finally starting to mean "the order was actually created."
Where it actually stands
I did not hit the milestone. Record → replay of one full checkout still isn't reliable end to end — the recording gap is the blocker, and the fingerprint resolver and confidence threshold aren't fully wired into the replayer yet. Everything I built today is still uncommitted.
And I'll be honest about the doubt I ended Day 1 with, because it got louder today, not quieter:
If the product quality is worse than just prompting a plain LLM, who would use it?
The bet is narrow and specific: zero-token deterministic replay + a real backend-state oracle is the thing a one-shot agent can't give you. Everything else is table stakes now. Today didn't prove the bet — it just made the gap I have to cross very concrete.
Next: fix recording so it captures every action, then get one real checkout to replay green, end to end.
If you record browser flows for testing, how do you keep your element targeting from silently matching the wrong thing? That's the part eating me right now.
Top comments (0)