DEV Community: Hyun Sung Cho

Day 2: The Demo Worked in One Prompt. Then Replay Fell Apart.

Hyun Sung Cho — Wed, 28 Oct 2026 21:44:31 +0000

Yesterday I pivoted my shelved testing project into edge-pilot: let an agent drive a browser once, then replay that flow forever at zero LLM cost, with the captured API calls as the oracle. The Day 1 milestone was small on purpose — an MCP skeleton that can record and replay one real checkout end to end.

Today I tried to build it. The demo worked immediately. Replaying it did not.

What worked

I stood up the MCP authoring server. An external agent (Claude Code, in my case) drives the browser through edge-pilot's tools — ep_navigate, ep_snapshot, ep_click, ep_fill, ep_save_scenario — and edge-pilot does the hardening behind each action.

From one prompt, the agent walked our real shop: opened the treatments page, added an Aromatherapy Massage, filled in guest, therapist, date, and time, entered a test card, and completed checkout. A real order came back. I froze the verified run and replayed it with no model calls at all — it created the next orders (I ended the day with test orders #513950001 through #513950006). The "drive once, replay free" loop exists.

Then I tried to make replay reliable, and every weak assumption fell out at once.

What broke

The recording was garbage. Replay kept failing because the recorder simply wasn't capturing every action the user took. There's no clever fix here — if the recording is lossy, nothing downstream can be trusted. My note to myself for the week:

Recording must be perfect. It has to save every single one of the user's browser actions.

The selector logic grabbed the wrong field. My old resolveTarget walked a fixed priority list — css → data-testid → name → aria → role → text — and judged each signal in isolation. On the real payment form (MUI), every signal was non-unique: three inputs shared #outlined-basic, there were no test IDs, everything was type=text. So it fell through to a shared CSS selector, .first() picked the wrong input, and a zip code (12342) got typed into the Address line. The order was rejected.

A test that silently fills the wrong field is worse than no test at all.

So I rebuilt element identity as a multi-signal fingerprint with a weighted scorer. Instead of trusting one signal, weak signals combine: role=textbox + a nearby label "Zip code" + nth-position-in-form can be uniquely identifying even when id, name, and type are all shared. If nothing clears a confidence threshold, edge-pilot refuses to guess and emits a testability warning ("add a data-testid here") instead of quietly doing the wrong thing. It's framework-agnostic — no data-testid required.

The live site drifted under me. A time slot I kept booking went SOLD OUT from my own repeated test orders, so a recorded step stopped matching. A cart-page dropdown came back with a different id mid-run and broke its selector.

Annoying — but also the entire reason edge-pilot exists. Selectable dates move, totals change, options disappear. This is the drift that makes me re-verify flows by hand every time. Watching it break my own replay on day two was oddly validating: this is the problem, reproduced.

The oracle, wired in

I also made the captured network calls into assertions on save. Replay no longer just checks that a button was clicked — it checks that POST /order returned 200 with an orderId in the body. "It didn't crash" is finally starting to mean "the order was actually created."

Where it actually stands

I did not hit the milestone. Record → replay of one full checkout still isn't reliable end to end — the recording gap is the blocker, and the fingerprint resolver and confidence threshold aren't fully wired into the replayer yet. Everything I built today is still uncommitted.

And I'll be honest about the doubt I ended Day 1 with, because it got louder today, not quieter:

If the product quality is worse than just prompting a plain LLM, who would use it?

The bet is narrow and specific: zero-token deterministic replay + a real backend-state oracle is the thing a one-shot agent can't give you. Everything else is table stakes now. Today didn't prove the bet — it just made the gap I have to cross very concrete.

Next: fix recording so it captures every action, then get one real checkout to replay green, end to end.

If you record browser flows for testing, how do you keep your element targeting from silently matching the wrong thing? That's the part eating me right now.

Day 1 Back: Someone Already Built the Thing I Was Stuck On, So I'm Pivoting

Hyun Sung Cho — Tue, 27 Oct 2026 00:16:07 +0000

I came back today from two months of paternity leave.

My team switched to an AI-agent workflow about nine months ago. Agents plan, implement, review, test, and deploy, and I haven't written much code by hand in months. Not much changed while I was gone, but the AI tooling had moved on.

This is the first post in a daily series about what I build, what breaks, and what being a developer means now.

What I did today

Caught up on two months in an hour

A Slack bot summarized the threads I'd missed, and I used Claude to dig through the commit history for the changes that actually mattered. Two months of context took about an hour.

Fixed a Node.js security issue

Our security-scanning agent flagged a vulnerable Node.js patch version. I opened a shell into a production pod to check what was actually running, and it was the affected version.

Our images used node:22-alpine and our CI used lts/*, so we had picked up the vulnerable patch without noticing. I had an agent pin every runtime Dockerfile and CI job to the same exact patched version:

chore(deploy): pin node across runtime images and CI

- Dockerfile base images: node:22-alpine → node:22.x.y-alpine
- CI node-version: 22 / lts/* → 22.x.y

I verified it on staging before release.

Lesson: floating tags trade reproducibility for convenience. If prod, CI, and local can each resolve a different patch version, pin them.

Debugged a state mismatch

An order showed as active, but its linked appointment didn't. I asked an agent to analyze it, and its explanation didn't make sense to me, so I read the code myself.

My current hypothesis is that two confirm requests raced each other. The first moved the appointment to active, and the second saw the state had already changed and skipped it. I still need to reproduce it before I'm sure.

Lesson: agents are fast, but you still have to judge whether their answer is plausible.

The moment that changed my plan

For years, the most painful part of development for me has been UI testing.

A single order goes through several pages. A guest's check-in-to-payment flow has a dozen variables. Unit and integration tests never fully reassured me, so I kept verifying by hand. I've created more fake orders than I can count.

Before AI tools existed, I built Playwright and Puppeteer harnesses again and again. I kept wondering why small teams like mine don't get good UI testing tools, and why I couldn't just say "here's the URL, go test it."

So I started building an AI-driven testing tool myself. I extracted DOM elements, sent them to an LLM, cleaned up the responses, and kept fighting to cut token usage. I burned out before it really worked and shelved it.

Today my mentor showed me an agent that drives a browser directly. I gave it a prompt, and it placed a test order flawlessly.

I was amazed and a little angry. Someone else had the same problem and solved it well. The base model was already good enough. I had spent my energy figuring out how to hand things to the LLM, when I should have let it do the work.

The pivot: edge-pilot

A browser-driving agent is great at exploring a flow once. But:

every rerun costs tokens and needs a human to prompt it,
the scripts it writes break when the UI changes,
and "the script didn't crash" doesn't mean the order was actually created.

So I'm turning my shelved project into edge-pilot:

Record by prompt, once. An agent drives the browser through edge-pilot's MCP tools (navigate, snapshot, click, fill, save_scenario).
Capture what lasts. Behind each action, edge-pilot replaces the snapshot's temporary ref with a durable, multi-signal element fingerprint and records the network effects.
Attach an oracle automatically. On save, the captured API calls become assertions, for example POST /order → 200 with an orderId in the response body.
Replay forever at zero LLM cost. Deterministic replay locally or in CI, and an alert when the revenue path actually breaks, not when a button moves.

Later, when a redesign breaks a step, only that one step goes back to the LLM to be re-fingerprinted.

The first milestone is small on purpose: an MCP server with navigate, snapshot, click, and fill, fingerprint capture, network capture, auto-assertions, and SQLite storage. If it can record and replay one real checkout flow end to end, the core idea works.

The harder question

The rule I'm adopting: ship something that solves the problem, not something perfect.

I also want to answer something I've avoided. Why do so many of my ideas stop at the idea stage? And what does a developer career look like when I'm no longer the one typing the code?

I don't have answers yet. That's what this series is for.

Next: get the edge-pilot MCP skeleton to record one real checkout flow.

If you've fought the same UI-testing battle on a small team, I'd love to hear how you handle it.