engineer-step-graph-execution-model

#ai

Stop Writing Spider Scripts. Draw State Machines.

Every scraper you've ever written is a lie. You write it as a script — open page, find element, click, extract — but the moment it meets a real website, it becomes something else: a hidden state machine of retries, waits, fallbacks, and error branches, expressed through the medium of try/except and vibes. Scrapewright makes the state machine the explicit model, and it changes how scrapers fail, heal, and get generated.

The unit of execution is a step, not a script

A Scrapewright service is a directed graph of named steps. Each step is a small script — generated by an LLM, hand-editable — with explicit control-flow edges:

{
  id: "wait_results",
  name: "Wait for search results",
  script: "return { done: $count('div.result') > 0 }",
  onSuccess: "extract_list",
  onFailure: "TERMINATE",
  maxIterations: 20
}

The interesting fields:

onSuccess — the next step when this one succeeds (content ready, data extracted), or TERMINATE.
onFailure — where control goes when the step fails or gives up: condition false, retry budget exhausted, or the script returned { failed: true } / { error: '...' }.
condition — an optional JS expression evaluated in the target tab; false means skip and follow onFailure. Free branching.
maxIterations — 1 (default) is a plain step. Greater than 1 opts the step into poll/retry semantics.

That last one is the design decision I most respect. Polling isn't a special API or a while loop buried in code — it's a property of the node. A wait step returns { done: false } to mean "not ready, run me again"; the orchestrator retries up to maxIterations, then follows onFailure. The orchestrator even auto-boosts maxIterations to the global cap for any step that's the target of a back-edge, so legitimate pagination loops aren't killed by a misremembered default.

Result signals are a deliberately tiny protocol, inspected only when maxIterations > 1:

Script returns	Orchestrator does
`{ done: false }`, `{ ready: false }`, `{ loading: true }` …	retry same step (poll)
data, `{ done: true }`	follow `onSuccess`
`{ failed: true }`, `{ error: 'msg' }`	follow `onFailure`

A normal step's return value is pure data and always advances via onSuccess — no accidental infinite loops because someone returned an object that looks vaguely not-ready.

State flows through injected globals

Steps communicate via __stepResults__ (a map of prior results by step id) and __lastResult__ (the previous step's result), injected into each step's execution context. Since __lastResult__ persists across a step's own retries, list iteration collapses into a single self-polling step: "open item i from the list in __lastResult__, extract, increment, not done yet" — rather than a hand-rolled outer loop in a host language you don't control.

The graphs are validated, not just executed

Because control flow is data, it can be checked at save time. Every persistence path — the wizard, service import, the HTTP step-CRUD endpoints — runs chain validation: every onSuccess/onFailure target must exist, no orphan steps, no duplicate ids, and no self-loop sentinels (an older onSuccess: 'SELF' convention was removed precisely because its semantics were a trap). Mutations go through relink helpers so inserting or deleting a step rewires the chain instead of silently dangling it.

This is the difference between a graph you draw and a graph you debug. A misconfigured poll step — say, onSuccess pointing onward but maxIterations: 1 — runs once and advances without retrying. That's a visible, debuggable failure, not a silent mis-execute.

Why this matters more than usual in 2026

The step-graph model isn't just tidier than scripts — it's the enabling substrate for LLM codegen you can trust:

Generated code stays small. The LLM writes leaf snippets ("wait for this selector", "extract these fields"), not control flow. Edges are structured data the wizard can validate before deploy. Small generation targets mean small blast radii when the model hallucinates.
Auto-repair has a unit to replace. When a deployed service fails, the failure is localized to a step; the repair loop feeds that step's script + error + sanitized DOM snapshot back to the LLM and swaps in the rewrite. You don't regenerate a monolith and pray — you patch a node and re-run the graph.
Execution is replayable and inspectable. The orchestrator returns every step's result and page snapshots per run. "Which step diverged after the redesign?" has a one-glance answer.
The same engine doubles as test automation. Click, type, wait, assert, branch — a step graph is a self-healing replayable web test. The project explicitly positions itself as a lightweight automation tool, not just scraping.

Where it runs

The steps execute in a sandboxed iframe reached through an offscreen document (MV3 CSP forbids eval in extension contexts; the declared sandbox page is the sanctioned hole), while $-prefixed DSL primitives ($click, $extract, $wait, $openTab, $extractWithHover, $scrollToBottom — 19 of them) relay to a content script in the target tab. DOM ops happen in the page; code runs quarantined; the orchestrator never blocks the UI thread of anything you care about.

Deployed, the whole graph is callable over a local HTTP API: POST /api/v1/services/{name}/execute → jobId → GET /jobs/{id}/wait. And at run time, none of it touches an LLM — generation and repair are build-time or on-failure events.

The takeaway

If your scraping codebase has accreted retry loops, sleep-and-pray waits, and per-site "utils", the problem isn't discipline — it's the execution model. Scripts hide the state machine; step graphs make it first-class, validatable, and machine-repairable. Scrapewright is a working, GPLv3, cross-platform implementation of that argument, with the LLM wiring included.

Clone it, import an example from examples/, and watch the wizard argue with a live website for a while. You'll know within an hour whether your next scraper should be a graph.

DEV Community