DEV Community: scrapewright

engineer-local-http-microservice

scrapewright — Thu, 20 Aug 2026 08:53:02 +0000

Every Scrape Becomes a Local Microservice

Most scraping tools end at "here's your JSON in a terminal." Scrapewright ends at "here's your endpoint." Every scraper you build becomes a named HTTP service on localhost, with a job model, queueing, health checks, and even step-level CRUD for CI. This is a tour of that surface — because the API design is half the product.

The two-call pattern

Deployed services live under http://localhost:8765/api/v1 (port and API key configurable; header X-API-Key on everything except /health):

# Submit — returns immediately with a jobId
JOB_ID=$(curl -s -X POST http://localhost:8765/api/v1/services/my-service/execute \
  -H "X-API-Key: dev-key" -H "Content-Type: application/json" \
  -d '{"input": {"query": "wireless mouse"}}' | jq -r '.jobId')

# Wait — blocks until done (timeout up to 300s)
curl -s "http://localhost:8765/api/v1/jobs/$JOB_ID/wait?timeout=120" \
  -H "X-API-Key: dev-key" | jq '.job.result'

The execute/wait split is the right async contract: submissions are cheap, results are pull-based (/jobs/{id} for non-blocking status), and both fit cron jobs, CI pipelines, and server handlers equally well.

The response is a contract, not a blob

Each service declares JSON Schemas on both ends — input and output. The result envelope is stable regardless of what the target site looks like:

{
  "job": {
    "status": "completed",
    "result": {
      "posts": [
        { "author": "…", "likes": "4", "sourcePageId": "page_0007_a1b2c3d4" }
      ]
    },
    "pages": [
      { "id": "page_0007_a1b2c3d4", "url": "…", "title": "…", "html": "…" }
    ]
  }
}

Note pages[] and sourcePageId: every extracted record is stamped with the id of the page it came from, and every page is captured with URL, title, and cleaned HTML. Provenance is part of the response envelope. When a number in your database looks wrong in March, you can trace it back to the exact page state that produced it.

Queueing is explicit

The host runs one execution at a time (a deliberate constraint of the single-browser architecture — see below), and instead of hiding that, the API surfaces it:

{ "success": true, "jobId": "…", "status": "queued", "queuePosition": 1 }

queuePosition: 0 means executing. Jobs can also be cancelled (POST /jobs/{id}/cancel) and listed (GET /jobs). No mysterious 429s — your place in line is a first-class field.

Errors are classified, and some repair themselves

The error taxonomy is small and actionable:

Error	Meaning
`ELEMENT_NOT_FOUND` / `SCRIPT_ERROR`	element missing / script failed — AI attempts auto-repair
`SCRIPT_TIMEOUT`	step exceeded its budget (default 60s)
`LOGIN_REQUIRED`	the site wants a human login; fail fast with a clear message

That first row is the interesting one: on element-not-found, the host's extension side feeds the failing step plus a DOM snapshot to the LLM and attempts an automatic rewrite before giving up. Your monitoring sees either a success or a classified failure — not a stack trace from a selector library.

Services are manageable over HTTP too

The API isn't just for calling services — it's for administering them:

GET    /api/v1/services                       # list services + their I/O schemas
POST   /api/v1/services/{name}/steps          # add a step
PUT    /api/v1/services/{name}/steps/{id}     # update a step's script/flow fields
DELETE /api/v1/services/{name}/steps/{id}     # delete (chain auto-relinks)
GET    /health                                # no-auth liveness, for LB/K8s probes

Step CRUD over HTTP means your pipeline tooling can adjust services programmatically — and every mutation re-validates the step graph (edges, no orphans, no dangling ids), so you can't save a broken service even by API.

One more export worth knowing: each service can emit its own Markdown API documentation — endpoints, schemas, examples — designed to be handed to other AI agents so they can build callers themselves. The docs your scraper needs are generated, not maintained.

The constraint, stated plainly

A single instance is a single browser: jobs serialize, and it depends on your Chrome being alive. For higher throughput the intended answer is horizontal — run multiple host instances on different ports behind a load balancer (the repo ships Docker and Kubernetes manifests plus a manager script), each with /health for probe-based routing. The architecture is honest about being a client-side platform: it wins on identity (your logins, your real browser), not on anonymous concurrency.

The upshot

If you've glued scrapers into systems before — cron + python + output files + prayer — the difference here is that the scrape is a service from the moment it's deployed: queued, observable, versioned, self-describing, and repair-aware. That's not a scraping feature. That's an integration feature, and it's why this repo is worth an hour of any backend engineer's time.

Repo: github.com/singhand-labs/scrapewright — GPLv3, macOS/Linux/Windows, Node ≥ 18.

engineer-step-graph-execution-model

scrapewright — Thu, 20 Aug 2026 08:43:30 +0000

Stop Writing Spider Scripts. Draw State Machines.

Every scraper you've ever written is a lie. You write it as a script — open page, find element, click, extract — but the moment it meets a real website, it becomes something else: a hidden state machine of retries, waits, fallbacks, and error branches, expressed through the medium of try/except and vibes. Scrapewright makes the state machine the explicit model, and it changes how scrapers fail, heal, and get generated.

The unit of execution is a step, not a script

A Scrapewright service is a directed graph of named steps. Each step is a small script — generated by an LLM, hand-editable — with explicit control-flow edges:

{
  id: "wait_results",
  name: "Wait for search results",
  script: "return { done: $count('div.result') > 0 }",
  onSuccess: "extract_list",
  onFailure: "TERMINATE",
  maxIterations: 20
}

The interesting fields:

onSuccess — the next step when this one succeeds (content ready, data extracted), or TERMINATE.
onFailure — where control goes when the step fails or gives up: condition false, retry budget exhausted, or the script returned { failed: true } / { error: '...' }.
condition — an optional JS expression evaluated in the target tab; false means skip and follow onFailure. Free branching.
maxIterations — 1 (default) is a plain step. Greater than 1 opts the step into poll/retry semantics.

That last one is the design decision I most respect. Polling isn't a special API or a while loop buried in code — it's a property of the node. A wait step returns { done: false } to mean "not ready, run me again"; the orchestrator retries up to maxIterations, then follows onFailure. The orchestrator even auto-boosts maxIterations to the global cap for any step that's the target of a back-edge, so legitimate pagination loops aren't killed by a misremembered default.

Result signals are a deliberately tiny protocol, inspected only when maxIterations > 1:

Script returns	Orchestrator does
`{ done: false }`, `{ ready: false }`, `{ loading: true }` …	retry same step (poll)
data, `{ done: true }`	follow `onSuccess`
`{ failed: true }`, `{ error: 'msg' }`	follow `onFailure`

A normal step's return value is pure data and always advances via onSuccess — no accidental infinite loops because someone returned an object that looks vaguely not-ready.

State flows through injected globals

Steps communicate via __stepResults__ (a map of prior results by step id) and __lastResult__ (the previous step's result), injected into each step's execution context. Since __lastResult__ persists across a step's own retries, list iteration collapses into a single self-polling step: "open item i from the list in __lastResult__, extract, increment, not done yet" — rather than a hand-rolled outer loop in a host language you don't control.

The graphs are validated, not just executed

Because control flow is data, it can be checked at save time. Every persistence path — the wizard, service import, the HTTP step-CRUD endpoints — runs chain validation: every onSuccess/onFailure target must exist, no orphan steps, no duplicate ids, and no self-loop sentinels (an older onSuccess: 'SELF' convention was removed precisely because its semantics were a trap). Mutations go through relink helpers so inserting or deleting a step rewires the chain instead of silently dangling it.

This is the difference between a graph you draw and a graph you debug. A misconfigured poll step — say, onSuccess pointing onward but maxIterations: 1 — runs once and advances without retrying. That's a visible, debuggable failure, not a silent mis-execute.

Why this matters more than usual in 2026

The step-graph model isn't just tidier than scripts — it's the enabling substrate for LLM codegen you can trust:

Generated code stays small. The LLM writes leaf snippets ("wait for this selector", "extract these fields"), not control flow. Edges are structured data the wizard can validate before deploy. Small generation targets mean small blast radii when the model hallucinates.
Auto-repair has a unit to replace. When a deployed service fails, the failure is localized to a step; the repair loop feeds that step's script + error + sanitized DOM snapshot back to the LLM and swaps in the rewrite. You don't regenerate a monolith and pray — you patch a node and re-run the graph.
Execution is replayable and inspectable. The orchestrator returns every step's result and page snapshots per run. "Which step diverged after the redesign?" has a one-glance answer.
The same engine doubles as test automation. Click, type, wait, assert, branch — a step graph is a self-healing replayable web test. The project explicitly positions itself as a lightweight automation tool, not just scraping.

Where it runs

The steps execute in a sandboxed iframe reached through an offscreen document (MV3 CSP forbids eval in extension contexts; the declared sandbox page is the sanctioned hole), while $-prefixed DSL primitives ($click, $extract, $wait, $openTab, $extractWithHover, $scrollToBottom — 19 of them) relay to a content script in the target tab. DOM ops happen in the page; code runs quarantined; the orchestrator never blocks the UI thread of anything you care about.

Deployed, the whole graph is callable over a local HTTP API: POST /api/v1/services/{name}/execute → jobId → GET /jobs/{id}/wait. And at run time, none of it touches an LLM — generation and repair are build-time or on-failure events.

The takeaway

If your scraping codebase has accreted retry loops, sleep-and-pray waits, and per-site "utils", the problem isn't discipline — it's the execution model. Scripts hide the state machine; step graphs make it first-class, validatable, and machine-repairable. Scrapewright is a working, GPLv3, cross-platform implementation of that argument, with the LLM wiring included.

Clone it, import an example from examples/, and watch the wizard argue with a live website for a while. You'll know within an hour whether your next scraper should be a graph.

enthusiast-local-first-ai

scrapewright — Thu, 20 Aug 2026 08:39:41 +0000

An AI Scraper That Lives in Your Browser — and Never Calls the Cloud at Run Time

Most "AI-powered" tools share an uncomfortable design: your data goes to their cloud, their model looks at it, and you pay per interaction forever. Scrapewright inverts that model in a way worth paying attention to: the AI only helps you build a scraper. Running it afterwards involves zero AI, zero tokens, and zero cloud.

What it is

Scrapewright is an open-source (GPLv3) platform with two parts:

a Chrome extension that does the scraping — inside the Chrome you already use, with your existing logins and sessions;
a small Node.js host service that exposes everything as a local HTTP API on your machine, installed as a proper OS background service (systemd on Linux, launchd on macOS, a scheduled task on Windows).

You describe a scraping task in natural language. The built-in wizard's AI opens the target page, analyzes its structure, writes the extraction script, test-runs it in front of you, and deploys it as a reusable HTTP endpoint. From that point on, calling the service is just:

curl -X POST http://localhost:8765/api/v1/services/my-service/execute \
  -H "X-API-Key: dev-key" -H "Content-Type: application/json" \
  -d '{"input": {"query": "hello"}}'

Why "local-first" matters here

Your data doesn't leave your machine at run time. After deployment, the service never calls the LLM again. The pages it visits, the data it extracts — all of it stays on your box. The LLM only ever saw page structure during the build phase, if you let it.

Bring your own model. The wizard works with OpenAI, Anthropic, Moonshot Kimi, GLM, or any OpenAI-compatible endpoint you point it at — including a local gateway. You choose the intelligence; you hold the key.

The economics are backwards (in a good way). Agent-style scrapers that have an LLM drive a browser on every page turn think tokens per page. Scrapewright thinks tokens per scraper. Build once for a few cents; run it a thousand times for free. For anything you scrape repeatedly, the cost curve crosses zero almost immediately.

No automation fingerprint. Because the extension runs in your genuine daily Chrome, requests look like... you. No headless-browser markers, no synthetic fingerprints to maintain. If you can see the page, the scraper can read it — including JavaScript-rendered content, nested iframes, hover popups, and lazy-loaded feeds.

The self-healing part is the sleeper feature

Every scraper eventually breaks — sites redesign. The usual outcome is an afternoon of re-debugging someone else's selector logic. Scrapewright's Auto-Fix takes the failure, a snapshot of the new DOM, and its own script, and asks the LLM to rewrite the broken part. Config time or run time, same mechanism: repair beats rewrite.

Worth knowing before you dive in

It's not magic for every job — the README is refreshingly honest about this. One browser means one execution lane: a single instance serializes jobs, and 10,000-URL firehoses are better served by server-side farms. Where it shines is the huge middle ground of scraping: logged-in dashboards, intranets, paid content, list-plus-detail-page drill-downs, and low-frequency high-value lookups — precisely the jobs cloud scrapers fumble because they can't be you.

Try it

Everything is on GitHub: github.com/singhand-labs/scrapewright

The 60-second quick start gets you from clone to first deployed service in one sitting, and the repo ships importable example services so you can see the shape of a finished scraper before building your own. If you've been waiting for AI tooling that treats your machine as the platform instead of a terminal — this one's worth a weekend.

stem-students-lab-data

scrapewright — Thu, 20 Aug 2026 08:23:59 +0000

Get the Data for Your Lab Report Before Midnight

Let's be honest about how the data-collection phase of a course project actually goes. You budget two hours. You spend the first on a scraping tutorial, the second on installing a headless browser that doesn't quite launch, and then the site you needed renders everything through JavaScript anyway, and now it's 23:40 and your "dataset" is 40 rows of hand-pasted text with inconsistent date formats.

Scrapewright is an open-source Chrome extension built around a simple observation: the part that's hard for you (writing robust extraction code against a messy page) is exactly the part modern AI is good at, and the part that's easy (knowing which fields you need) is the part you already know. You describe the target — "search this component-supplier site for the part number I input, open each result, return name, price, stock, and specs" — and its wizard opens the page, figures out the structure, writes the extraction steps, and test-runs them while you watch. Deploy it, and you get a stable local HTTP endpoint any script, notebook, or report pipeline can call:

curl -X POST http://localhost:8765/api/v1/services/parts-lookup/execute \
  -H "X-API-Key: dev-key" -H "Content-Type: application/json" \
  -d '{"input": {"part": "LM358"}}'

That last part matters more than it looks. Most no-code tools hand you a CSV and stop. Scrapewright hands you an API — JSON in, JSON out, with declared input/output schemas — so your Python/MATLAB/Julia analysis code can call the live source directly instead of importing a stale export. For a lab that means one less manual step between "sample measured" and "supplier data joined". For a thesis it means your data pipeline is re-runnable in April when the professor says "also do last year's parts".

The realistic failure modes of student scraping are handled rather than hand-waved:

JavaScript-rendered pages and nested iframes — it reads the DOM in your real Chrome, so what you see is what it extracts.
Infinite scroll / lazy loading — handled, including edge cases where sites ignore programmatic scrolling.
Login-gated course systems and supplier portals — it uses your existing browser session, so if you can see the data, it can reach it.
"It worked yesterday" — sites redesign; the Auto-Fix button has the AI repair the extraction steps against the new layout instead of you starting over.

Costs, in student terms: you need an API key from some LLM provider (OpenAI, Anthropic, Moonshot Kimi, GLM — student/cheap tiers fine) to power the building step, which costs cents per service. Running a deployed service costs nothing — it never calls the AI again — and everything runs locally. Setup is: load the extension in Chrome, run one install command for its background service. Node ≥ 18 required.

One caution worth writing down: it's a single-browser, single-job-at-a-time tool. If your project genuinely needs a 100k-page crawl, use a server-side framework and cite this repo for the prototype phase instead. Scrapewright's sweet spot is structured, repeated queries against pages you personally can access — which is what most course projects, lab pipelines, and hackathon backends actually are.

Repo and quick start (60 seconds to read, an evening to first working service): github.com/singhand-labs/scrapewright. The examples/ folder has importable sample services — useful both as a shortcut and as a template for what a finished one looks like.

The deadline hasn't moved. But the data part just got a lot shorter.

agent-builders-stable-tools

scrapewright — Thu, 20 Aug 2026 08:21:11 +0000

Give Your AI Agent a Stable Tool Instead of a Steering Wheel

The 2026 default for "AI + web" is agentic browsing: give the model a browser, let it look-click-wait-extract per task. Watching it work feels like the future. Putting it in production teaches you its actual economics: every run re-perceives the same pages, every run pays tokens per page-turn, every run is a fresh roll of the dice on whether it clicks the same button it clicked yesterday. For exploration, wonderful. For repeated extraction — the daily dashboard pull, the per-item lookup, the scheduled monitor — it's the wrong shape entirely.

The older engineering instinct says: repeated work belongs in a tool, not in a steering wheel. Scrapewright is that instinct, productized — an open-source platform where an LLM builds a scraping service once, and everything afterward is a deterministic HTTP call your agent can make.

The division of labor

Build time (LLM, once): you describe the extraction in natural language. The wizard's AI opens the target page in your real browser, researches its structure in rounds (candidate selectors confirmed against actual element HTML, not guessed), generates a step-graph program against a 19-primitive DSL, test-runs it, and self-repairs on failure. Result: a deployed service with declared input/output JSON Schemas.

Run time (no LLM, ever): your agent — or cron, or your app — calls:

POST /api/v1/services/{name}/execute   {"input": {"query": "..."}}
GET  /api/v1/jobs/{id}/wait?timeout=120

Structured JSON comes back, every record stamped with the page it came from. Fast (no model latency), free (no tokens), deterministic (same steps every run).

Repair time (LLM, on failure): when a site redesigns and the service errors, Auto-Fix feeds the failing step + DOM snapshot back to the model and rewrites it. The tool heals itself without a human or an agent re-learning the page.

The agent-facing detail I most want you to notice

Every Scrapewright service can export its own Markdown API documentation — endpoint, schemas, examples, generated from the service definition. The intended workflow is explicit in the README: hand that document to your agent (the docs name Hermes Agent, WorkBuddy, and Lobster as examples), and let the agent construct its own tool wrapper for calling the service. That's a cleaner pattern than most "agent + scraper" integrations, because it separates two jobs that keep getting conflated:

Deciding what data is needed and when → the agent (good at judgment, bad at repetition).
Getting the data → a stable HTTP service (good at repetition, free at repetition).

The agent doesn't need to see the target site's DOM, burn context on page perception, or hold a browser session open. It needs one tool call with a schema it was handed. Your context window — and your token bill — stay spent on the reasoning you actually hired the model for.

Why "client-side" makes the tool layer stronger

The service runs in a real Chrome via an extension — your logins, your sessions, your fingerprint. So the tool layer can cover sources that server-side scrapers structurally can't: the SaaS dashboard behind SSO, the paid archive, the internal portal. When your agent needs "yesterday's numbers from the ops dashboard," the honest options used to be (a) agentic browsing with stored credentials, or (b) an integration that doesn't exist. Now there's (c): a deterministic local endpoint that is your credentialed access, wrapped in a schema. Self-hosted, one API key on localhost, nothing in the runtime path phoning any cloud — LLM or otherwise.

The boundary

Compile-time tooling assumes the task repeats. Genuinely exploratory, one-off, every-site-is-different work is still agentic-browsing territory, and the project's own docs say so. The interesting design space is the hybrid — an agent that builds Scrapewright services on demand for tasks it detects are recurring, then stops thinking about them. The pieces for that are all in the box: a wizard API, service export/import as JSON, and per-service Markdown docs an agent can consume.

Repo: github.com/singhand-labs/scrapewright — GPLv3, quick start in the README, whitepaper for the architecture. If you're building agents that touch the web, spend one evening making a service and pointing your agent at its API doc — the drop in latency, cost, and flakiness is the whole argument.