scrapewright

Posted on Aug 20

engineer-local-http-microservice

#ai

Every Scrape Becomes a Local Microservice

Most scraping tools end at "here's your JSON in a terminal." Scrapewright ends at "here's your endpoint." Every scraper you build becomes a named HTTP service on localhost, with a job model, queueing, health checks, and even step-level CRUD for CI. This is a tour of that surface — because the API design is half the product.

The two-call pattern

Deployed services live under http://localhost:8765/api/v1 (port and API key configurable; header X-API-Key on everything except /health):

# Submit — returns immediately with a jobId
JOB_ID=$(curl -s -X POST http://localhost:8765/api/v1/services/my-service/execute \
  -H "X-API-Key: dev-key" -H "Content-Type: application/json" \
  -d '{"input": {"query": "wireless mouse"}}' | jq -r '.jobId')

# Wait — blocks until done (timeout up to 300s)
curl -s "http://localhost:8765/api/v1/jobs/$JOB_ID/wait?timeout=120" \
  -H "X-API-Key: dev-key" | jq '.job.result'

The execute/wait split is the right async contract: submissions are cheap, results are pull-based (/jobs/{id} for non-blocking status), and both fit cron jobs, CI pipelines, and server handlers equally well.

The response is a contract, not a blob

Each service declares JSON Schemas on both ends — input and output. The result envelope is stable regardless of what the target site looks like:

{
  "job": {
    "status": "completed",
    "result": {
      "posts": [
        { "author": "…", "likes": "4", "sourcePageId": "page_0007_a1b2c3d4" }
      ]
    },
    "pages": [
      { "id": "page_0007_a1b2c3d4", "url": "…", "title": "…", "html": "…" }
    ]
  }
}

Note pages[] and sourcePageId: every extracted record is stamped with the id of the page it came from, and every page is captured with URL, title, and cleaned HTML. Provenance is part of the response envelope. When a number in your database looks wrong in March, you can trace it back to the exact page state that produced it.

Queueing is explicit

The host runs one execution at a time (a deliberate constraint of the single-browser architecture — see below), and instead of hiding that, the API surfaces it:

{ "success": true, "jobId": "…", "status": "queued", "queuePosition": 1 }

queuePosition: 0 means executing. Jobs can also be cancelled (POST /jobs/{id}/cancel) and listed (GET /jobs). No mysterious 429s — your place in line is a first-class field.

Errors are classified, and some repair themselves

The error taxonomy is small and actionable:

Error	Meaning
`ELEMENT_NOT_FOUND` / `SCRIPT_ERROR`	element missing / script failed — AI attempts auto-repair
`SCRIPT_TIMEOUT`	step exceeded its budget (default 60s)
`LOGIN_REQUIRED`	the site wants a human login; fail fast with a clear message

That first row is the interesting one: on element-not-found, the host's extension side feeds the failing step plus a DOM snapshot to the LLM and attempts an automatic rewrite before giving up. Your monitoring sees either a success or a classified failure — not a stack trace from a selector library.

Services are manageable over HTTP too

The API isn't just for calling services — it's for administering them:

GET    /api/v1/services                       # list services + their I/O schemas
POST   /api/v1/services/{name}/steps          # add a step
PUT    /api/v1/services/{name}/steps/{id}     # update a step's script/flow fields
DELETE /api/v1/services/{name}/steps/{id}     # delete (chain auto-relinks)
GET    /health                                # no-auth liveness, for LB/K8s probes

Step CRUD over HTTP means your pipeline tooling can adjust services programmatically — and every mutation re-validates the step graph (edges, no orphans, no dangling ids), so you can't save a broken service even by API.

One more export worth knowing: each service can emit its own Markdown API documentation — endpoints, schemas, examples — designed to be handed to other AI agents so they can build callers themselves. The docs your scraper needs are generated, not maintained.

The constraint, stated plainly

A single instance is a single browser: jobs serialize, and it depends on your Chrome being alive. For higher throughput the intended answer is horizontal — run multiple host instances on different ports behind a load balancer (the repo ships Docker and Kubernetes manifests plus a manager script), each with /health for probe-based routing. The architecture is honest about being a client-side platform: it wins on identity (your logins, your real browser), not on anonymous concurrency.

The upshot

If you've glued scrapers into systems before — cron + python + output files + prayer — the difference here is that the scrape is a service from the moment it's deployed: queued, observable, versioned, self-describing, and repair-aware. That's not a scraping feature. That's an integration feature, and it's why this repo is worth an hour of any backend engineer's time.

Repo: github.com/singhand-labs/scrapewright — GPLv3, macOS/Linux/Windows, Node ≥ 18.

DEV Community