Recording is the rule: a different architecture for browser data collection

#webdev #opensource #scraping #go

Web scraping has two failure modes, and the industry keeps fixing the wrong one.

The first failure is writing the scraper. That got 100x easier: record with Playwright codegen, or just ask an LLM. The second failure is the eighteen months after writing it — the selector that quietly broke, the flow that changed, the "AI agent" that decided to click somewhere new today. Nobody has a good answer for that one, because the answer isn't more generation. It's governance of the artifact that runs.

This is the architecture we open-sourced as AegisCrawler. This post walks through the design decisions, not the marketing.

1. The recording is the source of truth, not a scaffold

A Chrome extension records semantic events — clicks, typing, scrolling, navigation — plus DOM snapshots at three phases (initial / pre-action / final). The recording is sanitized in the browser before it leaves, and the rule generator is deterministic: same recording in, same rule out.

That determinism is the whole ballgame. It means a rule can be:

reviewed — a human reads the steps and the YAML
replayed — the wizard runs the rule in a real browser and requires the replay to succeed before you can save
confirmed step by step — every action gets an explicit checkbox
frozen — saved as an immutable version with a checksum'd lineage

A rule looks like this (abridged from a real recording on a demo bookstore):

id: ext-1789889606822
entry: "https://books.toscrape.com/"
humanize:
  preDelay: 420
  postDelay: 380
  mousePath: curved
selectors:
  travel_category:
    role: link
    text: Travel
steps:
  - action: click
    target: { $ref: selectors.travel_category }
  - action: navigate
    url: "https://books.toscrape.com/catalogue/category/books/travel_2/"
  - action: extract
    fields:
      title: { type: text, path: "h1" }
      price: { type: text, path: ".price_color", regex: "£([\\d.]+)" }
  - action: sendResult
    payload: { title: "{{title}}", price: "{{price}}" }

A non-developer can read that in the Admin UI's structured view; a developer gets the DAG and the raw JSON. Same artifact.

2. The LLM gets a leash, not the keys

We're not anti-LLM — there's an optional enhancement path. But the contract is strict:

The LLM receives the recording and the deterministic baseline rule, and may only propose a patch
The patched rule goes through a security scan (no evaluate, no CAPTCHA solving, no exfiltration targets)
It lands as a pending rule — never active
A human sees a diff in the Admin UI and clicks apply or reject

Budgets, provider routes, and attempt caps are enforced server-side under an "enforced" policy mode; a generic API key path exists only for local development. If the LLM provider is down, the baseline rule runs anyway. The model is an accelerator for rule authoring, never a runtime dependency.

Compare that with the agent-style scrapers where the model chooses actions at execution time: you gain "zero setup" and lose determinism, auditability, and — when it inevitably does something odd — the ability to explain what happened. For anything that must run weekly for a year, that trade is backwards.

3. Execution is boring on purpose

The server (Go, SQLite, single binary) treats rules like a job queue with manners:

Lease-based claiming with heartbeat renewal; a sweeper reclaims expired leases
Retries to a per-task cap, then a dead-letter state that requires human triage — no infinite retry loops hammering a wounded target
Tiered rate limiting (global / per-worker / per-target-site) and a per-site circuit breaker
Execution attempts bind to immutable rule versions, so a task always runs against the exact rule that was approved, and results are idempotent per attempt
Prometheus metrics, health/readiness probes, trace IDs through every log line

The worker side is a userscript (runs under ScriptCat) executing in real browser tabs — real browser, real fingerprint, humanized timing — reporting results and logs back over an authenticated protocol.

How this compares

	Playwright codegen / Selenium IDE	LLM-agent scrapers	AegisCrawler
Authoring	Record a script	Prompt	Record → confirm (LLM optional)
Artifact	Code, developer-owned	Prompt + vibes	Immutable versioned rule, human-approved
Deterministic replay	Per-run	No (model decides)	Yes, validated before save
Scheduling/retries/DLQ	Bring your own	Varies	Built-in
Audit trail	Git (if you're lucky)	Logs of "thoughts"	Version + attempt + diff
Who can read the rule	Developers	Nobody really	Ops/business via step view + DAG

Try it

The README has a five-minute quick start (docker compose up -d, load the extension, record on any site you're authorized to collect). GPL-3.0-or-later, Docker images on GHCR, and the admin UI is currently Chinese with English on the roadmap — the docs are bilingual.

If you take one idea from this post, take this one: the artifact that runs in production should be something a human approved, versioned, and can diff. Generation — by recorder or by model — is the easy 10%. We built the other 90%.

DEV Community