Why your scraper breaks every time a site is redesigned — and how to make it heal itself

Sheng Shijie — Sat, 24 Oct 2026 07:39:03 +0000

Every scraper I've ever written died the same death. Not with an error — with
silence. The site shipped a redesign, my CSS selectors matched nothing, and the
script kept running and returning empty fields. I'd notice days later when the
data looked wrong.

The usual fixes all have a catch:

Hand-written selectors are free and fast, but every redesign means a manual rewrite. The maintenance never ends.
Cloud "AI scraping" services are robust, but they're expensive, they send the pages you're scraping through someone else's servers, and they run a model call on every single request — your bill scales with your crawl.

I wanted the robustness of the AI approach without paying for it on every page.
So I built scrapemend around one
idea: treat a broken selector as a cache miss, not a crash.

How it works

You describe what you want in plain language — no selectors:

from scrapemend import Scraper, Schema

schema = (
    Schema("product")
    .add("title", "the product name")
    .add("price", "the product price with currency")
)

scraper = Scraper()
result = scraper.extract("https://shop.example/p/1", schema)
print(result["title"], result["price"])

Under the hood:

extract(url, schema)
      │
      ├─ cached selectors exist? ──► apply them
      │                               ├─ all fields matched ──► return (free)
      │                               └─ some field missed ──┐
      │                                                      ▼
      └─ no cache ─────────────────────────────────► LLM heals selectors
                                                      apply, cache, return

The LLM only runs when selectors are missing or have stopped matching. In the
steady state, extraction is just parsing — zero tokens. You pay the model
exactly when a site actually changes, and the repaired selectors are cached so
the next run is free again.

Does it actually survive a redesign?

Here's a real run against two versions of the same page — same Schema, no code
changes:

Run 1 — first visit, no cache        -> HEALED,     178 tokens
Run 2 — same markup again            -> CACHE HIT,     0 tokens
Run 3 — site redesigned (new markup) -> HEALED,     190 tokens

Run 2 is the whole point: once selectors are known, you're not paying anything.
Run 3 is the other half: when the markup changed and the cached selectors broke,
it repaired itself and still returned the right data.

Before the LLM ever sees a page, the HTML is condensed — scripts, styles, and
noise attributes are stripped, keeping only structure and class/id. On a
typical bloated page that cut the prompt by over 90%, which keeps healing cheap.

Try it

pip install scrapemend

It ships with an offline stub so the pipeline runs with no API key, and
provider adapters for OpenAI and DeepSeek when you want real healing. DeepSeek
in particular makes healing almost free.

Code and docs: https://github.com/FengYue427/scrapemend

It's early (0.0.1) and I'd genuinely like feedback on where it breaks. If
you've ever lost a day to a silent scraper, I'd love to know whether this
approach holds up on the sites you work with.

DEV Community: Sheng Shijie

Why your scraper breaks every time a site is redesigned — and how to make it heal itself

How it works

Does it actually survive a redesign?

Try it