Building a Self-Healing Scraper: Let AI Fix Your Selectors When Sites Change

Ahmed Shamandy — Fri, 18 Sep 2026 00:57:06 +0000

Your scraper doesn't fail with a stack trace. It fails silently.

A site ships a redesign, renames one CSS class, and your .css("span.price") quietly starts returning None. Nothing crashes. The pipeline keeps running, writing rows of empty prices into your database, and you find out a week later when someone downstream asks why the dashboard is flat.

The hard part of scraping was never writing the first version. It's keeping it alive as the target keeps moving. In this tutorial we'll build a scraper that notices when its own selectors break, uses an LLM to find the data again and generate a fresh selector, then saves that selector so the next run is cheap again.

The key idea, before we write any code: the LLM is not doing the scraping. It only runs when the normal path breaks. That one constraint is what keeps this practical instead of a way to set money on fire.

What "self-healing" actually means

Here's the whole control flow:

request → try SAVED selectors
             │
     ┌───────┴────────┐
   works            fails validation
     │                    │
  return         trim HTML → ask LLM for value + new selector
                            │
                     verify the new selector actually works
                            │
                     save it → return

The expensive branch — trimming HTML, calling a model — is the exception, not the rule. On a stable site it never fires. It only kicks in on the day a site changes, and after it heals once, every following request is back on the fast path.

The baseline scraper

Start with something completely ordinary. httpx to fetch, parsel to parse, hardcoded selectors.

import httpx
from parsel import Selector

DEFAULT_SELECTORS = {
    "title": "h1.product-title",
    "price": "span.price",
}

def extract(html: str, selectors: dict) -> dict:
    tree = Selector(html)
    return {
        field: tree.css(f"{sel}::text").get()
        for field, sel in selectors.items()
    }

Nothing new here. This is the scraper you already write. The interesting part is how we decide whether its output is any good.

Detecting the break

Most people check for breakage with if result is None. That catches a selector that matches nothing, but it misses the nastier case: a selector that matches the wrong element and returns plausible-looking garbage.

A schema is a much stronger health check. Define what a valid result looks like with pydantic, and let validation be your detector:

from pydantic import BaseModel, ValidationError, field_validator

class Product(BaseModel):
    title: str
    price: float

    @field_validator("title")
    @classmethod
    def title_not_blank(cls, v: str) -> str:
        if not v or not v.strip():
            raise ValueError("empty title")
        return v.strip()

def validate(raw: dict) -> Product | None:
    clean = {}
    for key, value in raw.items():
        if value is None:
            return None
        if key == "price":
            value = "".join(c for c in value if c.isdigit() or c == ".")
        clean[key] = value
    try:
        return Product(**clean)
    except ValidationError:
        return None

Now "is my scraper healthy?" has a precise answer: validate(extract(...)) returns a Product, or it returns None. A price of "Add to cart" fails the float coercion. An empty title fails the validator. That None is our trigger.

The healing layer

When validation fails, we hand the problem to a model. Two things matter here: cost control and trust.

Cost control means never sending raw HTML. A product page is mostly <script>, <style>, and inline SVG — none of which the model needs, all of which you pay for by the token. Strip it first:

from bs4 import BeautifulSoup

def trim_html(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "svg", "noscript", "head"]):
        tag.decompose()
    return str(soup)

On a typical product page this cuts the payload by 60–80%.

Trust means the prompt asks for two things — the value and a selector — so we can verify the answer instead of taking it on faith:

import json
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY

HEAL_PROMPT = """You are a web-scraping repair tool.
Find one field in the HTML below and return a CSS selector that reaches it.

Field: {field}
What it is: {hint}

Return ONLY JSON, no prose:
{{"value": "<exact text you found>", "selector": "<CSS selector for the element>"}}

HTML:
{html}
"""

def heal_field(field: str, hint: str, html: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",          # cheap model is fine — see below
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{
            "role": "user",
            "content": HEAL_PROMPT.format(field=field, hint=hint, html=html),
        }],
    )
    return json.loads(resp.choices[0].message.content)

The model can absolutely hallucinate a selector that doesn't exist, or a value that isn't on the page. That's fine, because we never trust it directly. We verify.

Closing the loop

This is the section that makes the whole thing "self-healing" rather than "call an LLM every time."

Before saving a healed selector, we run it against the real HTML and confirm it produces the value the model claimed. Only a selector that actually works gets persisted:

from parsel import Selector

def verify(html: str, selector: str, expected: str) -> bool:
    got = Selector(html).css(f"{selector}::text").get()
    return got is not None and expected.strip() in got.strip()

And here's the loop that ties every piece together:

import json
from pathlib import Path

SELECTOR_STORE = Path("selectors.json")

FIELD_HINTS = {
    "title": "the product's name or title",
    "price": "the product's price, digits only",
}

def load_selectors() -> dict:
    if SELECTOR_STORE.exists():
        return json.loads(SELECTOR_STORE.read_text())
    return dict(DEFAULT_SELECTORS)

def save_selectors(selectors: dict) -> None:
    SELECTOR_STORE.write_text(json.dumps(selectors, indent=2))

def scrape(html: str) -> Product:
    selectors = load_selectors()

    # Fast path: the selectors we already have
    product = validate(extract(html, selectors))
    if product is not None:
        return product

    # Slow path: heal
    trimmed = trim_html(html)
    new_selectors = dict(selectors)
    values = {}
    for field in Product.model_fields:
        result = heal_field(field, FIELD_HINTS[field], trimmed)
        if verify(html, result["selector"], result["value"]):
            new_selectors[field] = result["selector"]  # keep only verified fixes
        values[field] = result["value"]

    product = validate(values)
    if product is None:
        raise ValueError("healing failed: LLM output did not match schema")

    save_selectors(new_selectors)  # <-- next run is cheap again
    return product

Trace one lifecycle:

Run 1 — the site is as you found it. Saved selectors match, validation passes, you return in milliseconds. No model call.
Run 2 — the site redesigned overnight. span.price misses, validation returns None, the healing branch fires, the model finds the price under its new markup, the selector verifies, and it's written to selectors.json.
Run 3 — same redesigned site, but now the healed selector is the saved one. Fast path again. The model doesn't run.

You pay for the LLM exactly once per break, not once per request. That's the whole trick.

The honest limitations

This pattern is genuinely useful, and it is not magic. Where it bites:

Cost still scales with churn. Healing only on failure is cheap on a stable site and expensive on a site that reshuffles its DOM daily. If your target ships A/B tests that rotate markup constantly, you'll be in the healing branch a lot. Measure your break rate before you assume this is free.

Validation is not optional. Everything here depends on the schema being strict enough to catch a wrong answer. A loose schema (title: str with no other checks) will happily accept the model's hallucination and persist a broken selector. Your validator is the only thing standing between "self-healing" and "self-corrupting." Spend time on it.

It fixes layout, not access. Self-healing repairs selectors when the structure changes. It does nothing about anti-bot blocking, JavaScript that hasn't rendered yet, or rate limits. Those are separate problems and this pattern won't touch them.

Sometimes you shouldn't. If the site is stable, plain selectors are simpler and free — don't add an LLM to a problem you don't have. And if there's an official API, that's almost always cheaper and more reliable than healing a scraper against a hostile frontend.

FAQ

Which LLM should I use?
A cheap one. gpt-4o-mini, Claude Haiku, or any small model is fine, because your pydantic validation catches bad output regardless of which model produced it. Reach for a bigger model only if the small one keeps failing verification on genuinely complex pages.

Does this work on JavaScript-heavy sites?
Only after the page is rendered. Healing fixes selectors; it doesn't execute JavaScript. Put a headless browser (Playwright) in front to get rendered HTML, then feed that into the same loop.

What about anti-bot protection?
Different problem. Self-healing handles layout changes, not blocking. If you're getting challenged or banned, that's an access issue to solve before extraction ever runs.

How much does one heal actually cost?
A trimmed product page is usually a few thousand tokens. At small-model prices that's a fraction of a cent per field, and it happens once per break — not per request. The expensive scenario is a high break rate, not a high request count.

Can I use XPath instead of CSS selectors?
Yes. Same loop — ask the model for an XPath, swap .css() for .xpath() in extract and verify. Nothing else changes.

Conclusion

A self-healing scraper isn't a smarter scraper — it's a scraper with a fallback that costs money, guarded by validation so it only spends that money when it has to, and a memory so it doesn't spend it twice for the same break. That combination — cheap by default, self-repairing on change, honest about its limits — is what turns a scraper from something you babysit into something that mostly runs itself.

The full runnable code is in the repo below. Clone it, point it at a site you actually maintain, break a selector on purpose, and watch it heal.

[https://github.com/ahmedshamndi/self-healing-scraper]

DEV Community: Ahmed Shamandy