Extraction Rules as Versioned Config: Monitoring 50 Retailers Without 50 Codebases

A while back I wrote about the general architecture of a non-Amazon retail price monitor: fetchers, schedulers, snapshot storage, diffing. This article zooms into the layer where most of these systems actually die — the extraction layer — and argues one specific point: per-site extraction rules should be declarative, versioned data, not code.

The N-codebases trap

The naive architecture for monitoring N retailers is N bespoke parsers. Each parser knows that retailer's HTML, its price element, its sale-price variant, its "out of stock" badge, its JSON blob buried in a script tag. The code is different every time because the sites are different every time.

This fails in predictable ways:

• Onboarding friction scales linearly. Adding retailer #51 means a new module, new tests, new deployment. At some point adding a retailer becomes a multi-day project instead of an afternoon.
• Silent breakage. When a retailer ships a redesign, your parser doesn't throw. soup.find("span", class_="price") returns None, you coerce or skip, and you only find out weeks later when a client asks why their competitor's price stopped updating. Custom parsers fail customly.
• Untestable combinatorics. 50 sites times 3 failure modes times quarterly site updates is a steady stream of incidents, each requiring a human to re-read bespoke code. There is no shared harness, no shared metric, no shared alert.

The fix is not better parsers. It's recognizing that the per-site knowledge — selectors, strategies, validation thresholds, stock semantics — is configuration. One generic extraction engine consumes it. Sites become data.

What goes in a site schema

Each retailer gets a YAML file (JSON works; YAML is easier to review in diffs). The schema is the contract between "what the site looks like" and "what a valid record is":

• Field specs. For each field (price, list_price, currency, title, availability), an ordered list of extraction strategies: a primary CSS selector, fallback selectors, an XPath, a JSON-LD (ld+json) path, a regex against an embedded JSON blob. Ordered matters — first match wins, and we record which strategy won.
• Required vs optional. A record without a title is degraded; a record without a price is garbage. The schema says which.
• Validation rules. Price sanity (0 < price <= 100000), expected currency, tolerated price delta vs the last snapshot (I use 80% — a bigger swing is more likely a parsing error than a real price change), and in-stock semantics (some sites use a class, some use meta tags, some put it only in JSON-LD).
• Schema version and changelog. A version field and a short changelog so every stored record can be traced back to the rules that produced it. When your data looks weird in three months, you need to know which schema version was running.

Here are two real-ish specs. Same engine, wildly different sites:

# schemas/nordtools.yml
version: "2.1.0"
site: nordtools.example
changelog:
  - "2.1.0: fallback to JSON-LD after Feb redesign"
  - "2.0.0: price moved into data-price attribute"
fields:
  price:
    required: true
    strategies:
      - type: css
        selector: "span.product-price[data-price]"
        attribute: "data-price"
      - type: ld+json
        path: "offers.price"
    validation:
      min: 0.01
      max: 100000
  currency:
    required: true
    strategies:
      - type: css
        selector: "meta[itemprop='priceCurrency']"
        attribute: "content"
    validation:
      enum: [USD, EUR, GBP]
  availability:
    required: false
    strategies:
      - type: css
        selector: "button.add-to-cart"
        present_means: "in_stock"

# schemas/fragattire.yml
version: "1.3.0"
site: fragattire.example
fields:
  price:
    required: true
    strategies:
      - type: regex-json
        # embedded window.__INITIAL_STATE__ blob
        pattern: '"price":\s*([0-9]+\.?[0-9]*)'
      - type: css
        selector: "div.pdp-price-now"
    validation:
      min: 0.01
      max: 100000
      max_delta_vs_snapshot: 0.80

The engine

The extractor below is the entire per-site "code" that remains — and it's site-agnostic. It loads a spec, walks strategies in order, validates the assembled record, and tags each record with the schema version plus which strategy won each field. It depends on beautifulsoup4, lxml (for XPath), and pyyaml.

"""Schema-driven extractor: one engine, N retailer specs."""
import json
import re
from dataclasses import dataclass, field
from typing import Any

import yaml
from bs4 import BeautifulSoup
from lxml import etree


@dataclass
class ExtractedRecord:
    data: dict
    schema_version: str
    winning_strategies: dict          # field -> strategy that won
    errors: list = field(default_factory=list)

    @property
    def valid(self) -> bool:
        return not self.errors


class SchemaDrivenExtractor:
    """Loads a site spec (YAML) and extracts fields generically."""

    def __init__(self, spec_path: str, last_snapshot: dict | None = None):
        with open(spec_path, encoding="utf-8") as fh:
            self.spec = yaml.safe_load(fh)
        self.version = self.spec["version"]
        self.last_snapshot = last_snapshot or {}
        # strategy dispatch table: type -> handler
        self._handlers = {
            "css": self._from_css,
            "xpath": self._from_xpath,
            "ld+json": self._from_ldjson,
            "regex-json": self._from_regex_json,
        }

    # ---------- strategy handlers ----------

    def _from_css(self, soup: BeautifulSoup, strat: dict) -> Any | None:
        node = soup.select_one(strat["selector"])
        if node is None:
            return None
        if "attribute" in strat:
            return node.get(strat["attribute"])
        return node.get_text(strip=True)

    def _from_xpath(self, soup: BeautifulSoup, strat: dict) -> Any | None:
        tree = etree.HTML(str(soup))   # lxml for XPath; BS4 has none
        hits = tree.xpath(strat["selector"])
        return hits[0].strip() if hits else None

    def _from_ldjson(self, soup: BeautifulSoup, strat: dict) -> Any | None:
        for tag in soup.find_all("script", type="application/ld+json"):
            try:
                blob = json.loads(tag.string or "")
            except (json.JSONDecodeError, TypeError):
                continue
            # dotted path walk, e.g. "offers.price"
            cur: Any = blob
            ok = True
            for part in strat["path"].split("."):
                if isinstance(cur, list):
                    cur = cur[0] if cur else {}
                if isinstance(cur, dict) and part in cur:
                    cur = cur[part]
                else:
                    ok = False
                    break
            if ok and cur is not None:
                return cur
        return None

    def _from_regex_json(self, _soup: BeautifulSoup, strat: dict,
                         raw_html: str = "") -> Any | None:
        m = re.search(strat["pattern"], raw_html)
        return m.group(1) if m else None

    # ---------- per-field extraction ----------

    def _extract_field(self, soup, raw_html: str, name: str, fspec: dict):
        for strat in fspec.get("strategies", []):
            handler = self._handlers[strat["type"]]
            value = handler(soup, strat, raw_html=raw_html) \
                if strat["type"] == "regex-json" else handler(soup, strat)
            if value is not None:
                return value, strat["type"]
        return None, None

    # ---------- validation ----------

    def _validate(self, record: ExtractedRecord, fspec: dict):
        rules = fspec.get("validation", {})
        value = record.data.get(fspec["_name"])

        if value is None:
            if fspec.get("required"):
                record.errors.append(f"{fspec['_name']}: missing (required)")
            return

        if "min" in rules or "max" in rules:
            try:
                num = float(re.sub(r"[^0-9.]", "", str(value)))
                record.data[fspec["_name"]] = num
            except ValueError:
                record.errors.append(f"{fspec['_name']}: not numeric: {value!r}")
                return
            if "min" in rules and num < rules["min"]:
                record.errors.append(f"{fspec['_name']}: {num} < min")
            if "max" in rules and num > rules["max"]:
                record.errors.append(f"{fspec['_name']}: {num} > max")
            # delta vs last snapshot: big swings are usually parse errors
            prev = self.last_snapshot.get(fspec["_name"])
            if prev and "max_delta_vs_snapshot" in rules and prev > 0:
                if abs(num - prev) / prev > rules["max_delta_vs_snapshot"]:
                    record.errors.append(
                        f"{fspec['_name']}: {num} vs snapshot {prev} "
                        f"exceeds {rules['max_delta_vs_snapshot']:.0%} delta")

        if "enum" in rules and value not in rules["enum"]:
            record.errors.append(f"{fspec['_name']}: {value!r} not in enum")

    # ---------- entry point ----------

    def extract(self, html: str) -> ExtractedRecord:
        soup = BeautifulSoup(html, "html.parser")
        record = ExtractedRecord(data={}, schema_version=self.version,
                                 winning_strategies={})
        for name, fspec in self.spec["fields"].items():
            fspec = {**fspec, "_name": name}
            value, won = self._extract_field(soup, html, name, fspec)
            if value is not None:
                record.data[name] = value
                record.winning_strategies[name] = won
            self._validate(record, fspec)
        return record


if __name__ == "__main__":
    ex = SchemaDrivenExtractor("schemas/nordtools.yml",
                               last_snapshot={"price": 49.99})
    rec = ex.extract(open("fixtures/nordtools/product.html", encoding="utf-8").read())
    print(rec.valid, rec.data, rec.winning_strategies)

Two details worth calling out. First, the present_means trick in the availability spec: some fields are boolean-by-presence, and the schema encodes the semantics so the engine stays dumb. Second, the snapshot-delta check lives in the engine, not in per-site code, because "price jumped 90% overnight" is almost always a parser bug regardless of retailer.

Selector drift detection

Fallbacks create a subtle failure mode: the primary selector dies, the fallback quietly takes over, and everything looks fine — until the fallback dies too, six months later, and you've lost months of higher-quality data (JSON-LD fallbacks, for instance, often lag behind the live DOM price).

The fix is to make fallback wins a first-class metric. The engine already tags every record with the winning strategy. On top of that, run a rolling counter per site per field: over the last 200 requests, if the primary selector's win rate drops below 85% — i.e., fallbacks win more than 15% of the time — emit a drift alert. Don't alert on a single miss; single misses are caching, A/B tests, regional variants. Alert on the sustained shift.

Drift alerts turn "silent degradation" into a ticket with a known remedy: inspect the site, bump the spec version, update the primary selector, move the old one to fallback. Which brings us to the workflow.

The payoff: schema PRs instead of deploys

Once sites are data, the operational loop changes shape:

• Adding a retailer is a new YAML file plus a validation run against 30–50 archived HTML fixtures for that site (you do keep fixtures, right?). The engine is unchanged. No deploy.
• Fixing breakage is a schema PR. The diff is reviewable by someone who never read the engine code: "primary selector changed from span.product-price to div[data-price]" is a one-line review. A 40-line parser patch is not.
• A/B validation before rollout: run the candidate schema and the current schema against the same fixture set and diff the extracted records. If 50 fixtures produce identical prices, titles, and availability, ship it. If two fixtures disagree, you've found the regression before your data pipeline did.
• Every stored record is traceable: schema version + winning strategy means you can answer "why did we record $12.99 on March 3" with evidence, not archaeology.

One operational note that sits outside the schema: fetching from 50 retailers means 50 different expected request patterns, and hammering all of them from one datacenter IP range gets you blocked long before your selectors matter. Route everything through a residential proxy gateway so per-site request rates, headers, and geography look like ordinary shoppers — a single generic endpoint like http://gateway.example:8080 with per-site rate limits in the same config, e.g. max_rpm: 6, min_interval_s: 9. That keeps the crawling concerns declarative too.

Honest limits

Config is not a panacea, and pretending otherwise is how these systems rot from the other direction.

Some sites genuinely need code. Heavy JS rendering means you're extracting from a headless browser, and while you can put "wait for selector X" in config, complex flows (login walls, anti-bot challenges, infinite scroll with nonce-guarded APIs) belong in a plugin. The boundary I've settled on: if a site can be handled by ordered strategies over fetched HTML, it's schema; if extraction requires driving the browser or replaying signed requests, it's a plugin that still emits the same record shape and schema version. The schema can even declare engine: plugin and name it.

Obfuscated sites — CSS classes that rotate per deploy, hashed attribute names — make static selectors worthless. Regex-on-JSON survives longer than CSS in these cases, which is why it earns a strategy slot, but eventually you're writing heuristics that are code in disguise. Accept it, isolate it in a plugin, and keep the validation and versioning uniform.

Finally, JSON-LD is not a silver bullet. Merchants generate it from the same broken feeds that generate their pages; I've seen offers.price disagree with the DOM by 20%. Treat it as a strategy and a cross-check, not as truth.

Wrapping up

The failure mode of multi-retailer monitoring is not crawling, proxies, or scheduling — it's 50 bespoke parsers each failing in its own private way. Move per-site extraction into versioned, declarative specs with explicit strategies and validation, and the engine becomes generic, breakage becomes measurable drift, fixes become reviewable diffs, and onboarding retailer #51 becomes an afternoon of YAML plus a fixture run. The code you keep is the code you write once.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele