Stop writing a parser per site. Run five and let confidence decide.

For a long time I ran product extraction off a database of custom selector configs. Hundreds of retailers, each with its own set of CSS selectors mapped field by field: price here, title there, image over there. It worked. It was also a treadmill. Every retailer that redesigned its frontend silently broke its config, and the maintenance tax grew with every retailer I added. Hundreds of configs is hundreds of things that can rot without telling you, usually right before someone downstream asks why a brand's prices went null.

At some point I stopped feeding it. The scraping platform I ran at my last engagement fed a 20M+ product catalogue at millions of pages a day, and the thing that made that survivable wasn't better per-site config. It was leaning on almost no per-site config at all.

The pattern

Instead of one careful parser per site, run several cheap generic extractors on every page, in parallel:

1. JSON-LD. A huge share of e-commerce pages ship a schema.org/Product block because Google rewards it. It has name, price, currency, availability, images. It's the closest thing to a public API hiding in the HTML.
2. OpenGraph tags. og:title, og:image, product:price:amount. Lower quality than JSON-LD, but present on sites too lazy for structured data, because everyone wants pretty link previews.
3. Microdata / RDFa. Older sites, still surprisingly common outside the US.
4. Embedded state. __NEXT_DATA__, window.__INITIAL_STATE__ and friends. A JSON blob the site's own frontend hydrates from.
5. Heuristics. A price-shaped string near a currency symbol inside the main content region. The largest above-the-fold image. The h1. Dumb, and dumb works more often than you'd think.

None of these is reliable alone. OG price tags go stale, JSON-LD sometimes describes the wrong variant, heuristics grab the crossed-out "was" price. The trick is you don't pick an extractor. You pick per field, and you let agreement between sources carry the decision.

Confidence merge

Each extractor emits candidates with a prior based on how trustworthy that source usually is for that field. Then candidates that agree with each other get boosted, and the top score wins per field:

function merge(results, fields) {
  const out = {};
  for (const field of fields) {
    const candidates = results
      .filter((r) => r.data[field] != null)
      .map((r) => ({
        value: r.data[field],
        source: r.name,
        score: PRIORS[r.name][field] ?? 0.3,
      }));

    for (const c of candidates) {
      const agreeing = candidates.filter(
        (o) => o !== c && normalize(o.value) === normalize(c.value)
      ).length;
      c.score *= 1 + 0.4 * agreeing;
    }

    candidates.sort((a, b) => b.score - a.score);
    out[field] = candidates[0] ?? null;
  }
  return out;
}

normalize matters more than the scoring. "1 299,00 zł" and "1299.00 PLN" have to compare equal or your agreement boost never fires. I spent more time on value normalization than on everything else in this system combined.

Add a validation pass after the merge: does the price parse, is the currency a real ISO code, is the title longer than three characters. A merged record that fails validation is a signal, not a result. Log it, sample those pages weekly, and you'll find out which extractor is drifting before your customers do.

Why bother

Because the long tail is where per-site config dies. The top few hundred retailers can justify hand-tuned selectors, and I maintained exactly that for years. Site number 4,000 does not, and with an ensemble it doesn't need any. A new domain comes in, four of five extractors fire, three agree on price, and you have a usable record with zero engineering hours spent. Across that platform we held roughly 95% extraction success at about 3 seconds a page, and most of the domains involved had nobody who had ever looked at their HTML.

The honest limitation: variant-heavy product pages are still rough. When one URL carries twelve colorways at different prices, the ensemble happily returns a confident answer to the wrong question, because every source on the page describes the default variant. I handle that with a separate variant expansion step, and it's the least elegant code I own.

But for "give me name, price, image, availability from an arbitrary shop URL," the ensemble beat every hand-written parser I've replaced with it. Write extractors for formats, not for websites. Formats change maybe once a decade. Websites change on Tuesdays.

This approach is the core of what I'm building at Cartpie, an e-commerce product-data scraping platform, free tier live at cartpie.com if you want to throw some long-tail URLs at it.