A web scraper that works the day you ship it tells you almost nothing about whether it will work in October.
We have watched this pattern repeat across enough deployments to stop calling it bad luck. Roughly a third of the URLs in a first-generation extraction pipeline stop returning usable data inside the first quarter. Not all at once. Almost never loudly.
The number matters less than the shape of the decay. If you know which parts rot first, you can instrument for them before they cost you a customer demo. What follows is a retrospective on the three decay curves we see most often, why none of them page anyone, and what we would build into the first ninety days if we were starting over.
The three decay curves
Extraction pipelines do not fail on a single axis. They fail on three, and the three run on different clocks.
Selector drift is the fastest and the most visible. A class name changes, a wrapper div appears, a product card gets a new layout for a promotion. Time to first failure is usually two to six weeks. This is the failure everyone budgets for, because it is the one that shows up in a tutorial.
Anti-bot escalation is slower and steeper. Nothing happens for a month, then a source moves behind a challenge, or starts fingerprinting, or quietly begins serving a degraded page to traffic it does not recognise. The curve is a cliff rather than a slope, and it tends to hit several sources at once because vendors sell the same defences to whole industries.
Source-side structural change is the slowest and does the most damage. A site consolidates two sections. A directory adds pagination it did not have. An organisation merges with another and its whole URL scheme is retired. Your scraper still runs, still returns 200, still parses. It is just describing a world that no longer exists.
A healthcare data company we work with was tracking provider directories across several thousand hospital sites. Their scrapers were healthy by every metric they had. What they could not see was that hospital acquisition activity had reshuffled the affiliation graph underneath them, and their affiliation data was lagging reality by weeks. The extraction was fine. The dataset was wrong.
That gap between "the job succeeded" and "the data is true" is where most of the ninety-day loss lives, and it is the reason enterprise extraction pipelines break in ways their own dashboards cannot see.
Expert Insight: Teams consistently over-invest in the first curve and under-invest in the third. Selector drift is annoying but self-announcing, and a junior engineer can fix it in an afternoon. Structural change is silent, compounds daily, and is usually discovered by a customer rather than a monitor. If you only have budget to instrument one curve properly, instrument the slow one.
Why none of it pages you
The uncomfortable part is not that scrapers break. It is that a broken scraper looks identical to a working one from the outside.
Three specific silences account for most of what we see.
The empty 200. The request succeeds, the response is well formed, the body contains a challenge page or a soft error rendered as normal HTML. Your parser finds no matches and writes zero rows. Zero rows is a valid outcome, so nothing fires.
Schema-valid, value-wrong. Extraction returns the right shape with the wrong contents. A price field picks up a strikethrough comparison price. A location field picks up the "nearby" module instead of the record. Every type check passes. Every value is garbage.
Coverage collapse hidden by averages. You scrape four hundred sources. Eleven of them die. Your aggregate row count moves two percent, well inside normal variance, and your dashboard shows a flat green line. The eleven that died happen to be the eleven your largest customer cares about.
None of these throw. That is the entire point. Standard pipeline observability, the kind you get free with an orchestrator, watches whether the task completed and how long it took. It has no opinion about whether the numbers are real. Judging that is the job of an extraction QA workflow, which is a separate discipline with a separate set of checks.
Expert Insight: The single highest-return check we deploy is also the least sophisticated: a per-source row-count floor with a comparison against the same source seven days earlier. It catches the empty 200 and coverage collapse on the same day they happen, and it takes an afternoon to build. Sophisticated anomaly detection can come later. Most teams skip the afternoon version and end up with neither.
What we would instrument in the first ninety days
If we were standing up a new pipeline tomorrow, five checks would go in before any additional source did. They are ordered by return on effort, not by elegance.
Check 1: per-source row-count floors, compared week over week. Absolute thresholds go stale. Relative comparison against the same source's recent history does not. It is the cheapest form of observability for external data pipelines and the one with the highest return per hour spent.
Check 2: field-level null-rate tracking. Not "did we get rows" but "did we get values". A field that was 4% null last week and is 60% null today is the earliest honest signal that a layout changed.
Check 3: a golden set per source. Twenty to fifty records you have verified by hand, re-extracted on every run and diffed. This is the only check that catches schema-valid, value-wrong extraction, and there is no clever substitute for it.
Check 4: response-shape fingerprinting. Hash the structural skeleton of the page, not the content. When the hash moves for a whole source at once, a redesign has shipped and you have days rather than weeks of warning.
Check 5: a freshness clock per record, not per job. Job-level timestamps tell you when you ran. Record-level timestamps tell you how old the truth is. Those diverge the moment a source starts serving cached or partial responses, and only the second one is a number you can put in front of a customer.
None of this is difficult engineering. It is unglamorous, and it competes for time with building the next twenty scrapers, which is why it usually loses. At Forage AI we run these checks as a standing part of delivery rather than an add-on, because the alternative is finding out from the person who bought the data. That is a decision about where the burden of proof sits, and it is worth making deliberately rather than by default.
Expert Insight: Golden sets are the check teams argue about most and regret skipping most. The objection is always maintenance cost. In practice a fifty-record golden set per source costs a few hours to build and about twenty minutes a quarter to keep honest, and it is the only mechanism that distinguishes "we got data" from "we got the right data". Every team we have seen build one has kept it.
What the ninety days actually cost
The cost of skipping this is rarely booked as a data cost. It shows up as headcount.
A scraper starts as a side project. It becomes a quarter of an engineer's week, then half, then a named owner, then a rotation with a pager. We have seen teams cross from "we built a scraper" to "we staff a scraping function" without a single conversation about whether they meant to. One to two full-time engineers, permanently, on maintenance that produces no new capability.
The three curves are going to bend whatever you do. Sources will redesign, defences will escalate, organisations will merge. What you control is whether you find out on day two or day ninety, and the difference between those two numbers is almost entirely a function of five checks you either built early or did not.
Worth asking on your own pipeline: if a third of your sources went quiet this morning, which dashboard would tell you before your customer did?
About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed web data extraction pipelines across more than 500M+ websites. We have spent 12+ years watching extraction pipelines decay in production and building the checks that catch it early. Learn more about Forage AI's work in web data extraction at forage.ai.
Top comments (0)