DEV Community: Taranpreet Kaur

Why Your AI-Generated Scraper Works in the Demo and Dies by Friday

Taranpreet Kaur — Sat, 08 Aug 2026 12:00:13 +0000

Something changed in 2026, and it is worth naming precisely rather than dismissing.

You can now describe a website to a model and get back a working scraper in under a minute. It handles the pagination. It gets the selectors right. It produces clean JSON on the first run. Ten years ago that was a day of work for a competent engineer, and the fact that it is now a minute is a real advance, not a party trick.

Then it runs for four days and quietly stops being right.

We have watched this cycle enough times in the last year to think the failure is structural rather than a prompt-quality issue. The gap is not between a good scraper and a bad one. It is between the problem that generation solves and the problem that production actually is. Here is where the seam sits, and what the generated code is genuinely good for once you see it.

Generation solves the part that was never expensive

Write down where the hours actually went on the last extraction project you staffed. Not the estimate. The hours.

For every team we have worked with, the distribution looks roughly like this. Writing the initial selectors and parse logic is a small share of total effort, usually well under a fifth. The rest goes to four categories that a code generator does not touch:

Source discovery and scoping. Working out which of the several hundred candidate sources actually carry the fields you need, in a usable form, at a usable refresh rate. This is judgement work against messy reality, and it happens before any code exists.
Anti-bot and access strategy. Not defeating defences, but understanding which sources will tolerate what access pattern, at what rate, from what path, and designing around it so you are a well-behaved consumer rather than an escalating one.
Normalisation and entity resolution. Getting the same company, provider or product to reconcile across sources that disagree about its name, address and identity. This is where most of the real difficulty lives and it is almost entirely invisible in a demo, because a demo has one source.
Continuous monitoring and repair. The permanent tax. Sources redesign, defences escalate, structures change.

Those four are the categories that actually decide the build versus buy question for web data, and a model generates category zero brilliantly. It has no view on the other four, because none of them are expressible as "write me code that reads this page".

Expert Insight: The most reliable predictor of whether a team is about to be surprised by extraction cost is whether their estimate included a line for maintenance. Teams that budget for build-only routinely find the work becoming a permanent one to two engineer commitment inside a year. It rarely arrives as a decision. It arrives as a side project that becomes a quarter of someone's week, then half, then a named owner, then a rotation with a pager.

Why Friday specifically

The four-day figure is a joke with a mechanism behind it.

Generated scrapers are fitted to the page as it existed at generation time, and they are fitted tightly. A model given one rendering of a page will produce selectors that work perfectly on that rendering, which sounds like the goal and is in fact the failure mode. Hand-written extraction code from an experienced engineer tends to be looser on purpose. It anchors on things that change slowly, tolerates missing optional fields, and is built by someone who has been burned by a promotional banner shifting the DOM.

Three specific things go wrong first, in this order.

The A/B variant. The site was already serving two layouts. The model saw one. Your extraction silently covers half the traffic and you have no way to know, because half of something is a plausible number.

The optional-field assumption. The example record had a value in every field. Roughly a fifth of real records do not. The generated parser treats absence as breakage, or worse, grabs the nearest structurally similar element and returns a well-formed wrong answer.

The first redesign. Whenever it lands, the generated code has no mechanism to notice. It returns zero rows, or partial rows, and zero rows is a valid output that fires no alert.

None of these are model failures. A page is a sample of one. Fitting tightly to a sample of one is what you asked for.

Expert Insight: There is a version of this that works, and we have seen teams get real leverage from it: use generation for the first draft, then have someone with production scars deliberately loosen it. Widen the selectors, make every field optional, add the null-rate check, add a golden set. That review takes twenty minutes and it is the difference between a four-day scraper and a four-month one. The teams that skip it are not being careless. They are responding rationally to code that looks finished.

What this actually changes

The honest read is that generation moved the bottleneck rather than removing it, and that is still worth something.

Before, standing up coverage of five sources took a competent engineer a week or two, which was enough friction that most teams scoped down and stayed there. Now it takes an afternoon. The constraint has shifted to the long tail, where it always belonged. We worked with a ticketing marketplace whose engineer had built five scrapers in six months and then discovered the market they were describing had more than five hundred sources in it. Generation makes those first five nearly free. It does not change that the remaining four hundred and ninety-five each carry their own access pattern, their own schema quirks and their own permanent maintenance line.

That is the reframe worth holding: AI collapsed the cost of the first scraper and left the cost of the five-hundredth almost untouched. If your data requirement is genuinely five sources, generation may have just solved your project, and you should take the win. If it is five hundred, generation has made the easy part free and left the expensive part exactly where it was. Most teams that reach for AI-assisted extraction are in the second situation and are pricing themselves as though they were in the first.

This is also, in fairness, the reason a managed extraction layer exists at all. At Forage AI the durable work is not writing extraction code, which we also generate where it makes sense. It is source discovery across the long tail, the access strategy per source, the normalisation, and the monitoring that runs forever. Those four have not become cheaper, for us or for anyone.

Use the generated scraper. Then ask what happens to it on Friday, and whether anyone would find out.

About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed extraction pipelines across more than 500M+ websites. We generate a fair amount of extraction code ourselves, and then we go and loosen it. Learn more about Forage AI's work in web data extraction at forage.ai.

The 1-in-3 Rule: What Actually Breaks in a Scraping Pipeline After 90 Days

Taranpreet Kaur — Fri, 31 Jul 2026 09:10:07 +0000

A web scraper that works the day you ship it tells you almost nothing about whether it will work in October.

We have watched this pattern repeat across enough deployments to stop calling it bad luck. Roughly a third of the URLs in a first-generation extraction pipeline stop returning usable data inside the first quarter. Not all at once. Almost never loudly.

The number matters less than the shape of the decay. If you know which parts rot first, you can instrument for them before they cost you a customer demo. What follows is a retrospective on the three decay curves we see most often, why none of them page anyone, and what we would build into the first ninety days if we were starting over.

The three decay curves

Extraction pipelines do not fail on a single axis. They fail on three, and the three run on different clocks.

Selector drift is the fastest and the most visible. A class name changes, a wrapper div appears, a product card gets a new layout for a promotion. Time to first failure is usually two to six weeks. This is the failure everyone budgets for, because it is the one that shows up in a tutorial.
Anti-bot escalation is slower and steeper. Nothing happens for a month, then a source moves behind a challenge, or starts fingerprinting, or quietly begins serving a degraded page to traffic it does not recognise. The curve is a cliff rather than a slope, and it tends to hit several sources at once because vendors sell the same defences to whole industries.
Source-side structural change is the slowest and does the most damage. A site consolidates two sections. A directory adds pagination it did not have. An organisation merges with another and its whole URL scheme is retired. Your scraper still runs, still returns 200, still parses. It is just describing a world that no longer exists.

A healthcare data company we work with was tracking provider directories across several thousand hospital sites. Their scrapers were healthy by every metric they had. What they could not see was that hospital acquisition activity had reshuffled the affiliation graph underneath them, and their affiliation data was lagging reality by weeks. The extraction was fine. The dataset was wrong.

That gap between "the job succeeded" and "the data is true" is where most of the ninety-day loss lives, and it is the reason enterprise extraction pipelines break in ways their own dashboards cannot see.

Expert Insight: Teams consistently over-invest in the first curve and under-invest in the third. Selector drift is annoying but self-announcing, and a junior engineer can fix it in an afternoon. Structural change is silent, compounds daily, and is usually discovered by a customer rather than a monitor. If you only have budget to instrument one curve properly, instrument the slow one.

Why none of it pages you

The uncomfortable part is not that scrapers break. It is that a broken scraper looks identical to a working one from the outside.

Three specific silences account for most of what we see.

The empty 200. The request succeeds, the response is well formed, the body contains a challenge page or a soft error rendered as normal HTML. Your parser finds no matches and writes zero rows. Zero rows is a valid outcome, so nothing fires.

Schema-valid, value-wrong. Extraction returns the right shape with the wrong contents. A price field picks up a strikethrough comparison price. A location field picks up the "nearby" module instead of the record. Every type check passes. Every value is garbage.

Coverage collapse hidden by averages. You scrape four hundred sources. Eleven of them die. Your aggregate row count moves two percent, well inside normal variance, and your dashboard shows a flat green line. The eleven that died happen to be the eleven your largest customer cares about.

None of these throw. That is the entire point. Standard pipeline observability, the kind you get free with an orchestrator, watches whether the task completed and how long it took. It has no opinion about whether the numbers are real. Judging that is the job of an extraction QA workflow, which is a separate discipline with a separate set of checks.

Expert Insight: The single highest-return check we deploy is also the least sophisticated: a per-source row-count floor with a comparison against the same source seven days earlier. It catches the empty 200 and coverage collapse on the same day they happen, and it takes an afternoon to build. Sophisticated anomaly detection can come later. Most teams skip the afternoon version and end up with neither.

What we would instrument in the first ninety days

If we were standing up a new pipeline tomorrow, five checks would go in before any additional source did. They are ordered by return on effort, not by elegance.

Check 1: per-source row-count floors, compared week over week. Absolute thresholds go stale. Relative comparison against the same source's recent history does not. It is the cheapest form of observability for external data pipelines and the one with the highest return per hour spent.

Check 2: field-level null-rate tracking. Not "did we get rows" but "did we get values". A field that was 4% null last week and is 60% null today is the earliest honest signal that a layout changed.

Check 3: a golden set per source. Twenty to fifty records you have verified by hand, re-extracted on every run and diffed. This is the only check that catches schema-valid, value-wrong extraction, and there is no clever substitute for it.

Check 4: response-shape fingerprinting. Hash the structural skeleton of the page, not the content. When the hash moves for a whole source at once, a redesign has shipped and you have days rather than weeks of warning.

Check 5: a freshness clock per record, not per job. Job-level timestamps tell you when you ran. Record-level timestamps tell you how old the truth is. Those diverge the moment a source starts serving cached or partial responses, and only the second one is a number you can put in front of a customer.

None of this is difficult engineering. It is unglamorous, and it competes for time with building the next twenty scrapers, which is why it usually loses. At Forage AI we run these checks as a standing part of delivery rather than an add-on, because the alternative is finding out from the person who bought the data. That is a decision about where the burden of proof sits, and it is worth making deliberately rather than by default.

Expert Insight: Golden sets are the check teams argue about most and regret skipping most. The objection is always maintenance cost. In practice a fifty-record golden set per source costs a few hours to build and about twenty minutes a quarter to keep honest, and it is the only mechanism that distinguishes "we got data" from "we got the right data". Every team we have seen build one has kept it.

What the ninety days actually cost

The cost of skipping this is rarely booked as a data cost. It shows up as headcount.

A scraper starts as a side project. It becomes a quarter of an engineer's week, then half, then a named owner, then a rotation with a pager. We have seen teams cross from "we built a scraper" to "we staff a scraping function" without a single conversation about whether they meant to. One to two full-time engineers, permanently, on maintenance that produces no new capability.

The three curves are going to bend whatever you do. Sources will redesign, defences will escalate, organisations will merge. What you control is whether you find out on day two or day ninety, and the difference between those two numbers is almost entirely a function of five checks you either built early or did not.

Worth asking on your own pipeline: if a third of your sources went quiet this morning, which dashboard would tell you before your customer did?

About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed web data extraction pipelines across more than 500M+ websites. We have spent 12+ years watching extraction pipelines decay in production and building the checks that catch it early. Learn more about Forage AI's work in web data extraction at forage.ai.