Something changed in 2026, and it is worth naming precisely rather than dismissing.
You can now describe a website to a model and get back a working scraper in under a minute. It handles the pagination. It gets the selectors right. It produces clean JSON on the first run. Ten years ago that was a day of work for a competent engineer, and the fact that it is now a minute is a real advance, not a party trick.
Then it runs for four days and quietly stops being right.
We have watched this cycle enough times in the last year to think the failure is structural rather than a prompt-quality issue. The gap is not between a good scraper and a bad one. It is between the problem that generation solves and the problem that production actually is. Here is where the seam sits, and what the generated code is genuinely good for once you see it.
Generation solves the part that was never expensive
Write down where the hours actually went on the last extraction project you staffed. Not the estimate. The hours.
For every team we have worked with, the distribution looks roughly like this. Writing the initial selectors and parse logic is a small share of total effort, usually well under a fifth. The rest goes to four categories that a code generator does not touch:
Source discovery and scoping. Working out which of the several hundred candidate sources actually carry the fields you need, in a usable form, at a usable refresh rate. This is judgement work against messy reality, and it happens before any code exists.
Anti-bot and access strategy. Not defeating defences, but understanding which sources will tolerate what access pattern, at what rate, from what path, and designing around it so you are a well-behaved consumer rather than an escalating one.
Normalisation and entity resolution. Getting the same company, provider or product to reconcile across sources that disagree about its name, address and identity. This is where most of the real difficulty lives and it is almost entirely invisible in a demo, because a demo has one source.
Continuous monitoring and repair. The permanent tax. Sources redesign, defences escalate, structures change.
Those four are the categories that actually decide the build versus buy question for web data, and a model generates category zero brilliantly. It has no view on the other four, because none of them are expressible as "write me code that reads this page".
Expert Insight: The most reliable predictor of whether a team is about to be surprised by extraction cost is whether their estimate included a line for maintenance. Teams that budget for build-only routinely find the work becoming a permanent one to two engineer commitment inside a year. It rarely arrives as a decision. It arrives as a side project that becomes a quarter of someone's week, then half, then a named owner, then a rotation with a pager.
Why Friday specifically
The four-day figure is a joke with a mechanism behind it.
Generated scrapers are fitted to the page as it existed at generation time, and they are fitted tightly. A model given one rendering of a page will produce selectors that work perfectly on that rendering, which sounds like the goal and is in fact the failure mode. Hand-written extraction code from an experienced engineer tends to be looser on purpose. It anchors on things that change slowly, tolerates missing optional fields, and is built by someone who has been burned by a promotional banner shifting the DOM.
Three specific things go wrong first, in this order.
The A/B variant. The site was already serving two layouts. The model saw one. Your extraction silently covers half the traffic and you have no way to know, because half of something is a plausible number.
The optional-field assumption. The example record had a value in every field. Roughly a fifth of real records do not. The generated parser treats absence as breakage, or worse, grabs the nearest structurally similar element and returns a well-formed wrong answer.
The first redesign. Whenever it lands, the generated code has no mechanism to notice. It returns zero rows, or partial rows, and zero rows is a valid output that fires no alert.
None of these are model failures. A page is a sample of one. Fitting tightly to a sample of one is what you asked for.
Expert Insight: There is a version of this that works, and we have seen teams get real leverage from it: use generation for the first draft, then have someone with production scars deliberately loosen it. Widen the selectors, make every field optional, add the null-rate check, add a golden set. That review takes twenty minutes and it is the difference between a four-day scraper and a four-month one. The teams that skip it are not being careless. They are responding rationally to code that looks finished.
What this actually changes
The honest read is that generation moved the bottleneck rather than removing it, and that is still worth something.
Before, standing up coverage of five sources took a competent engineer a week or two, which was enough friction that most teams scoped down and stayed there. Now it takes an afternoon. The constraint has shifted to the long tail, where it always belonged. We worked with a ticketing marketplace whose engineer had built five scrapers in six months and then discovered the market they were describing had more than five hundred sources in it. Generation makes those first five nearly free. It does not change that the remaining four hundred and ninety-five each carry their own access pattern, their own schema quirks and their own permanent maintenance line.
That is the reframe worth holding: AI collapsed the cost of the first scraper and left the cost of the five-hundredth almost untouched. If your data requirement is genuinely five sources, generation may have just solved your project, and you should take the win. If it is five hundred, generation has made the easy part free and left the expensive part exactly where it was. Most teams that reach for AI-assisted extraction are in the second situation and are pricing themselves as though they were in the first.
This is also, in fairness, the reason a managed extraction layer exists at all. At Forage AI the durable work is not writing extraction code, which we also generate where it makes sense. It is source discovery across the long tail, the access strategy per source, the normalisation, and the monitoring that runs forever. Those four have not become cheaper, for us or for anyone.
Use the generated scraper. Then ask what happens to it on Friday, and whether anyone would find out.
About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed extraction pipelines across more than 500M+ websites. We generate a fair amount of extraction code ourselves, and then we go and loosen it. Learn more about Forage AI's work in web data extraction at forage.ai.
Top comments (1)
Scrapers fail because the web is not a stable API. AI can help produce the first version, but the production system needs selectors with fallbacks, change detection, rate limits, retries, and a way to know when extraction became untrustworthy.