I ran a scraping platform that processed millions of pages a day at roughly 95% extraction success, around three seconds per page. The fetch-and-parse code, the part everyone thinks of as "the scraper", was a tiny slice of the whole thing. The years went into everything around it.
Here's what actually broke, more or less in the order it broke.
The queue with no manners
Crawling is bursty in a way that surprises you the first time. One category page fans out into a few hundred product URLs. A sitemap refresh dumps half a million at once. Meanwhile your parsers chew through pages at a steady rate that doesn't care about your ambitions.
Our first queue was effectively unbounded. It absorbed every burst happily, Redis memory climbed for two days, and then the whole thing fell over at once instead of slowing down gracefully. Lesson: if your queue can't say no, it's not a queue, it's a landfill. Bound it, and make producers block or shed when it's full. A crawler that pauses discovery for an hour is a non-event. A crawler that OOMs the broker is a weekend.
Retries that stampede
A target site starts throwing 500s for two minutes. Fine, that happens. Every failed page gets rescheduled with the same fixed backoff. Which means 40,000 pages land back on that site in the same ten-second window, it tips over again, and now their WAF has opinions about you. You caused the second outage yourself.
Two fixes, both mandatory: full jitter on backoff, and a per-domain circuit breaker so a struggling site stops receiving traffic entirely for a while.
import random
def backoff(attempt, base=2.0, cap=300.0):
# full jitter: spread retries across the whole window
return random.uniform(0, min(cap, base * 2 ** attempt))
That one-liner is the difference between "we retry politely" and "we DDoS people by accident".
Dedupe before fetch, not after
The same product reaches you through six URLs. Tracking params, color variant params, three different category paths, a mobile subdomain. If you dedupe after fetching, you paid for six fetches to keep one page. At millions of pages a day that's real money and real block-risk for zero data.
Canonicalize at enqueue time. Strip the junk params, pick one canonical form, check the seen-set before the URL ever enters the queue. And watch that seen-set: it grows forever unless you give it a TTL or accept a small false-positive rate with something bloom-filter-shaped.
Parser drift, the quiet one
Sites almost never break your parser loudly. They drip. A price moves from a DOM node into an embedded script blob. An image attribute gets renamed. Your extraction still "succeeds", the run is green, and one field quietly goes null on 30% of pages. Nobody notices until a downstream consumer asks why half the prices vanished last Tuesday.
Field-level fill rates saved us. For every field, track what fraction of pages yielded a value, keep a rolling baseline per site, alert on the delta. A green run only proves the code didn't crash.
The other thing that got us to 95%: never rely on a single extraction strategy. Embedded JSON, microdata, plain DOM selectors, run more than one and merge by confidence. When a site redesign kills one of them, the others carry it while you fix things, instead of the number going to zero overnight.
The CPU step hiding in an I/O system
Scraping feels like an I/O problem. Fetch, wait, fetch, wait. So you build it all
async and assume the network is the only thing you're waiting on. Then you profile a
slow day and find the bottleneck is HTML parsing.
Loading a big product page into an HTML parser and running extractors over it is
CPU-bound and synchronous. On a single-threaded runtime, every parse blocks the event
loop, which means it blocks the very downloads you thought were the slow part. Your
concurrency number looks healthy and your throughput doesn't match it, because workers
are queued behind each other's parsing, not behind the network.
The fix is to move the CPU-heavy parse off the main thread into a worker pool, and fall
back to inline parsing when the pool is disabled or a job errors. Downloads keep flowing
while parsing happens elsewhere. Obvious in hindsight, invisible until you measure it.
It ended up one of the bigger single throughput wins I got at scale.
You will need to debug the past
The nastiest questions arrive late. "Why did brand X prices go missing on the 14th?" It's the 19th. The pages have changed since. Your logs say everything was fine.
We got out of that hole with a typed per-page event stream plus archived raw HTML. Every page emitted structured events (fetched, parsed, extracted, with typed payloads) and the raw response got stored. So "what happened on the 14th" became a replay: pull the archived pages, re-run the current parser against them, diff the output. Debugging in the past tense. Without replay you're reconstructing incidents from vibes.
The uncomfortable summary
None of this is glamorous. There's no clever algorithm in this post, and that's the point. At small scale, scraping is a parsing problem. At millions of pages a day it's a distributed systems problem wearing a parsing costume, and the failure modes are the boring ones: backpressure, thundering herds, silent data decay, no audit trail.
All of this is what I'm productizing at Cartpie, e-commerce product data as an API so you don't have to own any of the machinery above. Free tier is live.
Top comments (6)
This is one of the better descriptions of what scraping becomes at scale: not a parsing problem, but a distributed systems problem with HTML attached.
The field-level fill-rate point is especially important. A parser can keep returning “success” while the actual data quietly decays, which makes schema drift more dangerous than a hard failure.
I’d add two caveats:
Canonicalization before enqueue should usually be site-aware. Some query parameters are noise, but others represent real variants, locale, currency, or pricing context. A bad canonicalizer can reduce duplicates by deleting meaning.
And archived raw HTML is incredibly useful for replay, but at millions of pages per day it quickly becomes a storage, retention, PII, and compliance problem. Compression, content hashing, lifecycle policies, and selective retention become part of the architecture too.
Still, the line about debugging the past is exactly right. Without replayable inputs, incident analysis becomes archaeology.
That's very accurate, thanks for adding this important piece. I have a URL cleaner layer with generic config + per-site overrides. A big PITA to maintain, but I have not found a way to automate this meaningfully yet. I am working on a big layer of self-improvement for my scraping platform, but it's currently aimed at analysing pages that fail the extraction and understanding why. Which basically becomes a dynamic roadmap that answers questions like: when the page fails - is this because we hit a captcha, or extraction config drifted, or site is down etc etc. Which then, sorted by volume, tells me what is the next big thing I need to improve.
That sounds like the right direction. Turning extraction failures into a classified, volume-ranked roadmap is much more useful than improving the platform based on intuition. The important part will probably be preserving an “unknown” category instead of forcing every failure into captcha, drift, downtime, or another known bucket. Otherwise, a confident misclassification can quietly distort the roadmap. I’d also track confidence and sample some classifications for human verification. Over time, the disagreement between the classifier and manual review could become another signal for what the system still does not understand.
At that point, the platform is not only detecting failures. It is learning where its own diagnosis is weak, which is arguably the more interesting problem.
Also very accurate and important, thanks. Funnily enough, keeping the cloud cost low is one of my primary services, and a huge S3 bucket with years of (often useless) data is one of the first things I check for the clients 😃
In case of the scraping platform - my current setup is 30 days retention, then delete. No complicated lifecycle policy, no putting on ice.
That’s a sensible retention window. At that point I’d probably optimize for replayability rather than raw storage.
One idea I’ve found interesting is storing a lightweight “replay package” instead of treating every page equally: compressed HTML, normalized response metadata, extraction version, parser version, classifier decision, confidence score, and a content hash. Then replay becomes deterministic while duplicate pages can be deduplicated by hash.
The other benefit is that when a parser improves, you can selectively replay only the affected cohorts instead of reprocessing everything. It turns replay into a targeted migration rather than a bulk operation.
Very good idea @merbayerp 🙌