DEV Community

Kinuthia Matata
Kinuthia Matata

Posted on

Predictable Web Scraping with Web Extensions: A More Localized + Apify Approach

#etl #ai #architecture

The bottleneck was never the ETL

A few years back I built and ran a data pipeline for a previous employer, aggregating public real estate listings and people-search records at scale, from sites that really don't want to be scraped. Once you had the HTML, the rest was ordinary software engineering: normalize the listings, dedupe the records, load the warehouse. Nobody loses sleep over an upsert.

Getting the HTML in the first place was the actual job. Every decision upstream of "parse the response" was really a decision about one thing: not getting blocked. IP reputation, CAPTCHA solve rates, fingerprint plausibility; these weren't abstract security concerns, they were line items that set our unit economics directly.

What we built to solve it was a fleet of Docker containers in two tiers. A cheap tier did plain HTTP fetches for low-risk endpoints. An expensive tier ran full Playwright, real Chrome, Xvfb, and a VNC server, so a human could remote in and solve a CAPTCHA by hand when automation gave up. playwright-extra with the stealth plugin, rotating user agents, persistent cookie jars, human-like typing delays, residential proxies billed by rotation lifetime. All the standard moves.

It still wasn't reliable. Pulled straight from our own logs: about 63% success against one real-estate site, 8% against another. At one point we had a code path that shelled out to xdotool to literally hold the spacebar down through an X11 session, auto-solving a "press and hold" challenge. It worked, for a while. Then the vendor tweaked a signal and our success rate quietly dropped, and we were back to square one.

That's the industry's permanent condition. Apify's own recent look at where scraping is headed describes it as a straight arms race: Cloudflare updates its Turnstile payloads regularly, and a bypass that worked last week can fail today. Nobody wins that fight for good. You just try to stop needing to fight it at all, which is the actual thesis of this post.

None of this is a knock on Playwright or the stealth-plugin ecosystem. They're solving a genuinely hard problem from a structurally difficult position, and I'll get into exactly why in a minute. But it's why I stopped defaulting to headless-browser + proxy-pool for anything that needs to run predictably, every day, for months. This post is about the architecture I use instead: currently pointed at Zillow, and being built into an Apify actor.

Inconsistencies in the standard playbook

Most scraping advice collapses into four moves: make the browser look more human, make the mouse move like a human, make the IP look residential, or pay someone else to do all three. Each one has a specific reason it works less well than advertised.

Stealth plugins fight the wrong layer. Modern behavioral anti-bot vendors ship something they call Code Defender (PerimeterX, now sold under the HUMAN brand, is the one I've spent the most time fighting): a JS sensor that watches the page's own runtime for tampering, injected scripts, modified prototypes, function wrappers around APIs it cares about. In my own testing against it, that includes calling .toString() on sensitive accessors like navigator.webdriver, Canvas, WebGL, and AudioContext, and comparing the result against what native, unmodified browser code should return. Stealth plugins patch those same accessors with Object.defineProperty(), which changes what .toString() reports. So the exact thing meant to hide automation is the exact thing Code Defender is built to catch. Only patches applied below the JS layer, inside the browser engine's own source, survive.

Synthetic mouse movement makes things worse, not better. This is the advice I'd push back on hardest, since it's also the one I hear repeated most confidently. Any event you dispatch programmatically, el.dispatchEvent(new MouseEvent(...)), carries isTrusted: false. Real user input carries isTrusted: true. This is in fact part of the DOM event spec itself, reliably supported across every major browser since 2016. A JS-dispatched "human-like" mouse path doesn't fool anything; it just hands the detector a clean signal it wouldn't otherwise have had. Getting isTrusted: true requires driving input from outside the JS engine entirely, using native OS input drivers, a real technique, and a much heavier one than "add some jitter to your mouse movements" implies.

Residential proxies solve reputation and torch it at the same time. A pooled residential IP isn't a clean slate. It's a shared identity, quietly accumulating reputation damage from every other customer routed through it, scored by cross-customer systems (PerimeterX calls theirs "Collective Intelligence": a single IP's reputation is informed by behavior across every site running PerimeterX, not just the one you're hitting). And it costs real money: $50 to $500 a month for a pool good enough to matter, a rounding error on an enterprise contract and a business-model-breaking line item on a $40/month subscription product.

Commercial scraping APIs are a sanity check, not an answer. A ScrapeOps-run benchmark of seven scraping-API vendors against Zillow, written up by agenthustler, makes the point well. The best performer, ScraperAPI, hit about 98% success at an average of 6.1 seconds per request. Scrapfly came close on success but took nearly three times as long (18.2s). The worst performer, Scrapingdog, managed 61% at 19.4 seconds a request. Even at the top of that market, sustained latency sits somewhere between 6 and 20 seconds a request. The ceiling isn't proxy quality or fingerprint sophistication. It's pacing. The best-funded scrapers in the world aren't going fast. They're going carefully.

That last point is the thread the rest of this architecture pulls on.

The insight: stop imitating a human, be one

Every technique above tries to simulate a legitimate session from outside one: spoofing a fingerprint, faking an event, laundering an IP. There's a simpler move. Don't simulate the session. Use one.

A browser extension's content script executes inside an actual tab, on the page's real origin. When it calls fetch(), the request carries the tab's real cookies, real TLS fingerprint, real canvas/WebGL/AudioContext signatures, and navigator.webdriver === false. None of it is emulated: no CDP automation flag, no patched accessor for Code Defender to catch, no proxy IP to reputation-check.

I'd already validated this pattern on an unrelated project earlier this year: an extension reading live data out of an already-authenticated session on third-party sites, in near real time. (The pattern draws partly from Matt Frisbie's Building Browser Extensions, still the best single reference I've found for content-script mechanics.)

// content script, injected into the target page's own origin
async function fetchInPageContext(apiUrl, headers) {
  const res = await fetch(apiUrl, {
    credentials: 'include',   // attaches the page's real session cookies
    headers,
  });
  return res.json();
}
Enter fullscreen mode Exit fullscreen mode

That's a simplified version of the real pattern; the actual code targets a specific third-party API and does real request-building work this snippet skips. But the trick that matters is exactly this: a content-script fetch() with credentials: 'include'. Because the script runs in the page's own execution context, that one option attaches the real session. No CORS issue, no separate auth flow, because there's no bot: it's the user's own logged-in session, doing the same fetch a real interaction would trigger anyway.

Applying the same pattern to a WAF-protected, logged-out site like Zillow needed one more piece. Zillow doesn't require login to browse, but it runs PerimeterX plus AWS WAF behind CloudFront, actively scoring session behavior. The fingerprint problem disappears with this architecture. The pacing problem does not, and that turned out to be the entire remaining fight.

Architecture: an extension as the collection layer, Apify as the broker

The system I'm building (working name: the zillow-leads-property-data actor) has three moving pieces, plus Apify's own broker in between:

Architecture diagram{width=6.5in}

One thing that diagram compresses: the extension's own database is in-memory only, so nothing it collects survives a reload on its own. Every row gets buffered and flushed to the FastAPI runner over /ingest (batched, every 100 rows or 5 minutes, whichever comes first), and that flush is what makes the runner's cache durable. The "dispatch shortfall / ingest" arrow is a two-way street: work goes down, collected rows come back up.

Extension: drives search and detail collection through a real, cookie-bearing tab, never navigating a page per listing. Zillow server-renders the full property object, resoFacts and all, directly into the detail page's initial HTML, inside __NEXT_DATA__/gdpClientCache. One fetch() per listing, regex out the embedded JSON.

Runner: owns a durable SQLite cache the extension doesn't have. When a buyer's order comes in, the runner first checks whether it can already answer the request from that cache. If everything requested is already there, the order returns in seconds with no live collection at all, a cache hit. If not, the runner works out exactly what's missing and dispatches only that gap to the extension.

Actor: the buyer-facing layer. It writes an order into the shared KV store, polls for result parts as they land, charges per event (listing rate vs. enriched rate), and streams rows into the buyer's dataset while the order is still being fulfilled, not as one blocking dump at the end.

Here's the full order lifecycle, both paths at once: the fast cache-hit path down the left, and the cache-miss path that dispatches the extension, on the right. A single order can use both, some rows served instantly from cache while the rest are still being collected live:

Order fulfillment sequence diagram{width=6.5in}

The part I like most, from an infra-cost standpoint: at a small scale, there's no need to run any scraping infrastructure in the cloud at all. The actor and the KV broker are the only things Apify hosts. The thing doing the actual fetching is a browser extension on an ordinary machine with a residential connection, also the cheapest possible source of a trustworthy IP and a genuine fingerprint. No proxy pool. No headless-browser farm. No fingerprint-spoofing arms race to keep up with.

Behavioral pacing: the fight that's actually left

Killing the fingerprint problem doesn't kill the anti-bot problem. It just isolates it down to one layer: behavioral biometrics, PerimeterX's actual primary signal. It's not looking at what your browser is. It's watching how you use it: request timing, session coherence, whether your behavior looks like a person or a script.

Here's a mistake I made early on and want to be upfront about. I assumed Zillow's stack was Imperva, because that's what a couple of external write-ups claimed, and it was easy to take on faith without checking. Live inspection of the actual response headers and cookies told a different story: AWSALB and aws-waf-token (AWS WAF), _px3, _pxvid, pxcts (PerimeterX). No Imperva signature anywhere. That mistake mattered: an evasion strategy built around Imperva's token lifecycle would have targeted the wrong vendor entirely. Check the actual stack against live traffic before designing a strategy around it. Vendor identity isn't guessable from outside, and getting it wrong wastes time.

Once I had the target right, I ran a small battery of live, controlled tests against it myself rather than trusting secondhand claims about cookie lifetimes and rate thresholds; solid documentation on the exact numbers was hard to find, so I measured it myself. Two findings contradicted the numbers that do circulate online:

  • _px3 rotation is frequently cited online as expiring around every 60 seconds. In my own testing, it rotated roughly every 120 seconds, and didn't rotate at all across six fetches spaced 10 seconds apart. It self-refreshes through PerimeterX's own background sensor polling, so there's nothing to manage proactively.
  • pxcts isn't well documented publicly beyond being one of PerimeterX's session cookies. I'd seen it treated informally as meaning "a challenge is currently active." In practice it's a residual cookie present even on a healthy, unchallenged session; only a change in its value looked meaningful in my testing, not its presence.

What I landed on for production: gaussian-jittered delays (Box-Muller, not a fixed interval, since mechanically regular timing is itself a signal), a slower warmup at session start, one detail fetch at a time, periodic "organic" navigation to refresh telemetry, a 30-second cooldown between newly seeded jobs, and adaptive backoff that doubles every delay on a detected challenge and hard-pauses 45 minutes after three challenges in 30 minutes.

function jitteredDelay(meanMs, stddevMs, min, max) {
  // Box-Muller transform: normal distribution, not uniform.
  // Uniform jitter is itself a detectable pattern.
  const u1 = Math.random(), u2 = Math.random();
  const z = Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2);
  return Math.min(max, Math.max(min, meanMs + z * stddevMs));
}
Enter fullscreen mode Exit fullscreen mode

A 45-request, 5-minute live run at this pacing (mean interval about 6.7 seconds) produced zero challenges. Extrapolated, that's a sustained throughput of roughly 13,000 listings a day per browser profile. Not fast in the headless-fleet sense, but fast enough and reliable, the property that actually matters running this every day rather than once. Stack that against the commercial-API ceiling from earlier (best case about 6.1 seconds a request), and it's clear this isn't unusually conservative. It's roughly what the site actually tolerates, no matter who's asking.

Where an LLM actually helped (and where it wouldn't have)

"We used AI to help build this" doesn't tell you anything, so let me be specific.

The useful pattern was pairing with an LLM (Claude Code, here) as a debugger and pattern-recognizer across a codebase and its own history, always with a human directing what to check and verifying the result against live ground truth. Not an autonomous agent turned loose on "go get the data."

Part of what made that workable was tooling design, not prompting. I exposed the pipeline's internals to the model through narrow, composable tools over MCP: query the database directly, run the real detail-page extraction against one URL, dump the actual schema, get a live snapshot of a subsystem's internal state. None of these are "go read the website and figure it out" tools; each returns something verifiable, a real HTTP response, a real database row, a real process's actual state, not prose the model has to interpret and hope is accurate. The pattern generalizes: the more you can hand a model a narrow tool pointed at ground truth instead of a page it has to read and guess about, the more useful it gets as a debugger. Less "let it browse," more "let it query."

A couple of examples of what that actually caught. The runner's terminal showed a steady stream of successful /health checks, but the endpoints that actually mattered (/ingest, /orders/next, /cache/since) never showed up, even though the server-side route logic was independently correct. The logs looked healthy and were actively misleading. Instead of staring harder at server logs, I had Claude Code build a small tool, runner_bridge_debug, that runs a live probe from inside the extension's own JS context and reports what the extension itself actually sees. That surfaced the real cause: the browser was silently blocking the extension from reading its own successful responses, because of a missing CORS header, so its health check treated every call as a failure and gave up before trying the other three endpoints. No amount of reading server logs was going to show that; the tool had to run from the one vantage point that could see it.

Or the field-mapping document I'd been trusting, which claimed to enumerate every field in Zillow's response and looked authoritative. Rather than take its word for it, I had the model query the database directly with run_sql and separately re-fetch a live listing with fetch_zillow_detail, then diff both against the doc, field by field. It was wrong in specific, boring ways: missing real fields, one field name that doesn't exist anywhere in Zillow's schema, and several fields documented as strings that are actually arrays in live responses (which crashes a db write if you trust the doc over the data). None of that was guessable from reading the document alone.

What these share: each required grounding against live, specific, current data from one particular site, a captured HTTP response, a running process's actual behavior, a value diffed row by row. An LLM's training data is a snapshot with a cutoff date. It can't verify a claim against your running system unless you give it a way to check, and a tool to check with.

Worth addressing directly: Claude Code can drive a real, running browser too, and separately, this project exposes its own similar debug tooling over MCP (navigating a tab, running script in it, checking what URL it's on). I use both for debugging: confirming what a live page is actually doing right now, not for the collection pipeline itself. The upside is real, it's the only way to see genuinely current page state, which a training-data snapshot can't give you. The downside is the one this whole post has been about: driving a browser live and interactively is slow, and it's exactly the irregular, improvised pattern behavioral detection is built to flag. A good debugging tool, pointed by a human. A bad substitute for the paced, narrow-purpose fetch loop that actually does the collecting.

There's a split worth naming here, between directing a tool and being conscripted to serve one. Cory Doctorow's framing for it, from The Reverse Centaur's Guide to Life After AI, has stuck with me: a centaur chooses when and how to use a tool; a reverse centaur is conscripted into serving the tool's pace, on the tool's schedule. Debugging with an LLM in the loop, where you direct the investigation and verify every claim yourself, is centaur work. Handing an agent an open-ended "go extract this site's data" mandate and letting it improvise in real time is closer to reverse-centaur: you're paying, in tokens, in latency, in reliability, for the model to rediscover facts it has no privileged way of knowing, against a target actively trying to make that expensive.

The cost isn't abstract. One documented case of scraping a single Wikipedia article through a markdown-conversion API produced about 93,000 tokens of input for roughly 3,700 tokens of actual content (Dubey, 2026), a roughly 25x multiplier, almost entirely navigation chrome the model never needed. That's the tax an unstructured "let the model read the raw page" approach pays by default, and a decent argument for extracting a clean payload first and saving the LLM for reasoning about why something's broken. It's also a losing bet against targets now building defenses aimed specifically at AI consumption: the recent open-source "poisoned font" project swaps out roughly a quarter of a page's words at the font-rendering layer, so a scraper sees garbled nonsense while a human sees the real page. A preview of where this fight is headed.

Try it yourself: the pattern in miniature

You don't need the whole runner-actor-broker system to prove the core idea to yourself. The minimum viable version is one content script and one background script.

// manifest.json (MV3)
{
  "manifest_version": 3,
  "permissions": ["storage"],
  "host_permissions": ["https://www.example-target.com/*"],
  "content_scripts": [{
    "matches": ["https://www.example-target.com/*"],
    "js": ["content.js"],
    "run_at": "document_idle"
  }],
  "background": { "service_worker": "background.js" }
}
Enter fullscreen mode Exit fullscreen mode 
// content.js: runs in the page's own origin, inherits its session
browser.runtime.onMessage.addListener(async (msg) => {
  if (msg.type !== "FETCH_DETAIL") return;
  const res = await fetch(msg.url);         // same-origin, no CORS problem
  const html = await res.text();
  // Most server-rendered frameworks embed a state blob somewhere in
  // the initial HTML: Next.js uses __NEXT_DATA__, Nuxt uses __NUXT__,
  // plenty of custom stacks do their own thing. Find your target's
  // equivalent with devtools first, then swap in the real pattern here.
  const match = html.match(/<script id="__APP_STATE__"[^>]*>([\s\S]*?)<\/script>/);
  return match ? JSON.parse(match[1]) : null;
});
Enter fullscreen mode Exit fullscreen mode 
// background.js: owns pacing, never fetches directly
let lastFetch = 0;
async function fetchDetail(tabId, url) {
  const wait = jitteredDelay(4500, 1300, 2500, 9000) - (Date.now() - lastFetch);
  if (wait > 0) await new Promise(r => setTimeout(r, wait));
  lastFetch = Date.now();
  return browser.tabs.sendMessage(tabId, { type: "FETCH_DETAIL", url });
}
Enter fullscreen mode Exit fullscreen mode

That's the whole trick. The content script does the fetching, so it inherits the tab's real session and fingerprint. The background script owns pacing, so nothing fires faster than a real user plausibly would. Parsing targets whatever the server already embedded in the page, rather than replaying an API call you don't control and can't guarantee still exists next month. From here, the interesting engineering is almost entirely pacing discipline and getting your data model right, not fingerprint evasion, because there's no fingerprint left to evade.

What this doesn't solve

  • Throughput is bounded by session count, not compute. About 13,000 listings a day per browser profile is the ceiling for one real session running safely. Scaling means more real sessions, not more containers, a fundamentally different curve than a headless fleet, and one that doesn't get cheaper by renting more compute.
  • It needs a real, persistently running browser somewhere. This architecture is "localized" by necessity: something has to keep a real tab open with a real session, a genuine operational cost a pure-cloud actor doesn't have, traded against not needing a proxy budget at all.
  • It's specific to whatever the target server actually renders. This works cleanly against Zillow because Zillow server-renders its full payload into the initial HTML. A target that only populates data through authenticated, signed, or ephemeral client-side calls needs a different, though related, strategy.
  • This isn't a claim that scraping public data is risk-free. It's a claim about doing it predictably, once you've decided to. For what it's worth, I think Doctorow's position in his book, that well-behaved scraping of public information is fine and badly-behaved high-volume abuse is the actual problem, is basically right, and a useful frame for thinking about how to scrape, not just whether.

Closing

The point here isn't "web extensions beat headless browsers at everything." It's narrower, and I think more useful: if what you actually want is sustained, predictable extraction from a behaviorally-defended site, not a one-off scrape but a product you run every day, the fingerprint-evasion arms race is a fight you can mostly opt out of. Use a real browser instead of simulating one, and pacing discipline becomes the one problem worth solving carefully. Layer that onto Apify's Key-Value store as a broker and its actor runtime as the buyer-facing, billed surface, and you get a marketplace-ready product without ever running scraping infrastructure in the cloud. Real collection running locally, brokering and billing running in the cloud: that's the "more localized plus Apify" architecture in the title.

References

Top comments (0)