DEV Community: John Rooney

40% of pages are empty over plain HTTP

John Rooney — Tue, 08 Sep 2026 15:25:00 +0000

40.6% of popular landing pages need JavaScript to show you anything useful. That is from State of Web Access, an audit Zyte (where I work) ran on 11,100 landing pages. The test was blunt. Fetch the page over plain HTTP, fetch it again in a headless browser, and if rendering grew the meaningful HTML by more than half, count it as JavaScript-dependent.

On the pages that failed the test, rendering added 140,923 bytes on average. Three quarters of the final HTML did not exist until a browser ran the scripts. Restaurants and airlines top the list at 66%, travel at 65%.

The report calls this architecture rather than defence, and I agree. Single page apps won. The HTML that comes over the wire is a shell and the content arrives in a second round trip.

What rendering costs you

The instinct on seeing an empty response is to reach for Playwright. Here is what that does to the bill, using the audit's own tier data. Zyte's tiers map to the infrastructure needed to fetch a page reliably, and the report scored every site both ways:

Access method	Simple	Easy	Moderate	Complex	Advanced	Mean tier
Plain HTTP	85.2%	9.4%	2.5%	2.0%	0.9%	1.24
Headless browser	58.4%	31.3%	8.0%	1.8%	0.5%	1.55

The gap is not sites defending themselves. A browser carries a heavier fingerprint and a heavier compute cost even on a site with no barriers at all. Switching to rendering moves a quarter of all sites up a tier for no reason other than the method.

So before rendering, I look in three places. The data is very often already in the response.

First, measure

This is the audit's test, rewritten to compare visible text instead of HTML bytes, which is stricter. Run it once per website before you decide anything.

import asyncio
import sys

import httpx
from playwright.async_api import async_playwright
from selectolax.parser import HTMLParser

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"


def visible_text_len(html: str) -> int:
    tree = HTMLParser(html)
    for node in tree.css("script, style, noscript, template"):
        node.decompose()
    return len(tree.body.text(separator=" ", strip=True)) if tree.body else 0


async def js_dependence(url: str) -> dict:
    plain = httpx.get(url, headers={"User-Agent": UA}, follow_redirects=True, timeout=20).text
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page(user_agent=UA)
        await page.goto(url, wait_until="networkidle")
        rendered = await page.content()
        await browser.close()
    a, b = visible_text_len(plain), visible_text_len(rendered)
    return {"plain_chars": a, "rendered_chars": b, "js_dependent": b > a * 1.5}


if __name__ == "__main__":
    print(asyncio.run(js_dependence(sys.argv[1])))

If js_dependent is false, stop here. You have a plain HTTP site and a browser will only make it slower and more expensive.

Second, harvest the state the framework left behind

SPAs hydrate from data the server already embedded. Next.js puts it in a script tag with the id __NEXT_DATA__. Nuxt uses a global. Apollo, Redux and a dozen others assign to window.__SOMETHING__. E-commerce and news sites also carry JSON-LD for search engines, which is often cleaner than the visible page.

A Scrapy spider that pulls all of it from the plain response, with no browser involved:

import json
import re

import scrapy

STATE_SELECTORS = {
    "next": "script#__NEXT_DATA__::text",
    "jsonld": 'script[type="application/ld+json"]::text',
}

STATE_GLOBALS = [
    r"window\.__INITIAL_STATE__\s*=\s*",
    r"window\.__PRELOADED_STATE__\s*=\s*",
    r"window\.__APOLLO_STATE__\s*=\s*",
]


class EmbeddedStateSpider(scrapy.Spider):
    name = "embedded_state"

    def parse(self, response):
        for source, sel in STATE_SELECTORS.items():
            for raw in response.css(sel).getall():
                try:
                    yield {"source": source, "url": response.url, "data": json.loads(raw)}
                except json.JSONDecodeError:
                    self.logger.debug("unparseable %s block on %s", source, response.url)

        for script in response.css("script::text").getall():
            for pattern in STATE_GLOBALS:
                m = re.search(pattern, script)
                if m:
                    obj = self._leading_json(script[m.end():])
                    if obj is not None:
                        yield {"source": "global", "url": response.url, "data": obj}

    @staticmethod
    def _leading_json(s: str):
        # Parses the JSON object at the start of s and ignores whatever follows
        # (a semicolon, more script). Avoids trying to find the closing brace by hand.
        try:
            obj, _ = json.JSONDecoder().raw_decode(s)
            return obj
        except json.JSONDecodeError:
            return None

The raw_decode trick is the useful part. You never have to find the end of the object yourself. Nuxt is the awkward one, since its global is usually a function call rather than a literal, and you either evaluate it or fall through to the next step.

Third, replay the request the page makes

Open DevTools, Network tab, filter to Fetch/XHR, reload. The JSON endpoint the page calls is right there, and it usually returns cleaner data than the HTML ever will. Right click, copy as cURL, and check what it needs. If it works with a static header set, you have a plain HTTP scraper for a JavaScript site. If it needs a token that a script computes on the client, that is your signal to render, and now you know why.

When rendering is the right call

Three cases. The API is signed or needs a challenge token from a script. The content genuinely does not exist until a user interacts. Or the TLS probe and the 403 classifier from the earlier posts in this series put the site at Moderate or above, at which point you need a browser-shaped session anyway and rendering comes with it.

Everything else is a plain HTTP job wearing a React costume.

The full JavaScript breakdown by industry and site size is on the report page, and the tier tables are under scraping cost. The tier model is Zyte's own pricing model applied to Zyte's own data, so trust the ordering between industries more than the absolute percentages.

Disclosure: I work at Zyte. The audit is ours. The code and the opinions are mine.

One TLS handshake predicts the whole anti-bot stack

John Rooney — Mon, 07 Sep 2026 15:19:39 +0000

When I size up a new website I used to start with headers. Copy them out of DevTools, match the order, add the cookies, see what happens. Now I start with one TLS handshake, because it tells me more about the site than an afternoon of header work.

The reason is a set of numbers from State of Web Access, the audit Zyte (my employer) ran on 11,100 popular landing pages. 13.8% of them screen the TLS handshake. Look at what else those sites run compared to everyone else:

Barrier	TLS-fingerprinting sites	All other sites
Rate limiting	78.1%	13.2%
Antibot	45.0%	14.2%
CAPTCHA	37.3%	20.2%
JavaScript required	27.0%	42.8%

A TLS check is the strongest single predictor in the whole dataset. If the handshake is being inspected, you are almost certainly going to hit rate limiting too, and you have a coin-flip chance of a dedicated bot manager on top. If it is not being inspected, the site probably stops at a CDN firewall. One request and you know which world you are in.

The JavaScript row runs the other way, and I find it the most interesting. TLS-checking sites need a browser less often. They are server-rendered, performance-minded operations that enforce access at the connection, not in the page.

Why nobody notices this layer

TLS fingerprinting fails before HTTP starts. Your client sends a ClientHello announcing which TLS versions, cipher suites and extensions it supports, in a particular order. Python on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce different ones. A site can see "Chrome user agent, OpenSSL handshake" and know the client is lying before it reads a single header.

JA3 hashed those fields into a fingerprint in 2017. Chrome started shuffling its extension order in 2023, which broke JA3, and JA4 replaced it by sorting the fields first. The detail does not matter much for scraping. What matters is that the check is silent. Some sites return a bare 403 with an empty body. Some just close the connection, and your logs say "connection reset by peer", which looks like a network blip. Nothing in the failure says TLS.

Vendor attribution was only possible for 45% of the TLS-checking sites in the audit. The other 55% show the behaviour with no named signature. Where it could be attributed, a single bot management vendor was 88.8% of it. So this is mostly one vendor's check, bundled into a product a lot of sites bought for other reasons.

The probe

This is the method the report used, cut down. Two requests, identical headers, from the same IP. The only thing that differs is the handshake. aiohttp presents a standard Python TLS signature. curl_cffi presents Chrome's. If the first fails and the second passes, the site is looking at TLS.

import asyncio
import sys

import aiohttp
from curl_cffi.requests import AsyncSession

HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}


async def python_tls(url: str) -> tuple[int, int]:
    try:
        async with aiohttp.ClientSession(headers=HEADERS) as s:
            async with s.get(url, timeout=aiohttp.ClientTimeout(total=20)) as r:
                return r.status, len(await r.read())
    except aiohttp.ClientError:
        return 0, 0          # reset or refused: count it as a block


async def chrome_tls(url: str) -> tuple[int, int]:
    async with AsyncSession(impersonate="chrome", headers=HEADERS) as s:
        r = await s.get(url, timeout=20)
        return r.status_code, len(r.content)


async def probe(url: str) -> dict:
    py, ch = await asyncio.gather(python_tls(url), chrome_tls(url))
    filtered = py[0] != ch[0] and ch[0] < 400
    return {"python_tls": py, "chrome_tls": ch, "tls_filtered": filtered}


if __name__ == "__main__":
    print(asyncio.run(probe(sys.argv[1])))

Install with pip install aiohttp curl_cffi. The impersonate part follows whatever Chrome version curl_cffi ships, so it drifts less than a hand-maintained profile would.

Because both requests leave the same machine, IP reputation is held constant. This is a controlled experiment with one variable, which is more than most scraping diagnostics manage.

How I read the result

If tls_filtered is true, I plan for the stack the table above predicts. Browser-shaped TLS from request one, session handling, per-domain concurrency of one or two, content validation on every response. I do not spend a day on header tweaks because headers were never the problem.

If it is false and I am still blocked, I have ruled out one layer for the cost of one request. The block is IP class, headers, cookies or behaviour, and I look there instead.

If it is false and both requests pass, I have a Simple-tier site and I should stop reaching for Playwright. The audit's tier data puts 85% of landing pages at plain HTTP. Rendering when you do not need to is the most common way I see scraping budgets disappear.

One caveat the report makes and I will repeat. Everything here is landing pages, scanned once, from datacentre IPs. Search and product pages are usually defended harder than the front door. Treat the percentages as a floor.

The industry split is on the TLS page. Jewellery and luxury leads at 34%. Reference sites sit at 3%, because blocking crawlers would kill the thing they exist for.

Disclosure: I work at Zyte. The dataset is ours. The probe is a cut-down version of what the audit ran, and the opinions are mine.

When one vendor changes its mind, every scraper breaks the same morning

John Rooney — Mon, 07 Sep 2026 15:18:55 +0000

Two incidents look the same on most scraping dashboards.

In the first, one spider starts failing because the site changed a CSS class. In the second, forty spiders start failing in the same hour because a bot management vendor pushed a new model. Both show up as a drop in items and a rise in "blocked". The fix for the first is a code change. The fix for the second is to change identity and wait, and touching the spider code will make it worse.

The second kind is more common than it should be, and State of Web Access, the audit Zyte (where I work) ran on 11,100 landing pages, has the numbers on why.

One vendor decides what a browser is

18.5% of landing pages run a dedicated bot manager, the kind that looks at behaviour, canvas rendering and attribute consistency rather than just IPs. One vendor is 75.9% of those deployments. The next four combined are under a quarter.

The TLS layer is more concentrated still. Of the TLS-fingerprinting sites the audit could attribute, the same vendor is 88.8%.

And at the firewall layer, 61.9% of sites run a single WAF vendor in its default mode. 1.4% run more than one. Roughly three in ten sites run a WAF and nothing else, and that WAF arrived bundled with the CDN. Nobody chose it.

Put those together and for most of the defended web, one company's model of what a real browser looks like is the definition of a real browser. When that model changes, the change lands everywhere at once. Your spiders are not failing independently. They are failing together, for one reason, and the reason is not in your repository.

Why a "blocked" counter cannot see this

Most Scrapy projects I have looked at count blocks in one bucket. Maybe two, split by status code. That hides the two things you need to know when the pager goes off: which layer refused you, and whether it is one domain or all of them.

The audit found the same four layers over and over. TLS, which shows up as a connection reset or an empty 403. WAF, which shows up as a 403 with a block page. Challenge, which shows up as a 200 or 403 carrying a JavaScript challenge. And rendering, where the response is fine but the data is not in it. Each has a different fix. A single counter makes them all look like the same fire.

Middleware that counts by layer and domain

This is a Scrapy downloader middleware. It classifies each response by the wording of the page and increments a stat per layer and per domain. Connection-level failures get their own bucket, because that is where TLS filtering hides.

import re
from urllib.parse import urlparse

from scrapy import Request
from scrapy.http import Response
from twisted.internet.error import ConnectionLost, ConnectionRefusedError, TimeoutError

SIGNATURES = {
    "challenge/js":      [r"checking your browser", r"enable javascript and cookies",
                          r"just a moment", r"verify you are human"],
    "challenge/captcha": [r"captcha", r"are you a robot", r"unusual traffic"],
    "waf/block-page":    [r"access denied", r"request blocked", r"you have been blocked",
                          r"reference\s*(#|number|id)"],
}

CONNECTION_ERRORS = (ConnectionLost, ConnectionRefusedError, TimeoutError, OSError)


def classify(response: Response) -> str | None:
    body = response.text[:50_000] if hasattr(response, "text") else ""
    for label, patterns in SIGNATURES.items():
        if any(re.search(p, body, re.I) for p in patterns):
            return label
    if response.status == 429:
        return "ratelimit/429"
    if response.status == 403:
        return "waf/unattributed"
    if response.status == 200 and len(body) < 2_000:
        return "render/empty"
    return None


class BlockLayerStatsMiddleware:
    def __init__(self, stats):
        self.stats = stats

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.stats)

    def _record(self, layer: str, url: str) -> None:
        domain = urlparse(url).netloc
        self.stats.inc_value(f"blocked/{layer}")
        self.stats.inc_value(f"blocked/{layer}/{domain}")

    def process_response(self, request: Request, response: Response, spider):
        layer = classify(response)
        if layer:
            self._record(layer, response.url)
        return response

    def process_exception(self, request: Request, exception, spider):
        if isinstance(exception, CONNECTION_ERRORS):
            # TLS filtering often looks like a reset, not a status code.
            self._record("tls-or-network/reset", request.url)
        return None

Enable it in settings:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.BlockLayerStatsMiddleware": 543,
}

The stats land in the crawl's stats collector, so they end up wherever you already ship Scrapy stats. If you use Spidermon, that is where the rules below go.

Two rules that tell the incidents apart

The first rule is per spider. If blocked/* for one domain rises past a threshold and other domains are quiet, it is that site. Somebody changed a selector or added a rule. Open the spider.

The second rule is fleet-wide. If the same layer, say challenge/js, rises across three or more domains inside ten minutes, it is not your code. It is a vendor change. Do not deploy anything. Rotate identity, drop concurrency, and give it a few hours before anyone edits a spider. The person who owns the fetch layer gets paged, not the person who owns the parser.

The layer name in the stat key is what makes the second rule possible. blocked/403 across forty domains could be forty separate problems. blocked/challenge/js across forty domains is one problem.

The unattributed bucket is not a bug

You will see a lot of waf/unattributed. The audit could only attribute 45% of TLS-fingerprinting sites to any vendor, and plenty of block pages carry no recognisable wording at all. Keep the bucket. A spike in unattributed 403s across many domains is still a correlated failure, and the replay test from the first post in this series, one request from a different IP class, tells you whether it is reputation or something else.

The vendor shares by industry are on the antibot page and the WAF breakdown is on the WAF page. Adult content, furniture and gambling lead bot management adoption at 30% and above. Those are sectors with checkout flows and inventory, and the bot manager is guarding revenue, not text.

Disclosure: I work at Zyte. The audit is ours. The middleware and the opinions are mine.

Rendering Javascript pages without giving up Scrapy

John Rooney — Sun, 06 Sep 2026 08:00:00 +0000

A page can contain more JavaScript than HTML and still not need a browser. Often the data is already in the response, in a script tag, or behind a JSON endpoint the application calls itself. Reaching for browser automation before checking those places is an expensive habit.

The modern approach is to treat JavaScript rendering as an acquisition choice. Keep Scrapy for scheduling, retries, item processing, pipelines, logging, and monitoring. Use an ordinary Scrapy request when it is enough, call an authorized data endpoint when that is the real source, and render in a browser only when you genuinely need the browser's DOM or state.

A JavaScript application is not the same thing as a browser-only data source.

That distinction is the difference between a crawler that scales predictably and one that launches a browser for every pagination link.

Find the data source before choosing a renderer

Start by looking at the response Scrapy receives. The Scrapy guide to dynamically loaded content suggests saving a fetched response locally, then using browser developer tools to inspect the requests that actually supply the data.

scrapy fetch --nolog https://example.com/products > response.html

The following decision table is deliberately boring. That is a strength. It prevents a rendering problem from becoming the default architecture for the whole project.

What you find	First choice	Why
Data in the HTML response	Normal Scrapy request and selectors	Lowest operational cost and simplest failure mode
JSON-LD or serialized state in a script element	Normal Scrapy request and JSON parsing	The browser has nothing useful to add
A documented or otherwise authorized JSON endpoint	Scrapy request to that endpoint	You receive the data in its native form
Content appears only after client-side code runs	`scrapy-playwright` for those requests	A rendered DOM is the actual input
A public page needs interaction or browser-bound state	A targeted browser flow and explicit context management	State is part of acquisition, not an incidental header

The second and third rows are easy to miss when looking at a polished application UI. They are often the better solution.

Call the API with Scrapy when the API is the source of truth

Suppose the product page fetches a JSON document from an endpoint your project is permitted to use. Make that request with Scrapy, not with an ad hoc requests loop inside a callback. Scrapy can schedule it, retry it, log it, and feed the resulting items into the same pipeline and Spidermon checks from the earlier parts of this series.

JsonRequest serializes a JSON request body and sets the appropriate content type. The Scrapy request and response documentation also documents response.json(), which decodes the response body for you.

# catalog/spiders/products_api.py
import scrapy
from scrapy.http import JsonRequest

from catalog.items import Product


class ProductsApiSpider(scrapy.Spider):
    name = "products_api"
    api_url = "https://api.example.com/v1/products/search"

    async def start(self):
        yield JsonRequest(
            self.api_url,
            data={"category": "widgets", "page": 1},
            callback=self.parse_products,
        )

    def parse_products(self, response):
        payload = response.json()

        for product in payload["products"]:
            yield Product(
                name=product["name"],
                price=product["price"],
                description=product.get("description", ""),
                url=response.urljoin(product["url"]),
            )

Real APIs have their own pagination, rate limits, and terms. Treat those as part of the spider's contract. Do not treat a network-panel request as permission to use an endpoint.

This route also fits neatly with Part 2. The Product item stays the stable output contract even when the input changes from an HTML Page Object to an API response. That makes it possible to replace an acquisition strategy without rewriting exports, validation, or downstream consumers.

Add Playwright only for the requests that need a browser

When the page needs client-side rendering, scrapy-playwright is the useful integration point. It is a Scrapy download handler backed by Playwright, so rendered requests still go through Scrapy's regular scheduling and item-processing workflow.

Install the Python package and the browser binaries:

pip install scrapy-playwright
playwright install chromium

Then configure the handler and asyncio-compatible Twisted reactor. Recent Scrapy projects already use this reactor by default, but keeping it explicit removes an easy source of confusion when updating an older project.

# catalog/settings.py
DOWNLOAD_HANDLERS = {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

That setting does not mean every HTTPS request launches Chromium. A request uses Playwright only when it carries meta={"playwright": True}. Keep listing pages, robots-aware discovery, and API calls on Scrapy's normal downloader unless rendering is specifically necessary.

Make browser rendering the project default only when it really is the default

scrapy-playwright does not provide a setting that turns on Playwright for every request. Its download handler deliberately falls back to Scrapy's normal HTTP handler unless a request has the playwright meta flag.

If a narrowly scoped project truly needs browser rendering for every request, set that flag in a downloader middleware. Downloader middlewares run before Scrapy selects the download handler.

# catalog/middlewares.py
class EnablePlaywrightMiddleware:
    def process_request(self, request, spider):
        request.meta.setdefault("playwright", True)

# catalog/settings.py
DOWNLOADER_MIDDLEWARES = {
    "catalog.middlewares.EnablePlaywrightMiddleware": 543,
}

This preserves a request that has explicitly set playwright to False, which makes an opt-out possible for a public API or static asset endpoint. In most mixed crawls, the opposite policy is easier to operate: leave the default as False and mark only the rendered paths. The scrapy-playwright activation documentation explains why registering its handler alone does not enable rendering for every request.

Wait for a meaningful element, not an arbitrary number of seconds

The common first browser spider waits five seconds and hopes. The reliable version waits for an observable condition that says the field you need has appeared.

PageMethod lets the download handler perform an action before it returns the final rendered response. The callback then receives a normal Scrapy response whose selectors see the browser-rendered DOM.

# catalog/spiders/rendered_products.py
import scrapy
from scrapy_playwright.page import PageMethod

from catalog.items import Product


class RenderedProductsSpider(scrapy.Spider):
    name = "rendered_products"
    start_urls = ["https://example.com/widgets"]

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(
                href,
                callback=self.parse_product,
                meta={
                    "playwright": True,
                    "playwright_page_methods": [
                        PageMethod("wait_for_selector", ".product-price"),
                    ],
                },
            )

    def parse_product(self, response):
        yield Product(
            name=response.css("h1::text").get(default="").strip(),
            price=response.css(".product-price::text").get(default="").strip(),
            description=" ".join(
                response.css(".description *::text").getall()
            ).strip(),
            url=response.url,
        )

The example keeps the extraction short to show the boundary. In the project from Part 2, move those selectors into the appropriate scrapy-poet Page Object once the rendering path is understood. Browser acquisition should not become an excuse to put all extraction back in callbacks.

Avoid time.sleep() and fixed browser delays. They are slow when the page is fast and unreliable when the page is slow. A selector, response condition, or explicit page action describes what the spider needs rather than guessing how long the target will take.

Treat browser pages and contexts as limited resources

A browser context holds state such as cookies and local storage. A page consumes memory and a concurrency slot. Neither should be created casually.

Use a named context when public browsing needs a coherent state, such as a chosen locale, and cap the amount of browser work your crawler can do at once:

# catalog/settings.py
PLAYWRIGHT_CONTEXTS = {
    "catalog": {
        "locale": "en-GB",
    },
}

PLAYWRIGHT_MAX_CONTEXTS = 4
PLAYWRIGHT_MAX_PAGES_PER_CONTEXT = 2
PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT = 20 * 1000

# catalog/spiders/localized_products.py
import scrapy


class LocalizedProductsSpider(scrapy.Spider):
    name = "localized_products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/widgets",
            callback=self.parse_product,
            meta={
                "playwright": True,
                "playwright_context": "catalog",
            },
        )

    def parse_product(self, response):
        return {"url": response.url}

Only set playwright_include_page when the callback must directly use the Playwright Page, for example for a multi-step interaction that cannot be expressed as PageMethod calls. Otherwise let scrapy-playwright close it after producing the response. Leaving pages open is a quiet way to stall a crawl.

Connect to a managed browser when local Chromium is not the right runtime

Local launch is the simplest starting point. It is not the only option. scrapy-playwright can connect to a browser that another system owns, which is useful when browser lifecycle, isolation, or capacity is managed outside the spider process.

Connection	Setting	Use it when	Important constraint
Chrome DevTools Protocol	`PLAYWRIGHT_CDP_URL`	You have a remote Chromium browser exposing CDP	Chromium only; local launch options are ignored
Playwright WebSocket connection	`PLAYWRIGHT_CONNECT_URL`	You connect to a Playwright server endpoint	Client and server Playwright versions must be compatible

Here is a CDP connection to a remote Chromium instance:

# catalog/settings.py
PLAYWRIGHT_CDP_URL = "http://browser-worker.internal:9222"
PLAYWRIGHT_CDP_KWARGS = {
    "timeout": 20 * 1000,
}

Do not configure PLAYWRIGHT_CDP_URL and PLAYWRIGHT_CONNECT_URL together. They are alternative connection models, and both bypass PLAYWRIGHT_LAUNCH_OPTIONS. The scrapy-playwright connection settings explain the lifecycle and compatibility implications in detail.

The operational question is more important than the setting name: who restarts a disconnected browser, who limits capacity, and how are browser logs correlated with the Scrapy job? Answer those before moving a spider to remote browsers.

Use browser-provider integrations for more options

Current scrapy-playwright releases expose PLAYWRIGHT_BROWSER_PROVIDER, an extension point for a provider that owns browser startup, connection, and teardown. The default provider supports local launch, CDP, WebSocket connections, and persistent contexts. The project also documents examples for Playwright-compatible alternatives such as Patchright and Camoufox.

These are sometimes described as stealth-browser options. That phrase promises more than the setting does. They are browser-provider integrations, not a guarantee that a target will accept a crawl or that its access rules can be ignored.

Use a provider only when you can name the concrete browser behavior or runtime constraint it addresses, and test it as a separate acquisition implementation. Keep the same Items, Page Objects, logs, and monitors around it. Respect the site's terms, permissions, and applicable requirements regardless of which browser implementation is underneath.

Monitor rendered spiders as a separate operating mode

Part 3 used Spidermon to monitor output quality. Browser-backed spiders need those checks, plus a few browser-specific signals:

navigation timeouts and Playwright request failures;
an unexpected rise in error pages or interstitial pages;
a drop in rendered item count compared with the expected scope; and
pages or contexts left open long enough to reduce crawl throughput.

A rendered response can be syntactically valid HTML and still be the wrong page. Validate the resulting item, not just the HTTP status.

Start with an item-count monitor and required-field validation. Then add a periodic monitor when a long-running browser crawl can waste meaningful capacity after a failure. The Spidermon periodic-monitoring documentation covers that pattern.

Browser automation also changes the economics of a spider. It is worth keeping a normal Scrapy path for the pages that do not need rendering, then using monitoring to prove that the smaller browser surface still produces the expected data.

Recent work on using Web Scraping Copilot in VS Code can help with the initial mechanics of a Scrapy project. It does not replace the technical judgment in the table above. The same caution appears in a review of AI-generated Scrapy projects: a spider that runs is not necessarily one with a sound acquisition strategy. For teams that need a managed alternative to operating browser infrastructure, Zyte API's headless browser capability is another option to evaluate against your requirements.

Questions developers ask about Scrapy and JavaScript rendering

Do I need Playwright because a site uses JavaScript?

No. Inspect the Scrapy response and network requests first. If the data is in HTML, serialized state, or an authorized JSON response, use Scrapy without a browser. Use Playwright when the rendered DOM, client-side interaction, or browser state is genuinely required.

Can one spider mix ordinary and rendered requests?

Yes. That is usually the point. Register the download handler once, then set meta={"playwright": True} only on the individual requests that need it. The rest continue through Scrapy's standard downloader.

When should I use CDP instead of `PLAYWRIGHT_CONNECT_URL`?

Use CDP when the remote browser is Chromium and exposes a CDP endpoint. Use the Playwright connection URL for a Playwright server endpoint. Pick one connection model, document who owns browser lifecycle, and test reconnect behavior before relying on it in scheduled crawls.

Should I enable a stealth-oriented browser provider by default?

No. Start with the default provider and add another only for a defined, permitted requirement. A provider increases the number of moving parts, so it should earn its place through a concrete operational need, not a vague expectation that every modern site needs it.

Keep the browser at the edge of the architecture

The series began with Scrapy's core components, moved extraction into Page Objects, and added monitoring around the data contract. Browser rendering belongs at the edge of that architecture: it changes how a selected response is acquired, not how the rest of the project is designed.

For your next JavaScript-heavy target, inspect one product page and one listing page. Write down where each field comes from. Then choose the least expensive permitted acquisition method that gives you that source, and make the resulting Items pass the same checks as every other spider.

Spider monitoring made easy

John Rooney — Sat, 05 Sep 2026 08:00:00 +0000

A Scrapy job can exit cleanly after collecting zero products. It can also export 20,000 products with no prices. Neither failure needs to raise an exception in your spider, which means neither failure is visible if the only definition of success is an exit code of zero.

Spidermon gives a Scrapy project a place to define success in terms that matter to the data: how many items arrived, whether required fields are present, and what should happen when those expectations are not met.

A finished crawl tells you that the process stopped. A healthy crawl tells you that the data is still useful.

Part 1 put logging, pipelines, and feed exports in the right places. Part 2 moved page-specific extraction into typed Page Objects. This article closes the loop: it turns the resulting crawl statistics and item validation into monitors, then connects a failed monitor to an action that somebody can see.

What should a production spider prove?

Start with the failure modes of the specific spider, not a generic monitoring checklist. Our running catalog spider should not pass merely because it issued requests and wrote a file. It should show that it found enough products and that those products still meet the Product schema from Part 2.

Signal	What it catches	What it does not prove
Exit status	Unhandled process-level failure	That the expected pages or fields were collected
`item_scraped_count`	Empty or unexpectedly small output	That individual fields are valid
JSON Schema validation	Missing fields and incompatible values	That a price is commercially plausible
A Spidermon failure action	A failed expectation reaches the right place	That a person has responded

The last column matters. Monitoring is not a promise that data is perfect. It is a way to turn specific, known bad states into visible events before downstream users discover them in a dashboard or a customer-facing report.

Enable Spidermon before writing a monitor

Install Spidermon and jsonschema, which its JSON Schema validation feature uses:

pip install spidermon jsonschema

Enable the extension and tell it which monitor suite should run when the spider closes. Keep the configuration with the rest of your project settings, not hidden in a deployment script.

# catalog/settings.py
SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "catalog.monitors.CatalogCloseMonitorSuite",
)

Spidermon runs monitors in suites at defined points in the spider lifecycle. A spider-close suite is the natural first choice because the final Scrapy stats and the item-validation results are both available by then. The Spidermon getting-started guide documents the extension configuration and monitor lifecycle.

Do not begin by alerting on every retry or every 404. Start with the conditions under which you would not ship the resulting dataset.

Define a minimum item count that means something

The first monitor checks an intentionally modest, project-specific threshold. In a real catalog, derive it from the spider's normal output, its configured scope, and the consequences of partial data. A development crawl limited to one category should not be judged by the threshold of a nightly full crawl.

# catalog/monitors.py
from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Catalog crawl")
class CatalogStatsMonitor(Monitor):
    @monitors.name("Minimum product count")
    def test_minimum_product_count(self):
        item_count = getattr(self.data.stats, "item_scraped_count", 0)
        minimum_count = 100

        self.assertGreaterEqual(
            item_count,
            minimum_count,
            msg=(
                f"Expected at least {minimum_count} products, "
                f"but extracted {item_count}"
            ),
        )


class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [CatalogStatsMonitor]

Run the spider locally, then deliberately set minimum_count above the current output once to see the failure format in the logs:

scrapy crawl products

That small exercise is worth doing. A monitor that has never failed in development is only assumed to be connected to your runtime.

Scrapy already collects the stats that make monitors possible. The Scrapy stats documentation is useful when you need to inspect the exact counter name produced by a downloader, middleware, pipeline, or your own component.

Validate items at the point where they leave the spider

Part 2 gave our Page Object a typed Product item. That prevents plenty of accidental field mistakes in Python, but the data still comes from the web. Schema validation gives the project an independent check on the item it is about to export.

Place the validation pipeline after the project's own cleanup pipeline. Validating before a pipeline normalizes a price or URL creates false alarms, while validating before a later pipeline changes the item creates false confidence.

# catalog/settings.py
ITEM_PIPELINES = {
    "catalog.pipelines.ProductPipeline": 300,
    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800,
}

SPIDERMON_VALIDATION_SCHEMAS = ["schemas/product.json"]
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True

Here is a deliberately small schema. It requires all four fields from the Product item and rejects empty name and price values. Your schema should grow with your data contract, not with every possibility you can imagine on the first day.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "name": {"type": "string", "minLength": 1},
    "price": {"type": "string", "minLength": 1},
    "description": {"type": "string"},
    "url": {"type": "string", "format": "uri"}
  },
  "required": ["name", "price", "description", "url"]
}

With SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS enabled, invalid items carry validation information in an _validation field. That is useful while diagnosing a new rule. For a production export, decide whether downstream systems should receive flagged records or whether the pipeline should drop invalid items using SPIDERMON_VALIDATION_DROP_ITEMS_WITH_ERRORS.

The Spidermon item-validation guide recommends making the validation pipeline the last one, and documents the schema, error-field, and item-dropping settings.

Fail the crawl when validation errors become data loss

Adding validation errors to items is observability. A monitor turns those errors into a decision.

# catalog/monitors.py
from spidermon import Monitor, monitors
from spidermon.contrib.monitors.mixins import StatsMonitorMixin


@monitors.name("Catalog data quality")
class ProductValidationMonitor(Monitor, StatsMonitorMixin):
    @monitors.name("No product validation errors")
    def test_no_product_validation_errors(self):
        error_count = getattr(
            self.stats,
            "spidermon/validation/fields/errors",
            0,
        )

        self.assertEqual(
            error_count,
            0,
            msg=f"Found {error_count} product field validation errors",
        )

Add it to the suite from the first monitor:

class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [
        CatalogStatsMonitor,
        ProductValidationMonitor,
    ]

This rule is intentionally strict. It fits a catalog where a product without a price is not acceptable output. Other use cases may tolerate a small number of missing optional descriptions, or may care about a percentage rather than an absolute count. Spidermon provides validation-monitor helpers for field-level counts and percentages when that is the more honest definition of failure. Do not set a tolerance just to silence a monitor. Explain what the tolerated records mean for the people using the data.

Use periodic monitors to stop bad long-running crawls early

Spider-close monitors are necessary, but they arrive too late for every problem. If a broad crawl has been returning error pages for 45 minutes, waiting for it to complete wastes capacity and leaves an operator with a larger incident to untangle.

Periodic monitors run a normal monitor suite at a configured interval while the spider is still running. They are a good fit for failures that become expensive as they continue: a rising error count, no increase in extracted items, or a job running past its intended duration. Keep final data-quality validation in the spider-close suite, where the complete output is available.

A periodic monitor is a circuit breaker for a crawl, not a replacement for its final quality gate.

For the catalog spider, close the crawl after 20 logged errors. The number is an example, not a threshold to copy. Choose one that reflects the target site's normal failure behavior and the point at which continuing is no longer useful.

# catalog/actions.py
from spidermon.core.actions import Action


class CloseSpiderAction(Action):
    def run_action(self):
        spider = self.data["spider"]
        spider.logger.error("Closing spider after a periodic monitor failure")
        spider.crawler.engine.close_spider(
            spider,
            reason="closed_by_spidermon",
        )

# catalog/monitors.py
from spidermon import Monitor, MonitorSuite, monitors

from catalog.actions import CloseSpiderAction


@monitors.name("Catalog periodic health")
class CatalogPeriodicMonitor(Monitor):
    @monitors.name("Error count remains below the limit")
    def test_error_count(self):
        error_count = self.data.stats.get("log_count/ERROR", 0)
        maximum_errors = 20

        self.assertLessEqual(
            error_count,
            maximum_errors,
            msg=f"Crawl logged {error_count} errors, limit is {maximum_errors}",
        )


class CatalogPeriodicMonitorSuite(MonitorSuite):
    monitors = [CatalogPeriodicMonitor]
    monitors_failed_actions = [CloseSpiderAction]

Schedule the suite every 60 seconds:

# catalog/settings.py
SPIDERMON_PERIODIC_MONITORS = {
    "catalog.monitors.CatalogPeriodicMonitorSuite": 60,
}

Spidermon also provides built-in periodic monitors for execution time and item-count increase. Those are often a better first choice

A failed monitor should trigger an action

Logs are fine when you are running one spider in a terminal. They are not a monitoring strategy for scheduled jobs. Attach a failure action to the monitor suite so that a broken data contract reaches a channel where it can be triaged.

# catalog/monitors.py
from spidermon.contrib.actions.slack.notifiers import (
    SendSlackMessageSpiderFinished,
)


class CatalogCloseMonitorSuite(MonitorSuite):
    monitors = [
        CatalogStatsMonitor,
        ProductValidationMonitor,
    ]
    monitors_failed_actions = [
        SendSlackMessageSpiderFinished,
    ]

Keep credentials out of version control. For example, load the Slack configuration from the environment used to run the spider:

# catalog/settings.py
import os

SPIDERMON_SLACK_SENDER_TOKEN = os.environ["SPIDERMON_SLACK_SENDER_TOKEN"]
SPIDERMON_SLACK_SENDER_NAME = "catalog-monitor"
SPIDERMON_SLACK_RECIPIENTS = ["#data-alerts"]

Spidermon also supports actions for email, Telegram, Discord, Sentry, reports, and custom integrations. Start with one destination that has an owner. Alerting an unmonitored channel is only a more elaborate way to write to a log. The actions documentation lists the built-in and custom-action options.

Choose the action by the failure

Monitor result	First action	What to investigate
Zero or very few products	Alert the spider owner	Scope change, blocked listing page, pagination, or selector failure
A few invalid optional fields	Report and track the trend	A template variant or a cleanup rule
Invalid required fields	Alert and quarantine or drop affected output	A changed product page, response substitution, or broken extractor
A system-wide run failure	Escalate through the job platform	Credentials, deployment, networking, or a shared service

Do not make the first automated reaction "run the same spider again." Retrying can be sensible for a transient timeout. It cannot repair an HTML template that no longer has a .price element.

Keep the monitor suite small enough to trust

An early suite needs only a few checks that protect against known bad outcomes. For the catalog spider, start here:

At least the expected minimum number of products were extracted.
No required product fields failed schema validation.
A failed check sends one actionable notification to the owner.

Add checks when you can state the response to failure. For example, a sudden rise in retry count might warrant an alert only if it has a named owner and a clear threshold. Otherwise, it is a graph waiting to be ignored.

Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 can speed up the work of creating spiders and Page Objects. The review loop is still yours. The same is true of the concerns raised in this practical review of AI-generated Scrapy projects: code that runs needs checks that say whether it produced usable data.

Questions developers ask about Spidermon

Should every validation error fail the crawl?

No. Fail on the errors that make the dataset unfit for its purpose. A missing product URL is usually serious. A missing optional marketing description may deserve a report and a follow-up instead. Make the distinction explicit in the schema and monitor threshold.

Can Spidermon replace tests for Page Objects?

No. Page Object fixture tests catch a regression against a known response before deployment. Spidermon checks the behavior and output of a real crawl. Use both, because they fail at different points in the workflow.

Should invalid items be dropped immediately?

Sometimes. During development, attaching _validation errors to items makes debugging easier. In production, dropping an invalid record may be safer than exporting it, provided the monitor still alerts you to the loss. Do not silently drop records and call the job successful.

Make a failure visible before the next scheduled crawl

Add one item-count monitor, one required-field check, and one failure action to an existing spider. Then break each condition on purpose in a safe environment and follow the signal from the Scrapy stats to the monitor result to the notification.

That is the beginning of operating a spider as a data system. Part 4 will return to acquisition: how to choose between Scrapy, APIs, and browser automation when the target site is more complicated than static HTML.

Building maintainable spiders with scrapy-poet

John Rooney — Fri, 04 Sep 2026 08:00:00 +0000

A Scrapy callback can begin life as six lines of selectors and end up responsible for following links, interpreting a page, normalizing an item, deciding whether data is missing, and fetching a second endpoint when the price is loaded separately. It still runs, so it is easy to call that design successful.

Until the site changes. Then the code that needs changing is also the code that decides how to crawl, and a selector fix becomes a risky edit to the spider's control flow.

scrapy-poet is valuable because it gives extraction logic its own home. It integrates web-poet, a Page Object framework, with Scrapy so that a spider can concentrate on crawling while Page Objects describe how to turn a particular kind of page into a typed item.

This is not an argument that every 20-line spider needs another abstraction. It is an argument that the callback is the wrong long-term home for reusable extraction logic.

The callback starts simple, then quietly owns too much

Here is the product callback from Part 1 of this series, expanded with a description and a JSON-LD fallback for a price. Nothing here is outrageous. That is why this pattern lasts so long.

import json


def parse_product(self, response):
    name = response.css("h1::text").get(default="").strip()
    price = response.css(".price::text").get(default="").strip()

    if not price:
        structured_data = response.css(
            "script[type='application/ld+json']::text"
        ).get()
        if structured_data:
            data = json.loads(structured_data)
            price = data.get("offers", {}).get("price", "")

    yield {
        "name": name,
        "price": price,
        "description": " ".join(response.css(".description *::text").getall()).strip(),
        "url": response.url,
    }

Now imagine a second spider needs the same product interpretation, or that the catalog moves to a new domain with different selectors but the same output schema. Copying the callback is fast. Keeping copies aligned is where the cost arrives.

The boundary we want is straightforward:

The spider owns where to go and which links to follow.
A Page Object owns what one page means and how to extract it.
A pipeline still owns rules applied to every yielded item, such as the price validation from Part 1.

That division keeps an ordinary selector change out of the crawling strategy. It also makes the right unit of reuse obvious: the page, not a large utility function passed a response, a spider, and a few flags.

Put product extraction and its output contract in a Page Object

A web-poet Page Object receives page inputs, such as the downloaded response, and exposes an item. Use an attrs class instead of a dictionary for that item. It makes a misspelled field or a missing required value fail close to the extractor, rather than silently creating a new key in a result file.

Returns[Product] declares the item contract for the Page Object. Combined with @field, it lets web-poet build the Product instance automatically, so there is no hand-written to_item() dictionary to keep in sync with the model.

# catalog/items.py
from attrs import define


@define
class Product:
    name: str
    price: str
    description: str
    url: str

# catalog/pages/products.py
import json

from web_poet import Returns, WebPage, field, handle_urls

from catalog.items import Product


@handle_urls("example.com")
class ProductPage(WebPage, Returns[Product]):
    @field
    def name(self) -> str:
        return self.css("h1::text").get(default="").strip()

    @field
    def price(self) -> str:
        visible_price = self.css(".price::text").get(default="").strip()
        if visible_price:
            return visible_price

        structured_data = self.css(
            "script[type='application/ld+json']::text"
        ).get()
        if not structured_data:
            return ""

        data = json.loads(structured_data)
        return data.get("offers", {}).get("price", "")

    @field
    def description(self) -> str:
        return " ".join(self.css(".description *::text").getall()).strip()

    @field
    def url(self) -> str:
        return self.response.url

The Page Object is not magically more correct than the callback. Its advantage is that the responsibilities and output schema are visible. A reviewer looking at ProductPage can discuss selector behavior and the Product contract without first understanding pagination, request metadata, or retry policy.

Keep the item class deliberately dull. Reuse Product across sites that produce the same schema, and keep site-specific parsing and cleanup in Page Object fields. A typed item is a contract, not a second place to hide extraction logic.

This is also a useful place to be honest about the limitations. A page with invalid JSON-LD will raise JSONDecodeError in the example above. Whether that should fail the item, produce a warning, or fall back to a different source is a data-quality decision. Do not swallow it just to make the crawl look healthier. Part 3 will deal with making those failures observable.

The scrapy-poet basic tutorial shows the same separation with a smaller example. The web-poet items guide explains its recommendation to use item classes, while the Page Object documentation goes deeper into available page inputs.

Enable scrapy-poet as a Scrapy add-on

For Scrapy 2.10 and later, scrapy-poet is enabled through Scrapy's add-on system. Add-ons are designed to package related Scrapy component configuration, rather than asking every project to manually copy middleware and fingerprinter settings. Scrapy's add-on documentation explains how their priorities and overrides work.

pip install scrapy-poet

# catalog/settings.py
ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["catalog.pages"]

SCRAPY_POET_DISCOVER tells the integration where to find Page Objects that have URL rules. Even if your first Page Object is imported directly by a spider, set this up now. It makes the project layout predictable once you add site-specific Page Objects. The setup guide documents both settings and the configuration required for older Scrapy versions.

Ask scrapy-poet for the item the spider needs

With a URL rule and Returns[Product] in place, a spider callback can ask for Product directly. scrapy-poet chooses the Page Object that can produce that item for the requested URL, builds it, and injects the completed item into the callback. The callback no longer needs the response for extraction.

# catalog/spiders/products.py
import scrapy

from catalog.items import Product
from scrapy_poet import DummyResponse


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        links = response.css(".product-card a::attr(href)").getall()
        yield from response.follow_all(links, callback=self.parse_product)

    def parse_product(self, response: DummyResponse, product: Product):
        yield product

DummyResponse makes the dependency explicit: this callback consumes the item, not Scrapy's response object. If you need the Page Object itself, perhaps because the callback combines it with crawling logic, request the Page Object and await page.to_item() instead.

For the common case where a product page produces one item, callback_for() removes the callback entirely:

import scrapy

from catalog.pages.products import ProductPage
from scrapy_poet import callback_for


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    parse_product = callback_for(ProductPage)

    def parse(self, response):
        links = response.css(".product-card a::attr(href)").getall()
        yield from response.follow_all(links, callback=self.parse_product)

Store callback_for(ProductPage) on the spider rather than creating it inline in response.follow_all(). The scrapy-poet documentation notes that the inline form does not work with Scrapy disk queues. That is exactly the kind of implementation detail you would rather learn before a crawl needs persistent scheduling.

Use URL rules when one spider needs more than one page implementation

The Page Object above describes one site's product pages. Reuse becomes more interesting when the crawl strategy stays the same while the page markup changes by domain.

web-poet can select a Page Object implementation using URL rules. Annotate a concrete class with handle_urls(), keep the common Product contract in a base class, and let discovery register the rule.

# catalog/pages/example_store.py
from web_poet import handle_urls

from catalog.pages.products import ProductPage


@handle_urls("example-store.com", instead_of=ProductPage)
class ExampleStoreProductPage(ProductPage):
    @field
    def name(self) -> str:
        return self.css("h1.product-title::text").get(default="").strip()

This does not mean every multi-site project should become a generic crawler. Different sites often have genuinely different navigation, login, pagination, or data contracts. Share the crawler only where the strategy is actually the same, and keep site-specific work explicit where it is not.

The payoff is most visible when it is time to replace a selector. You edit the relevant Page Object, test it against its fixture, and leave the rest of the crawl alone. The rules section of the scrapy-poet tutorial covers the fuller pattern, including explicit overrides.

Test an extraction rule without running a whole crawl

Most spider tests either mock so much that they no longer resemble a page, or run a real crawl and wait for the network. Page Objects give you a better middle ground: save the inputs and expected output for a representative page, then test the parser locally.

scrapy-poet supplies savefixture to capture a Page Object's dependencies and its to_item() result. The generated fixture can then be discovered by the pytest plugin that comes with web-poet.

scrapy savefixture catalog.pages.products.ProductPage \
  "https://example.com/products/widget-42"

python -m pytest

The fixture becomes a regression test. When a selector change causes a price or name to move, the test can fail before a scheduled crawl produces a quiet column of empty values. The scrapy-poet testing guide and web-poet test documentation explain the fixture layout, expected output, and how individual fields are tested.

Fixtures are not a substitute for live checks. They tell you whether your extraction code still interprets a known response as expected. They cannot tell you that the site has changed, that a login page arrived instead, or that the listing page stopped linking to products. You need both fixture tests and a small, monitored live crawl.

When scrapy-poet is the wrong extension point

Use scrapy-poet when the problem is page interpretation, reusable extraction, or dependencies needed to interpret a page. Do not use it as a container for every Scrapy customization.

Use an item pipeline for project-wide item validation and persistence.
Use downloader middleware for request and response behavior that applies before the spider sees a page.
Use a Scrapy extension and signals for crawl-wide behavior, such as reporting or metrics.
Keep code in the spider when it is navigation strategy that only that spider owns.

This matters because a framework can make bad boundaries easier to decorate. A Page Object that schedules all requests, sends Slack messages, and writes to a database is still a callback with a nicer name.

If you are using AI-assisted tooling to create Page Objects, keep the review focused on the contract: which URL does this class handle, which inputs does it need, and exactly which item does it promise to produce? Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 both make Page Objects part of the development workflow. The gaps between runnable code and a shippable Scrapy project are also the subject of this recent practical review. None of these tools remove the need to decide where project behavior belongs.

Make one callback smaller this week

Pick the callback that makes you hesitate before editing it. Move only the page-specific extraction into a Page Object, leave crawling and pipelines where they are, and generate one fixture for the page that has caused trouble before.

If that leaves the spider with a clear description of how it moves through the site, you have found a useful boundary. In Part 3, we will use Spidermon to define what a healthy crawl looks like and act when the spider stops meeting that definition.

Only 18.5% of the web is open. What an audit of 24,898 sites tells scraper engineers

John Rooney — Thu, 03 Sep 2026 06:01:41 +0000

Zyte, where I work, has just published The State of Web Access, an audit of 24,898 popular landing pages that measures which technical barriers sites put in front of automated visitors. It is free, it is 55 pages, and the underlying data is browsable in a data explorer broken down by barrier, industry and country. I want to share it here because most of the writing about "the web is closing" is anecdote, and this is the first dataset I have seen that is big enough to argue with.

This post is my read of it from the point of view of someone who writes and debugs scrapers. I will go through the method first, then the six barriers with the numbers, and along the way I will add context from other sources, because the report measures adoption and does not always explain the mechanism underneath. Where I disagree with the report's framing, or think a number needs a caveat, I say so.

How it was measured

Numbers without a method are marketing, so here is the method.

The sample comes from Semrush's top website lists. The global list gives 11,100 landing page records, which is the base for the global percentages. A regional list adds 22,137 unique landing pages, snapshotted in October 2025, sampled across countries and categories. Deduplicated, that is 24,898 sites across 230 countries, mapped into 110 industries.

For each landing page the audit checked six barriers, each detected a different way:

Web Application Firewall (WAF). HTTP responses and server headers matched against the fingerprint database from wafw00f 2.3.2.
Antibot services. Vendor fingerprints in scripts, request patterns and endpoints that appear when a commercial bot mitigation product is active.
CAPTCHA. HTML snippets, script loads and network calls tied to known providers. This detects the infrastructure, not whether a challenge was actually served to the scanner.
JavaScript dependence. A plain HTTP fetch compared against a fully rendered page. If rendering grew the meaningful HTML by more than 50%, the page counts as JavaScript-dependent.
Rate limiting. A short burst of requests at mixed pacing, recording whether and how fast the site started rejecting them.
TLS fingerprinting. Inspecting how the site responds to the TLS handshake, to see whether it checks that the client looks like a standard browser before any HTTP request is processed.

robots.txt was fetched and parsed separately as a policy signal rather than a barrier. Scanning ran on Scrapy Cloud with a modified version of caniscrape 0.1.8 and Zyte's internal tier API.

The report is upfront about the limits, and they matter. Everything is landing pages, and plenty of sites leave the homepage open for SEO while defending search results and product pages far harder, so treat every figure here as a floor. "Present" means "detectable via the fingerprints we scanned for", so custom or niche systems are undercounted. And the regional dataset is the pages most visited from a country, not the pages of sites based there.

The headline numbers

Barrier	Share of 11,100 landing pages
WAF (named or generic)	92.4%
JavaScript-dependent	40.6%
CAPTCHA infrastructure	22.5%
Rate limiting	22.1%
Dedicated antibot	18.5%
TLS fingerprinting	13.8%
No detectable barrier	18.5%

Half of all pages run two or more barriers. The distribution by count is 18.5% none, 30.7% one, 25% two, 12.1% three, 5.4% four, 7.4% five and 0.8% all six. That 0.8% sits almost entirely in high-value commercial categories.

The "18.5% open" figure is the one that will get quoted, and I think the more useful one is the shape of that distribution. Most of the defended web is not hardened, it is layered, and each layer is cheap for the site and adds a different failure mode for you.

WAF: the default nobody chose

92.4% of pages sit behind a WAF, but only 63.5% run one that fingerprints to a named vendor, and the remaining 29% match generic WAF signatures. Cloudflare is 34.6% of all pages, generic or unidentified 31.6%, Amazon CloudFront 9.6%, Akamai Kona 4.9%, AWS Elastic Load Balancer 4.3%, and then a long tail of 13 more. The top three vendors hold 71.7%.

The detail I find more interesting is that 61.9% of sites run a single WAF vendor in standard mode, and only 1.4% run multiple vendors or behavioural analysis. The WAF arrives bundled with the CDN and most operators never touch the defaults. 18% of all URLs in the audit rely on a WAF and nothing else.

For context, W3Techs as of 3 September 2026 puts Cloudflare in front of 25.0% of all websites in its 10 million site crawl, which is 84.6% of sites with an identifiable reverse proxy. So among popular sites the market is actually less concentrated than across the whole web, but the same vendor is at the top either way, and the antibot market below is even more lopsided.

Antibot: one vendor decides what a browser is

18.5% of landing pages run a dedicated bot management product, and these are the systems that actually look at behaviour, mouse movement, keyboard events, canvas rendering, and browser attribute consistency, rather than just headers and IP reputation.

Cloudflare Bot Management is 75.9% of those deployments. Imperva is 9.1%, Akamai Bot Manager 7.7%, PerimeterX (now HUMAN) 5%, DataDome 2%. The report calls this a one-horse race, and it is, and I would put the implication more bluntly than the report does: for three quarters of the behaviourally defended web, one company's model of what a real browser looks like is the definition of a real browser. When that model changes, everyone's scrapers break on the same day.

Adoption by industry is what you would guess. Adult content 36%, furniture 33%, gambling 31%, beauty, computer hardware and tobacco at 30%. These are sectors with checkout flows, inventory and account fraud, so the antibot is protecting revenue, not content.

CAPTCHA: two vendors, and it skews small

22.5% of pages have CAPTCHA infrastructure. reCAPTCHA is 49.4% of that, Cloudflare Turnstile 39.4%, GeeTest 7.7%, hCaptcha 1.9%, Arkose FunCaptcha 1.5%. Two vendors cover 88.8% of CAPTCHA-enabled sites.

Two things worth knowing. First, a lot of this is invisible, reCAPTCHA v3 and Turnstile score you in the background and only escalate when the score is bad, so a "CAPTCHA present" site may never show you a puzzle if the rest of your fingerprint holds up. Second, CAPTCHA adoption is inversely correlated with traffic. Lower-traffic sites are more likely to use one than high-traffic sites, because a CAPTCHA is the cheapest defence to bolt on. Tobacco leads adoption at 52%, which the report attributes to age verification requirements rather than anti-scraping.

JavaScript: 141 KB of page that HTTP never sees

40.6% of landing pages need JavaScript to produce meaningful content, by the 50% growth definition above. On those pages, rendering adds 140,923 bytes on average, which is 75% of the final HTML payload. Restaurants and airlines lead at 66%, travel at 65%, pharmaceuticals and sports at 64%.

The report is careful to say this is mostly architecture, not defence, and I agree. Single page application frameworks won, so the HTML you get over plain HTTP is a shell. Do not confuse this figure with the HTTP Archive Web Almanac number of roughly 697 KB of JavaScript on the median home page, that is script bytes downloaded, this is DOM added by executing them, they measure different things and both point the same way.

There is a defensive correlation though. Sites that require JavaScript are more likely to also run a WAF and rate limiting, and high-traffic sites are more likely to require it. Your three options are still the same as they were five years ago: render in a browser, find the internal API the page calls and hit that instead, or accept partial data. The second is still the best when it works.

Rate limiting: it is really blocking, and it is really 403

22.1% of pages throttle bursts. What surprised me is how they do it. Of the sites that rate limit, 94% block after a single request in the burst pattern and 5% after a burst of eight. That is not rate limiting in any RFC sense, it is bot detection that happens to fire on request cadence.

The status codes confirm it. The first blocking response is 403 Forbidden on 92.6% of rate-limiting sites, 429 Too Many Requests on 5.7%, 503 on 1.2%, 401 on 0.4%. RFC 6585 has defined 429 since 2012, along with a Retry-After header telling you how long to back off, and almost nobody uses it.

This has a direct consequence for how you write retry logic. If your client backs off on 429 and gives up on 403, you are giving up on the majority of rate limits. A 403 from a WAF rule, a 403 from a rate limiter, and a 403 from an antibot verdict are indistinguishable from the status code alone, so you need to classify by body, headers and timing, and log which layer you think produced it. High-traffic sites are 66% more likely to rate limit than low-traffic ones. Apparel and fashion leads at 54%, sporting goods 49%, jewellery, photography and restaurants at around 42%.

TLS fingerprinting: the barrier before the request

13.8% of landing pages check the TLS handshake, and this is the one I think engineers understand least, because it fails before any HTTP happens and the error looks like a network problem.

The mechanism is that your client announces its supported TLS versions, cipher suites and extensions in the ClientHello, and different TLS libraries do this differently. Python's requests on OpenSSL, Go's crypto/tls and Chrome's BoringSSL all produce distinguishable handshakes, so a site can tell "Chrome user agent, OpenSSL handshake" is lying before it reads a single header. JA3, from Salesforce in 2017, turned this into an MD5 hash of those fields in order. Chrome then started randomising the order of its ClientHello extensions in January 2023, which changed the JA3 hash on every connection, and JA4 from FoxIO replaced it by sorting the fields before hashing and producing a human-readable string. Clients like curl-impersonate exist specifically to reproduce a browser's handshake from a non-browser client.

The report's co-occurrence data is the useful part. TLS-fingerprinting sites are 5.9x more likely to rate limit (78.1% of them do, against 13.2% of other sites), 3.2x more likely to run antibot and 1.8x more likely to run CAPTCHA. They are also less likely to require JavaScript, 27% against 42.8%, which the report reads as performance-focused, server-rendered sites. So a TLS check is a strong predictor that everything else is switched on too, which makes it a cheap first probe when you size up a new target.

How the barriers stack

The WAF is the only barrier that regularly stands alone. A third of WAF sites run nothing else. TLS fingerprinting appears alone on under 8% of TLS sites, JavaScript alone on 0.6%, rate limiting alone on 0.2%, CAPTCHA alone on 0.1%, and antibot alone on 0%. Two stacks dominate, WAF only at 31% of sites and WAF plus JavaScript at 20%. The report gives 3.3% for its "full-stack combination" of active defences, against 0.8% running all six measured barriers, and both cluster in the highest-value commercial categories.

The report's advice out of this is to plan for multi-path retrieval, instrument failures by layer, and treat sessions as an asset with reputation and state. That matches my experience, and I would add that "instrument by layer" is the one most teams skip, because a generic "blocked" counter hides whether you are losing at TLS, at the WAF, at rendering or at a challenge, and those have completely different fixes.

The cost view, with a disclosure

Zyte mapped every domain to its own five-tier access model, testing each once over plain HTTP and once through a headless browser and taking the higher of the two. The result is 59.3% Simple, 28.6% Easy, 7.9% Moderate, 3% Complex and 1.2% Advanced, so 88% of landing pages sit in the two cheapest tiers. Apparel and fashion has the highest mean tier at 2.86, retail 2.60, sporting goods 2.50, real estate 2.37.

This is Zyte's pricing model applied to Zyte's own data, so read it as "how hard is this for one particular unblocking stack" rather than an independent difficulty score. The relative ordering between industries is the part I would trust, the absolute "88% is cheap" framing is the part I would want to see reproduced by someone else.

robots.txt is turning into a negotiating position

75.8% of sites publish a valid robots.txt. Only 0.5% rely on it with no technical defence behind it, and 48.3% pair it with two or more technical barriers, so the honour system is now backed by enforcement almost everywhere.

The AI section is where the data gets interesting. 13.7% of sites name AI crawlers, GPTBot is disallowed on 8.4% of sites, CCBot 7.3%, ClaudeBot 6.3%, Google-Extended 5.6%, Bytespider 5.5%, meta-externalagent 4.4%, ChatGPT-User 4.3%, PerplexityBot 4.2%. Newspapers name AI agents on 69% of sites, publishing on 51%, mass media on 44%. Two thirds of newspapers name five or more crawlers.

The counter-intuitive finding is that SEO and search-indexing bots are named in more robots.txt files (19.2%) than AI crawlers (13.7%). The report reads this as publishers valuing search as a traffic partner and AI as a content taker, which fits Cloudflare's own data from July 2026 that 52% of crawler requests on its network are now for AI training, up from 22% in spring 2025, and Imperva's finding that automated traffic passed 51% of all web traffic in 2024.

The report predicts that "access will become more explicitly negotiated". I would say that has already started, and it is happening in the protocol layer, which the report does not cover. RFC 9309 finally standardised robots.txt in September 2022. Cloudflare switched new domains to blocking AI crawlers by default in July 2025 and started a pay-per-crawl beta. In September 2025 it proposed a Content Signals Policy that adds a line like Content-Signal: search=yes, ai-train=no to robots.txt to separate indexing from training from inference. And Web Bot Auth, now at the IETF as draft-meunier-web-bot-auth-architecture, lets a crawler sign its requests using RFC 9421 HTTP Message Signatures with a Signature-Agent header pointing at its public key directory, so a site can verify who a bot is instead of guessing from IP ranges. Put the report's numbers next to those standards and the direction is clear: the advisory layer is becoming an authenticated, priced access layer, and "good bot" is about to mean "bot with a key".

By industry

The 20-group heatmap in the report is worth a look on its own, a few rows stood out to me:

Media and publishing has the highest WAF adoption at 97% and by far the most AI blocking in robots.txt at 30%, but low antibot (14%) and rate limiting (16%). Publishers defend with paywalls and registration walls, not infrastructure.
Real estate and automotive stack rate limiting (39% and 38%) and JavaScript (48% and 44%) on top of the baseline WAF, because listing and pricing data is the business.
Retail and e-commerce is the contested middle, 93% WAF, 23% antibot, 27% CAPTCHA, 44% JavaScript, 31% rate limiting, needing to be visible to search and customers while opaque to competitors.
Public sector, agriculture and energy stop at the CDN-bundled WAF with single-digit AI blocking.

What I would do differently after reading it

Probe TLS first when sizing a new target. It costs one handshake and predicts most of the rest of the stack.
Stop treating 403 as terminal. Classify it. Most rate limits and most WAF blocks arrive as 403 with no Retry-After.
Count failures per layer, TLS, WAF, render, challenge, antibot verdict, not per site.
Look for the internal API before reaching for a browser on the 40% of pages that need JavaScript.
Read robots.txt as a statement of what the operator will tolerate, and expect that file to start carrying content signals and to be paired with signed-request verification.

What I would like to see next

A landing-page audit is a floor, and the interesting fights are on search and detail pages, so a deep-page follow-up would be the most valuable extension. The snapshot is October 2025 with no earlier baseline, so the "web is getting harder" claim is inferred rather than measured, and a second run in a year would turn it into a trend line. And the cost tiers need an outside reproduction.

Those are real gaps, but the dataset is still the biggest public measurement of access controls I know of, and the explorer lets you check any of the claims above against your own industry or country. The PDF is free with no form to fill in. If you scrape for a living, or you run a site and want to know what your peers are doing, it is worth an hour.

Chrome has a new potential fingerprint vector

John Rooney — Thu, 03 Sep 2026 05:56:52 +0000

On August 25, 2026, Chrome 152 ships navigator.cpuPerformance to every user on Windows, Mac, Linux, ChromeOS, and Android. It is a single read-only property that returns a small integer from zero to four, meant to tell a website roughly how powerful the visitor's CPU is so it can decide whether to serve a 1080p video call with a virtual background or a 240p stream with the effects turned off. That sounds harmless, and the W3C Web Incubator Community Group (WICG) spec goes to real lengths to make it sound harmless: five possible values, a stated privacy goal that each tier should cover at least 10% of devices, and an explicit design rule that the value must never leak raw CPU model or vendor information. For anyone who spends their time thinking about fingerprinting, bot detection, or building scrapers, the interesting part is not what the spec says the API should reveal. It is what Chrome's actual implementation reveals once you read the algorithm behind it.

What the API is supposed to do

navigator.cpuPerformance is modelled on the older Device Memory API: a coarse, static signal rather than a precise one. It lives only in secure (HTTPS) contexts on Window, and it is explicitly not meant to compete with the dynamic Compute Pressure API, which measures how hard the CPU is working right now. The two are meant to work together: cpuPerformance tells a site what to load first, and Compute Pressure tells it when to back off once the page is running under load.

The spec's own example is a video conferencing app. Tier one is barely usable for a call at all, QVGA resolution at 15 frames per second with no effects. Tier four has performance to spare: 1080p at 30 frames per second, plus a virtual background. The spec even instructs developers to treat tier zero, meaning the browser could not classify the device, the same as tier four, so an unknown device gets the benefit of the doubt rather than the worst experience.

Chrome is not doing this alone for altruistic reasons. Rick Byers, who has been driving the proposal, has been candid on the blink-dev mailing list about why this exists: mobile Gmail has quietly maintained a server-side database mapping user agent and device-model strings to capability classes for years, similar to Facebook's old "Year Class" system, so it could ship different HTML and JavaScript bundles to different phones. This API takes a practice that has existed invisibly for over a decade and makes it a standard, inspectable browser feature.

The algorithm nobody is talking about

Here is where things get interesting for anyone working on unblocking or headless automation. On July 21, 2026, the spec's editor opened pull request #19, which adds Chrome's actual classification logic to the spec as normative text. Reading it strips away most of the mystery.

The algorithm works roughly like this:

1. Read the logical core count from the OS.
2. No core count reported, or zero cores → tier 0 (unknown).
3. Exactly one core → tier 1.
4. Two to four cores → check the CPU model against a hardcoded
   list of "old or Atom-class" families (demote to tier 1) or
   "modern efficient quad-core" families like Ryzen and Gracemont
   (promote to tier 3). Everything else falls back on core count alone.
5. Five to ten cores → tier 4 if it is an Apple M-series chip or an
   Intel Core Ultra with eight or more cores, otherwise tier 3.
6. More than ten cores → tier 4, unconditionally, regardless of
   the CPU model.

Frequency is never consulted, despite the spec permitting it. In practice, the tier is hardwareConcurrency doing nearly all of the work, with a small demotion table for CPUs considered underpowered and a small promotion table for chips considered premium. The demotions cluster around Intel Atom-derived cores and very old Core 2 and Nehalem-era chips. The promotions cluster around AMD Ryzen, Intel's Gracemont efficiency cores, Apple silicon, and Intel Core Ultra.

That detail matters more than it looks. Because the model-matching table is public in the spec, the tier value leaks CPU family information in specific bands that no other browser API exposes directly. An eight-core device reporting tier four is very likely an Apple M-series machine or an Intel Core Ultra system: no other eight-core CPU family clears that bar. A four-core device reporting tier three is very likely a Ryzen or Gracemont-based chip, since the fallback path for a generic four-core CPU caps out at tier two. Marcos Caceres flagged the underlying concern in WebKit's standards-positions review, pointing out that a higher tier correlates with a more expensive device, and a more expensive device is itself a proxy for the visitor's disposable income. Ad targeting is not a hypothetical use case here either. It is listed as use case five in the WICG explainer itself, right alongside things like deciding whether to run an AI model locally.

Why this matters for scraping and bot detection

The entropy math looks reassuring on paper. Five possible values is roughly 2.3 bits, which sounds trivial next to the 15 to 25-plus bits that the Performance API family alone can expose through timing side channels. But cpuPerformance has a property those noisier signals do not: it is a single synchronous read, it costs no CPU time to compute, and the reproducibility requirement in the spec means it is guaranteed to return the same value across every session on the same device. Stacked with hardwareConcurrency, deviceMemory, screen metrics, and GPU adapter strings, it becomes one more piece of clean, free, stable entropy in the fingerprinting surface. Nothing about running traffic through a VPN or rotating IPs touches it.

For anti-bot vendors specifically, the more immediate use is not identification, it is coherence checking. A cloud VM with two virtual CPUs will report tier one, the same bucket the spec describes as "practically unusable for video calls." A four-vCPU instance lands at tier two. Meanwhile, nearly every real consumer laptop or phone from the last five years reports tier three or four. That split alone is a free, low-cost signal for separating datacenter traffic from real users, in the same category as the existing deviceMemory and hardware concurrency checks that detection vendors already fold into their scoring.

The more interesting failure mode shows up when a headless browser's reported cpuPerformance disagrees with everything else the page can measure about it. A patched Chromium instance claiming tier four while its actual JavaScript execution speed, hardwareConcurrency, and GPU signals all look like a two-vCPU cloud box is exactly the kind of internal inconsistency that coherence-based detection is built to catch. Zyte's own conversations with anti-bot vendors such as Castle at Extract Summit 2025 described this shift directly: detection has moved from a binary bot-or-not decision toward scoring an entire session for consistency across IP, TLS, browser fingerprint, and now, potentially, this new CPU tier.

Spoofing the value itself is trivial. It is one property on navigator, and Chrome even ships an enterprise policy, CpuPerformanceTierOverride, that lets administrators force it to any value from zero to four, which is itself a tacit admission that some deployments will not want the real number exposed. Overriding it in a headless context is a one-liner:

Object.defineProperty(navigator, 'cpuPerformance', {
  get: () => 4
});

The catch is that the value alone was never strong evidence in the first place. What actually matters is whether it lines up with everything else the page can observe. This is the same lesson that shows up across TLS fingerprinting and network-level signals: a single spoofed header or property is cheap to fake, but a whole profile that has to agree with itself is much harder to fabricate convincingly. curl-cffi's browser-shaped TLS fingerprint solves one layer of that problem; a coherent cpuPerformance value that matches the emulated hardware profile is now one more layer to get right.

What to actually do about it

We aren't worried about breaking changes. But that doesn't mean you shouldn't be aware of it and decide if you need to take some precautions.

For anyone running headless infrastructure, the practical checklist is short. Do not simply hardcode cpuPerformance to four and call it done. If a browser session is emulating an eight-core M-series Mac, the CPU tier, hardwareConcurrency, GPU renderer string, and JavaScript execution timing all need to agree with that story, or the mismatch becomes the tell. This is exactly the kind of consistency problem that unblocking infrastructure exists to solve for, since maintaining a coherent fingerprint across dozens of signals at scale is a different job from spoofing any single one of them.

The interoperability story is worth watching too. Mozilla has taken no position, and WebKit has marked itself opposed, which means this is very likely to remain a Chromium-only signal for the foreseeable future, at least on the browser side. That does not make it any less relevant for automation built on Chromium, which covers the overwhelming majority of headless scraping tooling in production today.

Chrome 152 goes to stable on August 25, 2026, the last four-week release cycle before Chrome moves to a two-week cadence with version 153. Anyone maintaining anti-detect profiles, custom browser builds, or fingerprint-consistency checks has about a month to decide how cpuPerformance fits into that picture before it is live for every Chrome user on the web.

Modern Scrapy for experienced developers: A new series

John Rooney — Thu, 03 Sep 2026 05:50:27 +0000

This is the first of a 4 part mini series to help bring you up to speed with Scrapy, how we use it to scrape the modern web, and how you can extend it to meet your needs. By the end of this series you will be able to write production grade Scrapy code ready for large scale web scraping.

f you have used Scrapy before, you probably remember the satisfying part: write a callback, pull a few fields out of a response, yield a dictionary, and watch a JSON file appear. That workflow still works. The trouble starts when the spider has to run again next week, on a larger catalog, with a missing price, a timeout, or a request that needs different behavior from the rest of the project.

The useful mental shift is to stop treating Scrapy as a convenient loop around requests and selectors. It is a crawler with places for configuration, logging, data processing, and request handling. Using those places is usually less work than rebuilding them in a spider callback.

A working spider is not yet a working Scrapy project

Consider a small catalog spider. It follows product links and extracts a name and price. There is nothing wrong with starting here, and the Scrapy tutorial remains a good way to refresh the basic flow.

# catalog/spiders/products.py
import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        for href in response.css(".product-card a::attr(href)").getall():
            yield response.follow(href, callback=self.parse_product)

    def parse_product(self, response):
        yield {
            "name": response.css("h1::text").get(default="").strip(),
            "price": response.css(".price::text").get(default="").strip(),
            "url": response.url,
        }

Run it with scrapy crawl products -O products.jsonl, and you have data. For a short-lived investigation, that may be all you need.

For a project you intend to keep, ask different questions. Which settings apply to every spider? How will you notice that a selector stopped matching? Where should price normalization live? What is the polite request rate for this site? How can you change the output destination without editing extraction code?

Scrapy already has answers. The rest of this article puts them in the places where they remain useful after the initial crawl.

Put crawl behavior in settings, where it can be reviewed and overridden

settings.py is not a graveyard of copied options. It is the project contract: the behavior that should be consistent across spiders unless one has a specific reason to differ. Scrapy defines a clear precedence order, with command-line settings above spider settings, which sit above project settings and defaults. That makes a temporary diagnostic run possible without committing a one-off change to the repository. See the settings documentation for the full order and built-in reference.

Here is a sensible starting point for the catalog project:

# catalog/settings.py
BOT_NAME = "catalog"

SPIDER_MODULES = ["catalog.spiders"]
NEWSPIDER_MODULE = "catalog.spiders"

ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 2
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 2

LOG_LEVEL = "INFO"
LOG_SHORT_NAMES = True

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

FEEDS = {
    "products.jsonl": {
        "format": "jsonlines",
        "encoding": "utf8",
        "fields": ["name", "price", "url"],
        "store_empty": False,
    },
}

The values are examples, not universal defaults. A site may publish crawl guidance that calls for a lower rate, and an internal API might support more concurrency. The important part is that the decision is explicit, visible in review, and separate from parsing code. AutoThrottle adjusts delay using the observed latency while respecting the limits you set. It is a better starting point than guessing a fixed delay, but it is not permission to crawl a site aggressively.

Use the command line when you need a temporary change. This run writes logs to a file without changing the project settings:

scrapy crawl products \
  -s LOG_LEVEL=DEBUG \
  -s LOG_FILE=products-debug.log

Do not use per-spider settings just because a setting is nearby. Use them when the difference belongs to that spider. For example, a small, high-value spider might need a separate feed while the project defaults remain unchanged:

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    custom_settings = {
        "FEEDS": {
            "products.jsonl": {
                "format": "jsonlines",
                "encoding": "utf8",
            }
        }
    }

The feed exports documentation covers storage backends, batching, field order, and other options that become relevant before you write your own exporter.

Make logs explain the crawl, not narrate it

The default crawl log already contains useful request, response, retry, and item statistics. Your spider should add the missing business context. self.logger is a normal Python logger configured by Scrapy, so it fits into the same output and log-level controls as the rest of the crawl. Scrapy's logging guide explains the project-wide configuration.

Logging every successful product is usually noise. Logging an extraction condition that may change the quality of the dataset is useful:

def parse_product(self, response):
    name = response.css("h1::text").get(default="").strip()
    price = response.css(".price::text").get(default="").strip()

    if not price:
        self.logger.warning(
            "Product has no visible price: name=%r url=%s",
            name,
            response.url,
        )

    yield {"name": name, "price": price, "url": response.url}

Keep the values as logger arguments rather than formatting the string yourself. Apart from being standard Python logging practice, it keeps the call easy to scan and lets the logging system decide whether the message needs formatting.

When a page is puzzling, use the Scrapy shell before changing the spider. It shortens the loop from "run a crawl, inspect output, edit code" to "inspect the exact response and selector."

scrapy shell "https://example.com/products/widget-42"

response.css(".price::text").get()
response.css("script[type='application/ld+json']::text").get()

If the shell does not contain the data you see in a browser, that is evidence about the page delivery path, not a reason to immediately add a browser. Part four of this series will cover that choice.

Use pipelines for data rules that apply to every item

It is tempting to clean prices in parse_product(). It is also how normalization gets copied into the next three spiders. A pipeline is the right boundary for item-by-item work that should be consistent across the project: validating required fields, normalizing formats, removing duplicates, or sending items to storage. Scrapy calls each enabled item pipeline in order after a spider yields an item.

This example normalizes a simple dollar price and drops items that cannot be used. In a real project, agree on the currency and numeric representation with the data consumer rather than silently assuming dollars.

# catalog/pipelines.py
from decimal import Decimal, InvalidOperation

from scrapy.exceptions import DropItem


class ProductPipeline:
    def process_item(self, item, spider):
        if not item.get("name"):
            raise DropItem(f"Missing product name: {item.get('url')}")

        raw_price = item.get("price", "").replace("$", "").replace(",", "").strip()
        try:
            item["price"] = Decimal(raw_price)
        except InvalidOperation:
            raise DropItem(f"Invalid price {raw_price!r}: {item.get('url')}")

        return item

Enable it in the project settings:

ITEM_PIPELINES = {
    "catalog.pipelines.ProductPipeline": 300,
}

The number controls order. That matters when one pipeline depends on work done by another, such as normalizing a URL before a deduplication pipeline sees it. Avoid creating a pipeline for every tiny transformation, though. Page-specific extraction stays in the spider; a rule shared by the project belongs here.

Know what middleware is for before you need it

Pipelines work on items. Middleware works on the crawl itself. Downloader middleware can modify requests before download or responses after download, which makes it a useful home for cross-cutting concerns such as project-wide headers, authentication, or response diagnostics. Spider middleware sits around a spider's input and output.

You do not need to write custom middleware to benefit from it. Scrapy's own retry, redirect, cookies, HTTP cache, and robot exclusion behavior use this component model. The practical lesson for a returning developer is to resist adding request behavior to every callback. First check whether a built-in setting covers it. If it does not, middleware is usually a cleaner home than a growing stack of if statements in the spider.

The same restraint applies to packages and generated code. Tools can get you to a runnable spider quickly, but a runnable spider still needs project-level decisions about output, rate, retries, and data rules. That distinction appears in this recent look at AI-generated Scrapy projects and is why a project structure matters even when an IDE helps write the first callback.

Keep the first project boring on purpose

There is a version of this article that adds a browser, a queue, a custom downloader, a database, and a monitoring system before the first crawl finishes. That is not the version to copy.

Start with a project setting for behavior you want to keep, a log message for a condition you will need to investigate, a pipeline for a data rule shared across spiders, and feed exports for ordinary output. Once those boundaries are in place, adding a specialized component is a design choice rather than a rescue operation.

If you want help building and maintaining the project from inside your editor, Web Scraping Copilot is designed for Scrapy workflows. Zyte's guide to bringing web scraping into the IDE and the Web Scraping Copilot 1.0 introduction both explore that workflow. The code still deserves the same review: inspect the settings, run the shell, and decide where the data rules live.

What comes next: make the spider easier to change

In the next article, we will take the product extraction out of the callback and use scrapy-poet Page Objects to make it reusable and testable. That is where Scrapy's extension ecosystem starts paying for itself, because the framework gives you a clear place to put project-wide behavior before you need a larger architecture.

For now, take one spider you already have and make four changes: move its stable behavior into settings.py, replace its important print() calls with logs, move one shared data rule into a pipeline, and configure a feed export. Those are small changes, but they are the difference between a script that scraped a site once and a Scrapy project you can trust to revisit.

Your scraping framework already exists: it's Scrapy

John Rooney — Sun, 26 Jul 2026 06:35:40 +0000

The first scraper is usually honest about what it is. It fetches a page, runs a few selectors, and writes some records. Then it needs pagination, so you add a queue. A transient failure loses a page, so you add retries. Two URLs produce the same content, so you add duplicate filtering. The site slows down, so you add per-domain concurrency and delays. Soon the script has lifecycle hooks, plugins, metrics, browser workers, and a home-grown configuration system.

At that point, you are no longer writing a scraper. You are designing a crawling framework, and many of the decisions in front of you have already been made, tested, and exposed as replaceable components in Scrapy.

This does not mean every 20-line script should become a Scrapy project. It means that when a collection of scripts starts acquiring a runtime, you should ask whether that runtime is the part worth inventing. In most cases, the source-specific discovery and extraction logic is yours. The scheduler, retry machinery, throttling, lifecycle, statistics, exports, and component boundaries do not need to be.

The useful way to think about Scrapy is not as a library that gives you response.css(). Parsel already does that. Scrapy is an event-driven crawling runtime with a stable place to put each concern.

The framework appears one fix at a time

Imagine that a scheduled product scraper is growing beyond its original loop. It now needs to remember which URLs it has seen, pause without losing its queue, retry transport failures, apply different concurrency limits to different domains, normalize every product, split large exports into batches, report statistics, and fail the run when output quality drops.

Those requirements tend to produce classes with names such as QueueManager, RetryingClient, PluginRegistry, and RunMonitor. None of those names is inherently a mistake, but together they are a sign that the project is recreating a framework around its parsing code.

Scrapy already divides that system along practical boundaries:

What you are about to build	Scrapy component
URL queue and priority rules	Scheduler
Duplicate URL tracking	Duplicate filter
Retries, redirects, cookies, proxies	Downloader middleware
HTTP, browser, or custom network transport	Download handler
Site navigation and discovery	Spider
Response-to-object extraction	Spider callback or `scrapy-poet` Page Object
Validation, normalization, persistence	Item pipeline
Lifecycle hooks and operational behavior	Signals and extensions
Counters and run metadata	Stats collector
JSON Lines, CSV, object storage, batching	Feed exports

The boundaries matter more than the names. They let you replace the network transport without replacing the scheduler, or add data-quality checks without importing monitoring code into every callback.

flowchart LR
    Spider["Spider<br/>discovery"] --> Engine["Execution engine"]
    Engine --> Scheduler["Scheduler<br/>queue + dupefilter"]
    Scheduler --> Engine
    Engine --> Middleware["Downloader middleware<br/>retry + redirect + cookies"]
    Middleware --> Handler{"Download handler"}
    Handler --> HTTP["Scrapy HTTP"]
    Handler --> Wreq["wreq"]
    Handler --> Browser["Playwright"]
    HTTP --> Response["Response"]
    Wreq --> Response
    Browser --> Response
    Response --> Poet["Spider / scrapy-poet"]
    Poet --> Pipeline["Item pipelines"]
    Pipeline --> Export["Feed export / database"]
    Signals["Signals + stats"] -. observe .-> Engine
    Signals -. assert .-> Monitor["Core extensions + Spidermon"]

Here is the spider at the centre of that runtime. It contains the site-specific parts: where to begin, how to find product links, and what to yield from a product page.

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products",
            callback=self.parse_listing,
        )

    def parse_listing(self, response):
        yield from response.follow_all(
            response.css("[data-product-link]::attr(href)"),
            callback=self.parse_product,
        )

        next_page = response.css("[rel='next']::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse_listing)

    def parse_product(self, response):
        yield {
            "url": response.url,
            "name": response.css("[data-product-name]::text").get(),
            "price": response.css("[data-price]::attr(data-price)").get(),
        }

Scrapy's selectors use Parsel, so selectors from a smaller Parsel-based scraper move across with little ceremony. The larger change is that callbacks yield work to an engine instead of performing that work directly. The engine can now schedule, deduplicate, retry, throttle, record, and resume it.

The examples below target the current Scrapy async component APIs and Python 3.11 or later, which the Python wreq package requires. In a real project, pin a tested set of versions rather than allowing every integration to update independently.

python -m pip install \
    Scrapy scrapy-poet spidermon scrapy-playwright wreq
playwright install chromium

Start with the boring infrastructure

Before adding an ecosystem package, it is worth seeing how much of the runtime ships with Scrapy itself. A small set of settings gives the product spider adaptive throttling, bounded per-domain concurrency, resumable state, item processing, and compressed batch exports.

# myproject/settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
RETRY_TIMES = 3

ITEM_PIPELINES = {
    "myproject.pipelines.ProductValidationPipeline": 300,
    "myproject.pipelines.ProductUpsertPipeline": 800,
}

FEEDS = {
    "exports/products-%(batch_id)05d.jsonl.gz": {
        "format": "jsonlines",
        "encoding": "utf-8",
        "batch_item_count": 50_000,
        "postprocessing": [
            "scrapy.extensions.postprocessing.GzipPlugin",
        ],
    },
}

AutoThrottle adjusts delay from observed latency for each download slot. Its target concurrency is an average, while CONCURRENT_REQUESTS_PER_DOMAIN remains the ceiling. RetryMiddleware handles configured response codes and download exceptions, and lowers retried requests in the queue. The feed exporter handles serialization, file naming, compression, and batches without making the spider know where output goes.

Resumability is a command-line setting away:

scrapy crawl products -s JOBDIR=crawls/products-2026-07-26

JOBDIR persists the scheduler queue, duplicate-filter state, and spider state so a cleanly stopped crawl can resume. It is not a distributed queue, and a job directory must belong to one spider run. Requests in a disk queue also need serializable callbacks and metadata. These limits are much easier to reason about than an undocumented queue format that grew inside a script.

The item pipeline gives extraction and data handling separate failure modes. The spider describes what it found; ordered pipeline components validate, normalize, drop, or store it.

from decimal import Decimal, InvalidOperation

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class ProductValidationPipeline:
    def process_item(self, item):
        product = ItemAdapter(item)

        if not product.get("name"):
            raise DropItem("missing product name")

        try:
            product["price"] = Decimal(product["price"])
        except (InvalidOperation, TypeError):
            raise DropItem("invalid product price")

        return item

The extension system sits alongside these request and item flows. An extension is normally created once for a crawl and connects to signals such as spider_opened, item_scraped, spider_error, and spider_closed. Scrapy's built-in extensions use the same surface for core stats, periodic log stats, memory limits, close conditions, throttling, feed exports, and other lifecycle behaviour.

That API is small enough that a project-specific extension does not need its own plugin framework:

from scrapy import signals


class ProductStatsExtension:
    def __init__(self, crawler):
        self.stats = crawler.stats

    @classmethod
    def from_crawler(cls, crawler):
        extension = cls(crawler)
        crawler.signals.connect(
            extension.item_scraped,
            signal=signals.item_scraped,
        )
        return extension

    def item_scraped(self, item, response, spider):
        if not item.get("price"):
            self.stats.inc_value("product/missing_price")

EXTENSIONS = {
    "myproject.extensions.ProductStatsExtension": 500,
}

The extension records a fact. It does not decide whether the run should fail. Keeping that distinction makes the counter useful to loggers, dashboards, and monitors without letting one callback quietly define the operational policy for the whole crawl.

scrapy-poet separates navigation from interpretation

As a project adds sources, callbacks often become a mixture of navigation, selectors, normalization, and object construction. scrapy-poet moves page interpretation into Page Objects and injects the right object into a typed callback.

Enable its add-on and tell it where Page Objects live:

ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["myproject.pages"]

The Page Object owns extraction from one kind of page:

# myproject/pages/product.py
from decimal import Decimal

from web_poet import WebPage, handle_urls


@handle_urls("example.com")
class ProductPage(WebPage):
    def to_item(self) -> dict:
        name = self.css("[data-product-name]::text").get()
        price = self.css("[data-price]::attr(data-price)").get()

        return {
            "url": self.url,
            "name": name.strip(),
            "price": Decimal(price),
        }

The spider keeps navigation and asks for the dependency it needs:

import scrapy

from myproject.pages.product import ProductPage


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products/1",
            callback=self.parse_product,
        )

    def parse_product(self, response, page: ProductPage):
        yield page.to_item()

The annotation is not decorative. scrapy-poet inspects it, selects the Page Object using its rules, builds the dependency, and supplies it. You can override rules for a source, inject additional dependencies through providers, and test the Page Object without running a crawl. The spider is left to answer one question: where should the crawler go next?

This separation pays off when several spiders encounter the same page type, or one crawl strategy spans several sites with different representations. It also prevents browser acquisition, API-backed inputs, or cached responses from leaking into extraction code, because those can be provided as dependencies.

One detail matters if you use JOBDIR: callbacks must be serializable. scrapy-poet offers callback_for() to remove boilerplate, but assign the generated callback as a spider method instead of constructing it inline if requests may be written to disk.

Spidermon defines what a successful crawl means

A process can exit with code zero after extracting no products. It can also produce the expected number of records while silently losing every price. Scrapy's stats tell you what happened; Spidermon turns those facts into executable assertions and actions.

Enable the extension and select a monitor suite for spider close:

SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.ProductSpiderCloseSuite",
)

Then express the run contract in terms of the resulting dataset:

from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Product output")
class ProductOutputMonitor(Monitor):
    @monitors.name("Enough products were extracted")
    def test_item_count(self):
        count = getattr(self.data.stats, "item_scraped_count", 0)
        self.assertGreaterEqual(count, 1_000)

    @monitors.name("Missing prices stayed below one percent")
    def test_missing_price_rate(self):
        scraped = getattr(self.data.stats, "item_scraped_count", 0)
        missing = getattr(self.data.stats, "product/missing_price", 0)

        self.assertGreater(scraped, 0)
        self.assertLess(missing / scraped, 0.01)


class ProductSpiderCloseSuite(MonitorSuite):
    monitors = [ProductOutputMonitor]

Spidermon can run suites at spider open, spider close, engine stop, or periodically. Suites can trigger actions after checks pass or fail, and built-in monitors cover common conditions such as item counts, field coverage, finish reasons, retries, errors, and unwanted status codes.

This is a better layer for run-level policy than scattered if count == 0 checks. The spider collects data, components record facts, and monitors decide whether those facts describe useful output.

scrapy-playwright makes the browser an acquisition choice

Some pages do need JavaScript execution or browser state. That does not require the crawler itself to become a browser script. scrapy-playwright implements a Scrapy download handler, so selected requests can use Playwright while the surrounding scheduler, retries, callbacks, pipelines, and monitoring stay in place.

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

DOWNLOAD_HANDLERS = {
    "https": (
        "scrapy_playwright.handler."
        "ScrapyPlaywrightDownloadHandler"
    ),
}

# Let the browser provide a User-Agent that matches itself.
USER_AGENT = None

Requests opt in through metadata. Unmarked requests continue through Scrapy's regular HTTP implementation.

import scrapy
from scrapy_playwright.page import PageMethod


yield scrapy.Request(
    "https://example.com/products",
    callback=self.parse_products,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod(
                "wait_for_selector",
                "[data-products-loaded='true']",
            ),
        ],
    },
)

That opt-in is an important scaling property. Listing pages that already contain the required HTML can remain cheap HTTP requests, while the few pages that need execution consume browser capacity. Prefer bounded PageMethod actions that return a normal Scrapy response. If you request the Playwright Page object directly with playwright_include_page, your callback and errback must close it. Leaked pages eventually exhaust the per-context page limit and stall the crawl.

Browser work also changes what your metrics mean. Browser subresources are not individual Scrapy requests, and Chromium runs in another process. scrapy-playwright provides a memory-usage extension that can account for the browser process where supported. These details are exactly why the browser belongs behind a defined acquisition boundary rather than inside an arbitrary callback.

If the browser integration itself becomes the bulk of the crawler, tools such as Camoufox and Patchright may become relevant for particular sources. They are browser runtimes rather than replacements for Scrapy's scheduling and data pipeline, so the same architectural question remains: can the specialized browser sit behind a narrow boundary, or has browser state become the application?

Write a wreq download handler when the network stack is the variable

TLS and HTTP/2 fingerprints are properties of the client implementation, not just a collection of headers. If a source expects a network profile that resembles a current browser, changing User-Agent in Scrapy's default client does not change its TLS ClientHello, cipher ordering, extension ordering, or HTTP/2 behaviour.

The Python bindings for wreq provide an async client backed by a browser-emulating network stack. A custom download handler lets that client fetch bytes while Scrapy continues to own the crawl.

The following handler targets the current async download-handler API in Scrapy and the Python wreq package. It is a useful starting point, but the caveats after the code are part of the implementation, not optional footnotes.

# myproject/download_handlers.py
from __future__ import annotations

from datetime import timedelta

import wreq.exceptions
from scrapy import Request
from scrapy.core.downloader.handlers.base import BaseDownloadHandler
from scrapy.exceptions import DownloadFailedError, DownloadTimeoutError
from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from wreq import Client, Emulation, Method, Proxy, redirect
from wreq.header import HeaderMap


WREQ_NETWORK_ERRORS = (
    wreq.exceptions.BodyError,
    wreq.exceptions.BuilderError,
    wreq.exceptions.ConnectionError,
    wreq.exceptions.ConnectionResetError,
    wreq.exceptions.DecodingError,
    wreq.exceptions.ProxyConnectionError,
    wreq.exceptions.RequestError,
    wreq.exceptions.TlsError,
)


class WreqDownloadHandler(BaseDownloadHandler):
    def __init__(self, crawler):
        super().__init__(crawler)
        self.default_timeout = crawler.settings.getfloat(
            "DOWNLOAD_TIMEOUT"
        )
        profile_name = crawler.settings.get(
            "WREQ_EMULATION",
            "Chrome149",
        )

        self.client = Client(
            emulation=getattr(Emulation, profile_name),
            # RedirectMiddleware should receive 30x responses.
            redirect=redirect.Policy.none(),
            # CookiesMiddleware remains the cookie owner.
            cookie_store=False,
        )

    async def download_request(self, request: Request) -> Response:
        method = getattr(Method, request.method, None)
        if method is None:
            raise DownloadFailedError(
                f"wreq does not support {request.method!r}"
            )

        headers = HeaderMap()
        for name, values in request.headers.items():
            header_name = name.decode("latin-1")
            for value in values:
                headers.append(
                    header_name,
                    value.decode("latin-1"),
                )

        timeout = request.meta.get(
            "download_timeout",
            self.default_timeout,
        )
        kwargs = {
            "headers": headers,
            "body": request.body,
            "timeout": timedelta(seconds=timeout),
        }

        if proxy_url := request.meta.get("proxy"):
            kwargs["proxy"] = Proxy.all(proxy_url)

        try:
            async with await self.client.request(
                method,
                request.url,
                **kwargs,
            ) as response:
                body = await response.bytes()
                response_headers = Headers(list(response.headers))
                response_url = str(response.url)
                response_status = response.status.as_int()
                response_protocol = str(response.version)
        except wreq.exceptions.TimeoutError as exc:
            raise DownloadTimeoutError(str(exc)) from exc
        except WREQ_NETWORK_ERRORS as exc:
            raise DownloadFailedError(str(exc)) from exc

        response_cls = responsetypes.from_args(
            headers=response_headers,
            url=response_url,
            body=body,
        )

        return response_cls(
            url=response_url,
            status=response_status,
            headers=response_headers,
            body=body,
            request=request,
            protocol=response_protocol,
        )

    async def close(self) -> None:
        self.client.close()

DOWNLOAD_HANDLERS = {
    "http": "myproject.download_handlers.WreqDownloadHandler",
    "https": "myproject.download_handlers.WreqDownloadHandler",
}

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

WREQ_EMULATION = "Chrome149"

# Do not pair a browser transport profile with Scrapy's UA.
USER_AGENT = None

There are several deliberate ownership decisions here. wreq does not follow redirects, because Scrapy's RedirectMiddleware should see the 302, enforce its limits, adjust priority, and record redirect statistics. wreq's cookie store is disabled, because Scrapy's CookiesMiddleware should remain the single source of cookie state. Transport errors become Scrapy download exceptions, so the existing retry middleware can handle them.

Headers need the same care. A Chrome TLS and HTTP/2 profile combined with contradictory User-Agent, Accept, Accept-Language, or client-hint headers is not a coherent Chrome request. In a real implementation, decide which browser-shaped headers wreq owns, then filter those headers out of the Scrapy request before building HeaderMap. Keep source-specific headers, authentication, and cookies. Test the actual wire output rather than assuming the two layers compose cleanly.

The example buffers the entire response. It therefore does not yet implement DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE, Scrapy's headers_received and bytes_received signals, or StopDownload. A production handler should stream wreq chunks, apply size checks before and during the body, and propagate cancellation. It should also map DNS, connection refusal, data loss, and unsupported methods to the most specific Scrapy exceptions available.

Compression deserves an explicit test too. Both the HTTP client and Scrapy middleware may believe they own gzip, Brotli, deflate, or zstd decoding. Choose one owner, then verify that the body is decoded once and Content-Encoding still describes the bytes Scrapy receives.

Before using the handler in a real crawl, run it against a local HTTP fixture that covers at least:

HTML response-class selection, duplicate response headers, and request attachment.
POST bodies and repeated request headers.
A redirect handled by Scrapy rather than wreq.
A retryable 500 and a transport timeout.
Per-request timeout and proxy metadata.
gzip and Brotli responses decoded exactly once.
A large streamed body stopped at the configured size.
Handler close and in-flight cancellation.

This list is longer than await client.get(url) because a download handler is a transport contract. That is also the point of the exercise. Scrapy lets you replace this one contract without throwing away the rest of the crawling runtime.

Scrapy is a runtime, not the whole platform

Scrapy does not make every architecture decision disappear. Its default scheduler is not a distributed queue shared by a fleet of workers. JOBDIR is resumability for one crawl, not global orchestration. A database pipeline still needs a data model, idempotent writes, and migration policy. Multiple processes still need shared traffic budgets if they target the same domain. Browser capacity, proxy health, deployment, secrets, and backfills still need an operating model.

Source-specific extraction remains real engineering too. scrapy-poet gives that work a clean home, but it cannot decide what a price means, whether two variants are the same product, or how to detect a plausible-looking error page.

The narrower claim is more useful: Scrapy provides a mature runtime and a set of extensible boundaries around those decisions. Build the parts that distinguish your data product. Extend a known scheduler, downloader, pipeline, and lifecycle before writing new versions of all four.

A practical migration path

Do not begin by rewriting a working scraper into a perfectly abstract project. Pick one scheduled script that has already accumulated reliability code and move it in layers:

Put discovery and parsing into a spider, leaving existing Parsel selectors intact.
Let the scheduler and duplicate filter own request flow, then enable JOBDIR for resumability.
Remove the local retry loop and use downloader middleware plus Scrapy download exceptions.
Move validation and persistence into ordered pipelines.
Record source-specific facts in stats and assert the run contract with Spidermon.
Move reusable extraction into scrapy-poet Page Objects.
Add Playwright or a wreq handler only for requests that prove they need a different acquisition path.

At each step, delete a piece of home-grown runtime. If the migration only adds Scrapy beside the old queue, retry client, plugin system, and exporter, it has missed the opportunity.

Frequently asked questions

Is Scrapy too old for modern Python and async code?

No. Current Scrapy has coroutine support across major component APIs and an async download-handler interface. Its Twisted history is visible, but it does not prevent integration with asyncio-based clients and browser tooling. Version compatibility still matters, so pin Scrapy and ecosystem packages together and test reactor configuration.

Can one spider mix normal HTTP, Playwright, and wreq?

Yes, but handler selection is scheme-based. scrapy-playwright registers an HTTPS handler that delegates unmarked requests to Scrapy's normal implementation. A project that also wants wreq for selected HTTPS requests needs one dispatching handler or separate schemes/domains routed by a custom handler. Two different classes cannot both occupy the same https entry in DOWNLOAD_HANDLERS.

When is a plain script still the better choice?

Use a plain script when the input is small and known, the run is disposable, failure has little cost, and no one needs to resume or operate it. Scrapy earns its place when the crawler develops state, repeated execution, multiple request paths, operational expectations, or reusable components.

Does Scrapy solve scaling across machines?

Not by itself. Scrapy gives each crawler process strong internal boundaries and replaceable components. A distributed deployment still needs orchestration, shared state where appropriate, and a policy for global traffic and data consistency. The framework makes those integrations narrower; it does not pretend they are free.

Build your scraper, not another crawling framework

In the previous post, the small scraper grew into a system as its failure cost increased. This is the natural next decision: once you need that system, decide which parts are genuinely specific to your source and which are generic crawling infrastructure.

Scrapy will not write the difficult selectors or define the meaning of your data. It will give those decisions a scheduler, lifecycle, retry path, extension model, statistics layer, and several escape hatches. In many projects, that is the framework you were about to spend the next year discovering one production incident at a time.

So you want to get data from the web

John Rooney — Sun, 26 Jul 2026 06:14:44 +0000

Getting data from one web page can take 20 lines of Python. Keeping a trustworthy dataset alive while the source changes, rate limits you, returns partial responses, and occasionally serves a consent page with a 200 OK is a different engineering problem, because the selector is rarely as difficult as knowing when the result is wrong.

What web scraping becomes once someone depends on it

Web scraping is automated retrieval and extraction of data from web resources. At the small end, that means an HTTP request followed by an HTML parser. At the other end, the same basic operation sits inside a distributed acquisition pipeline with scheduling, traffic controls, schema validation, raw snapshots, observability, and a recovery path. Both are web scraping, but they should not have the same architecture.

The easiest way to choose that architecture is to classify the job by its failure cost:

Workload	A reasonable starting point	The first problems you will meet
A few known pages, needed once	One process and local output	Missing fields, pagination, duplicate URLs, odd encodings
A scheduled crawl of one site	A repeatable job with persistent state	Timeouts, rate limits, reruns, template changes
A JavaScript application	HTTP plus targeted browser automation where required	Background requests, timing, cookies, session expiry
Many sources feeding another product	A queue-based data pipeline	Source drift, partial runs, backfills, data contracts, cost control

This is the useful complexity ladder. A one-off question does not need a queue, six workers, and an observability stack, while a revenue-critical feed should probably graduate from a laptop before print() becomes its monitoring strategy.

A small Python scraper should leave evidence behind

The first version should already separate fetching from parsing because that boundary lets you test selectors without hitting the site again and distinguish a network failure from an extraction failure. It also means that when the markup changes, you can fix the parser against a saved response instead of repeatedly fetching the same page while you debug it.

Here is a complete example for a static product page. It uses Parsel, expects semantic attributes such as data-product-name and data-price, saves the raw response to a path chosen by the caller, and returns a typed record. Parsel is also the selector layer underneath Scrapy, so the CSS and XPath expressions carry across when the crawler grows into a Scrapy project.

Install the two dependencies first:

python -m pip install requests parsel

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path
from urllib.parse import urldefrag

import requests
from parsel import Selector


@dataclass(frozen=True)
class Product:
    source_url: str
    name: str
    price: Decimal
    fetched_at: str


def parse_product(
    html: str,
    *,
    source_url: str,
    fetched_at: str,
) -> Product:
    selector = Selector(text=html)
    root = selector.css("[data-product]")

    if not root:
        raise ValueError("missing product container")

    name = root.css("[data-product-name]::text").get()
    raw_price = root.css("[data-price]::attr(data-price)").get()

    if name is None or raw_price is None:
        raise ValueError("missing required product field")

    name = name.strip()

    try:
        price = Decimal(raw_price)
    except InvalidOperation as exc:
        raise ValueError("invalid product price") from exc

    if not name or not price.is_finite() or price < 0:
        raise ValueError("product failed validation")

    return Product(
        source_url=source_url,
        name=name,
        price=price,
        fetched_at=fetched_at,
    )


def fetch_product(url: str, snapshot_path: Path) -> Product:
    response = requests.get(
        url,
        headers={"User-Agent": "ExampleDataCollector/1.0 contact@example.com"},
        timeout=(5, 20),
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")
    if "text/html" not in content_type:
        raise ValueError(f"expected HTML, received {content_type!r}")

    fetched_at = datetime.now(timezone.utc).isoformat()
    source_url, _fragment = urldefrag(response.url)

    snapshot_path.parent.mkdir(parents=True, exist_ok=True)
    snapshot_path.write_bytes(response.content)

    return parse_product(
        response.text,
        source_url=source_url,
        fetched_at=fetched_at,
    )

The timeout has separate connection and read limits, raise_for_status() prevents a 404 page from flowing into the parser, and the content-type check catches JSON, images, and some challenge pages before Parsel sees them. The raw body is stored before parsing, so an extraction error does not destroy the evidence needed to reproduce it.

There is no content hash here because this example does not need one. The caller can name snapshots by run ID, URL key, or whatever fits the job. Hashes become useful later, when the system needs to detect unchanged responses, deduplicate stored bodies, or connect a parsed record to the exact bytes that produced it. Introducing that machinery here would explain the production architecture before we have earned it.

The parser rejects invalid records instead of returning a half-populated dictionary. Returning None for a missing price feels convenient until it reaches an aggregate and quietly changes the answer.

The test does not need the network:

from decimal import Decimal


def test_parse_product() -> None:
    html = """
    <article data-product>
      <h1 data-product-name>Mechanical Keyboard</h1>
      <span data-price="89.95">£89.95</span>
    </article>
    """

    product = parse_product(
        html,
        source_url="https://example.com/products/keyboard",
        fetched_at="2026-07-26T10:00:00+00:00",
    )

    assert product.name == "Mechanical Keyboard"
    assert product.price == Decimal("89.95")

This is the first architectural boundary worth keeping: fetchers produce source material and metadata, parsers turn that material into validated records, and storage comes after validation.

JavaScript rendering should be an explicit escalation

Some pages return the data in the initial HTML, while others return an application shell and fetch the useful payload after JavaScript runs. Check the response body and browser network panel before reaching for a headless browser because structured data embedded in the document or a background JSON response is usually easier to operate than a full browser session.

Playwright is excellent when you genuinely need rendering or interaction. It also turns deterministic request code into a small distributed-systems problem living inside Chromium, complete with page lifecycle events, async UI updates, browser crashes, context cleanup, and much higher CPU and memory use.

When a browser is justified, wait for a condition that means the data is ready. networkidle is often a weak signal because analytics, polling, and streaming requests can keep the network busy indefinitely.

Install Playwright and its Chromium build before running the example:

python -m pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright


def fetch_rendered_html(url: str) -> str:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        page = browser.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            page.locator("[data-product-loaded='true']").wait_for(
                state="attached",
                timeout=10_000,
            )
            return page.content()
        finally:
            browser.close()

The selector is intentionally site-specific. Replace it with a marker that appears only when the content you need is present, or wait for the relevant response. Playwright's auto-waiting documentation explains what its actions wait for automatically, while its evaluation guide covers the boundary between Python and code running in the page.

Standard Playwright also has a recognisable automation surface. If a site reacts differently to an automated browser than to the corresponding normal browser, stealth-oriented projects are another rung on the complexity ladder. Camoufox is a Firefox fork with Playwright compatibility and browser-level fingerprint changes, while Patchright patches the Chromium side of Playwright and presents itself as a drop-in replacement. Both are moving projects with their own compatibility gaps, release cadence, and operational cost, so treat them as specialised browser runtimes rather than magic flags. We will leave the comparison there because each deserves a separate article.

Retry transient failures without turning a scrape into a traffic spike

Once a job runs on a schedule, failures become normal input: DNS lookups fail, connections reset, upstream services return 503, and rate limits change during the day. Retries help with those transient failures, but unbounded or immediate retries make incidents worse.

requests can use urllib3's retry policy through an adapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist={429, 500, 502, 503, 504},
        allowed_methods={"GET", "HEAD"},
        respect_retry_after_header=True,
        raise_on_status=False,
    )

    adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

With a backoff factor of 0.5, urllib3 applies increasing delays between eligible retries. The final response still needs raise_for_status(). A 403 or 404 is not in the retry list because repeating a permanent failure four times adds load without changing the answer.

HTTP 429 Too Many Requests may include a Retry-After header. RFC 6585 leaves the server free to count requests by credential, cookie, client, resource, or another policy, which means changing one network attribute does not necessarily reset the limit.

At more than one worker, enforce concurrency and delay per host across the whole worker pool. A local sleep(1) inside each worker gives you ten requests per second when you scale to ten workers. Put the host budget in shared state or route each host through its own queue.

TLS fingerprinting starts before the server sees your headers

Changing the User-Agent header does not make an HTTP client behave like a browser. Before the server receives the HTTP request, the client has already sent a TLS ClientHello containing an ordered set of cipher suites, extensions, supported groups, signature algorithms, and ALPN values. Those choices, along with their ordering, can be summarised as JA3 or JA4 fingerprints. HTTP/2 adds another layer through SETTINGS values, connection behaviour, and pseudo-header ordering.

This is why a request can claim to be Firefox in its headers while looking nothing like Firefox on the wire. A conventional Python client is designed to make correct HTTP requests, not to reproduce a browser's network stack. For many sites that distinction does not matter. On sites that classify traffic using several signals, it can explain why copying browser headers changes nothing.

wreq exposes Python bindings over the Rust client, so there is no reason to leave the Python stack just to control the network fingerprint. It uses BoringSSL and supports TLS, JA3/JA4, HTTP/2, and header-order profiles. The current package ships more than 100 browser emulation profiles and offers both async and blocking clients.

Install wreq alongside Parsel:

python -m pip install wreq parsel

import asyncio

from parsel import Selector
from wreq import Client, Emulation


async def fetch_title(url: str) -> str:
    client = Client(emulation=Emulation.Chrome149)
    response = await client.get(url)

    if response.status >= 400:
        raise RuntimeError(f"request failed with status {response.status}")

    selector = Selector(text=await response.text())
    title = selector.css("title::text").get()

    if title is None:
        raise ValueError("response did not contain a title")

    return title.strip()


if __name__ == "__main__":
    print(asyncio.run(fetch_title("https://example.com")))

The profile needs to be internally consistent. If the TLS handshake resembles one browser version while the headers, operating system hints, cookies, or JavaScript-visible properties describe another, the inconsistency becomes a signal of its own. Keep a client alive across requests so connection pooling and cookies behave consistently, pin the package and profile versions you have tested, and expect fingerprints to age as browsers and detection systems change.

wreq solves the network-stack part of the problem rather than JavaScript rendering. If the page needs a browser, use one; if the HTML is already in the response and the mismatch is at TLS or HTTP/2, launching Chromium adds a large amount of machinery to solve a lower-level issue.

A production scraper is a replayable data pipeline

Once the output feeds search, analytics, pricing, or a customer-facing feature, the system needs more than fetch workers. A practical design looks like this:

flowchart LR
    S["Scheduler: cron or Airflow"] --> Q["Host-aware URL queues: SQS or Redis"]
    Q --> F["HTTP fetchers: requests or wreq"]
    Q --> B["Browser fetchers: Playwright, Camoufox, or Patchright"]
    F --> RAW["Raw snapshots and metadata: S3"]
    B --> RAW
    F --> RETRY["Retry and review queue"]
    B --> RETRY
    RAW --> P["Versioned parser workers"]
    P --> V["Schema and run-quality gates"]
    V --> DB["Idempotent records: PostgreSQL"]
    DB --> C["Search, analytics, or product consumers"]
    F -. metrics .-> O["Metrics, logs, traces, and alerts"]
    B -. metrics .-> O
    P -. metrics .-> O
    V -. metrics .-> O

The names are examples, not requirements. A single VM, SQLite, and a filesystem can implement the same boundaries for a smaller workload. The boundaries matter because they let you retry and replace one stage without repeating every other stage.

Raw snapshots make parser changes cheap

Store response bytes with the source URL, final URL after redirects, fetch timestamp, status, content type, relevant response headers, and a content hash. Add the fetcher and parser version to the eventual record. At this point the hash has a clear job: it gives the raw body a stable identity, makes unchanged responses cheap to detect, and links each parsed record back to the exact bytes that produced it.

When a selector breaks, rerun the new parser against stored responses. This is faster, cheaper, and kinder to the source than fetching every URL again. It also lets you compare old and new parser output before a backfill reaches production.

Retention is a policy decision, especially if pages contain personal or sensitive data. An immutable archive is useful engineering infrastructure. It is not a reason to keep everything forever.

Idempotent writes make retries safe

A worker can fail after writing a record but before acknowledging its queue message. The message will be delivered again. If the second write creates another row, your retry policy has become a duplication policy.

Use a stable key, such as (source, external_id), and upsert. This PostgreSQL statement uses SQLAlchemy-style named parameters:

INSERT INTO products (
    source,
    external_id,
    name,
    price,
    fetched_at,
    content_hash,
    parser_version
)
VALUES (
    :source,
    :external_id,
    :name,
    :price,
    :fetched_at,
    :content_hash,
    :parser_version
)
ON CONFLICT (source, external_id) DO UPDATE SET
    name = EXCLUDED.name,
    price = EXCLUDED.price,
    fetched_at = EXCLUDED.fetched_at,
    content_hash = EXCLUDED.content_hash,
    parser_version = EXCLUDED.parser_version;

This gives you the latest record. If you also need history, write changes to an append-only version table or event log. Do not overload one table to be both current state and an audit trail.

Quality gates should fail the run, not merely decorate a dashboard

Transport metrics tell you whether requests completed, while data metrics tell you whether the output is believable.

Track at least:

attempted, succeeded, retried, and finally failed URLs by host
status-code distribution and request latency
records produced, required-field completeness, and uniqueness
new, changed, and missing records relative to the previous successful run
parser errors grouped by source and parser version

Then define stop conditions. If yesterday's crawl produced 48,000 products and today's produces 730, publishing the result should require an explanation. The threshold depends on the source, but the decision must exist in code rather than in someone's memory.

Here is a small quality gate with deliberately configurable limits:

from dataclasses import dataclass


@dataclass(frozen=True)
class RunStats:
    attempted: int
    succeeded: int
    records: int
    records_with_required_fields: int


def validate_run(
    current: RunStats,
    *,
    previous_records: int,
    min_success_rate: float = 0.95,
    min_completeness: float = 0.99,
    min_volume_ratio: float = 0.70,
) -> None:
    if current.attempted <= 0 or current.records <= 0:
        raise ValueError("run produced no usable work")

    success_rate = current.succeeded / current.attempted
    completeness = current.records_with_required_fields / current.records
    volume_ratio = current.records / max(previous_records, 1)

    failures = []
    if success_rate < min_success_rate:
        failures.append(f"success rate {success_rate:.1%}")
    if completeness < min_completeness:
        failures.append(f"completeness {completeness:.1%}")
    if volume_ratio < min_volume_ratio:
        failures.append(f"volume ratio {volume_ratio:.1%}")

    if failures:
        raise ValueError("run failed quality gates: " + ", ".join(failures))

Those defaults are examples, not universal scraping constants. A news index and a product catalogue have different expected churn. Start by observing normal variation, then choose thresholds that catch abnormal runs without paging someone every morning.

The shortcuts that fail when the workload grows

Several designs work just long enough to become expensive:

Putting fetch and parse logic in one function. You cannot test a parser without network traffic, and every parser fix requires another crawl.

Using a browser for every URL. It simplifies the first implementation when some pages need JavaScript, then spends far more CPU and memory on pages that do not. Keep HTTP and browser fetchers behind the same response interface, and route only the pages that need rendering.

Retrying every error. A timeout may recover. A malformed selector, an authentication failure, or a deleted page will not. Classify failures and send permanent or suspicious ones to review.

Scaling with global concurrency. One large domain and one tiny domain do not have the same capacity or rules. Global concurrency lets the busiest host consume the whole budget and makes request pressure unpredictable.

Treating row count as the only quality metric. A parser can return the expected number of rows with the wrong field in every row. Validate types, ranges, uniqueness, and changes in field-level completeness.

The common thread is lost evidence. Once the fetch, parser, and output are entangled, you cannot tell which part failed without repeating the entire job.

How to choose the smallest architecture that will survive

Use one script when all of these are true: the page set is small, the source is stable, the work is easy to rerun, and a bad result has limited consequences. Even then, save the raw input and validate required fields.

Add persistent state and scheduling when runs repeat. Add browser automation only for pages that require it. Add queues when one process no longer meets the throughput or isolation you need. Add separate parser workers when you want to replay stored responses without fetching again.

The forcing function is not URL count by itself. Ten authenticated, stateful workflows can be harder to operate than a million static pages from a stable bulk source.

Before increasing concurrency, write down:

What is the per-host request budget?
Which failures are safe to retry?
Which quality signal stops publication?
What raw evidence and version metadata will you retain?
How will you replay or backfill without duplicating records?

If those answers are vague, more workers will mostly help you fail faster.

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of `requests`?

Use requests or httpx when the URL set is small and you can manage scheduling, retries, deduplication, and persistence directly. A framework such as Scrapy earns its place when you need concurrent crawling, downloader middleware, per-domain controls, request deduplication, item pipelines, and a consistent extension model. A framework removes plumbing, not the need to understand the source.

When should I use Playwright for web scraping?

Use Playwright when data only appears after browser-side execution or requires an interaction that a plain HTTP client cannot reproduce. Do not use it by default for static pages. Inspect the initial HTML and background requests first, then route only the pages that require rendering through a browser pool.

How do I know whether a scraper is returning bad data?

Validate each record against a schema, then compare run-level metrics with the previous successful run. Required-field completeness, type and range checks, uniqueness, record volume, and the number of changed or missing entities catch different classes of failure. Keep raw snapshots so you can reproduce the parser's decision.

How do you scale web scraping without overwhelming a site?

Set concurrency and request rates per host across all workers, respect 429 and Retry-After, cache or fetch incrementally where appropriate, and use bounded retries for transient failures. Scale the scheduler and parser independently from the fetch rate. More parsing capacity does not require more traffic to the source when raw responses are replayable.

Start with a replay test

Take one representative page, save the raw response, and make the parser pass against that fixture. Then change or remove one required field and confirm the test fails for the reason you expect.

That small exercise tells you whether you are building a script that happens to return data today, or a system you can still explain after the source changes tomorrow.

How to use Camoufox with Scrapy (New EASY Method)

John Rooney — Sat, 25 Jul 2026 11:20:24 +0000

A question comes up often enough in the web scraping community that it has a familiar shape to it: I like working in scrapy-playwright, but I want to run camoufox, or another Python browser library, instead of the browser it ships with, so how do I actually wire that in? For a long time the honest answer was some version of "you can, but it isn't pretty." People raised it in GitHub issues and talked around it on forums, and the workarounds that existed all asked you to figure out a way to run that browser separately and use CDP.

That has changed with a small addition to scrapy-playwright: PLAYWRIGHT_BROWSER_PROVIDER, that hands the browser lifecycle to whatever Playwright-compatible package you want to use. It’s a small Python class that integrates into Scrapy and scrapy-playwright. It is the kind of change that reads like a one-line entry in a changelog and quietly removes a frustration a lot of people had learned to live with.

Why the stock browsers get caught

To see why this matters, it helps to remember what people are reaching for a different browser to do. Playwright is excellent at driving a browser, and scrapy-playwright wires that capability into Scrapy at the download handler so a rendered page arrives through the normal request and response cycle, asynchronously and quickly. The catch is the binary. The Chromium and Firefox builds that Playwright ships are the standard, well-known ones, which means even fairly basic anti-bot systems recognize them almost immediately, and the sites you most want to reach are rarely running basic anti-bot systems. These browsers were designed for automated testing, and no scraping specifically.

Getting around that recognition used to be a matter of patching the browser at runtime, but that has stopped being enough. The more effective projects now change the browser at the source and compile their modifications into the binary itself, instead of patching a stock build once it is running. Runtime patches tend to leave small inconsistencies behind, and modern fingerprinting is built to notice exactly those, in the same way that the network fingerprint has become a central battleground between scrapers and site operators. Each mismatch lowers the trust a site assigns to your session and makes a challenge more likely, so the goal is to present a browser with nothing out of place. Building the changes at compile time is harder to detect, and because Chromium and Firefox expose different things, what needs changing is different in each. This is one important part of how modern stealth browsers work, not the whole story, but it is enough to explain why swapping the browser is worth the effort, especially when anti-bot vendors now ship changes at a pace that punishes any setup relying on manual tuning.

The middle ground that was missing

scrapy-playwright already had one good answer for using a different browser: connect to one that is running somewhere else. Through PLAYWRIGHT_CDP_URL or Playwright's own connect endpoint, you point the plugin at an already-running browser over the network, and that route is genuinely useful when the browser lives on a paid service or on infrastructure you run yourself. I have written before about standing up a self-hosted browser service and connecting Playwright to it over a WebSocket, and it works well, but it is worth being clear about what it asks of you. You have to run the browser in debug mode, keep that instance alive, watch it for memory leaks, and restart it when it dies, which is a reasonable amount of operational surface for what started as "I just want to use camoufox locally."

That was the gap. Between the stock browser that gets caught and a full separate browser service that you connect to remotely, there was a large group of people who simply wanted a stealthier browser running inline with their spider, on their own machine, without building a service around it. The workarounds they were left with were the ugly ones. One approach was to swap the binary by hand: download the original, remove it, and drop the replacement in where Playwright expected to find it, which is as fragile as it sounds and breaks the moment anything updates. Another leaned on the fact that some of these browsers shipped an experimental built-in server you could expose and connect to, which was fine until you needed one of the browsers that had no such server, at which point you were back to no clean route at all. That last point bites hardest for the Firefox-based options, because the connect-to-a-running-browser story was never as clean for them as it was for Chromium.

How browser providers actually work

The new setting works by naming a provider, which is just a class that owns starting and stopping the browser. The default provider wraps vanilla Playwright and gives you everything the README already documented, and if you want a different backend you point PLAYWRIGHT_BROWSER_PROVIDER at a class of your own. The provider implements a small asynchronous lifecycle, start, launch_browser, launch_persistent_context, and close, and because packages like camoufox, patchright, and invisible_playwright keep the standard Browser, BrowserContext, and Page objects, everything downstream of the launch, your pages, contexts, and routing code, keeps working unchanged. Only how the browser starts and stops is different.

Here is the camoufox provider from the documentation, which is the shape almost any provider will take:

from contextlib import AsyncExitStack

from scrapy_playwright.handler import Config, PERSISTENT_CONTEXT_PATH_KEY

class CamoufoxBrowserProvider:
    def __init__(self, config: Config) -> None:
        self.config = config
        self.stack = AsyncExitStack()

    async def start(self) -> None:
        pass

    async def launch_browser(self):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(**self.config.launch_options)
        )

    async def launch_persistent_context(self, context_kwargs: dict):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(
                persistent_context=True,
                user_data_dir=context_kwargs[PERSISTENT_CONTEXT_PATH_KEY],
                **self.config.launch_options,
            )
        )

    async def close(self) -> None:
        await self.stack.aclose()

Point the settings at it, and because camoufox is Firefox-based, set the browser type to match:

# settings.py
PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CamoufoxBrowserProvider"
PLAYWRIGHT_BROWSER_TYPE = "firefox"

I ran this exact example from the documentation, and it works. That is roughly all there is to it: one class, two settings, and the stealth browser is running inline with your spider. Importing the third-party library lazily inside the methods that need it, as the example does, is a nice detail, because it means the setting can point at a provider whose backend is only installed in some of your environments without breaking the others.

Why this fits the way Scrapy is built

None of this is an accident of design. Scrapy is lean on purpose. It is a full framework, but the core deliberately avoids telling you which browser to use, which extraction approach to take, or how your infrastructure should look, and it pushes those decisions out to extensions instead, some official and some from the community. Browser providers extend that same principle right down to the choice of browser binary, which is a very Scrapy way to solve the problem: rather than blessing one stealth browser, the plugin adds a seam and lets you bring your own.

It is worth saying where scrapy-playwright sits in that picture, because it shapes why this addition feels significant rather than incidental. The plugin lives in the scrapy-plugins organization and has been driven largely by a colleague at Zyte, which puts it in an interesting spot, not a headline product with a marketing page, but not a random third-party bolt-on either. Seeing a genuinely useful capability land in a project like that, one that many people had quietly filed under "mature, does what it does," is a good reminder that the tools we lean on are still being improved by people who hit the same friction we do.

What it changes, and what it doesn't

I would put this on the "relatively big deal" side of the ledger rather than the nice-to-have side, precisely because of the group it serves. If you were already connecting to a remote browser service, your setup does not change, and if you only ever needed the stock browser, this will pass you by. But for the sizable middle, the people who wanted a stealthier browser running locally with scrapy-playwright and were stuck with fragile workarounds, this turns a hack into a configuration choice, and lowering the cost of that experiment is the real value here.

A couple of honest caveats belong alongside that. The provider examples in the documentation are offered as guidelines, not supported parts of scrapy-playwright, and the plugin is not affiliated with camoufox, patchright, or any of the other projects it references, so you should read each project's own documentation and evaluate any dependency yourself before you rely on it. Swapping in a stealth browser is also not a finish line. It removes one common way of being spotted, but it sits inside a larger system of proxies, behavior, and session management, and treating it as a single switch that solves blocking will lead you astray. When you would rather not run and maintain any of this yourself, Zyte API's headless browser handles the binary, the fingerprinting, and the proxy rotation behind a single request, which is the right answer for a different set of problems than the one browser providers solve. What providers give you is a clean way to make the choice yourself, and for a lot of people that is exactly what was missing.