DEV Community: John Rooney

Your scraping framework already exists: it's Scrapy

John Rooney — Sun, 26 Jul 2026 06:35:40 +0000

The first scraper is usually honest about what it is. It fetches a page, runs a few selectors, and writes some records. Then it needs pagination, so you add a queue. A transient failure loses a page, so you add retries. Two URLs produce the same content, so you add duplicate filtering. The site slows down, so you add per-domain concurrency and delays. Soon the script has lifecycle hooks, plugins, metrics, browser workers, and a home-grown configuration system.

At that point, you are no longer writing a scraper. You are designing a crawling framework, and many of the decisions in front of you have already been made, tested, and exposed as replaceable components in Scrapy.

This does not mean every 20-line script should become a Scrapy project. It means that when a collection of scripts starts acquiring a runtime, you should ask whether that runtime is the part worth inventing. In most cases, the source-specific discovery and extraction logic is yours. The scheduler, retry machinery, throttling, lifecycle, statistics, exports, and component boundaries do not need to be.

The useful way to think about Scrapy is not as a library that gives you response.css(). Parsel already does that. Scrapy is an event-driven crawling runtime with a stable place to put each concern.

The framework appears one fix at a time

Imagine that a scheduled product scraper is growing beyond its original loop. It now needs to remember which URLs it has seen, pause without losing its queue, retry transport failures, apply different concurrency limits to different domains, normalize every product, split large exports into batches, report statistics, and fail the run when output quality drops.

Those requirements tend to produce classes with names such as QueueManager, RetryingClient, PluginRegistry, and RunMonitor. None of those names is inherently a mistake, but together they are a sign that the project is recreating a framework around its parsing code.

Scrapy already divides that system along practical boundaries:

What you are about to build	Scrapy component
URL queue and priority rules	Scheduler
Duplicate URL tracking	Duplicate filter
Retries, redirects, cookies, proxies	Downloader middleware
HTTP, browser, or custom network transport	Download handler
Site navigation and discovery	Spider
Response-to-object extraction	Spider callback or `scrapy-poet` Page Object
Validation, normalization, persistence	Item pipeline
Lifecycle hooks and operational behavior	Signals and extensions
Counters and run metadata	Stats collector
JSON Lines, CSV, object storage, batching	Feed exports

The boundaries matter more than the names. They let you replace the network transport without replacing the scheduler, or add data-quality checks without importing monitoring code into every callback.

flowchart LR
    Spider["Spider<br/>discovery"] --> Engine["Execution engine"]
    Engine --> Scheduler["Scheduler<br/>queue + dupefilter"]
    Scheduler --> Engine
    Engine --> Middleware["Downloader middleware<br/>retry + redirect + cookies"]
    Middleware --> Handler{"Download handler"}
    Handler --> HTTP["Scrapy HTTP"]
    Handler --> Wreq["wreq"]
    Handler --> Browser["Playwright"]
    HTTP --> Response["Response"]
    Wreq --> Response
    Browser --> Response
    Response --> Poet["Spider / scrapy-poet"]
    Poet --> Pipeline["Item pipelines"]
    Pipeline --> Export["Feed export / database"]
    Signals["Signals + stats"] -. observe .-> Engine
    Signals -. assert .-> Monitor["Core extensions + Spidermon"]

Here is the spider at the centre of that runtime. It contains the site-specific parts: where to begin, how to find product links, and what to yield from a product page.

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products",
            callback=self.parse_listing,
        )

    def parse_listing(self, response):
        yield from response.follow_all(
            response.css("[data-product-link]::attr(href)"),
            callback=self.parse_product,
        )

        next_page = response.css("[rel='next']::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse_listing)

    def parse_product(self, response):
        yield {
            "url": response.url,
            "name": response.css("[data-product-name]::text").get(),
            "price": response.css("[data-price]::attr(data-price)").get(),
        }

Scrapy's selectors use Parsel, so selectors from a smaller Parsel-based scraper move across with little ceremony. The larger change is that callbacks yield work to an engine instead of performing that work directly. The engine can now schedule, deduplicate, retry, throttle, record, and resume it.

The examples below target the current Scrapy async component APIs and Python 3.11 or later, which the Python wreq package requires. In a real project, pin a tested set of versions rather than allowing every integration to update independently.

python -m pip install \
    Scrapy scrapy-poet spidermon scrapy-playwright wreq
playwright install chromium

Start with the boring infrastructure

Before adding an ecosystem package, it is worth seeing how much of the runtime ships with Scrapy itself. A small set of settings gives the product spider adaptive throttling, bounded per-domain concurrency, resumable state, item processing, and compressed batch exports.

# myproject/settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
RETRY_TIMES = 3

ITEM_PIPELINES = {
    "myproject.pipelines.ProductValidationPipeline": 300,
    "myproject.pipelines.ProductUpsertPipeline": 800,
}

FEEDS = {
    "exports/products-%(batch_id)05d.jsonl.gz": {
        "format": "jsonlines",
        "encoding": "utf-8",
        "batch_item_count": 50_000,
        "postprocessing": [
            "scrapy.extensions.postprocessing.GzipPlugin",
        ],
    },
}

AutoThrottle adjusts delay from observed latency for each download slot. Its target concurrency is an average, while CONCURRENT_REQUESTS_PER_DOMAIN remains the ceiling. RetryMiddleware handles configured response codes and download exceptions, and lowers retried requests in the queue. The feed exporter handles serialization, file naming, compression, and batches without making the spider know where output goes.

Resumability is a command-line setting away:

scrapy crawl products -s JOBDIR=crawls/products-2026-07-26

JOBDIR persists the scheduler queue, duplicate-filter state, and spider state so a cleanly stopped crawl can resume. It is not a distributed queue, and a job directory must belong to one spider run. Requests in a disk queue also need serializable callbacks and metadata. These limits are much easier to reason about than an undocumented queue format that grew inside a script.

The item pipeline gives extraction and data handling separate failure modes. The spider describes what it found; ordered pipeline components validate, normalize, drop, or store it.

from decimal import Decimal, InvalidOperation

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class ProductValidationPipeline:
    def process_item(self, item):
        product = ItemAdapter(item)

        if not product.get("name"):
            raise DropItem("missing product name")

        try:
            product["price"] = Decimal(product["price"])
        except (InvalidOperation, TypeError):
            raise DropItem("invalid product price")

        return item

The extension system sits alongside these request and item flows. An extension is normally created once for a crawl and connects to signals such as spider_opened, item_scraped, spider_error, and spider_closed. Scrapy's built-in extensions use the same surface for core stats, periodic log stats, memory limits, close conditions, throttling, feed exports, and other lifecycle behaviour.

That API is small enough that a project-specific extension does not need its own plugin framework:

from scrapy import signals


class ProductStatsExtension:
    def __init__(self, crawler):
        self.stats = crawler.stats

    @classmethod
    def from_crawler(cls, crawler):
        extension = cls(crawler)
        crawler.signals.connect(
            extension.item_scraped,
            signal=signals.item_scraped,
        )
        return extension

    def item_scraped(self, item, response, spider):
        if not item.get("price"):
            self.stats.inc_value("product/missing_price")

EXTENSIONS = {
    "myproject.extensions.ProductStatsExtension": 500,
}

The extension records a fact. It does not decide whether the run should fail. Keeping that distinction makes the counter useful to loggers, dashboards, and monitors without letting one callback quietly define the operational policy for the whole crawl.

scrapy-poet separates navigation from interpretation

As a project adds sources, callbacks often become a mixture of navigation, selectors, normalization, and object construction. scrapy-poet moves page interpretation into Page Objects and injects the right object into a typed callback.

Enable its add-on and tell it where Page Objects live:

ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["myproject.pages"]

The Page Object owns extraction from one kind of page:

# myproject/pages/product.py
from decimal import Decimal

from web_poet import WebPage, handle_urls


@handle_urls("example.com")
class ProductPage(WebPage):
    def to_item(self) -> dict:
        name = self.css("[data-product-name]::text").get()
        price = self.css("[data-price]::attr(data-price)").get()

        return {
            "url": self.url,
            "name": name.strip(),
            "price": Decimal(price),
        }

The spider keeps navigation and asks for the dependency it needs:

import scrapy

from myproject.pages.product import ProductPage


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products/1",
            callback=self.parse_product,
        )

    def parse_product(self, response, page: ProductPage):
        yield page.to_item()

The annotation is not decorative. scrapy-poet inspects it, selects the Page Object using its rules, builds the dependency, and supplies it. You can override rules for a source, inject additional dependencies through providers, and test the Page Object without running a crawl. The spider is left to answer one question: where should the crawler go next?

This separation pays off when several spiders encounter the same page type, or one crawl strategy spans several sites with different representations. It also prevents browser acquisition, API-backed inputs, or cached responses from leaking into extraction code, because those can be provided as dependencies.

One detail matters if you use JOBDIR: callbacks must be serializable. scrapy-poet offers callback_for() to remove boilerplate, but assign the generated callback as a spider method instead of constructing it inline if requests may be written to disk.

Spidermon defines what a successful crawl means

A process can exit with code zero after extracting no products. It can also produce the expected number of records while silently losing every price. Scrapy's stats tell you what happened; Spidermon turns those facts into executable assertions and actions.

Enable the extension and select a monitor suite for spider close:

SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.ProductSpiderCloseSuite",
)

Then express the run contract in terms of the resulting dataset:

from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Product output")
class ProductOutputMonitor(Monitor):
    @monitors.name("Enough products were extracted")
    def test_item_count(self):
        count = getattr(self.data.stats, "item_scraped_count", 0)
        self.assertGreaterEqual(count, 1_000)

    @monitors.name("Missing prices stayed below one percent")
    def test_missing_price_rate(self):
        scraped = getattr(self.data.stats, "item_scraped_count", 0)
        missing = getattr(self.data.stats, "product/missing_price", 0)

        self.assertGreater(scraped, 0)
        self.assertLess(missing / scraped, 0.01)


class ProductSpiderCloseSuite(MonitorSuite):
    monitors = [ProductOutputMonitor]

Spidermon can run suites at spider open, spider close, engine stop, or periodically. Suites can trigger actions after checks pass or fail, and built-in monitors cover common conditions such as item counts, field coverage, finish reasons, retries, errors, and unwanted status codes.

This is a better layer for run-level policy than scattered if count == 0 checks. The spider collects data, components record facts, and monitors decide whether those facts describe useful output.

scrapy-playwright makes the browser an acquisition choice

Some pages do need JavaScript execution or browser state. That does not require the crawler itself to become a browser script. scrapy-playwright implements a Scrapy download handler, so selected requests can use Playwright while the surrounding scheduler, retries, callbacks, pipelines, and monitoring stay in place.

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

DOWNLOAD_HANDLERS = {
    "https": (
        "scrapy_playwright.handler."
        "ScrapyPlaywrightDownloadHandler"
    ),
}

# Let the browser provide a User-Agent that matches itself.
USER_AGENT = None

Requests opt in through metadata. Unmarked requests continue through Scrapy's regular HTTP implementation.

import scrapy
from scrapy_playwright.page import PageMethod


yield scrapy.Request(
    "https://example.com/products",
    callback=self.parse_products,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod(
                "wait_for_selector",
                "[data-products-loaded='true']",
            ),
        ],
    },
)

That opt-in is an important scaling property. Listing pages that already contain the required HTML can remain cheap HTTP requests, while the few pages that need execution consume browser capacity. Prefer bounded PageMethod actions that return a normal Scrapy response. If you request the Playwright Page object directly with playwright_include_page, your callback and errback must close it. Leaked pages eventually exhaust the per-context page limit and stall the crawl.

Browser work also changes what your metrics mean. Browser subresources are not individual Scrapy requests, and Chromium runs in another process. scrapy-playwright provides a memory-usage extension that can account for the browser process where supported. These details are exactly why the browser belongs behind a defined acquisition boundary rather than inside an arbitrary callback.

If the browser integration itself becomes the bulk of the crawler, tools such as Camoufox and Patchright may become relevant for particular sources. They are browser runtimes rather than replacements for Scrapy's scheduling and data pipeline, so the same architectural question remains: can the specialized browser sit behind a narrow boundary, or has browser state become the application?

Write a wreq download handler when the network stack is the variable

TLS and HTTP/2 fingerprints are properties of the client implementation, not just a collection of headers. If a source expects a network profile that resembles a current browser, changing User-Agent in Scrapy's default client does not change its TLS ClientHello, cipher ordering, extension ordering, or HTTP/2 behaviour.

The Python bindings for wreq provide an async client backed by a browser-emulating network stack. A custom download handler lets that client fetch bytes while Scrapy continues to own the crawl.

The following handler targets the current async download-handler API in Scrapy and the Python wreq package. It is a useful starting point, but the caveats after the code are part of the implementation, not optional footnotes.

# myproject/download_handlers.py
from __future__ import annotations

from datetime import timedelta

import wreq.exceptions
from scrapy import Request
from scrapy.core.downloader.handlers.base import BaseDownloadHandler
from scrapy.exceptions import DownloadFailedError, DownloadTimeoutError
from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from wreq import Client, Emulation, Method, Proxy, redirect
from wreq.header import HeaderMap


WREQ_NETWORK_ERRORS = (
    wreq.exceptions.BodyError,
    wreq.exceptions.BuilderError,
    wreq.exceptions.ConnectionError,
    wreq.exceptions.ConnectionResetError,
    wreq.exceptions.DecodingError,
    wreq.exceptions.ProxyConnectionError,
    wreq.exceptions.RequestError,
    wreq.exceptions.TlsError,
)


class WreqDownloadHandler(BaseDownloadHandler):
    def __init__(self, crawler):
        super().__init__(crawler)
        self.default_timeout = crawler.settings.getfloat(
            "DOWNLOAD_TIMEOUT"
        )
        profile_name = crawler.settings.get(
            "WREQ_EMULATION",
            "Chrome149",
        )

        self.client = Client(
            emulation=getattr(Emulation, profile_name),
            # RedirectMiddleware should receive 30x responses.
            redirect=redirect.Policy.none(),
            # CookiesMiddleware remains the cookie owner.
            cookie_store=False,
        )

    async def download_request(self, request: Request) -> Response:
        method = getattr(Method, request.method, None)
        if method is None:
            raise DownloadFailedError(
                f"wreq does not support {request.method!r}"
            )

        headers = HeaderMap()
        for name, values in request.headers.items():
            header_name = name.decode("latin-1")
            for value in values:
                headers.append(
                    header_name,
                    value.decode("latin-1"),
                )

        timeout = request.meta.get(
            "download_timeout",
            self.default_timeout,
        )
        kwargs = {
            "headers": headers,
            "body": request.body,
            "timeout": timedelta(seconds=timeout),
        }

        if proxy_url := request.meta.get("proxy"):
            kwargs["proxy"] = Proxy.all(proxy_url)

        try:
            async with await self.client.request(
                method,
                request.url,
                **kwargs,
            ) as response:
                body = await response.bytes()
                response_headers = Headers(list(response.headers))
                response_url = str(response.url)
                response_status = response.status.as_int()
                response_protocol = str(response.version)
        except wreq.exceptions.TimeoutError as exc:
            raise DownloadTimeoutError(str(exc)) from exc
        except WREQ_NETWORK_ERRORS as exc:
            raise DownloadFailedError(str(exc)) from exc

        response_cls = responsetypes.from_args(
            headers=response_headers,
            url=response_url,
            body=body,
        )

        return response_cls(
            url=response_url,
            status=response_status,
            headers=response_headers,
            body=body,
            request=request,
            protocol=response_protocol,
        )

    async def close(self) -> None:
        self.client.close()

DOWNLOAD_HANDLERS = {
    "http": "myproject.download_handlers.WreqDownloadHandler",
    "https": "myproject.download_handlers.WreqDownloadHandler",
}

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

WREQ_EMULATION = "Chrome149"

# Do not pair a browser transport profile with Scrapy's UA.
USER_AGENT = None

There are several deliberate ownership decisions here. wreq does not follow redirects, because Scrapy's RedirectMiddleware should see the 302, enforce its limits, adjust priority, and record redirect statistics. wreq's cookie store is disabled, because Scrapy's CookiesMiddleware should remain the single source of cookie state. Transport errors become Scrapy download exceptions, so the existing retry middleware can handle them.

Headers need the same care. A Chrome TLS and HTTP/2 profile combined with contradictory User-Agent, Accept, Accept-Language, or client-hint headers is not a coherent Chrome request. In a real implementation, decide which browser-shaped headers wreq owns, then filter those headers out of the Scrapy request before building HeaderMap. Keep source-specific headers, authentication, and cookies. Test the actual wire output rather than assuming the two layers compose cleanly.

The example buffers the entire response. It therefore does not yet implement DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE, Scrapy's headers_received and bytes_received signals, or StopDownload. A production handler should stream wreq chunks, apply size checks before and during the body, and propagate cancellation. It should also map DNS, connection refusal, data loss, and unsupported methods to the most specific Scrapy exceptions available.

Compression deserves an explicit test too. Both the HTTP client and Scrapy middleware may believe they own gzip, Brotli, deflate, or zstd decoding. Choose one owner, then verify that the body is decoded once and Content-Encoding still describes the bytes Scrapy receives.

Before using the handler in a real crawl, run it against a local HTTP fixture that covers at least:

HTML response-class selection, duplicate response headers, and request attachment.
POST bodies and repeated request headers.
A redirect handled by Scrapy rather than wreq.
A retryable 500 and a transport timeout.
Per-request timeout and proxy metadata.
gzip and Brotli responses decoded exactly once.
A large streamed body stopped at the configured size.
Handler close and in-flight cancellation.

This list is longer than await client.get(url) because a download handler is a transport contract. That is also the point of the exercise. Scrapy lets you replace this one contract without throwing away the rest of the crawling runtime.

Scrapy is a runtime, not the whole platform

Scrapy does not make every architecture decision disappear. Its default scheduler is not a distributed queue shared by a fleet of workers. JOBDIR is resumability for one crawl, not global orchestration. A database pipeline still needs a data model, idempotent writes, and migration policy. Multiple processes still need shared traffic budgets if they target the same domain. Browser capacity, proxy health, deployment, secrets, and backfills still need an operating model.

Source-specific extraction remains real engineering too. scrapy-poet gives that work a clean home, but it cannot decide what a price means, whether two variants are the same product, or how to detect a plausible-looking error page.

The narrower claim is more useful: Scrapy provides a mature runtime and a set of extensible boundaries around those decisions. Build the parts that distinguish your data product. Extend a known scheduler, downloader, pipeline, and lifecycle before writing new versions of all four.

A practical migration path

Do not begin by rewriting a working scraper into a perfectly abstract project. Pick one scheduled script that has already accumulated reliability code and move it in layers:

Put discovery and parsing into a spider, leaving existing Parsel selectors intact.
Let the scheduler and duplicate filter own request flow, then enable JOBDIR for resumability.
Remove the local retry loop and use downloader middleware plus Scrapy download exceptions.
Move validation and persistence into ordered pipelines.
Record source-specific facts in stats and assert the run contract with Spidermon.
Move reusable extraction into scrapy-poet Page Objects.
Add Playwright or a wreq handler only for requests that prove they need a different acquisition path.

At each step, delete a piece of home-grown runtime. If the migration only adds Scrapy beside the old queue, retry client, plugin system, and exporter, it has missed the opportunity.

Frequently asked questions

Is Scrapy too old for modern Python and async code?

No. Current Scrapy has coroutine support across major component APIs and an async download-handler interface. Its Twisted history is visible, but it does not prevent integration with asyncio-based clients and browser tooling. Version compatibility still matters, so pin Scrapy and ecosystem packages together and test reactor configuration.

Can one spider mix normal HTTP, Playwright, and wreq?

Yes, but handler selection is scheme-based. scrapy-playwright registers an HTTPS handler that delegates unmarked requests to Scrapy's normal implementation. A project that also wants wreq for selected HTTPS requests needs one dispatching handler or separate schemes/domains routed by a custom handler. Two different classes cannot both occupy the same https entry in DOWNLOAD_HANDLERS.

When is a plain script still the better choice?

Use a plain script when the input is small and known, the run is disposable, failure has little cost, and no one needs to resume or operate it. Scrapy earns its place when the crawler develops state, repeated execution, multiple request paths, operational expectations, or reusable components.

Does Scrapy solve scaling across machines?

Not by itself. Scrapy gives each crawler process strong internal boundaries and replaceable components. A distributed deployment still needs orchestration, shared state where appropriate, and a policy for global traffic and data consistency. The framework makes those integrations narrower; it does not pretend they are free.

Build your scraper, not another crawling framework

In the previous post, the small scraper grew into a system as its failure cost increased. This is the natural next decision: once you need that system, decide which parts are genuinely specific to your source and which are generic crawling infrastructure.

Scrapy will not write the difficult selectors or define the meaning of your data. It will give those decisions a scheduler, lifecycle, retry path, extension model, statistics layer, and several escape hatches. In many projects, that is the framework you were about to spend the next year discovering one production incident at a time.

So you want to get data from the web

John Rooney — Sun, 26 Jul 2026 06:14:44 +0000

Getting data from one web page can take 20 lines of Python. Keeping a trustworthy dataset alive while the source changes, rate limits you, returns partial responses, and occasionally serves a consent page with a 200 OK is a different engineering problem, because the selector is rarely as difficult as knowing when the result is wrong.

What web scraping becomes once someone depends on it

Web scraping is automated retrieval and extraction of data from web resources. At the small end, that means an HTTP request followed by an HTML parser. At the other end, the same basic operation sits inside a distributed acquisition pipeline with scheduling, traffic controls, schema validation, raw snapshots, observability, and a recovery path. Both are web scraping, but they should not have the same architecture.

The easiest way to choose that architecture is to classify the job by its failure cost:

Workload	A reasonable starting point	The first problems you will meet
A few known pages, needed once	One process and local output	Missing fields, pagination, duplicate URLs, odd encodings
A scheduled crawl of one site	A repeatable job with persistent state	Timeouts, rate limits, reruns, template changes
A JavaScript application	HTTP plus targeted browser automation where required	Background requests, timing, cookies, session expiry
Many sources feeding another product	A queue-based data pipeline	Source drift, partial runs, backfills, data contracts, cost control

This is the useful complexity ladder. A one-off question does not need a queue, six workers, and an observability stack, while a revenue-critical feed should probably graduate from a laptop before print() becomes its monitoring strategy.

A small Python scraper should leave evidence behind

The first version should already separate fetching from parsing because that boundary lets you test selectors without hitting the site again and distinguish a network failure from an extraction failure. It also means that when the markup changes, you can fix the parser against a saved response instead of repeatedly fetching the same page while you debug it.

Here is a complete example for a static product page. It uses Parsel, expects semantic attributes such as data-product-name and data-price, saves the raw response to a path chosen by the caller, and returns a typed record. Parsel is also the selector layer underneath Scrapy, so the CSS and XPath expressions carry across when the crawler grows into a Scrapy project.

Install the two dependencies first:

python -m pip install requests parsel

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path
from urllib.parse import urldefrag

import requests
from parsel import Selector


@dataclass(frozen=True)
class Product:
    source_url: str
    name: str
    price: Decimal
    fetched_at: str


def parse_product(
    html: str,
    *,
    source_url: str,
    fetched_at: str,
) -> Product:
    selector = Selector(text=html)
    root = selector.css("[data-product]")

    if not root:
        raise ValueError("missing product container")

    name = root.css("[data-product-name]::text").get()
    raw_price = root.css("[data-price]::attr(data-price)").get()

    if name is None or raw_price is None:
        raise ValueError("missing required product field")

    name = name.strip()

    try:
        price = Decimal(raw_price)
    except InvalidOperation as exc:
        raise ValueError("invalid product price") from exc

    if not name or not price.is_finite() or price < 0:
        raise ValueError("product failed validation")

    return Product(
        source_url=source_url,
        name=name,
        price=price,
        fetched_at=fetched_at,
    )


def fetch_product(url: str, snapshot_path: Path) -> Product:
    response = requests.get(
        url,
        headers={"User-Agent": "ExampleDataCollector/1.0 contact@example.com"},
        timeout=(5, 20),
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")
    if "text/html" not in content_type:
        raise ValueError(f"expected HTML, received {content_type!r}")

    fetched_at = datetime.now(timezone.utc).isoformat()
    source_url, _fragment = urldefrag(response.url)

    snapshot_path.parent.mkdir(parents=True, exist_ok=True)
    snapshot_path.write_bytes(response.content)

    return parse_product(
        response.text,
        source_url=source_url,
        fetched_at=fetched_at,
    )

The timeout has separate connection and read limits, raise_for_status() prevents a 404 page from flowing into the parser, and the content-type check catches JSON, images, and some challenge pages before Parsel sees them. The raw body is stored before parsing, so an extraction error does not destroy the evidence needed to reproduce it.

There is no content hash here because this example does not need one. The caller can name snapshots by run ID, URL key, or whatever fits the job. Hashes become useful later, when the system needs to detect unchanged responses, deduplicate stored bodies, or connect a parsed record to the exact bytes that produced it. Introducing that machinery here would explain the production architecture before we have earned it.

The parser rejects invalid records instead of returning a half-populated dictionary. Returning None for a missing price feels convenient until it reaches an aggregate and quietly changes the answer.

The test does not need the network:

from decimal import Decimal


def test_parse_product() -> None:
    html = """
    <article data-product>
      <h1 data-product-name>Mechanical Keyboard</h1>
      <span data-price="89.95">£89.95</span>
    </article>
    """

    product = parse_product(
        html,
        source_url="https://example.com/products/keyboard",
        fetched_at="2026-07-26T10:00:00+00:00",
    )

    assert product.name == "Mechanical Keyboard"
    assert product.price == Decimal("89.95")

This is the first architectural boundary worth keeping: fetchers produce source material and metadata, parsers turn that material into validated records, and storage comes after validation.

JavaScript rendering should be an explicit escalation

Some pages return the data in the initial HTML, while others return an application shell and fetch the useful payload after JavaScript runs. Check the response body and browser network panel before reaching for a headless browser because structured data embedded in the document or a background JSON response is usually easier to operate than a full browser session.

Playwright is excellent when you genuinely need rendering or interaction. It also turns deterministic request code into a small distributed-systems problem living inside Chromium, complete with page lifecycle events, async UI updates, browser crashes, context cleanup, and much higher CPU and memory use.

When a browser is justified, wait for a condition that means the data is ready. networkidle is often a weak signal because analytics, polling, and streaming requests can keep the network busy indefinitely.

Install Playwright and its Chromium build before running the example:

python -m pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright


def fetch_rendered_html(url: str) -> str:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        page = browser.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            page.locator("[data-product-loaded='true']").wait_for(
                state="attached",
                timeout=10_000,
            )
            return page.content()
        finally:
            browser.close()

The selector is intentionally site-specific. Replace it with a marker that appears only when the content you need is present, or wait for the relevant response. Playwright's auto-waiting documentation explains what its actions wait for automatically, while its evaluation guide covers the boundary between Python and code running in the page.

Standard Playwright also has a recognisable automation surface. If a site reacts differently to an automated browser than to the corresponding normal browser, stealth-oriented projects are another rung on the complexity ladder. Camoufox is a Firefox fork with Playwright compatibility and browser-level fingerprint changes, while Patchright patches the Chromium side of Playwright and presents itself as a drop-in replacement. Both are moving projects with their own compatibility gaps, release cadence, and operational cost, so treat them as specialised browser runtimes rather than magic flags. We will leave the comparison there because each deserves a separate article.

Retry transient failures without turning a scrape into a traffic spike

Once a job runs on a schedule, failures become normal input: DNS lookups fail, connections reset, upstream services return 503, and rate limits change during the day. Retries help with those transient failures, but unbounded or immediate retries make incidents worse.

requests can use urllib3's retry policy through an adapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist={429, 500, 502, 503, 504},
        allowed_methods={"GET", "HEAD"},
        respect_retry_after_header=True,
        raise_on_status=False,
    )

    adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

With a backoff factor of 0.5, urllib3 applies increasing delays between eligible retries. The final response still needs raise_for_status(). A 403 or 404 is not in the retry list because repeating a permanent failure four times adds load without changing the answer.

HTTP 429 Too Many Requests may include a Retry-After header. RFC 6585 leaves the server free to count requests by credential, cookie, client, resource, or another policy, which means changing one network attribute does not necessarily reset the limit.

At more than one worker, enforce concurrency and delay per host across the whole worker pool. A local sleep(1) inside each worker gives you ten requests per second when you scale to ten workers. Put the host budget in shared state or route each host through its own queue.

TLS fingerprinting starts before the server sees your headers

Changing the User-Agent header does not make an HTTP client behave like a browser. Before the server receives the HTTP request, the client has already sent a TLS ClientHello containing an ordered set of cipher suites, extensions, supported groups, signature algorithms, and ALPN values. Those choices, along with their ordering, can be summarised as JA3 or JA4 fingerprints. HTTP/2 adds another layer through SETTINGS values, connection behaviour, and pseudo-header ordering.

This is why a request can claim to be Firefox in its headers while looking nothing like Firefox on the wire. A conventional Python client is designed to make correct HTTP requests, not to reproduce a browser's network stack. For many sites that distinction does not matter. On sites that classify traffic using several signals, it can explain why copying browser headers changes nothing.

wreq exposes Python bindings over the Rust client, so there is no reason to leave the Python stack just to control the network fingerprint. It uses BoringSSL and supports TLS, JA3/JA4, HTTP/2, and header-order profiles. The current package ships more than 100 browser emulation profiles and offers both async and blocking clients.

Install wreq alongside Parsel:

python -m pip install wreq parsel

import asyncio

from parsel import Selector
from wreq import Client, Emulation


async def fetch_title(url: str) -> str:
    client = Client(emulation=Emulation.Chrome149)
    response = await client.get(url)

    if response.status >= 400:
        raise RuntimeError(f"request failed with status {response.status}")

    selector = Selector(text=await response.text())
    title = selector.css("title::text").get()

    if title is None:
        raise ValueError("response did not contain a title")

    return title.strip()


if __name__ == "__main__":
    print(asyncio.run(fetch_title("https://example.com")))

The profile needs to be internally consistent. If the TLS handshake resembles one browser version while the headers, operating system hints, cookies, or JavaScript-visible properties describe another, the inconsistency becomes a signal of its own. Keep a client alive across requests so connection pooling and cookies behave consistently, pin the package and profile versions you have tested, and expect fingerprints to age as browsers and detection systems change.

wreq solves the network-stack part of the problem rather than JavaScript rendering. If the page needs a browser, use one; if the HTML is already in the response and the mismatch is at TLS or HTTP/2, launching Chromium adds a large amount of machinery to solve a lower-level issue.

A production scraper is a replayable data pipeline

Once the output feeds search, analytics, pricing, or a customer-facing feature, the system needs more than fetch workers. A practical design looks like this:

flowchart LR
    S["Scheduler: cron or Airflow"] --> Q["Host-aware URL queues: SQS or Redis"]
    Q --> F["HTTP fetchers: requests or wreq"]
    Q --> B["Browser fetchers: Playwright, Camoufox, or Patchright"]
    F --> RAW["Raw snapshots and metadata: S3"]
    B --> RAW
    F --> RETRY["Retry and review queue"]
    B --> RETRY
    RAW --> P["Versioned parser workers"]
    P --> V["Schema and run-quality gates"]
    V --> DB["Idempotent records: PostgreSQL"]
    DB --> C["Search, analytics, or product consumers"]
    F -. metrics .-> O["Metrics, logs, traces, and alerts"]
    B -. metrics .-> O
    P -. metrics .-> O
    V -. metrics .-> O

The names are examples, not requirements. A single VM, SQLite, and a filesystem can implement the same boundaries for a smaller workload. The boundaries matter because they let you retry and replace one stage without repeating every other stage.

Raw snapshots make parser changes cheap

Store response bytes with the source URL, final URL after redirects, fetch timestamp, status, content type, relevant response headers, and a content hash. Add the fetcher and parser version to the eventual record. At this point the hash has a clear job: it gives the raw body a stable identity, makes unchanged responses cheap to detect, and links each parsed record back to the exact bytes that produced it.

When a selector breaks, rerun the new parser against stored responses. This is faster, cheaper, and kinder to the source than fetching every URL again. It also lets you compare old and new parser output before a backfill reaches production.

Retention is a policy decision, especially if pages contain personal or sensitive data. An immutable archive is useful engineering infrastructure. It is not a reason to keep everything forever.

Idempotent writes make retries safe

A worker can fail after writing a record but before acknowledging its queue message. The message will be delivered again. If the second write creates another row, your retry policy has become a duplication policy.

Use a stable key, such as (source, external_id), and upsert. This PostgreSQL statement uses SQLAlchemy-style named parameters:

INSERT INTO products (
    source,
    external_id,
    name,
    price,
    fetched_at,
    content_hash,
    parser_version
)
VALUES (
    :source,
    :external_id,
    :name,
    :price,
    :fetched_at,
    :content_hash,
    :parser_version
)
ON CONFLICT (source, external_id) DO UPDATE SET
    name = EXCLUDED.name,
    price = EXCLUDED.price,
    fetched_at = EXCLUDED.fetched_at,
    content_hash = EXCLUDED.content_hash,
    parser_version = EXCLUDED.parser_version;

This gives you the latest record. If you also need history, write changes to an append-only version table or event log. Do not overload one table to be both current state and an audit trail.

Quality gates should fail the run, not merely decorate a dashboard

Transport metrics tell you whether requests completed, while data metrics tell you whether the output is believable.

Track at least:

attempted, succeeded, retried, and finally failed URLs by host
status-code distribution and request latency
records produced, required-field completeness, and uniqueness
new, changed, and missing records relative to the previous successful run
parser errors grouped by source and parser version

Then define stop conditions. If yesterday's crawl produced 48,000 products and today's produces 730, publishing the result should require an explanation. The threshold depends on the source, but the decision must exist in code rather than in someone's memory.

Here is a small quality gate with deliberately configurable limits:

from dataclasses import dataclass


@dataclass(frozen=True)
class RunStats:
    attempted: int
    succeeded: int
    records: int
    records_with_required_fields: int


def validate_run(
    current: RunStats,
    *,
    previous_records: int,
    min_success_rate: float = 0.95,
    min_completeness: float = 0.99,
    min_volume_ratio: float = 0.70,
) -> None:
    if current.attempted <= 0 or current.records <= 0:
        raise ValueError("run produced no usable work")

    success_rate = current.succeeded / current.attempted
    completeness = current.records_with_required_fields / current.records
    volume_ratio = current.records / max(previous_records, 1)

    failures = []
    if success_rate < min_success_rate:
        failures.append(f"success rate {success_rate:.1%}")
    if completeness < min_completeness:
        failures.append(f"completeness {completeness:.1%}")
    if volume_ratio < min_volume_ratio:
        failures.append(f"volume ratio {volume_ratio:.1%}")

    if failures:
        raise ValueError("run failed quality gates: " + ", ".join(failures))

Those defaults are examples, not universal scraping constants. A news index and a product catalogue have different expected churn. Start by observing normal variation, then choose thresholds that catch abnormal runs without paging someone every morning.

The shortcuts that fail when the workload grows

Several designs work just long enough to become expensive:

Putting fetch and parse logic in one function. You cannot test a parser without network traffic, and every parser fix requires another crawl.

Using a browser for every URL. It simplifies the first implementation when some pages need JavaScript, then spends far more CPU and memory on pages that do not. Keep HTTP and browser fetchers behind the same response interface, and route only the pages that need rendering.

Retrying every error. A timeout may recover. A malformed selector, an authentication failure, or a deleted page will not. Classify failures and send permanent or suspicious ones to review.

Scaling with global concurrency. One large domain and one tiny domain do not have the same capacity or rules. Global concurrency lets the busiest host consume the whole budget and makes request pressure unpredictable.

Treating row count as the only quality metric. A parser can return the expected number of rows with the wrong field in every row. Validate types, ranges, uniqueness, and changes in field-level completeness.

The common thread is lost evidence. Once the fetch, parser, and output are entangled, you cannot tell which part failed without repeating the entire job.

How to choose the smallest architecture that will survive

Use one script when all of these are true: the page set is small, the source is stable, the work is easy to rerun, and a bad result has limited consequences. Even then, save the raw input and validate required fields.

Add persistent state and scheduling when runs repeat. Add browser automation only for pages that require it. Add queues when one process no longer meets the throughput or isolation you need. Add separate parser workers when you want to replay stored responses without fetching again.

The forcing function is not URL count by itself. Ten authenticated, stateful workflows can be harder to operate than a million static pages from a stable bulk source.

Before increasing concurrency, write down:

What is the per-host request budget?
Which failures are safe to retry?
Which quality signal stops publication?
What raw evidence and version metadata will you retain?
How will you replay or backfill without duplicating records?

If those answers are vague, more workers will mostly help you fail faster.

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of `requests`?

Use requests or httpx when the URL set is small and you can manage scheduling, retries, deduplication, and persistence directly. A framework such as Scrapy earns its place when you need concurrent crawling, downloader middleware, per-domain controls, request deduplication, item pipelines, and a consistent extension model. A framework removes plumbing, not the need to understand the source.

When should I use Playwright for web scraping?

Use Playwright when data only appears after browser-side execution or requires an interaction that a plain HTTP client cannot reproduce. Do not use it by default for static pages. Inspect the initial HTML and background requests first, then route only the pages that require rendering through a browser pool.

How do I know whether a scraper is returning bad data?

Validate each record against a schema, then compare run-level metrics with the previous successful run. Required-field completeness, type and range checks, uniqueness, record volume, and the number of changed or missing entities catch different classes of failure. Keep raw snapshots so you can reproduce the parser's decision.

How do you scale web scraping without overwhelming a site?

Set concurrency and request rates per host across all workers, respect 429 and Retry-After, cache or fetch incrementally where appropriate, and use bounded retries for transient failures. Scale the scheduler and parser independently from the fetch rate. More parsing capacity does not require more traffic to the source when raw responses are replayable.

Start with a replay test

Take one representative page, save the raw response, and make the parser pass against that fixture. Then change or remove one required field and confirm the test fails for the reason you expect.

That small exercise tells you whether you are building a script that happens to return data today, or a system you can still explain after the source changes tomorrow.

How to use Camoufox with Scrapy (New EASY Method)

John Rooney — Sat, 25 Jul 2026 11:20:24 +0000

A question comes up often enough in the web scraping community that it has a familiar shape to it: I like working in scrapy-playwright, but I want to run camoufox, or another Python browser library, instead of the browser it ships with, so how do I actually wire that in? For a long time the honest answer was some version of "you can, but it isn't pretty." People raised it in GitHub issues and talked around it on forums, and the workarounds that existed all asked you to figure out a way to run that browser separately and use CDP.

That has changed with a small addition to scrapy-playwright: PLAYWRIGHT_BROWSER_PROVIDER, that hands the browser lifecycle to whatever Playwright-compatible package you want to use. It’s a small Python class that integrates into Scrapy and scrapy-playwright. It is the kind of change that reads like a one-line entry in a changelog and quietly removes a frustration a lot of people had learned to live with.

Why the stock browsers get caught

To see why this matters, it helps to remember what people are reaching for a different browser to do. Playwright is excellent at driving a browser, and scrapy-playwright wires that capability into Scrapy at the download handler so a rendered page arrives through the normal request and response cycle, asynchronously and quickly. The catch is the binary. The Chromium and Firefox builds that Playwright ships are the standard, well-known ones, which means even fairly basic anti-bot systems recognize them almost immediately, and the sites you most want to reach are rarely running basic anti-bot systems. These browsers were designed for automated testing, and no scraping specifically.

Getting around that recognition used to be a matter of patching the browser at runtime, but that has stopped being enough. The more effective projects now change the browser at the source and compile their modifications into the binary itself, instead of patching a stock build once it is running. Runtime patches tend to leave small inconsistencies behind, and modern fingerprinting is built to notice exactly those, in the same way that the network fingerprint has become a central battleground between scrapers and site operators. Each mismatch lowers the trust a site assigns to your session and makes a challenge more likely, so the goal is to present a browser with nothing out of place. Building the changes at compile time is harder to detect, and because Chromium and Firefox expose different things, what needs changing is different in each. This is one important part of how modern stealth browsers work, not the whole story, but it is enough to explain why swapping the browser is worth the effort, especially when anti-bot vendors now ship changes at a pace that punishes any setup relying on manual tuning.

The middle ground that was missing

scrapy-playwright already had one good answer for using a different browser: connect to one that is running somewhere else. Through PLAYWRIGHT_CDP_URL or Playwright's own connect endpoint, you point the plugin at an already-running browser over the network, and that route is genuinely useful when the browser lives on a paid service or on infrastructure you run yourself. I have written before about standing up a self-hosted browser service and connecting Playwright to it over a WebSocket, and it works well, but it is worth being clear about what it asks of you. You have to run the browser in debug mode, keep that instance alive, watch it for memory leaks, and restart it when it dies, which is a reasonable amount of operational surface for what started as "I just want to use camoufox locally."

That was the gap. Between the stock browser that gets caught and a full separate browser service that you connect to remotely, there was a large group of people who simply wanted a stealthier browser running inline with their spider, on their own machine, without building a service around it. The workarounds they were left with were the ugly ones. One approach was to swap the binary by hand: download the original, remove it, and drop the replacement in where Playwright expected to find it, which is as fragile as it sounds and breaks the moment anything updates. Another leaned on the fact that some of these browsers shipped an experimental built-in server you could expose and connect to, which was fine until you needed one of the browsers that had no such server, at which point you were back to no clean route at all. That last point bites hardest for the Firefox-based options, because the connect-to-a-running-browser story was never as clean for them as it was for Chromium.

How browser providers actually work

The new setting works by naming a provider, which is just a class that owns starting and stopping the browser. The default provider wraps vanilla Playwright and gives you everything the README already documented, and if you want a different backend you point PLAYWRIGHT_BROWSER_PROVIDER at a class of your own. The provider implements a small asynchronous lifecycle, start, launch_browser, launch_persistent_context, and close, and because packages like camoufox, patchright, and invisible_playwright keep the standard Browser, BrowserContext, and Page objects, everything downstream of the launch, your pages, contexts, and routing code, keeps working unchanged. Only how the browser starts and stops is different.

Here is the camoufox provider from the documentation, which is the shape almost any provider will take:

from contextlib import AsyncExitStack

from scrapy_playwright.handler import Config, PERSISTENT_CONTEXT_PATH_KEY

class CamoufoxBrowserProvider:
    def __init__(self, config: Config) -> None:
        self.config = config
        self.stack = AsyncExitStack()

    async def start(self) -> None:
        pass

    async def launch_browser(self):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(**self.config.launch_options)
        )

    async def launch_persistent_context(self, context_kwargs: dict):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(
                persistent_context=True,
                user_data_dir=context_kwargs[PERSISTENT_CONTEXT_PATH_KEY],
                **self.config.launch_options,
            )
        )

    async def close(self) -> None:
        await self.stack.aclose()

Point the settings at it, and because camoufox is Firefox-based, set the browser type to match:

# settings.py
PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CamoufoxBrowserProvider"
PLAYWRIGHT_BROWSER_TYPE = "firefox"

I ran this exact example from the documentation, and it works. That is roughly all there is to it: one class, two settings, and the stealth browser is running inline with your spider. Importing the third-party library lazily inside the methods that need it, as the example does, is a nice detail, because it means the setting can point at a provider whose backend is only installed in some of your environments without breaking the others.

Why this fits the way Scrapy is built

None of this is an accident of design. Scrapy is lean on purpose. It is a full framework, but the core deliberately avoids telling you which browser to use, which extraction approach to take, or how your infrastructure should look, and it pushes those decisions out to extensions instead, some official and some from the community. Browser providers extend that same principle right down to the choice of browser binary, which is a very Scrapy way to solve the problem: rather than blessing one stealth browser, the plugin adds a seam and lets you bring your own.

It is worth saying where scrapy-playwright sits in that picture, because it shapes why this addition feels significant rather than incidental. The plugin lives in the scrapy-plugins organization and has been driven largely by a colleague at Zyte, which puts it in an interesting spot, not a headline product with a marketing page, but not a random third-party bolt-on either. Seeing a genuinely useful capability land in a project like that, one that many people had quietly filed under "mature, does what it does," is a good reminder that the tools we lean on are still being improved by people who hit the same friction we do.

What it changes, and what it doesn't

I would put this on the "relatively big deal" side of the ledger rather than the nice-to-have side, precisely because of the group it serves. If you were already connecting to a remote browser service, your setup does not change, and if you only ever needed the stock browser, this will pass you by. But for the sizable middle, the people who wanted a stealthier browser running locally with scrapy-playwright and were stuck with fragile workarounds, this turns a hack into a configuration choice, and lowering the cost of that experiment is the real value here.

A couple of honest caveats belong alongside that. The provider examples in the documentation are offered as guidelines, not supported parts of scrapy-playwright, and the plugin is not affiliated with camoufox, patchright, or any of the other projects it references, so you should read each project's own documentation and evaluate any dependency yourself before you rely on it. Swapping in a stealth browser is also not a finish line. It removes one common way of being spotted, but it sits inside a larger system of proxies, behavior, and session management, and treating it as a single switch that solves blocking will lead you astray. When you would rather not run and maintain any of this yourself, Zyte API's headless browser handles the binary, the fingerprinting, and the proxy rotation behind a single request, which is the right answer for a different set of problems than the one browser providers solve. What providers give you is a clean way to make the choice yourself, and for a lot of people that is exactly what was missing.

New Zyte add-ons: Agent Skills, Codex, GitHub and more updates

John Rooney — Sat, 25 Jul 2026 11:17:57 +0000

Developers are changing how they build software, asking agents to plan, create, run, test, and improve real workflows. Web scraping is no exception.

Earlier this year, Zyte launched a Zyte plugin for Claude Code, our latest step toward production-ready, agentic web scraping. The idea was simple: give an AI coding agent the web expertise it needs to move from a prompt to a working Scrapy project.

Today, we’re expanding that approach with three new additions to the Zyte Agentic Web Data line-up:

Agent Skills - portable Zyte web scraping skills for any compatible AI coding agent.
Zyte skills for GitHub Copilot CLI - a dedicated plugin for GitHub Copilot CLI users.
Zyte plugin for Codex - an integration specifically for Codex workflows.

Together, these additions make Zyte’s web scraping expertise available in more of the agentic coding environments developers already use. The new integrations add to the existing line-up:

Zyte Claude Code plugin - Zyte know-how and skills in a dedicated Claude package.
Web Scraping Copilot - Visual Studio Code extension that generates extraction logic and helps with project creation right in your code editor.

Three new agent integrations

1. Agent Skills

The biggest addition is Agent Skills.

Agent Skills package Zyte’s web scraping knowledge into portable skills that can be used by any AI coding agent that supports the Agent Skills standard.

Install them with:

npx skills add zytedata/skills

You can also find them on:

This is the most agent-neutral way to benefit from Zyte’s scraping code know-how. If your team has standardized on an AI coding agent that supports Agent Skills, you can bring a spider-building workflow into that environment without switching tools.

With Agent Skills, the agent can help you move from a URL and a data requirement to a complete Scrapy project with web-poet page objects, pytest fixtures, and a wired-up spider.

With our skills, you can also simply ask your tools to deploy your spider and run it to get data.

Use Agent Skills if you're looking for a much faster and easier way to build a spider and you're using a coding agent which does not yet have a native integration (Claude Code, GitHub Co-Pilot CLI etc).

2. Codex and GitHub Copilot CLI plugins

The second and third additions are Zyte Agentic Web Data for Codex and GitHub Copilot CLI.

Codex is another important environment for agentic software development. OpenAI’s coding agent that now sits inside ChatGPT. With this integration, developers using Codex can access the same web scraping workflow.

We’re also adding integration with GitHub Copilot CLI. This brings Zyte’s agentic scraping workflow to developers who work with GitHub Copilot from the command line.

The experience is designed for a familiar developer flow: stay in the terminal, point the agent at a site, describe the data you need, and generate a working Scrapy project.

Codex - find out more:

Zyte docs: Codex plugin
GitHub: https://github.com/zytedata/codex-skills

GitHub Copilot CLI - find out more:

Zyte docs: GitHub Copilot CLI
GitHub: https://github.com/zytedata/copilot-skills

Agentic scraping: The big idea

We first launched Web Scraping Copilot for Visual Studio Code in 2025. That, plus our Claude Code plugin, proved something important: general-purpose coding agents can become dramatically more useful for web data work when they are given specialist scraping knowledge.

A coding agent may know Python, it may even know Scrapy - but production web scraping requires more than generating plausible code. Reliable spiders need the right project structure, extraction patterns, page objects, test fixtures, dependency handling, rendering decisions, anti-bot configuration, and a workflow that lets you inspect and improve the result.

That is what Zyte brings to agentic development.

Instead of starting from a blank prompt, Zyte gives the agent a structured web scraping workflow:

Explore the target site.
Define the data you want.
Propose and refine an extraction schema.
Generate an inspectable Scrapy project.
Run, test, deploy, and iterate.

You still get the speed of an AI agent. But you also get the structure, control, and reliability developers need when the output matters.

How it works

When you use Zyte’s agentic web data workflow, the agent can create two key artifacts:

An extraction spec - This captures the schema, saved pages, expected values, and decisions made during the exploration and validation process.
A generated Scrapy project - This includes web-poet page objects, pytest fixtures, and a spider ready to run locally or deploy to Scrapy Cloud.

This separation matters. The extraction spec gives you a durable description of the data you want, while the generated Scrapy project gives you inspectable code you can run and maintain.

That means the workflow is not just “ask an agent, get some output”. It is a path toward a real web data project.

Enhancements to skills

We have been improving the skills themselves, across all integrations.

Recent updates include:

Smarter Scrapy Cloud runs

When a Scrapy Cloud job starts, the skill can now wait for it to finish, report the results, and, if the output looks wrong, attempt to fix the problem and rerun the job. That means the workflow is moving beyond code generation toward run-check-improve loops.

More efficient extraction from list pages

If all required fields can be extracted from list pages, the skills can now generate code that does that without requesting detail pages unnecessarily. That can make spiders faster, simpler, and more efficient.

Better login handling

The Zyte login flow now uses OAuth and stores credentials in the project’s .env file for reuse by scraping skills. That reduces setup friction and makes repeated workflows easier.

More reliable Scrapy Cloud deployment

The skills now generate frozen requirements.txt files where needed, point scrapinghub.yml at them, and provide clearer guidance for refreshing dependencies. They also include smarter Scrapy stack selection and more troubleshooting guidance. In practice, this helps reduce the gap between “the agent generated a project” and “the project runs reliably in the cloud.”

The next step for agentic web data

The first wave of AI coding assistants helped developers write code faster. The next wave is about agents that can plan, execute, test, and improve real workflows.

For web data, that means moving from “generate me a scraper” to something more useful:

Describe the data.
Validate the output.
Generate the project.
Run it.
Improve it.
Operate it.

That is the direction we are building toward with our Zyte Agentic Web Data line-up.

And with Agent Skills, GitHub Copilot CLI, and Codex support, that workflow is now available in more places developers already work.

The compiler caught a lot. It didn't catch enough.

John Rooney — Sun, 31 May 2026 18:20:02 +0000

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile.

That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong.

How I worked

I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo, move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made.

That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't.

The bug that the toolchain told me wasn't there

I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in a running design journal it kept:

Added ctx.resolve(href) and ctx.resolve_and_visit(href) ... The examples were updated to use ctx.resolve_and_visit(&href).

cargo test passed. cargo check passed. I believed the journal. Here is the actual line in the feature-gated example:

.on_html("li.next a", |el, ctx| {
    if let Some(href) = el.attr("href") {
        ctx.resolve_and_visit(href);   // href is String; the method wants &str
    }
})

resolve_and_visit takes &str. href is a String. That does not compile — String doesn't coerce to &str in argument position, only &String does. The Zyte example got the &; the wreq one didn't. The model fixed one of the two call sites it claimed to have fixed, and reported both as done.

So why was the build green? Because the broken example is behind a feature flag, and cargo check and cargo test don't build it by default. You have to ask:

$ cargo check --all-targets --all-features
error[E0308]: mismatched types
  --> examples/books_direct.rs:20:39
   |
20 |                 ctx.resolve_and_visit(href);
   |                     ----------------- ^^^^ expected `&str`, found `String`

This is the part worth sitting with. The compiler would have caught it — it's a textbook E0308, the friendliest error rustc produces, complete with help: consider borrowing here. The type system did its job perfectly. It just never ran on that file, because the default build target set didn't include it, and nobody — not me, not the model — pointed it at the path where the error lived. The oracle was switched off for exactly the line that needed it, and the green checkmark covered the gap.

What the compiler buys you, honestly

I don't want to undersell the good part, because it's real and it's specific. The most interesting moment in the whole project was the model refactoring the Element type. Originally it stored matched HTML as a string and re-parsed it on every field access — three accessors, three parses of the same fragment. The fix was to parse once and store the scraper::Html. But Html is !Send, and the crawl loop runs callbacks inside spawned tasks. The model reasoned about this explicitly:

Html is !Send but this is safe: Element is only created and consumed inside synchronous callbacks and is never stored in an Arc or sent across threads.

And it structured the crawl loop to hold that invariant — parse the document, run every callback, drain the results into owned Vecs, and drop the Html before the first .await:

let (all_visits, all_items) = {
    let doc = Html::parse_document(&html);
    // ... run callbacks, collect owned results ...
    (all_visits, all_items)
    // doc dropped here, before any await
};

In Python this reasoning is a comment you hope stays true. In Rust, if the model had gotten it wrong — held the Html across the await, stuffed an Element into the task's captured state — it wouldn't compile. The Send bound is load-bearing. That's the version of "the compiler helps the AI" that actually holds up: it converts a class of concurrency mistakes into build failures, and the model can lean on that to attempt refactors it would otherwise have to be timid about.

The same goes for the ordinary stuff. Widening push_item(Value) to push_item<T: Serialize>, swapping a spin-polling semaphore loop for a JoinSet, threading a FetchError enum through the fetch path — these landed cleanly and idiomatically, because they're well-trodden Rust patterns and the types kept the model honest about the seams.

The class of bugs that compiles fine

Then I had a second model do a senior-review pass over the finished code, and it found seven things the green build was hiding. None of them are type errors. All of them compile.

The library panics on normal user mistakes — an invalid selector, a missing API key, a bad output path all unwrap/expect their way to a crash, even though run() returns a ScrapeResult as if failure were representable. concurrency(0) doesn't error; the inner while tasks.len() < 0 loop never spawns anything, tasks.is_empty() is immediately true, and the crawl exits having silently processed nothing. join_next().await discards its Result, so a panicking callback vanishes and the run still reports success. And the stats are quietly miscounted: fetch_errors never increments for HTTP 4xx/5xx — only for network and parse failures — so a crawl that 500s on every page can report fetch_errors: 0 and had_errors() == false. The README, meanwhile, promises that successful status codes are tracked; they aren't recorded at all.

Every one of these is the same shape of mistake: code that satisfies the types and the test suite while doing the wrong thing. The compiler has nothing to say about whether concurrency(0) is meaningful, whether a 500 counts as an error, or whether your docs match your behavior. Those are semantic claims, and semantics is exactly the layer where LLMs are fluent and confident and wrong — and it's the layer Rust doesn't police.

There's a tidy demonstration of this sitting in the repo: the design journal and the README both describe the intended state of the code, not its actual state. The journal says both examples were fixed; one wasn't. The README says status codes are tracked; they're partly not. The model writes the world as it meant to leave it, and prose has no type checker.

You can't review what you can't read

Go back through those seven findings and ask what it would have taken to catch each one by reading. concurrency(0) exiting silently: you'd have to know that while tasks.len() < 0 is vacuously false, then trace what an empty JoinSet does on the next line. The fetch_errors miscount: you'd have to be holding the intended meaning of "error" in your head and notice the HttpError arm bumps a status counter where you expected it to bump the error count. The &href that wouldn't compile: you'd have to know that String doesn't coerce to &str in argument position — the single Rust fact the entire bug turns on.

None of these are exotic. They're the things you internalize after enough hours in the language. But that's precisely the trap when a learner pairs with an AI: the model emits code that looks right, reads fluently, and compiles, and the only way to know it's wrong is to already know the thing you were hoping the model would handle for you. Knowing Rust well wouldn't have stopped the model from writing the concurrency(0) hole — it would have stopped me from nodding past it in review.

So the green build gets promoted past its pay grade. When you can't evaluate the semantics yourself, "it compiles" slides from not obviously broken to works, because compiling is the only check you're actually equipped to read. That promotion is the real hazard of writing an unfamiliar language with a model that writes it confidently. The bugs didn't come from the model being bad at Rust — it handles the syntax better than I do. They came from neither of us being positioned to see the gap between code that satisfies the types and code that does the right thing. The model can't see it because it's a semantic claim about intent; I couldn't see it because I didn't yet know what right looked like.

So, is Rust better for AI now?

For the bugs Rust can see, yes, unambiguously, and more than I expected before I watched the !Send refactor go through. The borrow checker and the trait system catch a real category of AI error at compile time, and that lets a model attempt more aggressive changes without silently corrupting state. The floor is genuinely higher than in a dynamic language.

But the floor isn't the problem. Most of what was wrong with ferrous compiled, passed its tests, and was described accurately by documentation that was itself false. The compiler raised the floor; it did nothing for the ceiling, and feature flags punched a hole in even the floor by hiding a target from the default build. "It compiles, the tests are green, and the model says it's done" turned out to be three independent false comforts stacked on top of each other.

The lesson I'd actually act on has two parts. The mechanical one is cheap: the green checkmark is scoped and the model doesn't know the scope, so run --all-features --all-targets on every check, read the diff instead of the summary of the diff, and treat anything the model asserts about its own output — "both examples updated," "status codes tracked" — as a claim to verify rather than a result to log.

The other part is slower and matters more. If you're using an AI to write a language you're still learning, the AI is not a substitute for learning it — it's the thing that makes the learning feel optional right up until a 500 gets counted as a success. The type system is a fast, narrow oracle that flags the bugs it can see and stays silent on the ones that matter most. Closing that silence is on you, and you can only close it by knowing the language well enough to read what the model wrote and see where it's quietly wrong. I came out of ferrous knowing more Rust than I went in with. That, more than the framework, was the point — and it's the only thing that would have caught the other six.

Why Your Scraper Breaks Without Warning (And How to Fix It)

John Rooney — Sun, 31 May 2026 18:08:26 +0000

Most scraper failures don't raise exceptions. The spider finishes, the pipeline writes a file, the process exits with code 0 — and the output contains 0 items, or 800 instead of 8,000, or fields that are all empty strings. A CSS selector stopped matching after a site redesign. A field that used to be present is now conditional. The "next page" link moved.

None of these produce tracebacks. Without explicit checks, you won't know until someone looks at the data.

This post covers three layers of defence: validating individual items before they reach storage, checking aggregate counts at the end of a run, and setting up logging that's actually useful.

Why selectors fail silently

BeautifulSoup and Scrapy's CSS selectors return None or an empty list when nothing matches — they don't raise. The problem propagates depending on what you do next.

# This raises AttributeError if price_element is None
price_element = book.find("p", class_="price_color")
price = price_element.text.strip()  # AttributeError: 'NoneType' has no 'text'

That's a loud failure, which is fine. But this version is silent:

price = book.css("p.price_color::text").get(default="")

get(default="") is convenient, but it means a changed selector produces empty strings in your output rather than an error. You end up with a file full of {"title": "Some Book", "price": "", "rating": ""} records that look complete until you actually check the values.

The fix is validation — checking that the fields you extracted are actually populated before writing anything.

Item validation

A validation function takes an item dict and raises if any required field is missing or empty:

REQUIRED_FIELDS = ["title", "price", "rating"]

def validate_item(item: dict) -> dict:
    problems = [f for f in REQUIRED_FIELDS if not item.get(f)]
    if problems:
        raise ValueError(f"Missing fields {problems}: {item}")
    return item

not item.get(f) catches None, "", [], and 0 — all values that get() defaults produce when a selector fails. If you want to allow 0 as a valid value, use item.get(f) is None instead.

In a standalone scraper:

import requests
from bs4 import BeautifulSoup
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)
logger = logging.getLogger(__name__)

REQUIRED_FIELDS = ["title", "price", "rating"]

def validate_item(item: dict) -> dict:
    problems = [f for f in REQUIRED_FIELDS if not item.get(f)]
    if problems:
        raise ValueError(f"Missing fields {problems}: {item}")
    return item

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

resp = session.get("https://books.toscrape.com/", timeout=15)
resp.encoding = "utf-8"
soup = BeautifulSoup(resp.text, "html.parser")

items = []
errors = 0

for book in soup.find_all("article", class_="product_pod"):
    raw = {
        "title":  book.find("h3").find("a")["title"],
        "price":  book.find("p", class_="price_color").text.strip(),
        "rating": book.find("p", class_="star-rating")["class"][1],
    }
    try:
        items.append(validate_item(raw))
    except ValueError as e:
        logger.warning("Skipping invalid item: %s", e)
        errors += 1

logger.info("Scraped %d valid items, %d errors", len(items), errors)

Whether to skip invalid items or abort depends on the use case. For a scrape where partial data is usable, log and skip. For a scrape where every record matters, raise immediately.

Validating in a Scrapy pipeline

In Scrapy, validation belongs in a pipeline that runs before the storage pipelines. The ItemAdapter class normalises access across Scrapy Items, dataclasses, and plain dicts:

# pipelines.py
from itemadapter import ItemAdapter
import logging

logger = logging.getLogger(__name__)

REQUIRED_FIELDS = {"title", "price", "rating"}


class ItemValidatorPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        missing = REQUIRED_FIELDS - {
            k for k, v in adapter.items()
            if v not in (None, "", [])
        }

        if missing:
            raise ValueError(
                f"Item missing required fields: {missing} | {dict(adapter)}"
            )

        return item

Raising ValueError (or Scrapy's DropItem) from a pipeline drops the item and logs the error. It doesn't stop the spider. Enable it at a lower priority number than your storage pipelines so it runs first:

# settings.py
ITEM_PIPELINES = {
    "myproject.pipelines.ItemValidatorPipeline": 100,
    "myproject.pipelines.JsonlPipeline":         300,
    "myproject.pipelines.SqlitePipeline":        400,
}

Count checking

Per-item validation catches bad fields, but not a situation where the spider simply stopped finding items at all. A selector change that breaks the outer find_all("article", class_="product_pod") call produces zero items with zero errors — everything looks fine.

A count check at the end of a run catches this:

MIN_EXPECTED = 10

logger.info("Scraped %d valid items, %d errors", len(items), errors)

if len(items) < MIN_EXPECTED:
    logger.error(
        "Item count %d below threshold %d — check selectors or site structure",
        len(items),
        MIN_EXPECTED,
    )

In Scrapy, this belongs in an extension that connects to the spider_closed signal:

# extensions.py
from scrapy import signals
import logging

logger = logging.getLogger(__name__)


class ItemCountChecker:
    MIN_ITEMS = 10

    @classmethod
    def from_crawler(cls, crawler):
        ext = cls()
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
        return ext

    def spider_opened(self, spider):
        self.item_count = 0
        spider.crawler.signals.connect(
            self.item_scraped, signal=signals.item_scraped
        )

    def item_scraped(self, item, spider):
        self.item_count += 1

    def spider_closed(self, spider):
        if self.item_count < self.MIN_ITEMS:
            logger.error(
                "Spider %s closed with only %d items (threshold: %d). "
                "Check selectors or site structure.",
                spider.name,
                self.item_count,
                self.MIN_ITEMS,
            )
        else:
            logger.info(
                "Spider %s: %d items scraped.", spider.name, self.item_count
            )

Enable in settings:

EXTENSIONS = {
    "myproject.extensions.ItemCountChecker": 500,
}

The threshold needs thought. A scraper that usually returns 800 items should probably alert below 700, not below 10. Track a few runs first, then set the threshold relative to your expected baseline.

Logging that's useful

Scrapy's default logging is verbose at INFO level — request counts, middleware messages, and stats flood the output along with the things you actually care about. Two settings improve it:

# settings.py
LOG_LEVEL = "WARNING"   # suppress INFO noise from Scrapy internals
LOG_FILE  = "scrape.log"

Set LOG_LEVEL = "WARNING" for Scrapy internals, but log your own pipeline and extension messages at INFO. Because Scrapy uses Python's standard logging module, you can configure your own loggers separately:

import logging
logging.getLogger("myproject.pipelines").setLevel(logging.INFO)
logging.getLogger("myproject.extensions").setLevel(logging.INFO)

For standalone scrapers, the format matters. The default %(message)s loses context. At minimum include the timestamp and level:

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
)

When something goes wrong at 2am, the timestamp is how you correlate a failure with a site change or a rate-limit event.

The three signals that something is broken

In practice, silent scraper failures show up three ways:

A count drop with no errors. Item count is 0 or far below baseline, error count is 0, the spider exited cleanly. This is a selector change — the outer container no longer matches.

A count that looks right but fields are empty. Item count is normal, but price and rating are empty strings across all records. An inner selector changed while the outer one still matches.

A count drop accompanied by validation errors. Item count is lower than expected, and the log contains Skipping invalid item messages. A field is now conditional — present on some items, absent on others, probably due to a new page layout.

Each failure mode points at a different selector level. Logging both counts and individual errors separately makes it easier to tell which one you're dealing with.

Putting it together

A scraper with validation, count checking, and useful logging doesn't need to be complex. These three additions together cover the most common silent failure modes:

Validate each item before writing — catches changed inner selectors
Check the final count against a threshold — catches broken outer selectors
Log counts and errors at the end of every run — makes failures visible without manual inspection

For a production scraper that runs on a schedule, route the logs somewhere you'll see them. LOG_FILE to a location your monitoring system watches, or send the error-level messages to a webhook. A scraper that runs silently and produces empty output for a week is worse than one that fails loudly on the first run.

Tags: python scrapy webscraping tutorial

How to Store What You Scrape (Without Making a Mess)

John Rooney — Sun, 31 May 2026 18:07:43 +0000

The default for a first scraper is usually printing to stdout or dumping everything into a JSON array. Both work for a single test run. Neither works reliably at scale — a JSON array can't be read incrementally, and stdout isn't a data format.

The format you choose matters more than it seems. The wrong one creates problems that appear later: a 2GB JSON file that has to be fully parsed before you can read a single record; a CSV that corrupts silently when a field contains a comma; a database that slows to a crawl because nobody added an index. This post covers the three formats that cover most scraping use cases, and when to reach for each.

JSONL over JSON arrays

A JSON array written to a file is convenient until the scrape crashes halfway through, at which point you have a truncated, unparseable file. You also can't append to it without reading the whole thing first.

JSONL (JSON Lines) writes one JSON object per line. Each line is independently parseable. A crashed scrape leaves a valid file up to the last complete line. Appending is open("file", "a"). Streaming a large file is a for line in f loop.

import json
import requests
from bs4 import BeautifulSoup
from pathlib import Path

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

output = Path("books.jsonl")

with output.open("w") as f:
    resp = session.get("https://books.toscrape.com/", timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        item = {
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        }
        f.write(json.dumps(item) + "\n")

# Reading back
with output.open() as f:
    records = [json.loads(line) for line in f]

print(f"Read {len(records)} records")
print(records[0])

Output:

Read 20 records
{'title': 'A Light in the Attic', 'price': '£51.77', 'rating': 'Three'}

For a paginated scrape, open the file in append mode ("a") and write after each page. If the scrape is interrupted on page 15 of 50, you still have the first 14 pages intact. You can resume from page 15 without re-scraping the ones you already have.

CSV with DictWriter

CSV is the right format when the output has a fixed, flat schema and you need to open it in a spreadsheet or feed it to a tool that expects CSV. For nested data or variable fields, JSONL is more flexible.

import csv
import requests
from bs4 import BeautifulSoup
from pathlib import Path

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

FIELDS = ["title", "price", "rating"]
output = Path("books.csv")

with output.open("w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=FIELDS)
    writer.writeheader()

    resp = session.get("https://books.toscrape.com/", timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        writer.writerow({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

Two things to get right here: newline="" in the open() call is required on Windows (without it, csv.writer produces double line endings); and DictWriter with explicit fieldnames means extra keys in your dict are ignored rather than causing a crash, which matters when scraped data is messier than expected.

DictWriter also handles quoting automatically — if a field contains a comma or a quote character, it wraps the field correctly. Manual string concatenation doesn't.

SQLite for deduplication and querying

SQLite is worth reaching for when you're doing repeated scrapes of the same site (price monitoring, availability checking) and need to avoid re-inserting data you already have. INSERT OR IGNORE skips any row that would violate a UNIQUE constraint, with no error raised.

import sqlite3
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

conn = sqlite3.connect("books.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS books (
        title  TEXT,
        price  TEXT,
        rating TEXT,
        UNIQUE(title)
    )
""")
conn.commit()

resp = session.get("https://books.toscrape.com/", timeout=15)
resp.encoding = "utf-8"
soup = BeautifulSoup(resp.text, "html.parser")

inserted = 0
skipped  = 0

for book in soup.find_all("article", class_="product_pod"):
    try:
        conn.execute(
            "INSERT INTO books (title, price, rating) VALUES (?, ?, ?)",
            (
                book.find("h3").find("a")["title"],
                book.find("p", class_="price_color").text.strip(),
                book.find("p", class_="star-rating")["class"][1],
            ),
        )
        inserted += 1
    except sqlite3.IntegrityError:
        skipped += 1  # already in the DB

conn.commit()
conn.close()

print(f"Inserted: {inserted}, skipped (duplicate): {skipped}")

The parameterised query (? placeholders) is not optional. String formatting SQL with scraped data is a SQL injection vulnerability even when you control the scraper — the scraped site controls the data.

For repeated scrapes where you want to track price history rather than deduplicate, swap the UNIQUE constraint and INSERT OR IGNORE for a timestamp column and a plain INSERT. Each run adds new rows without touching old ones.

Scrapy pipelines

In Scrapy, storage belongs in item pipelines. A pipeline is a class with a process_item method that receives each item as it comes off the spider. Pipelines can be chained — one validates, one writes to JSONL, one writes to SQLite — and each gets called in the order defined in ITEM_PIPELINES in your settings.

# pipelines.py
import json
import sqlite3
from itemadapter import ItemAdapter


class JsonlPipeline:
    def open_spider(self, spider):
        self.file = open("output.jsonl", "w")

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict())
        self.file.write(line + "\n")
        return item  # must return item for the next pipeline to receive it


class SqlitePipeline:
    def open_spider(self, spider):
        self.conn = sqlite3.connect("output.db")
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS items (
                title  TEXT,
                price  TEXT,
                rating TEXT,
                UNIQUE(title)
            )
        """)
        self.conn.commit()

    def close_spider(self, spider):
        self.conn.close()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.conn.execute(
            "INSERT OR IGNORE INTO items (title, price, rating) VALUES (?, ?, ?)",
            (adapter["title"], adapter["price"], adapter["rating"]),
        )
        self.conn.commit()
        return item

Enable in settings.py:

ITEM_PIPELINES = {
    "myproject.pipelines.JsonlPipeline":   300,
    "myproject.pipelines.SqlitePipeline":  400,
}

The integer values are priority — lower numbers run first. The return item at the end of each process_item is mandatory; if you forget it, the next pipeline in the chain receives None.

open_spider and close_spider are the correct place to open and close file handles and database connections. Opening them in __init__ means they persist for the lifetime of the pipeline object even when no spider is running.

Choosing a format

Format	Use when
JSONL	Default for most scrapes; variable fields; needs to survive interruption
CSV	Fixed flat schema; downstream tools expect CSV; spreadsheet output
SQLite	Repeated scrapes of the same targets; need to query across runs; deduplication
PostgreSQL / MySQL	Multiple scrapers writing concurrently; data volumes above ~10GB

The jump to a full database server is rarely needed for a single-scraper project. SQLite handles hundreds of millions of rows on modern hardware without issue, provided you're not writing from multiple processes simultaneously.

Tags: python scrapy webscraping tutorial

Scraping Paginated Sites Without Getting It Wrong

John Rooney — Sun, 31 May 2026 18:07:12 +0000

Pagination is where a lot of scrapers quietly go wrong — not with errors, but with missing data. A scraper that stops two pages early produces no exceptions. Neither does one that re-fetches the same page in a loop. The output just looks thin, and you may not notice until you check it against the actual record count.

There are three common pagination patterns. Identifying which one you're dealing with takes 30 seconds in DevTools; implementing it correctly takes another ten minutes. This post covers all three.

Pattern 1: Page number in the URL

The simplest pattern. The URL contains a page parameter — either as a query string (?page=2) or as part of the path (/catalogue/page-2.html). Increment it until you get a 404 or an empty result set.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://books.toscrape.com/catalogue/"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_books = []
page = 1

while True:
    url = f"https://books.toscrape.com/catalogue/page-{page}.html"
    resp = session.get(url, timeout=15)

    if resp.status_code == 404:
        break  # past the last page

    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")
    books = soup.find_all("article", class_="product_pod")

    if not books:
        break  # empty page — also done

    for book in books:
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    print(f"Page {page}: {len(books)} books")
    page += 1

print(f"\nTotal: {len(all_books)} books")

Two termination conditions, not one. Some sites return an empty 200 for out-of-range pages rather than a 404. Check both.

Pattern 2: Following the "next" link

A cleaner approach for HTML-paginated sites: let the page tell you where to go next, rather than constructing URLs yourself. Most paginated sites include a "Next" link in the HTML. Follow it until it disappears.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://books.toscrape.com/catalogue/"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

url = "https://books.toscrape.com/catalogue/page-1.html"
all_books = []

while url:
    resp = session.get(url, timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

print(f"Scraped {len(all_books)} books")

The urljoin(BASE, next_btn["href"]) call is worth noting. The href in a "next" link is often relative (page-2.html, ../page-2.html). urljoin resolves it against the base URL correctly regardless of what form the relative path takes. Concatenating strings instead will break on unusual relative paths.

Pattern 3: API cursor / continuation token

JSON APIs often paginate differently. Instead of page numbers, they return a token or flag telling you whether more results exist, and sometimes a cursor to pass back on the next request.

The simplest version: a has_next boolean.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_quotes = []
page = 1

while True:
    resp = session.get(
        "https://quotes.toscrape.com/api/quotes",
        params={"page": page},
        timeout=15,
    )
    resp.raise_for_status()
    data = resp.json()

    all_quotes.extend(data["quotes"])
    print(f"Page {page}: {len(data['quotes'])} quotes")

    if not data["has_next"]:
        break
    page += 1

print(f"\nTotal: {len(all_quotes)} quotes")

Some APIs use a cursor instead — the response includes a next_cursor or next_page_token field that you pass as a parameter on the subsequent request. The structure changes but the loop logic is the same: keep going until the cursor field is null or absent.

# Generic cursor pattern
params = {"limit": 100}

while True:
    resp = session.get("https://example.com/api/items", params=params, timeout=15)
    data = resp.json()

    items.extend(data["results"])

    cursor = data.get("next_cursor")
    if not cursor:
        break
    params["cursor"] = cursor

Rate limiting

Sending requests as fast as the network allows is not scraping — it's a load test. Most sites will rate-limit or block traffic that arrives faster than a human could generate it. A 1-2 second delay between pages is a reasonable starting point; adjust based on the site's response times and any explicit rate-limit headers it sends.

import time

while url:
    resp = session.get(url, timeout=15)
    # ... process page ...

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

    if url:
        time.sleep(1)  # only sleep if there's another request coming

Sleeping after the last page is unnecessary. Put the sleep before the next request or, as above, after confirming there is a next request.

For higher-volume work, time.sleep with a fixed value is blunt. A better approach uses a random delay within a range — time.sleep(random.uniform(0.5, 2.0)) — which avoids the metronomic request timing that fixed delays produce.

Duplicate URL detection

Some sites have inconsistent pagination — "next" links that eventually loop back, or page parameters that wrap around. A simple seen_urls set catches this before it turns into an infinite loop:

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

BASE = "https://books.toscrape.com/catalogue/"
url = "https://books.toscrape.com/catalogue/page-1.html"
seen_urls = set()
all_books = []

while url:
    if url in seen_urls:
        print(f"Loop detected at {url} — stopping")
        break
    seen_urls.add(url)

    resp = session.get(url, timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

print(f"Scraped {len(all_books)} books across {len(seen_urls)} pages")

Using Scrapy's CrawlSpider

If you're building on Scrapy, CrawlSpider handles link following automatically via rules. This is the idiomatic Scrapy approach for sites where pagination follows a consistent pattern:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class BooksSpider(CrawlSpider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    rules = (
        # Follow "next page" links
        Rule(
            LinkExtractor(restrict_css="li.next a"),
            callback="parse_page",
            follow=True,
        ),
    )

    def parse_page(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title":  book.css("h3 a::attr(title)").get(),
                "price":  book.css("p.price_color::text").get(default="").strip(),
                "rating": book.css("p.star-rating::attr(class)").get(default="").split()[-1],
            }

CrawlSpider deduplicates URLs by default (Scrapy's built-in duplicate filter handles it), respects DOWNLOAD_DELAY in your settings, and handles retries. For a site with straightforward pagination, it removes most of the boilerplate above.

One thing to know: CrawlSpider calls the rules on every response, including the ones your callback generates. If a page both contains items and a "next" link, the rule fires correctly — but if you override parse() directly on a CrawlSpider, you'll break the rule processing. Use a separate callback method, as above.

Quick decision guide

Situation	Approach
URL has `/page/2` or `?page=2`	Increment, stop on 404 or empty
Page has a "Next" link in HTML	Follow href with `urljoin`, stop when absent
JSON API with `has_next` flag	Loop until flag is false
JSON API with cursor/token	Pass cursor back each request, stop when null
Building on Scrapy	`CrawlSpider` + `LinkExtractor`

Tags: python scrapy webscraping tutorial

How to Handle JavaScript-Rendered Pages Without a Full Browser

John Rooney — Sun, 31 May 2026 18:06:27 +0000

The HTML that requests downloads is what the server sends before any JavaScript runs. For a large and growing number of sites, that document is nearly empty — a shell with <script> tags that populate the content after the browser executes them. BeautifulSoup finds nothing because there is nothing to find in the source.

Two approaches handle this. The first is to skip the HTML entirely and call the underlying API the JavaScript is already talking to. The second is to use a real browser. The first option is faster, more reliable, and available more often than people expect.

Identifying a JS-rendered page

The test is straightforward: compare what you see in "View Source" against what you see in DevTools' Elements panel. View Source shows the raw server response — exactly what requests receives. The Elements panel shows the live DOM after JavaScript has run.

If your target data appears in the Elements panel but not in View Source, it's JS-rendered.

The other common indicator: requests returns a 200 with a short body. A page that renders 50 product cards in the browser but returns 2KB of HTML to a plain HTTP request is loading its content dynamically.

Option 1: Find the underlying API

When a page loads content via JavaScript, that JavaScript has to get the data from somewhere. Usually it makes an XHR or Fetch request to a JSON API. That same API is available to your scraper — with no browser required.

To find it, open DevTools, go to the Network tab, filter by "Fetch/XHR", then reload the page. Watch for requests that return JSON. Click on one, check the "Preview" tab — if it contains your data, you've found the endpoint.

Here's a concrete example. quotes.toscrape.com/js/ renders its quotes via JavaScript. The raw HTML response contains zero quote elements:

import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

resp = session.get("https://quotes.toscrape.com/js/", timeout=15)
soup = BeautifulSoup(resp.text, "html.parser")
print(len(soup.find_all("div", class_="quote")))  # 0

But the JavaScript is fetching from /api/quotes. That endpoint returns clean JSON and supports pagination via a page parameter:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_quotes = []
page = 1

while True:
    resp = session.get(
        "https://quotes.toscrape.com/api/quotes",
        params={"page": page},
        timeout=15,
    )
    resp.raise_for_status()
    data = resp.json()

    all_quotes.extend(data["quotes"])
    print(f"Page {page}: {len(data['quotes'])} quotes")

    if not data["has_next"]:
        break
    page += 1

print(f"\nTotal: {len(all_quotes)} quotes")

Output:

Page 1: 10 quotes
Page 2: 10 quotes
...
Page 10: 10 quotes

Total: 100 quotes

This is faster than any browser-based approach and produces cleaner data. Change the Accept header to application/json when hitting JSON endpoints — some APIs check it.

When inspecting Network requests, look for:

URLs containing /api/, /graphql, /v1/, /data/, or .json
Requests where the Preview tab shows an object or array
Query parameters like page, offset, cursor, limit — these tell you the pagination model upfront

Not every site exposes a clean API. Some assemble their data server-side, some use GraphQL, some obfuscate the endpoints. When the API route isn't viable, use a browser.

Option 2: Playwright

Playwright drives a real browser, so what it sees is identical to what a user sees. It's slower than requests and consumes more memory, but it's the correct tool when JavaScript execution is unavoidable.

Install:

pip install playwright
python -m playwright install chromium

Basic scrape of the same JS-rendered page:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    quotes = page.locator("div.quote").all()
    for q in quotes:
        text   = q.locator("span.text").inner_text()
        author = q.locator("small.author").inner_text()
        print(f"{text[:70]}... — {author}")

    browser.close()

wait_until="networkidle" tells Playwright to wait until there are no ongoing network requests for at least 500ms — enough time for most JS-driven content to load. For pages with long-running background requests (analytics pings, chat widgets), "domcontentloaded" or a specific element wait is more reliable:

# Wait for a specific element rather than network quiet
page.goto("https://quotes.toscrape.com/js/")
page.wait_for_selector("div.quote")

page.locator() is preferable to page.query_selector_all(). Locators are lazy — they don't execute until you call a method on them — and they retry automatically if the element isn't immediately present. This makes them more tolerant of pages that render content in stages.

Blocking unnecessary resources

By default, Playwright fetches everything: stylesheets, images, fonts, analytics scripts. For scraping, none of that matters. Blocking it cuts load time noticeably:

from playwright.sync_api import sync_playwright

def block_non_essential(route):
    if route.request.resource_type in ("image", "font", "stylesheet", "media"):
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.route("**/*", block_non_essential)

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    texts = page.locator("div.quote span.text").all_inner_texts()
    print(f"Found {len(texts)} quotes")

    browser.close()

The page.route() call intercepts every request. For anything that matches image, font, stylesheet, or media, it calls route.abort() instead of letting it through. The data-carrying requests — the HTML document and any XHR calls — continue normally.

Using Playwright to discover hidden APIs

Even when you intend to use Playwright for the actual scraping, it can show you API calls you didn't know existed. Register a response handler before navigation:

from playwright.sync_api import sync_playwright

api_calls = []

def capture_api(response):
    content_type = response.headers.get("content-type", "")
    if "json" in content_type:
        api_calls.append(response.url)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.on("response", capture_api)

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    print("JSON responses observed:")
    for url in api_calls:
        print(f"  {url}")

    browser.close()

If any JSON responses appear, check whether they contain your target data. If they do, you can switch from Playwright to a plain requests call against that URL — no browser needed for subsequent runs.

Choosing between the two approaches

API scraping is worth the investigation time. A requests loop that pages through a JSON API runs 10-50x faster than Playwright, uses a fraction of the memory, and produces structured data without parsing HTML. If the endpoint exists and isn't authenticated in a way you can't replicate, use it.

Playwright is the right answer when: the page builds its content from multiple sources with no single API, authentication involves cookies set by JavaScript challenges, or content only appears after user interactions like scroll events or button clicks.

The two approaches also compose. Use Playwright to log in and capture session cookies, then hand those cookies to a requests.Session for the actual data collection:

from playwright.sync_api import sync_playwright
import requests

# Step 1: get session cookies via browser
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/login")
    page.fill("#username", "you@example.com")
    page.fill("#password", "your-password")
    page.click("button[type=submit]")
    page.wait_for_url("**/dashboard")
    cookies = page.context.cookies()
    browser.close()

# Step 2: transfer cookies to requests session
session = requests.Session()
for cookie in cookies:
    session.cookies.set(cookie["name"], cookie["value"], domain=cookie["domain"])

# Step 3: scrape with requests
resp = session.get("https://example.com/api/data", timeout=15)
print(resp.json())

Tags: python webscraping playwright tutorial

Why Your Scraper Works in the Browser But Fails in Python

John Rooney — Sun, 31 May 2026 18:05:05 +0000

When a requests.get() call returns a 403, or a 200 with an "Access Denied" body, the first instinct is usually to blame the site. But the more likely explanation is that the server received a request that doesn't look anything like what a browser sends — and responded accordingly.

HTTP servers see every header your client sends. A bare requests call sends four. Chrome sends around fifteen, and the values are specific enough that the gap is obvious server-side. This post covers what that gap looks like, why it matters, and how to close it.

What requests actually sends by default

Start a fresh Python session and inspect what requests puts on the wire:

import requests

session = requests.Session()
req = requests.Request("GET", "https://example.com")
prepared = session.prepare_request(req)

for header, value in prepared.headers.items():
    print(f"{header}: {value}")

Output:

User-Agent: python-requests/2.33.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

Four headers. That's it. A real Chrome browser on the same request would send around 15-20, and the content of each one is meaningfully different.

The User-Agent is the most obvious signal — python-requests/2.33.1 is not subtle — but it's also the least interesting one to fixate on, because it's rarely the only reason a request fails. The deeper issue is the overall fingerprint: the combination of which headers are present, in what order, and with what values.

What a browser actually sends

For a standard top-level page navigation, Chrome sends something like this:

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
Accept-Language: en-GB,en;q=0.9
Accept-Encoding: gzip, deflate, br
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Connection: keep-alive

A few things stand out:

Accept is specific. The browser advertises exactly which content types it can handle, with quality weights (q=0.9). The requests default of */* says "anything goes," which is an unusual declaration for something claiming to be a browser.

Sec-Fetch-* headers are a family added by Chrome in 2019. Sec-Fetch-Dest: document says the request is fetching a top-level document. Sec-Fetch-Mode: navigate says it's a user-initiated navigation. Sec-Fetch-Site: none says there's no referring site (i.e., it was typed directly or bookmarked). These headers don't affect what most sites return, but sites that check for them will immediately identify an absent set as non-browser traffic.

Accept-Language identifies the browser's locale. Absent entirely in a raw requests call.

Building a session with browser-like headers

The right approach is to set your headers once on a Session object, not on every individual request. Sessions also handle cookies automatically across requests, which matters as soon as you need to scrape pages behind a login or track state.

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}

session = requests.Session()
session.headers.update(HEADERS)

With this session, every session.get() and session.post() call will include these headers automatically. You can override individual headers per-request by passing a headers dict to the call — the per-request dict is merged with the session headers, with the per-request values winning on collision:

# Adds Referer for this request only; all other session headers still apply
resp = session.get(
    "https://example.com/products/",
    headers={"Referer": "https://example.com/"},
)

A working example

books.toscrape.com is a site built specifically for scraping practice. Here's a complete working scraper using the session approach:

import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}

session = requests.Session()
session.headers.update(HEADERS)

resp = session.get("https://books.toscrape.com/", timeout=15)
resp.raise_for_status()
resp.encoding = "utf-8"  # server omits charset in Content-Type; set it explicitly

soup = BeautifulSoup(resp.text, "html.parser")
books = soup.find_all("article", class_="product_pod")

for book in books:
    title  = book.find("h3").find("a")["title"]
    price  = book.find("p", class_="price_color").text.strip()
    rating = book.find("p", class_="star-rating")["class"][1]
    print(f"{title} | {price} | {rating} stars")

Output:

A Light in the Attic | £51.77 | Three stars
Tipping the Velvet | £53.74 | One stars
Soumission | £50.10 | One stars
Sharp Objects | £47.82 | Four stars
Sapiens: A Brief History of Humankind | £54.23 | Five stars
...

One note on resp.encoding = "utf-8": when a server sends Content-Type: text/html without a charset parameter, requests defaults to ISO-8859-1 per the HTTP spec. That produces garbled currency symbols (£ becomes Â£). Setting it explicitly to UTF-8 before accessing resp.text fixes it. Alternatively, pass resp.content (raw bytes) to BeautifulSoup and let it detect the encoding from the HTML meta tags — but that can fail on malformed pages, so explicit is safer.

When you need to scrape multiple pages concurrently

If you're pulling many pages and speed matters, httpx gives you an async-compatible API with the same session model. The header setup is identical; you just swap the client:

import asyncio
import httpx
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}

async def fetch_pages(urls: list[str]) -> list[httpx.Response]:
    async with httpx.AsyncClient(
        headers=HEADERS,
        timeout=15,
        follow_redirects=True,
    ) as client:
        tasks = [client.get(url) for url in urls]
        return await asyncio.gather(*tasks)

urls = [
    "https://books.toscrape.com/catalogue/page-1.html",
    "https://books.toscrape.com/catalogue/page-2.html",
    "https://books.toscrape.com/catalogue/page-3.html",
]

responses = asyncio.run(fetch_pages(urls))

for url, resp in zip(urls, responses):
    soup = BeautifulSoup(resp.text, "html.parser")
    books = soup.find_all("article", class_="product_pod")
    print(f"{url} -> {len(books)} books")

Output:

https://books.toscrape.com/catalogue/page-1.html -> 20 books
https://books.toscrape.com/catalogue/page-2.html -> 20 books
https://books.toscrape.com/catalogue/page-3.html -> 20 books

httpx.AsyncClient fetches all three pages concurrently rather than sequentially. For 3 pages the difference is negligible; for 50 it's significant.

One thing to watch: asyncio.gather() will raise an exception if any request fails. In production you'd want to wrap each client.get() call in a try/except or use a return_exceptions=True argument to gather() so a single failed request doesn't kill the whole batch.

A helper to catch silent failures

A 200 response doesn't mean you got what you wanted. Some sites return 200 with a block page or a challenge in the body. Add a quick check to your fetch function:

def fetch(session: requests.Session, url: str, **kwargs) -> requests.Response:
    resp = session.get(url, timeout=15, **kwargs)
    resp.raise_for_status()

    body = resp.text.lower()
    if "access denied" in body or "captcha" in body or "enable javascript" in body:
        raise ValueError(f"Block signal detected in response body for {url}")

    return resp

This won't catch every case — some challenge pages use different wording, and JavaScript-rendered content simply won't appear in the HTML at all — but it catches the obvious ones early and turns a silent data quality problem into a loud error.

Quick reference: headers you should always set

Header	Why it matters
`User-Agent`	Identifies the client; `python-requests/x.x` is an instant flag
`Accept`	Advertises supported content types; `/` is unusual for browser traffic
`Accept-Language`	Absent in raw `requests`; browsers always send it
`Upgrade-Insecure-Requests`	Tells the server you prefer HTTPS; browsers send this on HTTP requests
`Sec-Fetch-Dest`	Part of the Fetch metadata spec; absent headers are a signal
`Sec-Fetch-Mode`	As above
`Sec-Fetch-Site`	As above
`Sec-Fetch-User`	Indicates user-initiated navigation

Copy the values directly from your browser's DevTools for the site you're targeting. The values above are correct for Chrome on Linux navigating to a top-level page; they differ slightly for XHR requests, POST submissions, and sub-resource loads (images, scripts).

What this doesn't solve

Better headers get you further than the bare requests default, but they're not a complete solution. Two scenarios they won't help with:

JavaScript-rendered content. If the data you want is injected by JavaScript after the initial HTML loads, requests and httpx will never see it. The HTML response simply won't contain those elements. The fix for that is a real browser via Playwright or Selenium — or finding the underlying API call the JavaScript is making (often easier and more reliable). That's what the next post in this series covers.

TLS fingerprinting. The HTTP headers your scraper sends are one signal; the TLS handshake is another. Some detection systems check the cipher suite order and TLS extension profile your client presents, which differs from Chrome's even if your headers match exactly. requests uses Python's ssl module, which has a distinct fingerprint. Addressing that requires a library like curl_cffi that wraps libcurl's TLS stack and can impersonate Chrome's handshake.

For most sites and most scraping tasks, the session approach above is enough to get started. When it isn't, the failure mode is usually clear: you'll get 403s, CAPTCHAs, or suspiciously empty responses regardless of how you set your headers.

Tags: python scrapy webscraping tutorial

Building a self-hosted browser scraping service (is it more hassle than its worth?)

John Rooney — Wed, 27 May 2026 13:54:31 +0000

There is a version of this project that is not worth doing. If you need browser rendering for a handful of URLs, pointing Playwright at a local binary and running it is fine. If you need to scale to thousands of requests and you want someone else to manage infrastructure, fingerprinting, proxies, and binary maintenance, Zyte API's headless browser handles all of that without any of what follows.

But if you want to understand exactly how a browser scraping service works at the infrastructure level, or you have a steady workload that you want running on hardware you already own, building one yourself teaches you things that matter. This article documents what that build required, the decisions behind each part of it, and the places where I would reach for Zyte API instead.

The architecture: separating the browser from the code that drives it

The foundational decision is understanding that Playwright is a control library, not a browser. It speaks Chrome DevTools Protocol (CDP) to whatever binary you point it at, and the binary is entirely separate from the library. This distinction is what makes a remote browser service possible.

# Local: Playwright launches and manages the browser itself
browser = await p.chromium.launch()

# Remote: Playwright connects to a browser running elsewhere
browser = await p.chromium.connect("ws://192.168.1.100:3000")

# From here, the API is identical
context = await browser.new_context()
page = await context.new_page()
await page.goto("https://example.com")

When you call playwright.connect(), the library stays on your machine and the browser runs on the server. Your scraping scripts become clients of a persistent browser service, which means multiple projects can share one browser instance, and the hardware running the browser can be completely separate from the hardware running your code.

The finished setup is four things working together: a patched Chromium binary (CloakBrowser), a virtual framebuffer so the browser runs headed on a machine with no display (Xvfb), the Playwright server process that accepts WebSocket connections, and Docker with supervisord managing the whole thing.

I am running this on a HP ProDesk 405 G6 with a Ryzen 4650G and 32GB of RAM. It is a small form factor desktop that draws very little power, runs Linux natively, and handles 16 concurrent browser contexts without difficulty.

Why the choice of binary matters

When a browser is put into automation mode, it is supposed to advertise that fact. navigator.webdriver = true is in the W3C WebDriver spec, not an incidental side effect of Playwright. Detection is not finding a bug in your setup; it is reading a flag the spec requires.

The detection surface has three distinct layers. At the JavaScript level there are visible properties: navigator.webdriver, the shape of navigator.plugins, and the presence or absence of window.chrome. These can be overridden before the page loads, but the overrides are detectable because the property descriptor and prototype chain look different from what a native property would produce. At the binary level there are internal automation flags and the CDP debugging port being open on localhost, which pages can probe via timing differences in connection failures. At the network level, TLS handshake characteristics and HTTP/2 settings are compiled into the browser's network stack and cannot be changed from JS or from Playwright settings.

# navigator.webdriver in a standard Playwright browser
> navigator.webdriver
true

# In a patched binary, the property is removed at the source
> navigator.webdriver
undefined

Projects like CloakBrowser patch Chromium at the C++ level before compilation, which means the signals are never emitted rather than overridden after the fact. A JS-level patch leaves something to detect; a binary-level patch does not. This is the reason patched binaries exist rather than simply using playwright-stealth or similar libraries.

Getting CloakBrowser into the container requires a specific step: Playwright maintains two Chromium slot directories, and you need to replace both.

# Replace both the full Chromium slot and the headless shell slot
RUN npx playwright install chromium \
    && SLOT=$(ls /root/.cache/ms-playwright/ | grep '^chromium-') \
    && CHROME_DIR="/root/.cache/ms-playwright/$SLOT/chrome-linux64" \
    && rm -rf "$CHROME_DIR" \
    && cp -r /browsers/chromium/. "$CHROME_DIR/" \
    && chmod +x "$CHROME_DIR/chrome" \
    # Playwright prefers the headless shell for headless mode
    # Replace this slot too or Playwright will ignore your patched binary
    && HS_SLOT=$(ls /root/.cache/ms-playwright/ | grep '^chromium_headless_shell-') \
    && HS_DIR="/root/.cache/ms-playwright/$HS_SLOT/chrome-headless-shell-linux64" \
    && rm -rf "$HS_DIR" \
    && cp -r /browsers/chromium/. "$HS_DIR/" \
    && mv "$HS_DIR/chrome" "$HS_DIR/chrome-headless-shell" \
    && chmod +x "$HS_DIR/chrome-headless-shell"

The second slot (chromium_headless_shell) is what Playwright uses when it runs in headless mode. If you only replace the first slot, Playwright silently falls back to its bundled binary and your patched version is never used. This took several hours to diagnose, and the only way to catch it was watching ps aux during an active browser session to read the actual binary path in the process arguments.

Why headed mode, and why that requires Xvfb

Headless mode is one of the more reliable detection signals available. The browser reports different screen properties, WebGL behaves differently, the font rendering pipeline changes, and the User-Agent string typically contains HeadlessChrome rather than Chrome. The fingerprint for headless Chromium has been studied for years by antibot vendors.

Running the browser headed via Xvfb (X Virtual Framebuffer) removes this entire category of signal. Xvfb provides a virtual display that the browser renders into without needing a physical monitor. The browser has no idea it is running on a headless machine; its screen properties, rendering pipeline, and UA string all reflect a genuine headed session.

# Install Xvfb alongside browser dependencies
RUN apt-get install -y xvfb

# Set the display environment variable
ENV DISPLAY=:99

The Playwright server startup script starts Xvfb on display :99 before launching the server:

#!/bin/bash
export DISPLAY=":99"
export PLAYWRIGHT_CHROMIUM_USE_HEADLESS_NEW=0
export PW_TEST_HEADED=1
exec npx playwright run-server --port 3000 --host 0.0.0.0

The tradeoff is slightly higher memory per context compared to headless mode. On 32GB of RAM running 16 concurrent contexts, this is not a practical constraint.

Why supervisord rather than a simpler process setup

A browser service is not one process. It is Xvfb, the Playwright server, and eventually many browser child processes. Docker's default model is one foreground process per container, which does not fit. A shell script with basic process management works until something crashes out of order; supervisord handles ordering, monitoring, and restart behavior cleanly.

[program:xvfb]
command=Xvfb :99 -screen 0 1920x1080x24 -ac
autorestart=true
priority=10

[program:playwright]
command=/start-playwright.sh
autorestart=true
priority=20
startsecs=3
stdout_logfile=/var/log/supervisor/playwright.log
stderr_logfile=/var/log/supervisor/playwright.log

The priority ordering ensures Xvfb is running before Playwright starts. If either process crashes, supervisord restarts them in the correct sequence. One detail worth noting: environment variables set in supervisord's environment directive do not reliably propagate into child processes. The Playwright startup script sets them directly with export to avoid this.

Concurrency: one browser instance, many contexts

A single browser instance runs multiple isolated contexts. Each context has separate cookies, separate session storage, and separate state, so contexts behave like independent browser profiles sharing one process. For most scraping workloads, one instance with a pool of contexts is the right model: you avoid the startup cost of launching a new process for each request while maintaining clean isolation between sessions.

The async queue pattern works well here. Workers pull URLs from the queue, create a context, scrape, close the context, and immediately pick up the next URL. A 403 response requeues the URL with a backoff delay and frees the worker to continue with other jobs.

async def worker(worker_id, browser, queue, results):
    while True:
        url, attempt = queue.get_nowait()
        result, should_retry = await scrape(browser, url)

        if result:
            results.append(result)
        elif should_retry and attempt < MAX_RETRIES:
            await asyncio.sleep(2 * attempt)
            await queue.put((url, attempt + 1))

        queue.task_done()

# Spin up N workers against the same browser instance
semaphore = asyncio.Semaphore(CONCURRENCY)
workers = [
    asyncio.create_task(worker(i, browser, queue, results))
    for i in range(CONCURRENCY)
]
await queue.join()

Proxy credentials go in new_context() per context, not at the browser level. Using residential proxies with sticky sessions means the same exit IP handles the full page load and all subresource requests, which matters for sites that correlate requests within a session.

context = await browser.new_context(
    proxy={
        "server": "http://proxy-provider:port",
        "username": "your-username",
        "password": "your-password",
    },
    locale="en-US",
    timezone_id="America/New_York",
    viewport={"width": 1920, "height": 1080},
)

# Block unnecessary resource types to reduce proxy bandwidth
await context.route(
    "**/*",
    lambda route: route.abort()
    if route.request.resource_type in ("image", "media", "font", "stylesheet")
    else route.continue_()
)

Blocking images, fonts, and stylesheets at the context level cuts proxy bandwidth significantly without affecting the data you are trying to extract. At 16 concurrent contexts on the ProDesk, throughput is limited by proxy response time rather than CPU or memory.

What this setup requires of you

The list of things you need to manage and maintain: binary updates as antibot vendors adapt to CloakBrowser, Docker image rebuilds when Playwright updates and the slot structure changes, proxy provider accounts and rotation logic, Xvfb stability under load, supervisord configuration, and the ongoing work of tuning context settings for new target sites.

This is not a set-it-and-forget-it infrastructure. It is a platform that requires active maintenance, and the engineering time that goes into it is real. As the challenges of scaling Playwright and Puppeteer make clear, the operational surface of a browser scraping operation grows quickly once you move beyond a single machine.

When Zyte API is the better answer

For many use cases, Zyte API removes the operational overhead described above entirely. Zyte API's headless browser is a purpose-built scraping browser with proxy management, session handling, and unblocking built in. You make a request, you get a rendered page. The binary maintenance, the fingerprint tuning, the proxy rotation, and the infrastructure management are handled for you.

The comparison with a self-hosted setup comes down to three questions.

Scale and cost. At low to medium volume, a home server with existing hardware costs only proxy fees and electricity. At high volume, the per-request pricing of a managed service can be more economical than the engineering time required to maintain and scale your own infrastructure.

Maintenance tolerance. Antibot vendors update their detection logic continuously. Staying ahead of them with a self-hosted binary means tracking binary releases, testing against real targets, and rebuilding regularly. Zyte API abstracts this.

Integration depth. If you are already working in the Scrapy ecosystem, Scrapy Cloud and the Zyte API Scrapy integration give you a managed pipeline with monitoring, scheduling, and data delivery. Building the equivalent from scratch on self-hosted infrastructure is a significant project.

A working self-hosted setup with a patched binary, residential proxies, and 16 concurrent contexts gets through a meaningful range of real targets. For targets that require it and for workloads that justify the maintenance overhead, it is a legitimate option. For everything else, start with Zyte API for free and skip the part where you watch ps aux for 30 minutes trying to figure out why Playwright is launching the wrong binary.

The repo

The complete setup, including the Dockerfile, docker-compose configuration, and supervisord setup, is available at [github link]. CloakBrowser is sourced separately from their releases page and is not included in the repository. The Dockerfile handles replacing both Playwright Chromium slots with the patched build once you have the tarball in place.

How to write and publish a Python package to PyPI

John Rooney — Mon, 11 May 2026 09:57:41 +0000

I wanted to publish my Scrapy download handler to PyPi - UV made it incredibly easy. Here's how.

At some point, every scraping developer writes the same middleware twice.

The first time is in project A. It works well, so when project B comes along you copy it across. Then project C. Then a colleague asks for it. You email them the file. They make changes. Now there are four versions of the same code living in four different repositories, diverging slowly, none of them getting each other's bug fixes.

The solution is a package. Publish it once to the Python Package Index (PyPI) and every project that needs it can install it with pip install your-package. Updates go to everyone at once. The code has a home.

This guide walks through the full process using uv, a fast, modern Python toolchain that replaces pip, virtualenv, pip-tools, twine, and build with a single tool. We will write a reusable Scrapy download handler, structure it as a proper Python package, test it, and publish it to PyPI.

By the end, the package will be installable with:

uv add scrapy-random-delay

And usable in any Scrapy project with two lines in settings.py:

DOWNLOAD_HANDLERS = {
    "http":  "scrapy_random_delay.RandomDelayHandler",
    "https": "scrapy_random_delay.RandomDelayHandler",
}
RANDOM_DELAY_RANGE = (0.5, 3.0)  # seconds

Installing uv

If you do not have uv installed:

# macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# Or via pip if you prefer
pip install uv

Verify the installation:

uv --version
# uv 0.5.0 (or later)

uv is written in Rust and is dramatically faster than pip for dependency resolution and installation: typically 10 to 100 times faster. More importantly for this guide, it provides a complete project management workflow that makes creating and publishing packages significantly simpler than the traditional pip, build, and twine toolchain.

What we are building

A Scrapy download handler that adds a random delay drawn from a configurable range before each request, with optional per-domain configuration. It is realistic, self-contained, and exactly the kind of code worth packaging: useful across multiple projects but simple enough to understand completely.

How Scrapy download handlers work

A download handler is responsible for a URL scheme. Scrapy's default HTTP handler takes a request, makes the network call, and returns the response. A custom handler wraps this: add headers, impose a delay, swap the underlying HTTP library. The API is simple:

class MyHandler:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    async def download_request(self, request, spider):
        # return a Response
        pass

    def close(self):
        pass

Our handler delegates actual HTTP work to Scrapy's built-in HTTPDownloadHandler and adds the delay logic around it.

Creating the project

uv init scaffolds a new package project in one command:

uv init --package scrapy-random-delay
cd scrapy-random-delay

The --package flag tells uv to create a proper importable package rather than a single-script project. The generated structure looks like this:

scrapy-random-delay/
├── src/
│   └── scrapy_random_delay/
│       └── __init__.py
├── pyproject.toml
└── README.md

uv uses the src/ layout by default, placing your package inside src/ so it cannot be accidentally imported from the project root during development, which can mask packaging errors.

Look at what uv generated in pyproject.toml:

[project]
name = "scrapy-random-delay"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []

[project.scripts]
scrapy-random-delay = "scrapy_random_delay:main"

[build-system]
requires = ["uv_build>=0.11.7,<0.12.0"]
build-backend = "uv_build"

Two things to note: the [project.scripts] entry is scaffolding you can delete, since the package exports a handler class rather than a command-line entry point. The build backend is uv_build, uv's own backend introduced in recent versions. We will fill in the rest of the metadata as we go.

Adding dependencies

Add Scrapy as a runtime dependency:

uv add scrapy

Add development dependencies — pytest and pytest-asyncio for testing:

uv add --dev pytest pytest-asyncio

uv writes these into pyproject.toml automatically and creates a uv.lock lockfile that pins exact versions for reproducible installs. You do not need to manually edit pyproject.toml for dependencies.

The resulting pyproject.toml dependencies section:

[project]
dependencies = [
    "scrapy>=2.12",
]

[dependency-groups]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.24",
]

Writing the package

Add the handler module:

touch src/scrapy_random_delay/handler.py

# src/scrapy_random_delay/handler.py
import asyncio
import random
import logging
from typing import Tuple, Optional

from scrapy import Request
from scrapy.crawler import Crawler
from scrapy.http import Response
from scrapy.settings import Settings
from scrapy.core.downloader.handlers.http11 import HTTP11DownloadHandler as HTTPDownloadHandler

logger = logging.getLogger(__name__)


class RandomDelayHandler:
    """
    A Scrapy download handler that adds a random delay before each request.

    Settings
    --------
    RANDOM_DELAY_RANGE : tuple[float, float]
        (min_seconds, max_seconds) delay range. Default: (0.5, 2.0)

    RANDOM_DELAY_PER_DOMAIN : dict[str, tuple[float, float]]
        Per-domain overrides. Keys are domain strings, values are (min, max) tuples.
        Example: {"api.example.com": (1.0, 4.0), "cdn.example.com": (0.0, 0.1)}

    RANDOM_DELAY_VERBOSE : bool
        Log the actual delay chosen for each request. Default: False
    """

    DEFAULT_RANGE: Tuple[float, float] = (0.5, 2.0)

    def __init__(self, settings: Settings, crawler: Crawler):
        self._settings = settings
        self._range = self._parse_range(
            settings.get("RANDOM_DELAY_RANGE", self.DEFAULT_RANGE)
        )
        self._per_domain: dict = settings.getdict("RANDOM_DELAY_PER_DOMAIN", {})
        self._verbose: bool = settings.getbool("RANDOM_DELAY_VERBOSE", False)
        self._delegate = HTTPDownloadHandler(settings, crawler)

    @classmethod
    def from_crawler(cls, crawler: Crawler) -> "RandomDelayHandler":
        return cls(crawler.settings, crawler)

    def _parse_range(self, value) -> Tuple[float, float]:
        try:
            low, high = float(value[0]), float(value[1])
        except (TypeError, ValueError, IndexError) as e:
            raise ValueError(
                f"RANDOM_DELAY_RANGE must be a two-element sequence of numbers, got: {value!r}"
            ) from e

        if low < 0 or high < 0:
            raise ValueError("RANDOM_DELAY_RANGE values must be non-negative")
        if low > high:
            raise ValueError(
                f"RANDOM_DELAY_RANGE minimum ({low}) must not exceed maximum ({high})"
            )
        return low, high

    def _get_delay_range(self, request: Request) -> Tuple[float, float]:
        from urllib.parse import urlparse
        domain = urlparse(request.url).netloc
        if domain in self._per_domain:
            return self._parse_range(self._per_domain[domain])
        return self._range

    async def download_request(self, request: Request, spider) -> Response:
        low, high = self._get_delay_range(request)
        delay = random.uniform(low, high)

        if self._verbose:
            logger.debug(f"Random delay: {delay:.2f}s before {request.url}")

        await asyncio.sleep(delay)
        return await self._delegate.download_request(request, spider)

    def close(self):
        self._delegate.close()

Update __init__.py to export the handler and set the version:

# src/scrapy_random_delay/__init__.py
from scrapy_random_delay.handler import RandomDelayHandler

__version__ = "0.1.0"
__all__ = ["RandomDelayHandler"]

Writing the tests

Create the test directory:

mkdir tests
touch tests/__init__.py tests/conftest.py tests/test_handler.py

# tests/conftest.py
import pytest
from unittest.mock import MagicMock
from scrapy.http import Request, Response
from scrapy.utils.test import get_crawler


@pytest.fixture
def crawler():
    return get_crawler()


@pytest.fixture
def make_request():
    def _make(url="https://example.com/products", **kwargs):
        return Request(url=url, **kwargs)
    return _make


@pytest.fixture
def mock_response():
    return MagicMock(spec=Response)

# tests/test_handler.py
import pytest
from unittest.mock import AsyncMock, MagicMock, patch

from scrapy.utils.test import get_crawler
from scrapy_random_delay import RandomDelayHandler


def make_handler(settings_dict: dict = None) -> RandomDelayHandler:
    crawler = get_crawler(settings_dict=settings_dict or {})
    with patch("scrapy_random_delay.handler.HTTPDownloadHandler"):
        return RandomDelayHandler(crawler.settings, crawler)


class TestRangeValidation:
    def test_default_range_applied_when_not_configured(self):
        handler = make_handler()
        assert handler._range == (0.5, 2.0)

    def test_custom_range_accepted(self):
        handler = make_handler({"RANDOM_DELAY_RANGE": (1.0, 5.0)})
        assert handler._range == (1.0, 5.0)

    def test_equal_values_accepted(self):
        handler = make_handler({"RANDOM_DELAY_RANGE": (2.0, 2.0)})
        assert handler._range == (2.0, 2.0)

    def test_zero_range_accepted(self):
        handler = make_handler({"RANDOM_DELAY_RANGE": (0.0, 0.0)})
        assert handler._range == (0.0, 0.0)

    def test_inverted_range_raises(self):
        with pytest.raises(ValueError, match="minimum.*must not exceed maximum"):
            make_handler({"RANDOM_DELAY_RANGE": (5.0, 1.0)})

    def test_negative_range_raises(self):
        with pytest.raises(ValueError, match="non-negative"):
            make_handler({"RANDOM_DELAY_RANGE": (-1.0, 2.0)})

    def test_invalid_type_raises(self):
        with pytest.raises(ValueError, match="two-element sequence"):
            make_handler({"RANDOM_DELAY_RANGE": "bad-value"})


class TestPerDomainOverrides:
    def test_per_domain_range_returned_for_matching_domain(self, make_request):
        handler = make_handler({
            "RANDOM_DELAY_RANGE": (0.5, 1.0),
            "RANDOM_DELAY_PER_DOMAIN": {"api.example.com": [2.0, 4.0]},
        })
        request = make_request(url="https://api.example.com/products")
        assert handler._get_delay_range(request) == (2.0, 4.0)

    def test_default_range_returned_for_unmatched_domain(self, make_request):
        handler = make_handler({
            "RANDOM_DELAY_RANGE": (0.5, 1.0),
            "RANDOM_DELAY_PER_DOMAIN": {"api.example.com": [2.0, 4.0]},
        })
        request = make_request(url="https://other.example.com/products")
        assert handler._get_delay_range(request) == (0.5, 1.0)


class TestDownloadRequest:
    @pytest.mark.asyncio
    async def test_delegates_to_inner_handler(self, make_request, mock_response):
        handler = make_handler({"RANDOM_DELAY_RANGE": (0.0, 0.0)})
        handler._delegate.download_request = AsyncMock(return_value=mock_response)

        result = await handler.download_request(make_request(), MagicMock())

        assert result is mock_response
        handler._delegate.download_request.assert_awaited_once()

    @pytest.mark.asyncio
    async def test_delay_is_within_range(self, make_request, mock_response):
        import time
        handler = make_handler({"RANDOM_DELAY_RANGE": (0.05, 0.1)})
        handler._delegate.download_request = AsyncMock(return_value=mock_response)

        start = time.monotonic()
        await handler.download_request(make_request(), MagicMock())
        elapsed = time.monotonic() - start

        assert 0.05 <= elapsed <= 0.5  # generous upper bound for slow CI

    @pytest.mark.asyncio
    async def test_verbose_mode_logs_delay(self, make_request, mock_response, caplog):
        import logging
        handler = make_handler({
            "RANDOM_DELAY_RANGE": (0.0, 0.0),
            "RANDOM_DELAY_VERBOSE": True,
        })
        handler._delegate.download_request = AsyncMock(return_value=mock_response)

        with caplog.at_level(logging.DEBUG, logger="scrapy_random_delay.handler"):
            await handler.download_request(make_request(), MagicMock())

        assert "Random delay" in caplog.text


class TestClose:
    def test_close_delegates_to_inner_handler(self):
        handler = make_handler()
        handler._delegate.close = MagicMock()
        handler.close()
        handler._delegate.close.assert_called_once()

Configure pytest in pyproject.toml by adding this section:

[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths    = ["tests"]

Run the tests with uv:

uv run pytest

uv run executes the command inside the project's virtual environment, which uv manages automatically. No source .venv/bin/activate needed.

Completing pyproject.toml

Fill in the project metadata. uv has already set the basics: add the rest:

[project]
name            = "scrapy-random-delay"
version         = "0.1.0"
description     = "A Scrapy download handler that adds a configurable random delay before each request."
readme          = "README.md"
license         = { file = "LICENSE" }
requires-python = ">=3.9"
authors         = [
    { name = "Your Name", email = "you@example.com" },
]
keywords        = ["scrapy", "web-scraping", "download-handler", "rate-limiting"]
classifiers     = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Framework :: Scrapy",
    "Topic :: Internet :: WWW/HTTP",
    "Topic :: Software Development :: Libraries :: Python Modules",
]
dependencies = [
    "scrapy>=2.8",
]

[dependency-groups]
dev = [
    "pytest>=8.0",
    "pytest-asyncio>=0.24",
]

[project.urls]
Homepage      = "https://github.com/yourname/scrapy-random-delay"
Documentation = "https://github.com/yourname/scrapy-random-delay#readme"
Repository    = "https://github.com/yourname/scrapy-random-delay.git"
Issues        = "https://github.com/yourname/scrapy-random-delay/issues"

[build-system]
requires      = ["uv_build>=0.11.7,<0.12.0"]
build-backend = "uv_build"

[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths    = ["tests"]

The classifiers list is used by PyPI for filtering and discovery. The Framework :: Scrapy classifier ensures your package appears when someone browses the Scrapy ecosystem. Browse the full list at pypi.org/classifiers.

Writing the README

# scrapy-random-delay

A Scrapy download handler that adds a configurable random delay before each request.

Scrapy's built-in `DOWNLOAD_DELAY` setting adds a fixed delay between requests.
This package draws a delay from a uniform distribution between a minimum and maximum,
which produces more natural request timing and is configurable per domain.

## Installation

    uv add scrapy-random-delay
    # or
    pip install scrapy-random-delay

## Usage

Add to your Scrapy `settings.py`:

    DOWNLOAD_HANDLERS = {
        "http":  "scrapy_random_delay.RandomDelayHandler",
        "https": "scrapy_random_delay.RandomDelayHandler",
    }

    RANDOM_DELAY_RANGE = (0.5, 3.0)   # seconds (min, max). Default: (0.5, 2.0)

### Per-domain configuration

    RANDOM_DELAY_PER_DOMAIN = {
        "api.example.com": (1.0, 4.0),   # slower for sensitive endpoints
        "cdn.example.com": (0.0, 0.1),   # faster for static assets
    }

### Verbose logging

    RANDOM_DELAY_VERBOSE = True   # logs the delay chosen for each request

## Settings reference

| Setting | Type | Default | Description |
|---|---|---|---|
| `RANDOM_DELAY_RANGE` | `(float, float)` | `(0.5, 2.0)` | Min and max delay in seconds |
| `RANDOM_DELAY_PER_DOMAIN` | `dict` | `{}` | Per-domain delay overrides |
| `RANDOM_DELAY_VERBOSE` | `bool` | `False` | Log each delay to DEBUG |

## Compatibility

- Python 3.9+
- Scrapy 2.8+

## License

MIT

Publishing to PyPI

Create accounts

You need two accounts: test.pypi.org for the test registry, and pypi.org for the real registry that pip install and uv add use. Use the test registry first, since it resets periodically and will not pollute the real index with test uploads. Enable two-factor authentication on both, as PyPI requires it for publishing.

Publish to Test PyPI

uv has publishing built in via uv publish. No need to install twine or build separately: uv handles the build step automatically.

uv publish --publish-url https://test.pypi.org/legacy/

You will be prompted for your Test PyPI username (__token__) and your API token. Create a token at test.pypi.org/manage/account/token.

Verify the Test PyPI installation

uv add scrapy-random-delay --index https://test.pypi.org/simple/ --extra-index https://pypi.org/simple/

The --extra-index flag allows uv to find dependencies like scrapy from the real PyPI, since they will not be on Test PyPI.

Verify it works:

import scrapy_random_delay
print(scrapy_random_delay.__version__)   # 0.1.0
from scrapy_random_delay import RandomDelayHandler
print(RandomDelayHandler)

Publish to the real PyPI

Once satisfied:

uv publish

Your package is live. Within a few minutes anyone can install it:

uv add scrapy-random-delay
# or
pip install scrapy-random-delay

Using a token stored in the environment

Rather than typing your token on every publish, store it in an environment variable:

export UV_PUBLISH_TOKEN=pypi-your-token-here
uv publish

Or configure it per-project in a .env file (never commit this):

# .env
UV_PUBLISH_TOKEN=pypi-your-token-here

# .gitignore
.env

Automating releases with GitHub Actions

Manually publishing works fine, but automating it means a new version ships by pushing a version tag, with no manual steps:

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  push:
    tags:
      - "v*"   # triggers on tags like v0.1.0, v1.2.3

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Run tests
        run: uv run pytest

  publish:
    needs: test
    runs-on: ubuntu-latest
    environment: pypi

    permissions:
      id-token: write   # required for trusted publishing

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Publish to PyPI
        run: uv publish
        # Uses trusted publishing via OIDC — no token stored in secrets needed
        # Configure at pypi.org/manage/account/publishing/

The workflow is notably short compared to the traditional pip, build, and twine approach, since uv handles the build and publish in a single command.

Trusted publishing is worth setting up. It authenticates the GitHub Actions workflow directly with PyPI using OpenID Connect, so you never need to store a PyPI token as a GitHub secret. Configure it once at pypi.org under your account's Publishing settings, specifying your GitHub username, repository name, and workflow file name.

To cut a release:

# Update version in pyproject.toml and src/scrapy_random_delay/__init__.py
git commit -am "Release v0.2.0"
git tag v0.2.0
git push && git push --tags

The workflow runs, tests pass, the package publishes.

Bumping versions

uv does not yet have a built-in version bump command, but the process is straightforward. The version lives in two places: pyproject.toml and __init__.py, and both need updating together.

For a small package, doing this by hand is fine. For a larger project, bump2version or python-semantic-release automate the version increment, changelog update, git commit, and tag in one command:

uv add --dev bump2version
uv run bump2version patch   # 0.1.0 → 0.1.1
uv run bump2version minor   # 0.1.0 → 0.2.0
uv run bump2version major   # 0.1.0 → 1.0.0

Configure it in pyproject.toml:

[tool.bumpversion]
current_version = "0.1.0"
commit          = true
tag             = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"

[[tool.bumpversion.files]]
filename = "src/scrapy_random_delay/__init__.py"

Now bumping the version, committing, tagging, and triggering the publish workflow is a single command.

Testing against multiple Python versions

Add a test matrix to the workflow to catch version-specific issues before they reach users:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.9", "3.10", "3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v4

      - name: Run tests on Python ${{ matrix.python-version }}
        run: uv run --python ${{ matrix.python-version }} pytest

uv handles Python version management directly: --python 3.9 downloads and uses that Python version without any additional tooling like pyenv.

Keeping your package maintainable

Pin your minimum Scrapy version. scrapy>=2.8 in dependencies sets a floor. Check which Scrapy version introduced the APIs you use and do not allow anything older. uv add scrapy will pick the current latest: tighten the floor to the oldest version you have tested against.

Keep a CHANGELOG. Users upgrading a dependency want to know if anything changed. Keep a CHANGELOG.md with an ## Unreleased section at the top that becomes the next version entry when you tag a release.

Be conservative with dependencies. Every package in dependencies becomes a constraint users must satisfy alongside their own dependencies. A Scrapy extension that depends on pandas and numpy is painful to integrate. Keep the list to what is genuinely required.

Handle deprecation cleanly. When you remove a setting or change an API, keep the old behavior working for at least one minor version and log a deprecation warning. Users who update without reading the changelog will thank you.

Lock files are for applications, not libraries. Commit uv.lock if this is an application. Do not commit it if this is a library, since your users' projects will resolve their own dependency versions and a committed lockfile does not help them.

The complete file tree

After following this guide, your project looks like this:

scrapy-random-delay/
├── src/
│   └── scrapy_random_delay/
│       ├── __init__.py          # exports RandomDelayHandler, sets __version__
│       └── handler.py           # the handler implementation
├── tests/
│   ├── __init__.py
│   ├── conftest.py              # shared fixtures
│   └── test_handler.py          # test suite
├── .github/
│   └── workflows/
│       └── publish.yml          # automated test and publish workflow
├── pyproject.toml               # project metadata, dependencies, tool config
├── uv.lock                      # locked dependency versions (omit for libraries)
├── README.md                    # installation and usage documentation
├── LICENSE                      # MIT or whichever license you choose
└── .gitignore                   # include .env, __pycache__, dist/, *.egg-info/

Next steps

Publishing a Scrapy extension to PyPI is the same process regardless of what the extension does: the download handler is just the example. The same pyproject.toml structure, the same uv publish command, and the same GitHub Actions workflow apply to middlewares, pipelines, item types, and extensions. If you want to go deeper on Scrapy's component system and understand where each type of extension fits, the Modern Scrapy Developer's Guide covers the full architecture from spiders through to deployment on Scrapy Cloud.

DEV Community: John Rooney

Your scraping framework already exists: it's Scrapy

The framework appears one fix at a time

Start with the boring infrastructure

scrapy-poet separates navigation from interpretation

Spidermon defines what a successful crawl means

scrapy-playwright makes the browser an acquisition choice

Write a wreq download handler when the network stack is the variable

Scrapy is a runtime, not the whole platform

A practical migration path

Frequently asked questions

Is Scrapy too old for modern Python and async code?

Can one spider mix normal HTTP, Playwright, and wreq?

When is a plain script still the better choice?

Does Scrapy solve scaling across machines?

Build your scraper, not another crawling framework

So you want to get data from the web

What web scraping becomes once someone depends on it

A small Python scraper should leave evidence behind

JavaScript rendering should be an explicit escalation

Retry transient failures without turning a scrape into a traffic spike

TLS fingerprinting starts before the server sees your headers

A production scraper is a replayable data pipeline

Raw snapshots make parser changes cheap

Idempotent writes make retries safe

Quality gates should fail the run, not merely decorate a dashboard

The shortcuts that fail when the workload grows

How to choose the smallest architecture that will survive

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of requests?

When should I use Playwright for web scraping?

How do I know whether a scraper is returning bad data?

How do you scale web scraping without overwhelming a site?

Start with a replay test

How to use Camoufox with Scrapy (New EASY Method)

Why the stock browsers get caught

The middle ground that was missing

How browser providers actually work

Why this fits the way Scrapy is built

What it changes, and what it doesn't

New Zyte add-ons: Agent Skills, Codex, GitHub and more updates

Three new agent integrations

1. Agent Skills

2. Codex and GitHub Copilot CLI plugins

Agentic scraping: The big idea

How it works

Enhancements to skills

Smarter Scrapy Cloud runs

More efficient extraction from list pages

Better login handling

More reliable Scrapy Cloud deployment

The next step for agentic web data

The compiler caught a lot. It didn't catch enough.

How I worked

The bug that the toolchain told me wasn't there

What the compiler buys you, honestly

The class of bugs that compiles fine

You can't review what you can't read

So, is Rust better for AI now?

Why Your Scraper Breaks Without Warning (And How to Fix It)

Why selectors fail silently

Item validation

Validating in a Scrapy pipeline

Count checking

Logging that's useful

The three signals that something is broken

Putting it together

How to Store What You Scrape (Without Making a Mess)

JSONL over JSON arrays

CSV with DictWriter

SQLite for deduplication and querying

Scrapy pipelines

Choosing a format

Scraping Paginated Sites Without Getting It Wrong

Pattern 1: Page number in the URL

Pattern 2: Following the "next" link

Pattern 3: API cursor / continuation token

Rate limiting

Duplicate URL detection

Using Scrapy's CrawlSpider

When should I use a web scraping framework instead of `requests`?