Your scraping framework already exists: it's Scrapy

The first scraper is usually honest about what it is. It fetches a page, runs a few selectors, and writes some records. Then it needs pagination, so you add a queue. A transient failure loses a page, so you add retries. Two URLs produce the same content, so you add duplicate filtering. The site slows down, so you add per-domain concurrency and delays. Soon the script has lifecycle hooks, plugins, metrics, browser workers, and a home-grown configuration system.

At that point, you are no longer writing a scraper. You are designing a crawling framework, and many of the decisions in front of you have already been made, tested, and exposed as replaceable components in Scrapy.

This does not mean every 20-line script should become a Scrapy project. It means that when a collection of scripts starts acquiring a runtime, you should ask whether that runtime is the part worth inventing. In most cases, the source-specific discovery and extraction logic is yours. The scheduler, retry machinery, throttling, lifecycle, statistics, exports, and component boundaries do not need to be.

The useful way to think about Scrapy is not as a library that gives you response.css(). Parsel already does that. Scrapy is an event-driven crawling runtime with a stable place to put each concern.

The framework appears one fix at a time

Imagine that a scheduled product scraper is growing beyond its original loop. It now needs to remember which URLs it has seen, pause without losing its queue, retry transport failures, apply different concurrency limits to different domains, normalize every product, split large exports into batches, report statistics, and fail the run when output quality drops.

Those requirements tend to produce classes with names such as QueueManager, RetryingClient, PluginRegistry, and RunMonitor. None of those names is inherently a mistake, but together they are a sign that the project is recreating a framework around its parsing code.

Scrapy already divides that system along practical boundaries:

What you are about to build	Scrapy component
URL queue and priority rules	Scheduler
Duplicate URL tracking	Duplicate filter
Retries, redirects, cookies, proxies	Downloader middleware
HTTP, browser, or custom network transport	Download handler
Site navigation and discovery	Spider
Response-to-object extraction	Spider callback or scrapy-poet Page Object
Validation, normalization, persistence	Item pipeline
Lifecycle hooks and operational behavior	Signals and extensions
Counters and run metadata	Stats collector
JSON Lines, CSV, object storage, batching	Feed exports

The boundaries matter more than the names. They let you replace the network transport without replacing the scheduler, or add data-quality checks without importing monitoring code into every callback.

flowchart LR
    Spider["Spider<br/>discovery"] --> Engine["Execution engine"]
    Engine --> Scheduler["Scheduler<br/>queue + dupefilter"]
    Scheduler --> Engine
    Engine --> Middleware["Downloader middleware<br/>retry + redirect + cookies"]
    Middleware --> Handler{"Download handler"}
    Handler --> HTTP["Scrapy HTTP"]
    Handler --> Wreq["wreq"]
    Handler --> Browser["Playwright"]
    HTTP --> Response["Response"]
    Wreq --> Response
    Browser --> Response
    Response --> Poet["Spider / scrapy-poet"]
    Poet --> Pipeline["Item pipelines"]
    Pipeline --> Export["Feed export / database"]
    Signals["Signals + stats"] -. observe .-> Engine
    Signals -. assert .-> Monitor["Core extensions + Spidermon"]

Here is the spider at the centre of that runtime. It contains the site-specific parts: where to begin, how to find product links, and what to yield from a product page.

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products",
            callback=self.parse_listing,
        )

    def parse_listing(self, response):
        yield from response.follow_all(
            response.css("[data-product-link]::attr(href)"),
            callback=self.parse_product,
        )

        next_page = response.css("[rel='next']::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse_listing)

    def parse_product(self, response):
        yield {
            "url": response.url,
            "name": response.css("[data-product-name]::text").get(),
            "price": response.css("[data-price]::attr(data-price)").get(),
        }

Scrapy's selectors use Parsel, so selectors from a smaller Parsel-based scraper move across with little ceremony. The larger change is that callbacks yield work to an engine instead of performing that work directly. The engine can now schedule, deduplicate, retry, throttle, record, and resume it.

The examples below target the current Scrapy async component APIs and Python 3.11 or later, which the Python wreq package requires. In a real project, pin a tested set of versions rather than allowing every integration to update independently.

python -m pip install \
    Scrapy scrapy-poet spidermon scrapy-playwright wreq
playwright install chromium

Start with the boring infrastructure

Before adding an ecosystem package, it is worth seeing how much of the runtime ships with Scrapy itself. A small set of settings gives the product spider adaptive throttling, bounded per-domain concurrency, resumable state, item processing, and compressed batch exports.

# myproject/settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
RETRY_TIMES = 3

ITEM_PIPELINES = {
    "myproject.pipelines.ProductValidationPipeline": 300,
    "myproject.pipelines.ProductUpsertPipeline": 800,
}

FEEDS = {
    "exports/products-%(batch_id)05d.jsonl.gz": {
        "format": "jsonlines",
        "encoding": "utf-8",
        "batch_item_count": 50_000,
        "postprocessing": [
            "scrapy.extensions.postprocessing.GzipPlugin",
        ],
    },
}

AutoThrottle adjusts delay from observed latency for each download slot. Its target concurrency is an average, while CONCURRENT_REQUESTS_PER_DOMAIN remains the ceiling. RetryMiddleware handles configured response codes and download exceptions, and lowers retried requests in the queue. The feed exporter handles serialization, file naming, compression, and batches without making the spider know where output goes.

Resumability is a command-line setting away:

scrapy crawl products -s JOBDIR=crawls/products-2026-07-26

JOBDIR persists the scheduler queue, duplicate-filter state, and spider state so a cleanly stopped crawl can resume. It is not a distributed queue, and a job directory must belong to one spider run. Requests in a disk queue also need serializable callbacks and metadata. These limits are much easier to reason about than an undocumented queue format that grew inside a script.

The item pipeline gives extraction and data handling separate failure modes. The spider describes what it found; ordered pipeline components validate, normalize, drop, or store it.

from decimal import Decimal, InvalidOperation

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class ProductValidationPipeline:
    def process_item(self, item):
        product = ItemAdapter(item)

        if not product.get("name"):
            raise DropItem("missing product name")

        try:
            product["price"] = Decimal(product["price"])
        except (InvalidOperation, TypeError):
            raise DropItem("invalid product price")

        return item

The extension system sits alongside these request and item flows. An extension is normally created once for a crawl and connects to signals such as spider_opened, item_scraped, spider_error, and spider_closed. Scrapy's built-in extensions use the same surface for core stats, periodic log stats, memory limits, close conditions, throttling, feed exports, and other lifecycle behaviour.

That API is small enough that a project-specific extension does not need its own plugin framework:

from scrapy import signals


class ProductStatsExtension:
    def __init__(self, crawler):
        self.stats = crawler.stats

    @classmethod
    def from_crawler(cls, crawler):
        extension = cls(crawler)
        crawler.signals.connect(
            extension.item_scraped,
            signal=signals.item_scraped,
        )
        return extension

    def item_scraped(self, item, response, spider):
        if not item.get("price"):
            self.stats.inc_value("product/missing_price")

EXTENSIONS = {
    "myproject.extensions.ProductStatsExtension": 500,
}

The extension records a fact. It does not decide whether the run should fail. Keeping that distinction makes the counter useful to loggers, dashboards, and monitors without letting one callback quietly define the operational policy for the whole crawl.

scrapy-poet separates navigation from interpretation

As a project adds sources, callbacks often become a mixture of navigation, selectors, normalization, and object construction. scrapy-poet moves page interpretation into Page Objects and injects the right object into a typed callback.

Enable its add-on and tell it where Page Objects live:

ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["myproject.pages"]

The Page Object owns extraction from one kind of page:

# myproject/pages/product.py
from decimal import Decimal

from web_poet import WebPage, handle_urls


@handle_urls("example.com")
class ProductPage(WebPage):
    def to_item(self) -> dict:
        name = self.css("[data-product-name]::text").get()
        price = self.css("[data-price]::attr(data-price)").get()

        return {
            "url": self.url,
            "name": name.strip(),
            "price": Decimal(price),
        }

The spider keeps navigation and asks for the dependency it needs:

import scrapy

from myproject.pages.product import ProductPage


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products/1",
            callback=self.parse_product,
        )

    def parse_product(self, response, page: ProductPage):
        yield page.to_item()

The annotation is not decorative. scrapy-poet inspects it, selects the Page Object using its rules, builds the dependency, and supplies it. You can override rules for a source, inject additional dependencies through providers, and test the Page Object without running a crawl. The spider is left to answer one question: where should the crawler go next?

This separation pays off when several spiders encounter the same page type, or one crawl strategy spans several sites with different representations. It also prevents browser acquisition, API-backed inputs, or cached responses from leaking into extraction code, because those can be provided as dependencies.

One detail matters if you use JOBDIR: callbacks must be serializable. scrapy-poet offers callback_for() to remove boilerplate, but assign the generated callback as a spider method instead of constructing it inline if requests may be written to disk.

Spidermon defines what a successful crawl means

A process can exit with code zero after extracting no products. It can also produce the expected number of records while silently losing every price. Scrapy's stats tell you what happened; Spidermon turns those facts into executable assertions and actions.

Enable the extension and select a monitor suite for spider close:

SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.ProductSpiderCloseSuite",
)

Then express the run contract in terms of the resulting dataset:

from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Product output")
class ProductOutputMonitor(Monitor):
    @monitors.name("Enough products were extracted")
    def test_item_count(self):
        count = getattr(self.data.stats, "item_scraped_count", 0)
        self.assertGreaterEqual(count, 1_000)

    @monitors.name("Missing prices stayed below one percent")
    def test_missing_price_rate(self):
        scraped = getattr(self.data.stats, "item_scraped_count", 0)
        missing = getattr(self.data.stats, "product/missing_price", 0)

        self.assertGreater(scraped, 0)
        self.assertLess(missing / scraped, 0.01)


class ProductSpiderCloseSuite(MonitorSuite):
    monitors = [ProductOutputMonitor]

Spidermon can run suites at spider open, spider close, engine stop, or periodically. Suites can trigger actions after checks pass or fail, and built-in monitors cover common conditions such as item counts, field coverage, finish reasons, retries, errors, and unwanted status codes.

This is a better layer for run-level policy than scattered if count == 0 checks. The spider collects data, components record facts, and monitors decide whether those facts describe useful output.

scrapy-playwright makes the browser an acquisition choice

Some pages do need JavaScript execution or browser state. That does not require the crawler itself to become a browser script. scrapy-playwright implements a Scrapy download handler, so selected requests can use Playwright while the surrounding scheduler, retries, callbacks, pipelines, and monitoring stay in place.

Register its HTTPS handler and the asyncio reactor:

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

DOWNLOAD_HANDLERS = {
    "https": (
        "scrapy_playwright.handler."
        "ScrapyPlaywrightDownloadHandler"
    ),
}

# Let the browser provide a User-Agent that matches itself.
USER_AGENT = None

Requests opt in through metadata. Unmarked requests continue through Scrapy's regular HTTP implementation.

import scrapy
from scrapy_playwright.page import PageMethod


yield scrapy.Request(
    "https://example.com/products",
    callback=self.parse_products,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod(
                "wait_for_selector",
                "[data-products-loaded='true']",
            ),
        ],
    },
)

That opt-in is an important scaling property. Listing pages that already contain the required HTML can remain cheap HTTP requests, while the few pages that need execution consume browser capacity. Prefer bounded PageMethod actions that return a normal Scrapy response. If you request the Playwright Page object directly with playwright_include_page, your callback and errback must close it. Leaked pages eventually exhaust the per-context page limit and stall the crawl.

Browser work also changes what your metrics mean. Browser subresources are not individual Scrapy requests, and Chromium runs in another process. scrapy-playwright provides a memory-usage extension that can account for the browser process where supported. These details are exactly why the browser belongs behind a defined acquisition boundary rather than inside an arbitrary callback.

If the browser integration itself becomes the bulk of the crawler, tools such as Camoufox and Patchright may become relevant for particular sources. They are browser runtimes rather than replacements for Scrapy's scheduling and data pipeline, so the same architectural question remains: can the specialized browser sit behind a narrow boundary, or has browser state become the application?

Write a wreq download handler when the network stack is the variable

TLS and HTTP/2 fingerprints are properties of the client implementation, not just a collection of headers. If a source expects a network profile that resembles a current browser, changing User-Agent in Scrapy's default client does not change its TLS ClientHello, cipher ordering, extension ordering, or HTTP/2 behaviour.

The Python bindings for wreq provide an async client backed by a browser-emulating network stack. A custom download handler lets that client fetch bytes while Scrapy continues to own the crawl.

The following handler targets the current async download-handler API in Scrapy and the Python wreq package. It is a useful starting point, but the caveats after the code are part of the implementation, not optional footnotes.

# myproject/download_handlers.py
from __future__ import annotations

from datetime import timedelta

import wreq.exceptions
from scrapy import Request
from scrapy.core.downloader.handlers.base import BaseDownloadHandler
from scrapy.exceptions import DownloadFailedError, DownloadTimeoutError
from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from wreq import Client, Emulation, Method, Proxy, redirect
from wreq.header import HeaderMap


WREQ_NETWORK_ERRORS = (
    wreq.exceptions.BodyError,
    wreq.exceptions.BuilderError,
    wreq.exceptions.ConnectionError,
    wreq.exceptions.ConnectionResetError,
    wreq.exceptions.DecodingError,
    wreq.exceptions.ProxyConnectionError,
    wreq.exceptions.RequestError,
    wreq.exceptions.TlsError,
)


class WreqDownloadHandler(BaseDownloadHandler):
    def __init__(self, crawler):
        super().__init__(crawler)
        self.default_timeout = crawler.settings.getfloat(
            "DOWNLOAD_TIMEOUT"
        )
        profile_name = crawler.settings.get(
            "WREQ_EMULATION",
            "Chrome149",
        )

        self.client = Client(
            emulation=getattr(Emulation, profile_name),
            # RedirectMiddleware should receive 30x responses.
            redirect=redirect.Policy.none(),
            # CookiesMiddleware remains the cookie owner.
            cookie_store=False,
        )

    async def download_request(self, request: Request) -> Response:
        method = getattr(Method, request.method, None)
        if method is None:
            raise DownloadFailedError(
                f"wreq does not support {request.method!r}"
            )

        headers = HeaderMap()
        for name, values in request.headers.items():
            header_name = name.decode("latin-1")
            for value in values:
                headers.append(
                    header_name,
                    value.decode("latin-1"),
                )

        timeout = request.meta.get(
            "download_timeout",
            self.default_timeout,
        )
        kwargs = {
            "headers": headers,
            "body": request.body,
            "timeout": timedelta(seconds=timeout),
        }

        if proxy_url := request.meta.get("proxy"):
            kwargs["proxy"] = Proxy.all(proxy_url)

        try:
            async with await self.client.request(
                method,
                request.url,
                **kwargs,
            ) as response:
                body = await response.bytes()
                response_headers = Headers(list(response.headers))
                response_url = str(response.url)
                response_status = response.status.as_int()
                response_protocol = str(response.version)
        except wreq.exceptions.TimeoutError as exc:
            raise DownloadTimeoutError(str(exc)) from exc
        except WREQ_NETWORK_ERRORS as exc:
            raise DownloadFailedError(str(exc)) from exc

        response_cls = responsetypes.from_args(
            headers=response_headers,
            url=response_url,
            body=body,
        )

        return response_cls(
            url=response_url,
            status=response_status,
            headers=response_headers,
            body=body,
            request=request,
            protocol=response_protocol,
        )

    async def close(self) -> None:
        self.client.close()

Register it for both HTTP schemes and choose a profile available in the pinned wreq version:

DOWNLOAD_HANDLERS = {
    "http": "myproject.download_handlers.WreqDownloadHandler",
    "https": "myproject.download_handlers.WreqDownloadHandler",
}

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

WREQ_EMULATION = "Chrome149"

# Do not pair a browser transport profile with Scrapy's UA.
USER_AGENT = None

There are several deliberate ownership decisions here. wreq does not follow redirects, because Scrapy's RedirectMiddleware should see the 302, enforce its limits, adjust priority, and record redirect statistics. wreq's cookie store is disabled, because Scrapy's CookiesMiddleware should remain the single source of cookie state. Transport errors become Scrapy download exceptions, so the existing retry middleware can handle them.

Headers need the same care. A Chrome TLS and HTTP/2 profile combined with contradictory User-Agent, Accept, Accept-Language, or client-hint headers is not a coherent Chrome request. In a real implementation, decide which browser-shaped headers wreq owns, then filter those headers out of the Scrapy request before building HeaderMap. Keep source-specific headers, authentication, and cookies. Test the actual wire output rather than assuming the two layers compose cleanly.

The example buffers the entire response. It therefore does not yet implement DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE, Scrapy's headers_received and bytes_received signals, or StopDownload. A production handler should stream wreq chunks, apply size checks before and during the body, and propagate cancellation. It should also map DNS, connection refusal, data loss, and unsupported methods to the most specific Scrapy exceptions available.

Compression deserves an explicit test too. Both the HTTP client and Scrapy middleware may believe they own gzip, Brotli, deflate, or zstd decoding. Choose one owner, then verify that the body is decoded once and Content-Encoding still describes the bytes Scrapy receives.

Before using the handler in a real crawl, run it against a local HTTP fixture that covers at least:

1. HTML response-class selection, duplicate response headers, and request attachment.
2. POST bodies and repeated request headers.
3. A redirect handled by Scrapy rather than wreq.
4. A retryable 500 and a transport timeout.
5. Per-request timeout and proxy metadata.
6. gzip and Brotli responses decoded exactly once.
7. A large streamed body stopped at the configured size.
8. Handler close and in-flight cancellation.

This list is longer than await client.get(url) because a download handler is a transport contract. That is also the point of the exercise. Scrapy lets you replace this one contract without throwing away the rest of the crawling runtime.

Scrapy is a runtime, not the whole platform

Scrapy does not make every architecture decision disappear. Its default scheduler is not a distributed queue shared by a fleet of workers. JOBDIR is resumability for one crawl, not global orchestration. A database pipeline still needs a data model, idempotent writes, and migration policy. Multiple processes still need shared traffic budgets if they target the same domain. Browser capacity, proxy health, deployment, secrets, and backfills still need an operating model.

Source-specific extraction remains real engineering too. scrapy-poet gives that work a clean home, but it cannot decide what a price means, whether two variants are the same product, or how to detect a plausible-looking error page.

The narrower claim is more useful: Scrapy provides a mature runtime and a set of extensible boundaries around those decisions. Build the parts that distinguish your data product. Extend a known scheduler, downloader, pipeline, and lifecycle before writing new versions of all four.

A practical migration path

Do not begin by rewriting a working scraper into a perfectly abstract project. Pick one scheduled script that has already accumulated reliability code and move it in layers:

1. Put discovery and parsing into a spider, leaving existing Parsel selectors intact.
2. Let the scheduler and duplicate filter own request flow, then enable JOBDIR for resumability.
3. Remove the local retry loop and use downloader middleware plus Scrapy download exceptions.
4. Move validation and persistence into ordered pipelines.
5. Record source-specific facts in stats and assert the run contract with Spidermon.
6. Move reusable extraction into scrapy-poet Page Objects.
7. Add Playwright or a wreq handler only for requests that prove they need a different acquisition path.

At each step, delete a piece of home-grown runtime. If the migration only adds Scrapy beside the old queue, retry client, plugin system, and exporter, it has missed the opportunity.

Frequently asked questions

Is Scrapy too old for modern Python and async code?

No. Current Scrapy has coroutine support across major component APIs and an async download-handler interface. Its Twisted history is visible, but it does not prevent integration with asyncio-based clients and browser tooling. Version compatibility still matters, so pin Scrapy and ecosystem packages together and test reactor configuration.

Can one spider mix normal HTTP, Playwright, and wreq?

Yes, but handler selection is scheme-based. scrapy-playwright registers an HTTPS handler that delegates unmarked requests to Scrapy's normal implementation. A project that also wants wreq for selected HTTPS requests needs one dispatching handler or separate schemes/domains routed by a custom handler. Two different classes cannot both occupy the same https entry in DOWNLOAD_HANDLERS.

When is a plain script still the better choice?

Use a plain script when the input is small and known, the run is disposable, failure has little cost, and no one needs to resume or operate it. Scrapy earns its place when the crawler develops state, repeated execution, multiple request paths, operational expectations, or reusable components.

Does Scrapy solve scaling across machines?

Not by itself. Scrapy gives each crawler process strong internal boundaries and replaceable components. A distributed deployment still needs orchestration, shared state where appropriate, and a policy for global traffic and data consistency. The framework makes those integrations narrower; it does not pretend they are free.

Build your scraper, not another crawling framework

In the previous post, the small scraper grew into a system as its failure cost increased. This is the natural next decision: once you need that system, decide which parts are genuinely specific to your source and which are generic crawling infrastructure.

Scrapy will not write the difficult selectors or define the meaning of your data. It will give those decisions a scheduler, lifecycle, retry path, extension model, statistics layer, and several escape hatches. In many projects, that is the framework you were about to spend the next year discovering one production incident at a time.