DEV Community: Extract by Zyte

Your scraping framework already exists: it's Scrapy

John Rooney — Sun, 26 Jul 2026 06:35:40 +0000

The first scraper is usually honest about what it is. It fetches a page, runs a few selectors, and writes some records. Then it needs pagination, so you add a queue. A transient failure loses a page, so you add retries. Two URLs produce the same content, so you add duplicate filtering. The site slows down, so you add per-domain concurrency and delays. Soon the script has lifecycle hooks, plugins, metrics, browser workers, and a home-grown configuration system.

At that point, you are no longer writing a scraper. You are designing a crawling framework, and many of the decisions in front of you have already been made, tested, and exposed as replaceable components in Scrapy.

This does not mean every 20-line script should become a Scrapy project. It means that when a collection of scripts starts acquiring a runtime, you should ask whether that runtime is the part worth inventing. In most cases, the source-specific discovery and extraction logic is yours. The scheduler, retry machinery, throttling, lifecycle, statistics, exports, and component boundaries do not need to be.

The useful way to think about Scrapy is not as a library that gives you response.css(). Parsel already does that. Scrapy is an event-driven crawling runtime with a stable place to put each concern.

The framework appears one fix at a time

Imagine that a scheduled product scraper is growing beyond its original loop. It now needs to remember which URLs it has seen, pause without losing its queue, retry transport failures, apply different concurrency limits to different domains, normalize every product, split large exports into batches, report statistics, and fail the run when output quality drops.

Those requirements tend to produce classes with names such as QueueManager, RetryingClient, PluginRegistry, and RunMonitor. None of those names is inherently a mistake, but together they are a sign that the project is recreating a framework around its parsing code.

Scrapy already divides that system along practical boundaries:

What you are about to build	Scrapy component
URL queue and priority rules	Scheduler
Duplicate URL tracking	Duplicate filter
Retries, redirects, cookies, proxies	Downloader middleware
HTTP, browser, or custom network transport	Download handler
Site navigation and discovery	Spider
Response-to-object extraction	Spider callback or `scrapy-poet` Page Object
Validation, normalization, persistence	Item pipeline
Lifecycle hooks and operational behavior	Signals and extensions
Counters and run metadata	Stats collector
JSON Lines, CSV, object storage, batching	Feed exports

The boundaries matter more than the names. They let you replace the network transport without replacing the scheduler, or add data-quality checks without importing monitoring code into every callback.

flowchart LR
    Spider["Spider<br/>discovery"] --> Engine["Execution engine"]
    Engine --> Scheduler["Scheduler<br/>queue + dupefilter"]
    Scheduler --> Engine
    Engine --> Middleware["Downloader middleware<br/>retry + redirect + cookies"]
    Middleware --> Handler{"Download handler"}
    Handler --> HTTP["Scrapy HTTP"]
    Handler --> Wreq["wreq"]
    Handler --> Browser["Playwright"]
    HTTP --> Response["Response"]
    Wreq --> Response
    Browser --> Response
    Response --> Poet["Spider / scrapy-poet"]
    Poet --> Pipeline["Item pipelines"]
    Pipeline --> Export["Feed export / database"]
    Signals["Signals + stats"] -. observe .-> Engine
    Signals -. assert .-> Monitor["Core extensions + Spidermon"]

Here is the spider at the centre of that runtime. It contains the site-specific parts: where to begin, how to find product links, and what to yield from a product page.

import scrapy


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products",
            callback=self.parse_listing,
        )

    def parse_listing(self, response):
        yield from response.follow_all(
            response.css("[data-product-link]::attr(href)"),
            callback=self.parse_product,
        )

        next_page = response.css("[rel='next']::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse_listing)

    def parse_product(self, response):
        yield {
            "url": response.url,
            "name": response.css("[data-product-name]::text").get(),
            "price": response.css("[data-price]::attr(data-price)").get(),
        }

Scrapy's selectors use Parsel, so selectors from a smaller Parsel-based scraper move across with little ceremony. The larger change is that callbacks yield work to an engine instead of performing that work directly. The engine can now schedule, deduplicate, retry, throttle, record, and resume it.

The examples below target the current Scrapy async component APIs and Python 3.11 or later, which the Python wreq package requires. In a real project, pin a tested set of versions rather than allowing every integration to update independently.

python -m pip install \
    Scrapy scrapy-poet spidermon scrapy-playwright wreq
playwright install chromium

Start with the boring infrastructure

Before adding an ecosystem package, it is worth seeing how much of the runtime ships with Scrapy itself. A small set of settings gives the product spider adaptive throttling, bounded per-domain concurrency, resumable state, item processing, and compressed batch exports.

# myproject/settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

CONCURRENT_REQUESTS_PER_DOMAIN = 4
RETRY_TIMES = 3

ITEM_PIPELINES = {
    "myproject.pipelines.ProductValidationPipeline": 300,
    "myproject.pipelines.ProductUpsertPipeline": 800,
}

FEEDS = {
    "exports/products-%(batch_id)05d.jsonl.gz": {
        "format": "jsonlines",
        "encoding": "utf-8",
        "batch_item_count": 50_000,
        "postprocessing": [
            "scrapy.extensions.postprocessing.GzipPlugin",
        ],
    },
}

AutoThrottle adjusts delay from observed latency for each download slot. Its target concurrency is an average, while CONCURRENT_REQUESTS_PER_DOMAIN remains the ceiling. RetryMiddleware handles configured response codes and download exceptions, and lowers retried requests in the queue. The feed exporter handles serialization, file naming, compression, and batches without making the spider know where output goes.

Resumability is a command-line setting away:

scrapy crawl products -s JOBDIR=crawls/products-2026-07-26

JOBDIR persists the scheduler queue, duplicate-filter state, and spider state so a cleanly stopped crawl can resume. It is not a distributed queue, and a job directory must belong to one spider run. Requests in a disk queue also need serializable callbacks and metadata. These limits are much easier to reason about than an undocumented queue format that grew inside a script.

The item pipeline gives extraction and data handling separate failure modes. The spider describes what it found; ordered pipeline components validate, normalize, drop, or store it.

from decimal import Decimal, InvalidOperation

from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem


class ProductValidationPipeline:
    def process_item(self, item):
        product = ItemAdapter(item)

        if not product.get("name"):
            raise DropItem("missing product name")

        try:
            product["price"] = Decimal(product["price"])
        except (InvalidOperation, TypeError):
            raise DropItem("invalid product price")

        return item

The extension system sits alongside these request and item flows. An extension is normally created once for a crawl and connects to signals such as spider_opened, item_scraped, spider_error, and spider_closed. Scrapy's built-in extensions use the same surface for core stats, periodic log stats, memory limits, close conditions, throttling, feed exports, and other lifecycle behaviour.

That API is small enough that a project-specific extension does not need its own plugin framework:

from scrapy import signals


class ProductStatsExtension:
    def __init__(self, crawler):
        self.stats = crawler.stats

    @classmethod
    def from_crawler(cls, crawler):
        extension = cls(crawler)
        crawler.signals.connect(
            extension.item_scraped,
            signal=signals.item_scraped,
        )
        return extension

    def item_scraped(self, item, response, spider):
        if not item.get("price"):
            self.stats.inc_value("product/missing_price")

EXTENSIONS = {
    "myproject.extensions.ProductStatsExtension": 500,
}

The extension records a fact. It does not decide whether the run should fail. Keeping that distinction makes the counter useful to loggers, dashboards, and monitors without letting one callback quietly define the operational policy for the whole crawl.

scrapy-poet separates navigation from interpretation

As a project adds sources, callbacks often become a mixture of navigation, selectors, normalization, and object construction. scrapy-poet moves page interpretation into Page Objects and injects the right object into a typed callback.

Enable its add-on and tell it where Page Objects live:

ADDONS = {
    "scrapy_poet.Addon": 300,
}

SCRAPY_POET_DISCOVER = ["myproject.pages"]

The Page Object owns extraction from one kind of page:

# myproject/pages/product.py
from decimal import Decimal

from web_poet import WebPage, handle_urls


@handle_urls("example.com")
class ProductPage(WebPage):
    def to_item(self) -> dict:
        name = self.css("[data-product-name]::text").get()
        price = self.css("[data-price]::attr(data-price)").get()

        return {
            "url": self.url,
            "name": name.strip(),
            "price": Decimal(price),
        }

The spider keeps navigation and asks for the dependency it needs:

import scrapy

from myproject.pages.product import ProductPage


class ProductsSpider(scrapy.Spider):
    name = "products"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/products/1",
            callback=self.parse_product,
        )

    def parse_product(self, response, page: ProductPage):
        yield page.to_item()

The annotation is not decorative. scrapy-poet inspects it, selects the Page Object using its rules, builds the dependency, and supplies it. You can override rules for a source, inject additional dependencies through providers, and test the Page Object without running a crawl. The spider is left to answer one question: where should the crawler go next?

This separation pays off when several spiders encounter the same page type, or one crawl strategy spans several sites with different representations. It also prevents browser acquisition, API-backed inputs, or cached responses from leaking into extraction code, because those can be provided as dependencies.

One detail matters if you use JOBDIR: callbacks must be serializable. scrapy-poet offers callback_for() to remove boilerplate, but assign the generated callback as a spider method instead of constructing it inline if requests may be written to disk.

Spidermon defines what a successful crawl means

A process can exit with code zero after extracting no products. It can also produce the expected number of records while silently losing every price. Scrapy's stats tell you what happened; Spidermon turns those facts into executable assertions and actions.

Enable the extension and select a monitor suite for spider close:

SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    "myproject.monitors.ProductSpiderCloseSuite",
)

Then express the run contract in terms of the resulting dataset:

from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Product output")
class ProductOutputMonitor(Monitor):
    @monitors.name("Enough products were extracted")
    def test_item_count(self):
        count = getattr(self.data.stats, "item_scraped_count", 0)
        self.assertGreaterEqual(count, 1_000)

    @monitors.name("Missing prices stayed below one percent")
    def test_missing_price_rate(self):
        scraped = getattr(self.data.stats, "item_scraped_count", 0)
        missing = getattr(self.data.stats, "product/missing_price", 0)

        self.assertGreater(scraped, 0)
        self.assertLess(missing / scraped, 0.01)


class ProductSpiderCloseSuite(MonitorSuite):
    monitors = [ProductOutputMonitor]

Spidermon can run suites at spider open, spider close, engine stop, or periodically. Suites can trigger actions after checks pass or fail, and built-in monitors cover common conditions such as item counts, field coverage, finish reasons, retries, errors, and unwanted status codes.

This is a better layer for run-level policy than scattered if count == 0 checks. The spider collects data, components record facts, and monitors decide whether those facts describe useful output.

scrapy-playwright makes the browser an acquisition choice

Some pages do need JavaScript execution or browser state. That does not require the crawler itself to become a browser script. scrapy-playwright implements a Scrapy download handler, so selected requests can use Playwright while the surrounding scheduler, retries, callbacks, pipelines, and monitoring stay in place.

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

DOWNLOAD_HANDLERS = {
    "https": (
        "scrapy_playwright.handler."
        "ScrapyPlaywrightDownloadHandler"
    ),
}

# Let the browser provide a User-Agent that matches itself.
USER_AGENT = None

Requests opt in through metadata. Unmarked requests continue through Scrapy's regular HTTP implementation.

import scrapy
from scrapy_playwright.page import PageMethod


yield scrapy.Request(
    "https://example.com/products",
    callback=self.parse_products,
    meta={
        "playwright": True,
        "playwright_page_methods": [
            PageMethod(
                "wait_for_selector",
                "[data-products-loaded='true']",
            ),
        ],
    },
)

That opt-in is an important scaling property. Listing pages that already contain the required HTML can remain cheap HTTP requests, while the few pages that need execution consume browser capacity. Prefer bounded PageMethod actions that return a normal Scrapy response. If you request the Playwright Page object directly with playwright_include_page, your callback and errback must close it. Leaked pages eventually exhaust the per-context page limit and stall the crawl.

Browser work also changes what your metrics mean. Browser subresources are not individual Scrapy requests, and Chromium runs in another process. scrapy-playwright provides a memory-usage extension that can account for the browser process where supported. These details are exactly why the browser belongs behind a defined acquisition boundary rather than inside an arbitrary callback.

If the browser integration itself becomes the bulk of the crawler, tools such as Camoufox and Patchright may become relevant for particular sources. They are browser runtimes rather than replacements for Scrapy's scheduling and data pipeline, so the same architectural question remains: can the specialized browser sit behind a narrow boundary, or has browser state become the application?

Write a wreq download handler when the network stack is the variable

TLS and HTTP/2 fingerprints are properties of the client implementation, not just a collection of headers. If a source expects a network profile that resembles a current browser, changing User-Agent in Scrapy's default client does not change its TLS ClientHello, cipher ordering, extension ordering, or HTTP/2 behaviour.

The Python bindings for wreq provide an async client backed by a browser-emulating network stack. A custom download handler lets that client fetch bytes while Scrapy continues to own the crawl.

The following handler targets the current async download-handler API in Scrapy and the Python wreq package. It is a useful starting point, but the caveats after the code are part of the implementation, not optional footnotes.

# myproject/download_handlers.py
from __future__ import annotations

from datetime import timedelta

import wreq.exceptions
from scrapy import Request
from scrapy.core.downloader.handlers.base import BaseDownloadHandler
from scrapy.exceptions import DownloadFailedError, DownloadTimeoutError
from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
from wreq import Client, Emulation, Method, Proxy, redirect
from wreq.header import HeaderMap


WREQ_NETWORK_ERRORS = (
    wreq.exceptions.BodyError,
    wreq.exceptions.BuilderError,
    wreq.exceptions.ConnectionError,
    wreq.exceptions.ConnectionResetError,
    wreq.exceptions.DecodingError,
    wreq.exceptions.ProxyConnectionError,
    wreq.exceptions.RequestError,
    wreq.exceptions.TlsError,
)


class WreqDownloadHandler(BaseDownloadHandler):
    def __init__(self, crawler):
        super().__init__(crawler)
        self.default_timeout = crawler.settings.getfloat(
            "DOWNLOAD_TIMEOUT"
        )
        profile_name = crawler.settings.get(
            "WREQ_EMULATION",
            "Chrome149",
        )

        self.client = Client(
            emulation=getattr(Emulation, profile_name),
            # RedirectMiddleware should receive 30x responses.
            redirect=redirect.Policy.none(),
            # CookiesMiddleware remains the cookie owner.
            cookie_store=False,
        )

    async def download_request(self, request: Request) -> Response:
        method = getattr(Method, request.method, None)
        if method is None:
            raise DownloadFailedError(
                f"wreq does not support {request.method!r}"
            )

        headers = HeaderMap()
        for name, values in request.headers.items():
            header_name = name.decode("latin-1")
            for value in values:
                headers.append(
                    header_name,
                    value.decode("latin-1"),
                )

        timeout = request.meta.get(
            "download_timeout",
            self.default_timeout,
        )
        kwargs = {
            "headers": headers,
            "body": request.body,
            "timeout": timedelta(seconds=timeout),
        }

        if proxy_url := request.meta.get("proxy"):
            kwargs["proxy"] = Proxy.all(proxy_url)

        try:
            async with await self.client.request(
                method,
                request.url,
                **kwargs,
            ) as response:
                body = await response.bytes()
                response_headers = Headers(list(response.headers))
                response_url = str(response.url)
                response_status = response.status.as_int()
                response_protocol = str(response.version)
        except wreq.exceptions.TimeoutError as exc:
            raise DownloadTimeoutError(str(exc)) from exc
        except WREQ_NETWORK_ERRORS as exc:
            raise DownloadFailedError(str(exc)) from exc

        response_cls = responsetypes.from_args(
            headers=response_headers,
            url=response_url,
            body=body,
        )

        return response_cls(
            url=response_url,
            status=response_status,
            headers=response_headers,
            body=body,
            request=request,
            protocol=response_protocol,
        )

    async def close(self) -> None:
        self.client.close()

DOWNLOAD_HANDLERS = {
    "http": "myproject.download_handlers.WreqDownloadHandler",
    "https": "myproject.download_handlers.WreqDownloadHandler",
}

TWISTED_REACTOR = (
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
)

WREQ_EMULATION = "Chrome149"

# Do not pair a browser transport profile with Scrapy's UA.
USER_AGENT = None

There are several deliberate ownership decisions here. wreq does not follow redirects, because Scrapy's RedirectMiddleware should see the 302, enforce its limits, adjust priority, and record redirect statistics. wreq's cookie store is disabled, because Scrapy's CookiesMiddleware should remain the single source of cookie state. Transport errors become Scrapy download exceptions, so the existing retry middleware can handle them.

Headers need the same care. A Chrome TLS and HTTP/2 profile combined with contradictory User-Agent, Accept, Accept-Language, or client-hint headers is not a coherent Chrome request. In a real implementation, decide which browser-shaped headers wreq owns, then filter those headers out of the Scrapy request before building HeaderMap. Keep source-specific headers, authentication, and cookies. Test the actual wire output rather than assuming the two layers compose cleanly.

The example buffers the entire response. It therefore does not yet implement DOWNLOAD_MAXSIZE, DOWNLOAD_WARNSIZE, Scrapy's headers_received and bytes_received signals, or StopDownload. A production handler should stream wreq chunks, apply size checks before and during the body, and propagate cancellation. It should also map DNS, connection refusal, data loss, and unsupported methods to the most specific Scrapy exceptions available.

Compression deserves an explicit test too. Both the HTTP client and Scrapy middleware may believe they own gzip, Brotli, deflate, or zstd decoding. Choose one owner, then verify that the body is decoded once and Content-Encoding still describes the bytes Scrapy receives.

Before using the handler in a real crawl, run it against a local HTTP fixture that covers at least:

HTML response-class selection, duplicate response headers, and request attachment.
POST bodies and repeated request headers.
A redirect handled by Scrapy rather than wreq.
A retryable 500 and a transport timeout.
Per-request timeout and proxy metadata.
gzip and Brotli responses decoded exactly once.
A large streamed body stopped at the configured size.
Handler close and in-flight cancellation.

This list is longer than await client.get(url) because a download handler is a transport contract. That is also the point of the exercise. Scrapy lets you replace this one contract without throwing away the rest of the crawling runtime.

Scrapy is a runtime, not the whole platform

Scrapy does not make every architecture decision disappear. Its default scheduler is not a distributed queue shared by a fleet of workers. JOBDIR is resumability for one crawl, not global orchestration. A database pipeline still needs a data model, idempotent writes, and migration policy. Multiple processes still need shared traffic budgets if they target the same domain. Browser capacity, proxy health, deployment, secrets, and backfills still need an operating model.

Source-specific extraction remains real engineering too. scrapy-poet gives that work a clean home, but it cannot decide what a price means, whether two variants are the same product, or how to detect a plausible-looking error page.

The narrower claim is more useful: Scrapy provides a mature runtime and a set of extensible boundaries around those decisions. Build the parts that distinguish your data product. Extend a known scheduler, downloader, pipeline, and lifecycle before writing new versions of all four.

A practical migration path

Do not begin by rewriting a working scraper into a perfectly abstract project. Pick one scheduled script that has already accumulated reliability code and move it in layers:

Put discovery and parsing into a spider, leaving existing Parsel selectors intact.
Let the scheduler and duplicate filter own request flow, then enable JOBDIR for resumability.
Remove the local retry loop and use downloader middleware plus Scrapy download exceptions.
Move validation and persistence into ordered pipelines.
Record source-specific facts in stats and assert the run contract with Spidermon.
Move reusable extraction into scrapy-poet Page Objects.
Add Playwright or a wreq handler only for requests that prove they need a different acquisition path.

At each step, delete a piece of home-grown runtime. If the migration only adds Scrapy beside the old queue, retry client, plugin system, and exporter, it has missed the opportunity.

Frequently asked questions

Is Scrapy too old for modern Python and async code?

No. Current Scrapy has coroutine support across major component APIs and an async download-handler interface. Its Twisted history is visible, but it does not prevent integration with asyncio-based clients and browser tooling. Version compatibility still matters, so pin Scrapy and ecosystem packages together and test reactor configuration.

Can one spider mix normal HTTP, Playwright, and wreq?

Yes, but handler selection is scheme-based. scrapy-playwright registers an HTTPS handler that delegates unmarked requests to Scrapy's normal implementation. A project that also wants wreq for selected HTTPS requests needs one dispatching handler or separate schemes/domains routed by a custom handler. Two different classes cannot both occupy the same https entry in DOWNLOAD_HANDLERS.

When is a plain script still the better choice?

Use a plain script when the input is small and known, the run is disposable, failure has little cost, and no one needs to resume or operate it. Scrapy earns its place when the crawler develops state, repeated execution, multiple request paths, operational expectations, or reusable components.

Does Scrapy solve scaling across machines?

Not by itself. Scrapy gives each crawler process strong internal boundaries and replaceable components. A distributed deployment still needs orchestration, shared state where appropriate, and a policy for global traffic and data consistency. The framework makes those integrations narrower; it does not pretend they are free.

Build your scraper, not another crawling framework

In the previous post, the small scraper grew into a system as its failure cost increased. This is the natural next decision: once you need that system, decide which parts are genuinely specific to your source and which are generic crawling infrastructure.

Scrapy will not write the difficult selectors or define the meaning of your data. It will give those decisions a scheduler, lifecycle, retry path, extension model, statistics layer, and several escape hatches. In many projects, that is the framework you were about to spend the next year discovering one production incident at a time.

So you want to get data from the web

John Rooney — Sun, 26 Jul 2026 06:14:44 +0000

Getting data from one web page can take 20 lines of Python. Keeping a trustworthy dataset alive while the source changes, rate limits you, returns partial responses, and occasionally serves a consent page with a 200 OK is a different engineering problem, because the selector is rarely as difficult as knowing when the result is wrong.

What web scraping becomes once someone depends on it

Web scraping is automated retrieval and extraction of data from web resources. At the small end, that means an HTTP request followed by an HTML parser. At the other end, the same basic operation sits inside a distributed acquisition pipeline with scheduling, traffic controls, schema validation, raw snapshots, observability, and a recovery path. Both are web scraping, but they should not have the same architecture.

The easiest way to choose that architecture is to classify the job by its failure cost:

Workload	A reasonable starting point	The first problems you will meet
A few known pages, needed once	One process and local output	Missing fields, pagination, duplicate URLs, odd encodings
A scheduled crawl of one site	A repeatable job with persistent state	Timeouts, rate limits, reruns, template changes
A JavaScript application	HTTP plus targeted browser automation where required	Background requests, timing, cookies, session expiry
Many sources feeding another product	A queue-based data pipeline	Source drift, partial runs, backfills, data contracts, cost control

This is the useful complexity ladder. A one-off question does not need a queue, six workers, and an observability stack, while a revenue-critical feed should probably graduate from a laptop before print() becomes its monitoring strategy.

A small Python scraper should leave evidence behind

The first version should already separate fetching from parsing because that boundary lets you test selectors without hitting the site again and distinguish a network failure from an extraction failure. It also means that when the markup changes, you can fix the parser against a saved response instead of repeatedly fetching the same page while you debug it.

Here is a complete example for a static product page. It uses Parsel, expects semantic attributes such as data-product-name and data-price, saves the raw response to a path chosen by the caller, and returns a typed record. Parsel is also the selector layer underneath Scrapy, so the CSS and XPath expressions carry across when the crawler grows into a Scrapy project.

Install the two dependencies first:

python -m pip install requests parsel

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path
from urllib.parse import urldefrag

import requests
from parsel import Selector


@dataclass(frozen=True)
class Product:
    source_url: str
    name: str
    price: Decimal
    fetched_at: str


def parse_product(
    html: str,
    *,
    source_url: str,
    fetched_at: str,
) -> Product:
    selector = Selector(text=html)
    root = selector.css("[data-product]")

    if not root:
        raise ValueError("missing product container")

    name = root.css("[data-product-name]::text").get()
    raw_price = root.css("[data-price]::attr(data-price)").get()

    if name is None or raw_price is None:
        raise ValueError("missing required product field")

    name = name.strip()

    try:
        price = Decimal(raw_price)
    except InvalidOperation as exc:
        raise ValueError("invalid product price") from exc

    if not name or not price.is_finite() or price < 0:
        raise ValueError("product failed validation")

    return Product(
        source_url=source_url,
        name=name,
        price=price,
        fetched_at=fetched_at,
    )


def fetch_product(url: str, snapshot_path: Path) -> Product:
    response = requests.get(
        url,
        headers={"User-Agent": "ExampleDataCollector/1.0 contact@example.com"},
        timeout=(5, 20),
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")
    if "text/html" not in content_type:
        raise ValueError(f"expected HTML, received {content_type!r}")

    fetched_at = datetime.now(timezone.utc).isoformat()
    source_url, _fragment = urldefrag(response.url)

    snapshot_path.parent.mkdir(parents=True, exist_ok=True)
    snapshot_path.write_bytes(response.content)

    return parse_product(
        response.text,
        source_url=source_url,
        fetched_at=fetched_at,
    )

The timeout has separate connection and read limits, raise_for_status() prevents a 404 page from flowing into the parser, and the content-type check catches JSON, images, and some challenge pages before Parsel sees them. The raw body is stored before parsing, so an extraction error does not destroy the evidence needed to reproduce it.

There is no content hash here because this example does not need one. The caller can name snapshots by run ID, URL key, or whatever fits the job. Hashes become useful later, when the system needs to detect unchanged responses, deduplicate stored bodies, or connect a parsed record to the exact bytes that produced it. Introducing that machinery here would explain the production architecture before we have earned it.

The parser rejects invalid records instead of returning a half-populated dictionary. Returning None for a missing price feels convenient until it reaches an aggregate and quietly changes the answer.

The test does not need the network:

from decimal import Decimal


def test_parse_product() -> None:
    html = """
    <article data-product>
      <h1 data-product-name>Mechanical Keyboard</h1>
      <span data-price="89.95">£89.95</span>
    </article>
    """

    product = parse_product(
        html,
        source_url="https://example.com/products/keyboard",
        fetched_at="2026-07-26T10:00:00+00:00",
    )

    assert product.name == "Mechanical Keyboard"
    assert product.price == Decimal("89.95")

This is the first architectural boundary worth keeping: fetchers produce source material and metadata, parsers turn that material into validated records, and storage comes after validation.

JavaScript rendering should be an explicit escalation

Some pages return the data in the initial HTML, while others return an application shell and fetch the useful payload after JavaScript runs. Check the response body and browser network panel before reaching for a headless browser because structured data embedded in the document or a background JSON response is usually easier to operate than a full browser session.

Playwright is excellent when you genuinely need rendering or interaction. It also turns deterministic request code into a small distributed-systems problem living inside Chromium, complete with page lifecycle events, async UI updates, browser crashes, context cleanup, and much higher CPU and memory use.

When a browser is justified, wait for a condition that means the data is ready. networkidle is often a weak signal because analytics, polling, and streaming requests can keep the network busy indefinitely.

Install Playwright and its Chromium build before running the example:

python -m pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright


def fetch_rendered_html(url: str) -> str:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        page = browser.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            page.locator("[data-product-loaded='true']").wait_for(
                state="attached",
                timeout=10_000,
            )
            return page.content()
        finally:
            browser.close()

The selector is intentionally site-specific. Replace it with a marker that appears only when the content you need is present, or wait for the relevant response. Playwright's auto-waiting documentation explains what its actions wait for automatically, while its evaluation guide covers the boundary between Python and code running in the page.

Standard Playwright also has a recognisable automation surface. If a site reacts differently to an automated browser than to the corresponding normal browser, stealth-oriented projects are another rung on the complexity ladder. Camoufox is a Firefox fork with Playwright compatibility and browser-level fingerprint changes, while Patchright patches the Chromium side of Playwright and presents itself as a drop-in replacement. Both are moving projects with their own compatibility gaps, release cadence, and operational cost, so treat them as specialised browser runtimes rather than magic flags. We will leave the comparison there because each deserves a separate article.

Retry transient failures without turning a scrape into a traffic spike

Once a job runs on a schedule, failures become normal input: DNS lookups fail, connections reset, upstream services return 503, and rate limits change during the day. Retries help with those transient failures, but unbounded or immediate retries make incidents worse.

requests can use urllib3's retry policy through an adapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist={429, 500, 502, 503, 504},
        allowed_methods={"GET", "HEAD"},
        respect_retry_after_header=True,
        raise_on_status=False,
    )

    adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

With a backoff factor of 0.5, urllib3 applies increasing delays between eligible retries. The final response still needs raise_for_status(). A 403 or 404 is not in the retry list because repeating a permanent failure four times adds load without changing the answer.

HTTP 429 Too Many Requests may include a Retry-After header. RFC 6585 leaves the server free to count requests by credential, cookie, client, resource, or another policy, which means changing one network attribute does not necessarily reset the limit.

At more than one worker, enforce concurrency and delay per host across the whole worker pool. A local sleep(1) inside each worker gives you ten requests per second when you scale to ten workers. Put the host budget in shared state or route each host through its own queue.

TLS fingerprinting starts before the server sees your headers

Changing the User-Agent header does not make an HTTP client behave like a browser. Before the server receives the HTTP request, the client has already sent a TLS ClientHello containing an ordered set of cipher suites, extensions, supported groups, signature algorithms, and ALPN values. Those choices, along with their ordering, can be summarised as JA3 or JA4 fingerprints. HTTP/2 adds another layer through SETTINGS values, connection behaviour, and pseudo-header ordering.

This is why a request can claim to be Firefox in its headers while looking nothing like Firefox on the wire. A conventional Python client is designed to make correct HTTP requests, not to reproduce a browser's network stack. For many sites that distinction does not matter. On sites that classify traffic using several signals, it can explain why copying browser headers changes nothing.

wreq exposes Python bindings over the Rust client, so there is no reason to leave the Python stack just to control the network fingerprint. It uses BoringSSL and supports TLS, JA3/JA4, HTTP/2, and header-order profiles. The current package ships more than 100 browser emulation profiles and offers both async and blocking clients.

Install wreq alongside Parsel:

python -m pip install wreq parsel

import asyncio

from parsel import Selector
from wreq import Client, Emulation


async def fetch_title(url: str) -> str:
    client = Client(emulation=Emulation.Chrome149)
    response = await client.get(url)

    if response.status >= 400:
        raise RuntimeError(f"request failed with status {response.status}")

    selector = Selector(text=await response.text())
    title = selector.css("title::text").get()

    if title is None:
        raise ValueError("response did not contain a title")

    return title.strip()


if __name__ == "__main__":
    print(asyncio.run(fetch_title("https://example.com")))

The profile needs to be internally consistent. If the TLS handshake resembles one browser version while the headers, operating system hints, cookies, or JavaScript-visible properties describe another, the inconsistency becomes a signal of its own. Keep a client alive across requests so connection pooling and cookies behave consistently, pin the package and profile versions you have tested, and expect fingerprints to age as browsers and detection systems change.

wreq solves the network-stack part of the problem rather than JavaScript rendering. If the page needs a browser, use one; if the HTML is already in the response and the mismatch is at TLS or HTTP/2, launching Chromium adds a large amount of machinery to solve a lower-level issue.

A production scraper is a replayable data pipeline

Once the output feeds search, analytics, pricing, or a customer-facing feature, the system needs more than fetch workers. A practical design looks like this:

flowchart LR
    S["Scheduler: cron or Airflow"] --> Q["Host-aware URL queues: SQS or Redis"]
    Q --> F["HTTP fetchers: requests or wreq"]
    Q --> B["Browser fetchers: Playwright, Camoufox, or Patchright"]
    F --> RAW["Raw snapshots and metadata: S3"]
    B --> RAW
    F --> RETRY["Retry and review queue"]
    B --> RETRY
    RAW --> P["Versioned parser workers"]
    P --> V["Schema and run-quality gates"]
    V --> DB["Idempotent records: PostgreSQL"]
    DB --> C["Search, analytics, or product consumers"]
    F -. metrics .-> O["Metrics, logs, traces, and alerts"]
    B -. metrics .-> O
    P -. metrics .-> O
    V -. metrics .-> O

The names are examples, not requirements. A single VM, SQLite, and a filesystem can implement the same boundaries for a smaller workload. The boundaries matter because they let you retry and replace one stage without repeating every other stage.

Raw snapshots make parser changes cheap

Store response bytes with the source URL, final URL after redirects, fetch timestamp, status, content type, relevant response headers, and a content hash. Add the fetcher and parser version to the eventual record. At this point the hash has a clear job: it gives the raw body a stable identity, makes unchanged responses cheap to detect, and links each parsed record back to the exact bytes that produced it.

When a selector breaks, rerun the new parser against stored responses. This is faster, cheaper, and kinder to the source than fetching every URL again. It also lets you compare old and new parser output before a backfill reaches production.

Retention is a policy decision, especially if pages contain personal or sensitive data. An immutable archive is useful engineering infrastructure. It is not a reason to keep everything forever.

Idempotent writes make retries safe

A worker can fail after writing a record but before acknowledging its queue message. The message will be delivered again. If the second write creates another row, your retry policy has become a duplication policy.

Use a stable key, such as (source, external_id), and upsert. This PostgreSQL statement uses SQLAlchemy-style named parameters:

INSERT INTO products (
    source,
    external_id,
    name,
    price,
    fetched_at,
    content_hash,
    parser_version
)
VALUES (
    :source,
    :external_id,
    :name,
    :price,
    :fetched_at,
    :content_hash,
    :parser_version
)
ON CONFLICT (source, external_id) DO UPDATE SET
    name = EXCLUDED.name,
    price = EXCLUDED.price,
    fetched_at = EXCLUDED.fetched_at,
    content_hash = EXCLUDED.content_hash,
    parser_version = EXCLUDED.parser_version;

This gives you the latest record. If you also need history, write changes to an append-only version table or event log. Do not overload one table to be both current state and an audit trail.

Quality gates should fail the run, not merely decorate a dashboard

Transport metrics tell you whether requests completed, while data metrics tell you whether the output is believable.

Track at least:

attempted, succeeded, retried, and finally failed URLs by host
status-code distribution and request latency
records produced, required-field completeness, and uniqueness
new, changed, and missing records relative to the previous successful run
parser errors grouped by source and parser version

Then define stop conditions. If yesterday's crawl produced 48,000 products and today's produces 730, publishing the result should require an explanation. The threshold depends on the source, but the decision must exist in code rather than in someone's memory.

Here is a small quality gate with deliberately configurable limits:

from dataclasses import dataclass


@dataclass(frozen=True)
class RunStats:
    attempted: int
    succeeded: int
    records: int
    records_with_required_fields: int


def validate_run(
    current: RunStats,
    *,
    previous_records: int,
    min_success_rate: float = 0.95,
    min_completeness: float = 0.99,
    min_volume_ratio: float = 0.70,
) -> None:
    if current.attempted <= 0 or current.records <= 0:
        raise ValueError("run produced no usable work")

    success_rate = current.succeeded / current.attempted
    completeness = current.records_with_required_fields / current.records
    volume_ratio = current.records / max(previous_records, 1)

    failures = []
    if success_rate < min_success_rate:
        failures.append(f"success rate {success_rate:.1%}")
    if completeness < min_completeness:
        failures.append(f"completeness {completeness:.1%}")
    if volume_ratio < min_volume_ratio:
        failures.append(f"volume ratio {volume_ratio:.1%}")

    if failures:
        raise ValueError("run failed quality gates: " + ", ".join(failures))

Those defaults are examples, not universal scraping constants. A news index and a product catalogue have different expected churn. Start by observing normal variation, then choose thresholds that catch abnormal runs without paging someone every morning.

The shortcuts that fail when the workload grows

Several designs work just long enough to become expensive:

Putting fetch and parse logic in one function. You cannot test a parser without network traffic, and every parser fix requires another crawl.

Using a browser for every URL. It simplifies the first implementation when some pages need JavaScript, then spends far more CPU and memory on pages that do not. Keep HTTP and browser fetchers behind the same response interface, and route only the pages that need rendering.

Retrying every error. A timeout may recover. A malformed selector, an authentication failure, or a deleted page will not. Classify failures and send permanent or suspicious ones to review.

Scaling with global concurrency. One large domain and one tiny domain do not have the same capacity or rules. Global concurrency lets the busiest host consume the whole budget and makes request pressure unpredictable.

Treating row count as the only quality metric. A parser can return the expected number of rows with the wrong field in every row. Validate types, ranges, uniqueness, and changes in field-level completeness.

The common thread is lost evidence. Once the fetch, parser, and output are entangled, you cannot tell which part failed without repeating the entire job.

How to choose the smallest architecture that will survive

Use one script when all of these are true: the page set is small, the source is stable, the work is easy to rerun, and a bad result has limited consequences. Even then, save the raw input and validate required fields.

Add persistent state and scheduling when runs repeat. Add browser automation only for pages that require it. Add queues when one process no longer meets the throughput or isolation you need. Add separate parser workers when you want to replay stored responses without fetching again.

The forcing function is not URL count by itself. Ten authenticated, stateful workflows can be harder to operate than a million static pages from a stable bulk source.

Before increasing concurrency, write down:

What is the per-host request budget?
Which failures are safe to retry?
Which quality signal stops publication?
What raw evidence and version metadata will you retain?
How will you replay or backfill without duplicating records?

If those answers are vague, more workers will mostly help you fail faster.

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of `requests`?

Use requests or httpx when the URL set is small and you can manage scheduling, retries, deduplication, and persistence directly. A framework such as Scrapy earns its place when you need concurrent crawling, downloader middleware, per-domain controls, request deduplication, item pipelines, and a consistent extension model. A framework removes plumbing, not the need to understand the source.

When should I use Playwright for web scraping?

Use Playwright when data only appears after browser-side execution or requires an interaction that a plain HTTP client cannot reproduce. Do not use it by default for static pages. Inspect the initial HTML and background requests first, then route only the pages that require rendering through a browser pool.

How do I know whether a scraper is returning bad data?

Validate each record against a schema, then compare run-level metrics with the previous successful run. Required-field completeness, type and range checks, uniqueness, record volume, and the number of changed or missing entities catch different classes of failure. Keep raw snapshots so you can reproduce the parser's decision.

How do you scale web scraping without overwhelming a site?

Set concurrency and request rates per host across all workers, respect 429 and Retry-After, cache or fetch incrementally where appropriate, and use bounded retries for transient failures. Scale the scheduler and parser independently from the fetch rate. More parsing capacity does not require more traffic to the source when raw responses are replayable.

Start with a replay test

Take one representative page, save the raw response, and make the parser pass against that fixture. Then change or remove one required field and confirm the test fails for the reason you expect.

That small exercise tells you whether you are building a script that happens to return data today, or a system you can still explain after the source changes tomorrow.

How to use Camoufox with Scrapy (New EASY Method)

John Rooney — Sat, 25 Jul 2026 11:20:24 +0000

A question comes up often enough in the web scraping community that it has a familiar shape to it: I like working in scrapy-playwright, but I want to run camoufox, or another Python browser library, instead of the browser it ships with, so how do I actually wire that in? For a long time the honest answer was some version of "you can, but it isn't pretty." People raised it in GitHub issues and talked around it on forums, and the workarounds that existed all asked you to figure out a way to run that browser separately and use CDP.

That has changed with a small addition to scrapy-playwright: PLAYWRIGHT_BROWSER_PROVIDER, that hands the browser lifecycle to whatever Playwright-compatible package you want to use. It’s a small Python class that integrates into Scrapy and scrapy-playwright. It is the kind of change that reads like a one-line entry in a changelog and quietly removes a frustration a lot of people had learned to live with.

Why the stock browsers get caught

To see why this matters, it helps to remember what people are reaching for a different browser to do. Playwright is excellent at driving a browser, and scrapy-playwright wires that capability into Scrapy at the download handler so a rendered page arrives through the normal request and response cycle, asynchronously and quickly. The catch is the binary. The Chromium and Firefox builds that Playwright ships are the standard, well-known ones, which means even fairly basic anti-bot systems recognize them almost immediately, and the sites you most want to reach are rarely running basic anti-bot systems. These browsers were designed for automated testing, and no scraping specifically.

Getting around that recognition used to be a matter of patching the browser at runtime, but that has stopped being enough. The more effective projects now change the browser at the source and compile their modifications into the binary itself, instead of patching a stock build once it is running. Runtime patches tend to leave small inconsistencies behind, and modern fingerprinting is built to notice exactly those, in the same way that the network fingerprint has become a central battleground between scrapers and site operators. Each mismatch lowers the trust a site assigns to your session and makes a challenge more likely, so the goal is to present a browser with nothing out of place. Building the changes at compile time is harder to detect, and because Chromium and Firefox expose different things, what needs changing is different in each. This is one important part of how modern stealth browsers work, not the whole story, but it is enough to explain why swapping the browser is worth the effort, especially when anti-bot vendors now ship changes at a pace that punishes any setup relying on manual tuning.

The middle ground that was missing

scrapy-playwright already had one good answer for using a different browser: connect to one that is running somewhere else. Through PLAYWRIGHT_CDP_URL or Playwright's own connect endpoint, you point the plugin at an already-running browser over the network, and that route is genuinely useful when the browser lives on a paid service or on infrastructure you run yourself. I have written before about standing up a self-hosted browser service and connecting Playwright to it over a WebSocket, and it works well, but it is worth being clear about what it asks of you. You have to run the browser in debug mode, keep that instance alive, watch it for memory leaks, and restart it when it dies, which is a reasonable amount of operational surface for what started as "I just want to use camoufox locally."

That was the gap. Between the stock browser that gets caught and a full separate browser service that you connect to remotely, there was a large group of people who simply wanted a stealthier browser running inline with their spider, on their own machine, without building a service around it. The workarounds they were left with were the ugly ones. One approach was to swap the binary by hand: download the original, remove it, and drop the replacement in where Playwright expected to find it, which is as fragile as it sounds and breaks the moment anything updates. Another leaned on the fact that some of these browsers shipped an experimental built-in server you could expose and connect to, which was fine until you needed one of the browsers that had no such server, at which point you were back to no clean route at all. That last point bites hardest for the Firefox-based options, because the connect-to-a-running-browser story was never as clean for them as it was for Chromium.

How browser providers actually work

The new setting works by naming a provider, which is just a class that owns starting and stopping the browser. The default provider wraps vanilla Playwright and gives you everything the README already documented, and if you want a different backend you point PLAYWRIGHT_BROWSER_PROVIDER at a class of your own. The provider implements a small asynchronous lifecycle, start, launch_browser, launch_persistent_context, and close, and because packages like camoufox, patchright, and invisible_playwright keep the standard Browser, BrowserContext, and Page objects, everything downstream of the launch, your pages, contexts, and routing code, keeps working unchanged. Only how the browser starts and stops is different.

Here is the camoufox provider from the documentation, which is the shape almost any provider will take:

from contextlib import AsyncExitStack

from scrapy_playwright.handler import Config, PERSISTENT_CONTEXT_PATH_KEY

class CamoufoxBrowserProvider:
    def __init__(self, config: Config) -> None:
        self.config = config
        self.stack = AsyncExitStack()

    async def start(self) -> None:
        pass

    async def launch_browser(self):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(**self.config.launch_options)
        )

    async def launch_persistent_context(self, context_kwargs: dict):
        from camoufox.async_api import AsyncCamoufox

        return await self.stack.enter_async_context(
            AsyncCamoufox(
                persistent_context=True,
                user_data_dir=context_kwargs[PERSISTENT_CONTEXT_PATH_KEY],
                **self.config.launch_options,
            )
        )

    async def close(self) -> None:
        await self.stack.aclose()

Point the settings at it, and because camoufox is Firefox-based, set the browser type to match:

# settings.py
PLAYWRIGHT_BROWSER_PROVIDER = "myproject.providers.CamoufoxBrowserProvider"
PLAYWRIGHT_BROWSER_TYPE = "firefox"

I ran this exact example from the documentation, and it works. That is roughly all there is to it: one class, two settings, and the stealth browser is running inline with your spider. Importing the third-party library lazily inside the methods that need it, as the example does, is a nice detail, because it means the setting can point at a provider whose backend is only installed in some of your environments without breaking the others.

Why this fits the way Scrapy is built

None of this is an accident of design. Scrapy is lean on purpose. It is a full framework, but the core deliberately avoids telling you which browser to use, which extraction approach to take, or how your infrastructure should look, and it pushes those decisions out to extensions instead, some official and some from the community. Browser providers extend that same principle right down to the choice of browser binary, which is a very Scrapy way to solve the problem: rather than blessing one stealth browser, the plugin adds a seam and lets you bring your own.

It is worth saying where scrapy-playwright sits in that picture, because it shapes why this addition feels significant rather than incidental. The plugin lives in the scrapy-plugins organization and has been driven largely by a colleague at Zyte, which puts it in an interesting spot, not a headline product with a marketing page, but not a random third-party bolt-on either. Seeing a genuinely useful capability land in a project like that, one that many people had quietly filed under "mature, does what it does," is a good reminder that the tools we lean on are still being improved by people who hit the same friction we do.

What it changes, and what it doesn't

I would put this on the "relatively big deal" side of the ledger rather than the nice-to-have side, precisely because of the group it serves. If you were already connecting to a remote browser service, your setup does not change, and if you only ever needed the stock browser, this will pass you by. But for the sizable middle, the people who wanted a stealthier browser running locally with scrapy-playwright and were stuck with fragile workarounds, this turns a hack into a configuration choice, and lowering the cost of that experiment is the real value here.

A couple of honest caveats belong alongside that. The provider examples in the documentation are offered as guidelines, not supported parts of scrapy-playwright, and the plugin is not affiliated with camoufox, patchright, or any of the other projects it references, so you should read each project's own documentation and evaluate any dependency yourself before you rely on it. Swapping in a stealth browser is also not a finish line. It removes one common way of being spotted, but it sits inside a larger system of proxies, behavior, and session management, and treating it as a single switch that solves blocking will lead you astray. When you would rather not run and maintain any of this yourself, Zyte API's headless browser handles the binary, the fingerprinting, and the proxy rotation behind a single request, which is the right answer for a different set of problems than the one browser providers solve. What providers give you is a clean way to make the choice yourself, and for a lot of people that is exactly what was missing.

GPT-5.6, Fable 5, and GLM-5.2 went for a crawl and got hit by The Rate Limit - My new fav foundational model

Ayan Pahwa — Fri, 10 Jul 2026 13:56:05 +0000

A while back I wrote about adding GLM-5.2 to my agentic arsenal, and the part that stuck was not the model, it was the method. I keep a small harness of real scraping tasks and an OpenRouter key so that when something new ships I can throw it at genuine work the same week. GLM-5.2 came out of that as a cheap open-weight workhorse I still reach for.
So when OpenAI shipped GPT-5.6 and Anthropic's Fable 5 was sitting at the top of the price list, my question was not "which one wins a leaderboard." It was the one I actually pay for: for the scraping I do, how much model do I need to buy? GPT-5.6 makes that a sharp question, because it ships as a price ladder, three tiers of the same generation. Add the cheap challenger below it and the frontier model above it and you get a clean spread on the cost that actually dominates a scraping bill, output tokens: from $1.76 per million on GLM-5.2 to $50 on Fable 5, nearly 30 times more. Five contenders walk into the same bar. Who is worth their tab?

The setup, kept light

The mechanics are deliberately simple. I send the identical prompt to each model so nothing is biased by wording, and everything runs through OpenRouter so I am comparing real charged dollars rather than guessing from a rate card. Where I can score a result objectively I do: does the generated code compile, does it run against a real page and return the right values, does the extracted JSON cover the schema. Then I read every output myself, because the automatic scores miss things. Five models, a handful of real scraping tasks, one afternoon. This is a first-hand read, not a benchmark, so take the small sample for what it is.
The five, cheapest to priciest:

GLM-5.2 (z-ai) at $0.54 / $1.76 per million in/out tokens, the cheap open-weight challenger.
GPT-5.6 Luna at $1 / $6, OpenAI's fast, cost-efficient tier.
GPT-5.6 Terra at $2.50 / $15, the balanced middle.
GPT-5.6 Sol at $5 / $30, the flagship.
Claude Fable 5 at $10 / $50, the most capable, most expensive model on the menu.

The one table that answers the question

Model	Price (in/out)	Suite cost	Tasks passed	Cost per success	Reasoning tokens	Tool loop	Suite speed	Landing page
GLM-5.2	$0.54 / $1.76	$0.045	3/4	$0.0099	4,585	7/8	176s	10.5 min
GPT-5.6 Luna	$1 / $6	$0.044	3/4	$0.0087	1,781	8/8	29s	18s
GPT-5.6 Terra	$2.50 / $15	$0.118	4/4	$0.020	1,692	8/8	51s	32s
GPT-5.6 Sol	$5 / $30	$0.206	4/4	$0.037	1,716	8/8	68s	36s
Claude Fable 5	$10 / $50	~$0.38	2/4	$0.119	109	8/8	57s	63s

Read down the "tasks passed" column and the story tells itself. Capability climbs as you pay more, right up to Terra in the middle. Then it stops climbing. Then, at the top, it falls. The curve is not a staircase where more money buys more model. It is a hump, and the peak is in the cheap seats.

What every dollar tier gets right: extraction

Start with the good news, which is that most scraping is boring, and boring is where everyone ties. Pulling a fixed schema of clean JSON out of a messy product page, and pulling every book off a listing page into an array, is bread-and-butter structured extraction. All five models did it perfectly. Same valid JSON, same full schema coverage, same 20 books off the listing, no drift.
That is the single most important line in the whole exercise, because extraction is the bulk of real scraping. On the work you do most, the cheapest model in the room is indistinguishable from the one that costs 30 times more. The only thing that separates them here is the bill, and the bill says use the cheap one: Luna and GLM cost under a cent per successful answer, and Sol costs four times that for output you could not pick out of a lineup.

What separates the cheap tier: currency

There is one task where the models split, and it is not the one I expected. I ask each model to wire a Scrapy project to run behind the Zyte API the way Zyte actually recommends today, which is a one-line addon. It is a quiet test of whether a model is working from current knowledge or a stale training snapshot.
The line fell inside a price band, not between vendors. Terra and Sol got it right, reaching straight for the modern one-line addon, no manual plumbing. GLM-5.2 and Luna both got it wrong, hand-wiring the deprecated download handlers and middleware, the textbook symptom of training-data lag. They are the two cheapest models, and they share the same blind spot.
So the freshness you are paying for is real, but it is not much of a moat. Honestly, you can pretty much solve it by giving the agent access to fresh docs through an MCP server like Context7, so it is not a biggie. Out of the box the gap closes at the Terra tier, and you never have to pay flagship prices to clear it.

What the flagship premium buys: nothing I could measure

Here is where the money stops working. Sol passed everything Terra passed, four for four, and matched it on every other axis: same clean addon, same extraction, same eight-for-eight in the agent loop, an indistinguishable landing page. It cost roughly double per successful task for zero additional capability I could detect. On this workload the flagship is a mid-tier model with a bigger price tag. Whatever Sol's extra headroom is for, everyday scraping is not it.

What the frontier model does: it refuses the job

And then the curve falls off a cliff. Fable 5 is a Mythos-class model, Anthropic's most capable tier, the kind of thing you bring in to orchestrate long-horizon, multi-step work and coordinate other models, not to write a parse function. Reaching for it on a scraping task is overkill before you even start. It has the raw capability, and it proved that in the agent loop with a clean sweep. But point it at actual scraping and its safety classifiers start declining the work.
It refused the anti-bot plan outright, returning nothing. Worse, it began writing a perfectly benign CSS-selector function, the kind of "extract the title and price from this product page" code the cheap models wrote without blinking, and then its classifier killed the response mid-function. Even the addon task, which it did complete, came back hedged: it gave the modern addon and then bolted the deprecated manual middleware on beside it, which is exactly the muddle you do not want a junior copying.
So the most expensive model in the field scored the lowest, two of four, and cost the most per successful answer, roughly 14 times Luna. Its reasoning-token count looks impressively low, but that is not efficiency, it is what refusing two tasks looks like. You are paying frontier prices for a model that treats your core workload as a threat. For scraping it is not overqualified. It is unavailable.

The one task I judged by eye: a landing-page build-off

Everything so far has a right answer a script can check. Design does not, so for this one I gave each model the same brief, build a single self-contained landing page for a fictional scraping API, everything inline so it renders offline, and then I put three of them side by side and ranked them blind before revealing which model built which.

Same brief, three models, judged blind. Left to right: GPT-5.6 Sol, Claude Fable 5, GLM-5.2.
I ranked them Sol first, Fable a close second, GLM last, and the honest headline is that they were all close. Two things stood out anyway. First, design quality is a wash you cannot buy your way out of: the GPT tiers and Fable all produced clean, shippable, modern pages I genuinely could not tell apart by maker, while GLM went its own way with a warm editorial "field manual" look that has real personality but reads as less of a conversion page. You are not buying a better landing page by spending more.
Second, the cost of getting there was wildly uneven. The two pages I ranked highest took 36 seconds (Sol) and 63 seconds (Fable). The one I ranked last took 10.5 minutes and 22,000 reasoning tokens (GLM). And Fable, which flatly refused to write a scraper a few tasks earlier, cheerfully built a polished page to market one, at roughly double Sol's cost for a page I ranked below it. The frontier model will sell the product it will not build.

The stuff that does not show up in a price table

A few dimensions matter as much as correctness and never appear on the menu.
Speed tracks thinking, and thinking tracks the bill. The GPT tiers ripped through the suite in 29 to 68 seconds; GLM took 176. On the landing page the gap turns absurd: GLM spent 10.5 minutes and 22,000 reasoning tokens building a page the GPT tiers produced in half a minute with almost no reasoning at all. Within the GPT family, cheaper is faster, because cheaper thinks less. One honest caveat on GLM's wall-clock: through OpenRouter, GLM gets routed to whichever third-party host is cheapest at the moment, Morph and StreamLake across my runs, each with its own tokens-per-second, while GPT-5.6 and Fable 5 were served by OpenAI and Anthropic directly, which tend to be faster. So some of GLM's slowness is a routing artifact, not the model itself. Either way, if your scraping is interactive or latency-sensitive, slow is slow at request time.

Self-healing is a family trait, not a premium. I gave every model a tool that always fails and watched whether it gave up gracefully or hammered it in a retry storm that burns tokens on every pointless turn. All of them behaved: one attempt, then a plain "this file does not exist," no storm. Good discipline is table stakes now, top to bottom.
Agentic discipline, same story. Every GPT tier, Luna included, went a perfect eight for eight on the tool-calling probe: correct tool selection, multi-hop chains in the right order, parallel fan-out of independent lookups in a single turn, and restraint on the trivial cases (answering "what is 2 plus 2" without reaching for the calculator). GLM was the only one that stumbled, an occasional reach for a tool it did not need. So even the cheapest OpenAI tier is a solid agent-loop citizen. One honest limit on that claim: this probes single-agent tool discipline, not multi-agent orchestration, the coordinating-a-fleet-of-subagents-across-a-long-job skill where a Mythos-class model like Fable is built to shine. That is a real axis, it is just not one a scraping run exercises.
Reporting and restraint differ by personality. GLM writes long and explains everything; the mid GPT tiers are clean and to the point; Fable narrates and hedges, wrapping answers in caveats. On the one judgment task, a build-versus-buy 403 plan, the cheap model gave the sharpest answer: GLM named a concrete threshold for when to stop hand-rolling proxies and reach for a managed API, named the tools, and threw in the trick I would give a junior, which is to check for a hidden JSON endpoint before scaling anything. Sol's plan was thorough but hedged; Fable refused to write one at all.

The five personas

GLM-5.2, the scrappy street-smart veteran. Knows the craft cold and gives the most useful advice in the room, but works from a slightly dated playbook (it missed the modern Zyte addon) and thinks the hardest and slowest of anyone. Cheapest on the menu, genuinely valuable on well-trodden ground, a little undisciplined in an agent loop and a little behind the times.
Luna, the fast, cheap, tidy intern. Quick, lean, does exactly what is asked, perfect in the agent loop, and astonishingly cheap. Its one blind spot is currency, the same stale-wiring gap as GLM. Ideal for high-volume, well-defined work; do not hand it the cutting-edge integration.
Terra, the current, competent mid-level engineer, and the sweet spot. Everything the flagship gets right, at half the price and with no drama: the modern addon, every extraction, eight-for-eight tools, the richest landing page of the bunch, fresh on best practices and frugal on reasoning. If you make one default choice for scraping, this is it.
Sol, the principal engineer you do not need for this work. The most cautious and, on paper, the most capable, with a reasoning-effort dial for genuinely hard problems. But on everyday scraping it is overqualified: the same passes as Terra for double the price, and cranking its effort dial higher on the addon task doubled the cost for zero gain in correctness.
Fable 5, the brilliant specialist who will not take your case. Untouchable on general reasoning and a clean sweep in the agent loop, but its safety classifiers refuse actual scraping: it declined the unblocking plan and killed a harmless selector function mid-sentence. Frontier prices for a model that will not do half the job.

A verdict you can act on

If your job is	Reach for	Why
High-volume structured extraction	Luna or GLM-5.2	Everyone ties on correctness; the cheapest wins on cost per answer. The flagship here is four times the price for identical JSON.
Everyday scraping code that must be current	Terra	The freshness line switches on at Terra. It writes today's Zyte addon, not last year's middleware, at half Sol's cost.
Agent-loop / tool-calling work	Any GPT-5.6 tier, Luna if cost matters	Discipline is a family trait; all three went eight-for-eight. Avoid GLM here.
Build-versus-buy and architecture calls	GLM-5.2	The cheap model gave the most directly useful plan. Sol over-hedged; Fable refused.
One-shot UI and landing pages	Luna	Family-level quality; two cents, not 16.
Genuinely frontier-hard, correctness-critical work	Sol with effort dialed up	The only place its ceiling might earn the premium.
Anything scraping-adjacent on a safety-tuned frontier model	not Fable 5	It refuses the work.

The menu lies, and here is how

The pricing page frames the flagship as a 17 times decision over GLM. In real charged dollars it is closer to four, and two things do the compressing.
First, token efficiency. A low per-token rate on a model that thinks twice as hard is not as cheap as it looks. GLM burned 4,585 reasoning tokens across the suite to the GPT tiers' roughly 1,700, and on the landing page it spent 22,000 reasoning tokens to their 50. You pay for tokens, not for the sticker.
Second, the menu is a list price, not your bill. GLM's real charged cost ran above its headline rate once you route it through a real provider, while the GPT tiers' frugality pulled their effective cost well under their sticker multiples.
Two framing points matter more than any per-token number. Measure cost per successful task, not per call, because a cheap wrong answer is not cheap; GLM and Luna "saved" money on the addon and produced code you would have to rewrite. And the effort dial is an invisible multiplier: Sol at high effort cost twice as much as low effort for the same correct answer, and the menu never warns you.
At scale it all gets concrete. A million product-page extractions runs about $4,700 on Luna, $6,000 on GLM, $11,000 on Terra, $20,000 on Sol, and roughly $56,000 on Fable, for JSON you cannot tell apart, and Fable would refuse a chunk of the work anyway.

Where this falls short

If I stopped at whichever model looked cheapest I would be selling you something, so here is the honest part. This is a small, first-hand sample, one run per task, not a benchmark; a model that wins a task once might not win it reliably, and Fable's refusals in particular could soften or harden with prompt wording. The cost gaps are real but provider routing and token counts move them around, so treat the ranking as directional and the method as the point. And I only tested the scraping work I actually do; your tasks may pull the curve into a different shape, which is exactly why you should run your own.

The real takeaway: know your models, then buy the least you need

The lesson is not "GPT-5.6 beats GLM" or "the frontier model is bad." It is that the price ladder is not a quality ladder. Capability rose to the middle, plateaued at the flagship, and inverted at the frontier, where safety tuning made the most expensive model the least useful for this domain. The smart move is not to buy the most model you can afford. It is to know which model is good for which task and buy the least that clears your bar, which for most scraping is Terra in the middle, and for pure extraction is whatever is cheapest that week.
The durable asset here is the harness, not the verdict. The leaderboard will reshuffle again before this post ages, and when it does the question is not which model I trust today, it is how fast I can prove which one fits which job. If your default model got repriced or deprecated tomorrow, could you answer that in an afternoon?

That harness is nothing fancy, a small custom rig I built for exactly this: it fires the identical prompt at every model through one API, compiles and runs the code they hand back against a real page, checks the JSON, and leaves the taste calls like the landing pages to me. Build your own version, keep it around, and the next launch is an afternoon's work instead of a leap of faith.

This post was originally published on the Zyte blog.

5 lessons from 5 interviews on AI agents and web scraping

Neha Setia — Tue, 23 Jun 2026 08:46:14 +0000

Over the last few months, I’ve been interviewing people at Zyte about AI, agents, web scraping, data quality, and what actually changes when LLMs enter developer workflows.

These were not abstract “AI will change everything” conversations. They were grounded in the daily work of people building and maintaining scraping systems, QA workflows, AI-assisted developer tools, and agentic infrastructure.

Five conversations stayed with me:

Adrian Chaves on Web Scraping Copilot and AI-generated scraping code
Julia Medina and Mihaela Popova on how LLMs changed their day-to-day development work
Javier on why AI feels like superpowers only when you already have judgment
Tomasz Lesiak on why AI cannot fix data quality until you define what “good” means
Konstantin Lopukhin on agent skills, scraping, and why the best skill often says the least

Here are the five lessons I’m taking from them.

1. Simpler code is usually better scraping code

In my interview with Adrian Chaves, we talked about what “correct” AI-generated scraping code actually means.

The obvious answer is: it extracts the right data.

But Adrian added a second standard that I keep coming back to: the code also has to be simple.

That matters because web scraping is not the kind of software where you write code once and forget about it. Websites change. Layouts move. Fields disappear. New variants appear. Pagination breaks. A selector that worked last month may quietly fail next month.

AI-generated code can easily become too defensive. It tries to handle every possible future case, and the result is often harder for a human to read, debug, and maintain.

The better standard is not “write code that imagines every future website change.”

It is:

Write the simplest code that gets the right value now, and surround it with tests and breakage detection.

That feels especially important in an AI-assisted workflow. If an agent generates a spider, the human still needs to understand what it produced. Maintainability is not a nice-to-have. It is part of correctness.

2. Developers are moving from coding-first to design-first

In my conversation with Julia Medina and Mihaela Popova, both described a similar shift in how they work with LLMs.

The work is less about typing every line yourself and more about deciding what should exist.

That sounds small. It is not.

When an AI assistant can generate code, the developer’s leverage moves upstream:

What are we trying to build?
What should the finished output look like?
What examples should the agent learn from?
What schema defines success?
What tests will tell us the output is wrong?
What should we not automate yet?

Julia described Web Scraping Copilot as a test-driven loop: give it a schema, create fixtures, generate expected output, write extraction code, run the tests, and iterate when the output is wrong.

Mihaela described LLMs inside production scraping workflows, where the question is not just “can the model extract this?” but where in the pipeline LLM judgment is actually useful.

That is the real shift I heard: developers are not disappearing. They are moving into design, evaluation, review, and system boundaries.

When agents can produce more code, the human job becomes more about defining what good looks like.

3. AI multiplies judgment. It does not replace it.

Javier described his experience with a line I loved:

“I have superpowers now.”

But the interesting part was what those superpowers were made of.

Not magic. Not blind delegation. Not “the AI knows everything now.”

His superpower was years of engineering judgment moving faster.

He uses AI to discuss approaches, document decisions, preserve context, and accelerate tedious parts of the work. But the quality still comes from knowing what is good, what is risky, what deserves skepticism, and when the assistant is simply being too agreeable.

That distinction matters.

A junior developer and a senior developer can use the same AI tool and get very different results. The difference is not only prompt quality. It is taste, context, pattern recognition, and the ability to evaluate the answer.

AI can compress the time between idea and artifact. But it does not remove the need to know whether the artifact is any good.

If anything, it makes that judgment more visible.

4. Data quality begins before AI enters the workflow

My conversation with Tomasz Lesiak was a useful antidote to AI abstraction.

Before asking whether AI can fix data quality, he brought the problem back to three questions:

Did we catch every item?
Did we fill every required field?
Is the information actually correct?

Every QA process, whether manual, automated, or AI-assisted, is trying to answer some version of those questions.

This is easy to forget because AI can make messy output look polished. It can summarize incomplete data. It can normalize wrong values. It can make a broken extraction feel more usable than it is.

But if the crawl missed 40% of the products, a beautiful summary does not help.

If the price belongs to the wrong product, valid JSON does not help.

If a required field is empty even though the website had the information, the workflow failed.

So before AI enters the quality process, the team still needs a schema, field definitions, expected outputs, manual review, and tests. LLMs can help with fields that are hard to check using regex or simple rules, but they cannot define “good data” on behalf of the project.

That definition has to come first.

5. The best agent skill does not over-explain

The fifth conversation, with Konstantin Lopukhin, helped me name something I had been noticing in agentic workflows.

A good skill is not the longest possible instruction manual.

In fact, too much instruction can make an agent worse. If you tell it exactly what path to follow in every case, it may miss the simpler and more reliable answer sitting in front of it.

For web scraping, that matters a lot.

A general-purpose coding agent may try to scrape visible HTML when the page has a clean embedded API. It may use curl and get blocked. It may pick the first visible price without checking whether it belongs to the right product. It may produce a one-time answer when the user actually needs a repeatable spider.

A useful scraping skill gives the agent the domain knowledge it needs:

access is part of the problem
the least fragile data source is usually best
structured output must be checked
tests matter
evaluation stays foundational
repeatability matters more than a one-off answer

But it should not over-script the agent into brittle behavior.

The best skill says enough to prevent obvious mistakes, but not so much that it prevents good judgment.

The pattern across all five

The pattern I keep seeing is this:

AI is not making web scraping less human.

It is moving the human work.

From typing to designing.

From scraping once to building loops.

From code generation to evaluation.

From “can the agent do it?” to “can we trust the system?”

That is the thread I want to keep following.

Because the future of web scraping is not just agents that scrape.

It is developers designing reliable web-data systems where agents, tools, infrastructure, tests, and human judgment all know their place.

Why everyone is talking about loop-engineering and how is it changing agentic ai workflows? Claude Code and Web Scraping examples

Ayan Pahwa — Wed, 10 Jun 2026 16:15:44 +0000

A couple of weeks ago I published a walkthrough of my agentic coding setup, the plan-first discipline, the four-agent team, the model routing, the CLAUDE.md files that teach agents to remember between sessions. I stand by every word of it, and yet parts of it already read like a snapshot of a moving target, because that is the pace of the agentic AI world right now: Claude Fable 5 shipped on Tuesday, June 9, 2026, new primitives for autonomous work seem to arrive with every release, and workflows that felt cutting-edge in May, like babysitting a pull request while an agent chews through review comments, are quietly becoming things you design once and then stop doing by hand. The ground is moving under all of us, and it is moving weekly.
Then a clip went viral that put precise words to the shift. Boris Cherny, the creator of Claude Code at Anthropic, said in a recent interview: "I don't prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops." That is not a throwaway line from a futurist; it is the person who builds the most widely used agentic coding tool describing how he actually works, someone who by his own account went a month without opening an IDE while Claude Code wrote every line across 259 pull requests. He no longer prompts the model, he builds loops around it, and the uncomfortable, exciting implication for the rest of us is that increasingly, neither should you.
Addy Osmani has given the practice a name: loop engineering. Instead of steering a model one prompt at a time, you design a system where the agent runs, gets graded against explicit criteria, revises, and repeats until the criteria pass, all without you touching the keyboard. You write the definition of done once. The loop does the rest.
I have been sitting with this for a few days, and the more I turn it over, the more convinced I am that web scraping is not just another domain where loop engineering applies. I think it is one of the best-fit domains there is, because the hardest part of building a good loop is something our community solved years ago. This is an opinion piece, so consider everything that follows an invitation to argue with me.

What loop engineering actually is

Strip away the buzz and a well-designed loop has three parts. There is a generator, the agent doing the work. There is an evaluator, a separate agent or program that grades the output against a rubric of checkable criteria. And there is the loop itself, which feeds the evaluator's report back to the generator until the rubric passes or a budget runs out.

Source: https://code.claude.com/docs/en/agent-sdk/agent-loop
The one rule that everyone building these systems agrees on is that the generator must never grade its own work. Anthropic's engineering team wrote about this directly in their post on harness design for long-running tasks: when a single agent evaluates its own output, it confidently praises mediocre work, and "tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work." Lance Martin at Anthropic reported the same pattern in his experiments with Fable 5, where verifier sub-agents running in independent context windows consistently outperformed self-critique, and where a rubric-driven loop let the model improve a training pipeline roughly six times more than the previous generation managed on the same task.
So the recipe is: write a rubric, separate the maker from the checker, and let the loop run. The whole pattern fits in one diagram, and it is worth a long look, because every idea in the rest of this piece is a variation of it.

The core loop engineering pattern: a human writes the rubric once, then a generator agent and a separate evaluator agent iterate until the rubric passes or the budget runs out, escalating to a human otherwise
That diagram raises the obvious question of where the rubric comes from, because for most domains, turning "good output" into machine-checkable criteria is genuinely hard. For us, it is not.

Web scraping already has the hard part

Think about what a mature scraping project already contains. There is a schema that every item must validate against. There are field coverage thresholds, because a run where only 60% of products have prices is a failed run no matter what the exit code says. There are expected item counts, error rate ceilings, and finish reason checks. In the Scrapy world we even have a dedicated framework for all of this, and I wrote about it earlier this year in my post on giving spidey-senses to your spiders with Spidermon.
Here is the reframe that I cannot stop thinking about: a Spidermon monitor suite is a rubric. Our community spent a decade encoding "what good data looks like" into machine-checkable criteria, because silent failure is scraping's oldest enemy, the spider that runs green for three weeks while quietly shipping garbage. We built the evaluator long before we had a generator capable of acting on its feedback. Every other field adopting loop engineering has to invent its definition of done from scratch. We just have to plug ours in.
The missing piece was never detection. It was what happens after detection, which until now was a human reading an alert, opening the site, sighing at the redesign, and rewriting selectors. Models like Fable 5, which Anthropic says can work autonomously far longer than any previous Claude model, are finally good enough to sit inside that gap. John Rooney saw early versions of this pattern when he built scraping agents for 30 days, and the lesson that stuck with me from his series is that agents fail not from lack of capability but from lack of structure around them. Loops are that structure.

The smallest self-healing spider I could build

I wanted to feel the shape of this before writing about it, so I built the most minimal version possible: a 20-line spider, a deterministic rubric, and a shell loop. No framework, no orchestration platform, nothing you could not reproduce in ten minutes.
The rubric is plain Python that reads items and exits nonzero with a report when quality drops:

REQUIRED = ["name", "price", "url"]
MIN_ITEMS = 5
MIN_FILL_RATE = 0.95
items = [json.loads(line) for line in sys.stdin if line.strip()]
failures = []
if len(items) < MIN_ITEMS:
    failures.append(f"item count {len(items)} < {MIN_ITEMS}")
for field in REQUIRED:
    filled = sum(1 for i in items if i.get(field))
    rate = filled / len(items) if items else 0.0
    if rate < MIN_FILL_RATE:
        failures.append(f"field '{field}' fill rate {rate:.0%} < {MIN_FILL_RATE:.0%}")

The loop runs the spider, grades it, and on failure hands the report to Claude Code in headless mode with permission to read the page and edit the spider, then grades again:

for attempt in 1 2 3; do
  report=$(python3 spider.py | python3 rubric.py)
  if [ $? -eq 0 ]; then
    echo "$report" && exit 0
  fi
  claude -p --allowedTools "Read Edit" <<PROMPT
The spider in spider.py failed its data quality rubric. Report:
$report
Read site/current.html, find why extraction fails, and fix the
selectors. Do not change the output schema or modify the rubric.
PROMPT
done
echo "Still failing after 3 attempts. Escalating to a human."

Then I simulated a site redesign by swapping in a rewritten version of the page, with every class renamed and the structure reorganized. The spider's fill rate dropped to 0% across all three fields, the loop kicked in, and Claude diagnosed the markup change, mapped each old selector to its new equivalent, and the rubric passed on the first healing attempt.
One detail from the run delighted me. The healing agent tried to verify its own fix and was denied permission to execute anything, so the independent rubric re-run in the outer loop was the only judge of whether the patch worked. The maker-checker separation that Anthropic recommends was not something I prompted for. It fell out of the loop's structure. That is the whole point of loop engineering: the guarantees live in the harness, not in the model's good intentions.
A toy, obviously. The page was local, the redesign was synthetic, and three attempts against a fixture is not production engineering. But the shape is real, and the shape is what I want to talk about.
Scaled up honestly, with real scheduled jobs, an independent evaluator, capped attempts, and memory that compounds, the same shape becomes the architecture I keep coming back to:

The self-healing spider loop: a scheduled job runs the spider, the monitor suite validates output, failures trigger a healing agent that patches the spider, an independent evaluator re-grades it, passing fixes deploy and distill a lesson into per-site memory, and exhausted attempts escalate to a human

Loops I want to see the community build

This is the breadth-first part, and the reason I wrote this piece. None of these are tutorials. They are shapes I think are now buildable, and I would genuinely love to see people run with them before I get to all of them myself.

Self-healing spider fleets

The demo above, scaled honestly: a monitor failure on a scheduled job triggers a healing agent that receives the failure report, the cached HTML from the last good run, and the current page. It patches the spider, an evaluator re-runs it against sample URLs, and only a passing grade deploys. Everything else escalates to a human with the diagnosis already written. The rubric is your existing monitor suite, which means teams running Scrapy Cloud with Spidermon already have the trigger and the grader in place.

Spider factories with a definition of done

Generation, not just repair. Instead of prompting an agent to "write a spider for this site," you hand it a goal: extract this schema from these 100 sample URLs with at least a 95% fill rate on every required field. The agent drafts, runs, reads its own fill rates, and iterates, and it does not get to declare victory, because the evaluator holds the rubric. This turns spider development from a conversation into a batch job.

Per-site memory that compounds

Lance Martin describes a memory progression that strong models complete in a loop: fail, investigate why, verify the diagnosis, distill it into a general rule, and consult that rule next time instead of re-deriving it. Map that onto fleet maintenance and you get per-site dossiers: "prices render via JavaScript after scroll," "this storefront migrated platforms in March," "the JSON API behind this listing page is more stable than the HTML." Every healing cycle deposits a lesson, and future cycles start by reading the dossier. Run a consolidation pass across the fleet periodically and cross-site patterns emerge, like a dozen sites sharing a storefront template that all break the same week. That is a scraping team's tribal knowledge, made durable and queryable.

Cost-aware escalation loops

Scraping has a dimension most agent domains lack: every retry has a price tag, and the difference between an HTTP request and a headless browser render is a multiple, not a rounding error. A well-designed loop should climb the escalation ladder only when the evaluator confirms the cheaper tier actually fails, and should record the cheapest configuration that passes the rubric as the new default. The loop optimizes for cost per record, not just for fill rate. I find this idea particularly exciting because it points at loops that do not just maintain quality but actively drive your unit economics down while you sleep.

Plausibility graders that catch the lies

Schema validation catches missing data. It does not catch plausible garbage: prices scraped from the related-items carousel, descriptions truncated at the first comma, currency symbols that quietly changed. A second-tier evaluator, an LLM judge that samples a handful of records per run and compares them against the live page, catches the failure mode that has burned every scraping team I have ever talked to. This grader is cheap because it samples, and it only needs to answer one question: would a human looking at this page agree with this record?

Schema drift scouts

Loops that propose, rather than repair. An evaluator that notices recurring data on pages that your schema does not capture, a new "fulfilled by" field, a sustainability badge, a member price, and files a suggested schema addition with sample evidence. Your extraction quietly keeps up with what the web is publishing instead of freezing at whatever the schema looked like on day one.

Coverage loops for discovery

Item-level quality is one rubric, but corpus-level coverage is another: did we find all the products, all the locations, all the listings? A discovery agent that expands the crawl frontier, graded on coverage against known totals and on duplicate rate, turns the vaguest part of scraping, "are we even seeing everything?", into a number that a loop can push upward.

How I plan to use it

My own starting point is the rubric side, because I think that is where the leverage is. I already maintain a Claude skill that generates Spidermon monitor suites from sample items, which means the grading criteria for any spider can themselves be generated in minutes, and in my agentic coding setup I have been gating agent-written scraping code on objective metrics like fill rate for months without calling it loop engineering. The next step for me is wiring the healing loop into real scheduled jobs, with Zyte API handling access so the loop's failures are genuinely about extraction logic rather than about blocking, and Spidermon actions as the trigger. That experiment deserves its own write-up with real numbers, costs, and the inevitable embarrassing failure cases.
One caution belongs here, and it is the part of the trend I think our industry needs to hold onto hardest. Osmani ends his piece warning against cognitive surrender, accepting whatever the loop produces because it is comfortable, and scraping has a version of this with sharper edges than most fields: an autonomous loop that patches spiders can also patch its way into data you did not intend to collect, from places you did not intend to touch. Compliance review, robots and terms awareness, and the judgment about what should be scraped at all do not go inside the loop. They stay with us. Build the loop, but stay the engineer.

Argue with me

I have shown you the smallest possible version and sketched seven bigger ones, and I am certain the list is incomplete, which is the point of publishing it. If you run spiders in production, you already own the hardest artifact in loop engineering, a battle-tested definition of done, and the only question is what you connect it to. So tell me: which of these loops would you trust in production first, and which one would you never let run unattended? I am easy to find, and I would rather be corrected in public than confident in private.
Originally published on the Zyte blog.

My agentic coding setup: Claude Code, multi-agent orchestration, and more

Ayan Pahwa — Mon, 01 Jun 2026 10:34:45 +0000

Every new agentic coding tool arrives with a version of the same implicit promise: this one will change how you build. I spent a good part of last year installing tools on that promise, configuring them, hitting their limits, and then either reaching for the next release or quietly uninstalling and going back to basics. The result, for a while, was a collection of half-configured assistants that each needed babysitting before they could help with anything.
What I have now is the setup that survived that process, not because the tools are exceptional in isolation, but because I made deliberate choices about what each one is actually for, what it is not for, and how they hand off to each other. I work on main projects at Zyte, on side projects, and on web scraping work, and this setup handles all three without requiring reconfiguration between them. This is that setup: the tools I kept, the habits that hold it together, and the reasoning behind each decision.

My unfair advantage

Before diving into the setup itself, I want to say something about what actually makes an agentic setup work, because it is not the tools.
I have been writing code for more than a decade, starting with Embedded C and C++ before gradually moving to higher-level languages and more recently into Python and web scraping. That background means I can usually tell when an agent is on the right track, when it is confidently producing something plausible but wrong, and when it is about to do something I will spend the next hour undoing. I do not need to read every line it writes to know whether the approach is sound. That accumulated judgment is the unfair advantage: years of building a mental model of how code actually behaves, which now applies directly to supervising what an agent produces.
But this is my advantage, not a universal prescription. Yours is different, and your setup should reflect that. If you have spent years in SEO, your unfair advantage is knowing precisely what good output looks like, what a manipulable signal looks like, and what an agent is getting subtly wrong before the metrics catch it. There are already excellent SEO-specific Claude skills available, and building a team of sub-agents around them (one for technical audits, one for content, one for structured data) with your domain knowledge as the quality filter is a genuinely powerful setup. If your background is in data engineering, you know what a clean pipeline looks like and what a silently broken one looks like, which is exactly the kind of judgment an agent cannot supply for itself. If you come from finance, security, or product management, the same principle holds.
The point is not that deep coding experience is required. Agentic tools amplify whatever domain judgment you already have. Think about where that knowledge lives in your case, and build your setup around it rather than copying someone else's wholesale. Everything in this post is what works for me and my context. Take what fits and ignore the rest.

The workspace that opens itself

The first friction point I fixed was the startup ritual. Every morning I was opening VS Code, arranging panels, launching a terminal, opening Claude Code, and getting everything positioned before I could do anything useful. Five minutes of overhead that was really ten minutes once you account for the mental cost of doing it on autopilot.
I now have a cw function in my ~/.zshrc:

cw() {
  local dir="${1:-.}"
  code "$dir"
  sleep 3 && osascript -e '
    tell application "Visual Studio Code" to activate
    delay 0.5
    tell application "System Events"
      key code 53 using {command down, shift down}
      delay 0.3
      key code 50 using {control down}
    end tell
  ' &
}

Type cw in any directory and VS Code opens in that directory, then AppleScript fires after three seconds to focus the window and drop you straight into the terminal panel where Claude Code is waiting. One command, full workspace ready.
The layout is fixed: file explorer on the left, terminal strip at the bottom, the agent chat panel in the middle, and outputs on the right. The chat pane is centered, not tucked in the sidebar, and that placement is deliberate. When the agent is your primary collaborator, putting it in the sidebar demotes it spatially. The center position is a constant reminder that orchestration is the primary activity and everything else supports it. For inline code review without breaking context, the Codex plugin for Claude runs directly in the editor.

Plan first, always

The habit that improved my output more than any tool was deciding that every task, regardless of scale, starts in plan mode before any file is touched. No exceptions.
Plan mode forces the agent to surface its assumptions, propose a concrete approach, and wait for sign-off before it touches anything. The default behavior of most agentic tools is to start executing immediately, and high-confidence execution in the wrong direction is the failure mode I have run into most. The agent is rarely wrong because the code is bad; it is wrong because it interpreted the brief differently than I intended, and three minutes of planning would have caught the gap.
I did not arrive at this habit entirely on my own. Dario Amodei, CEO of Anthropic, mentioned in a podcast that he spends the majority of his time in plan mode when working with Claude, and that once the plan is solid the actual execution becomes relatively straightforward. That framing stuck with me. If the person building the model treats planning as the primary activity, it is probably worth taking seriously.
For tasks where the problem is still fuzzy in my own head, I dictate into plan mode using WisprFlow. This is not just a comfort choice. I have noticed consistently that my spoken prompts produce better results than my typed ones: speaking forces me to construct a full sentence rather than tapping out a telegraphic shorthand, and that extra formality in the brief translates directly into more precise agent output. Describing the problem, the likely approach, and any constraints out loud usually clarifies the brief before the agent has responded, which means the plan mode exchange is a confirmation rather than a negotiation.
Something I have been testing recently on the planning side: Claude Code's /goal command. The idea is straightforward: before anything else in a session, you set a high-level goal that the agent holds as a persistent north star throughout all its subsequent actions. Where plan mode answers "how do we approach this specific task", /goal answers "what is this entire session ultimately in service of." I came across the same concept in Codex and liked the forcing function it created: it keeps a long session from gradually drifting away from what you actually opened it to achieve. I am still finding the edges of how to use it well, but the principle is sound: the more clearly you can state what done looks like before the first message, the less corrective steering you need to do mid-session. If you try it, be specific: "refactor the auth module to remove the session token storage" will serve you better than "clean up the auth code."

Two tools, different jobs

One thing worth saying before getting into the specifics: the agentic coding space is moving faster than almost any other area of software right now. Every major provider is shipping changes to tooling, pricing, context limits, and model capabilities on a cadence that would have seemed unrealistic two years ago. That pace is exciting, but it also means that going completely all-in on a single vendor is a real risk. If one provider changes pricing, deprecates a model, or ships a breaking change to their CLI, a setup that depends entirely on them stops working. The practical response is to not let that happen: maintain flexibility, keep alternatives warm, and make sure switching costs stay low.

My primary tool is Claude Code CLI with official Anthropic models, and it is where the majority of my serious work happens. I run it on the Claude API pay-per-usage plan, which means I pay for what I use and nothing beyond that. No monthly seat fee accumulating on days when I am not writing code, and no "I am already paying for it" pressure to keep the agent running past the point of diminishing returns. I also keep Codex in the mix via the Mac desktop app, not as a replacement, but as a parallel tool I use enough to stay current with how it is developing.
For model experimentation and usage overflow, I use OpenCode with OpenRouter via bring-your-own-key (BYOK). This is the experimentation layer and, frankly, the hedge against vendor lock-in. When a new model appears and I want to try it against a real task before committing to it in my main workflow, I reach for OpenCode. My ~/.aliases file has eight model shortcuts that make switching a single word in the terminal:

alias oc='opencode --model openrouter/anthropic/claude-sonnet-4-6'
alias oc-ds='opencode --model openrouter/deepseek/deepseek-chat-v3-1'
alias oc-free='opencode --model openrouter/deepseek/deepseek-r1:free'
alias oc-qwen='opencode --model openrouter/qwen/qwen3-coder'
alias oc-gemini='opencode --model openrouter/google/gemini-2.5-pro'
alias oc-opus='opencode --model openrouter/anthropic/claude-opus-4-7'
alias oc-cost='opencode --usage'
alias oc-stats='opencode --stats'

oc-cost and oc-stats are not cosmetic shortcuts. The pay-per-usage model only works as a cost discipline if you can see what you are spending.
OpenRouter also has an auto-router option (openrouter/auto) that selects the best available model for each prompt automatically, which is genuinely useful when you are unsure which model fits a task and do not want to think about it. My opencode.json config defines three routing entries that cover the main scenarios:

"openrouter/auto": {
  "name": "Auto Router (picks best model for prompt)",
  "tool_call": true,
  "limit": { "context": 200000, "output": 8192 }
},
"openrouter/free": {
  "name": "Free Router (picks best free model)",
  "tool_call": true,
  "limit": { "context": 128000, "output": 4096 }
},
"openrouter/pareto-code": {
  "name": "Pareto Code Router (auto-routes coding tasks)",
  "tool_call": true,
  "limit": { "context": 200000, "output": 8192 }
}

auto is the general-purpose router. free picks the best available free model, which is what I reach for when testing something throwaway. pareto-code is a coding-specific router that OpenRouter maintains separately, optimized for code tasks rather than general prompts.
Beyond Claude overflow, I use OpenRouter's free tier for testing with tools like OpenClaw and the Hermes agent, where spending money on a model I am just kicking the tires on makes no sense. This side of the setup is almost entirely for side projects and learning how different models actually behave on real tasks rather than benchmarks. My personal ranking after a fair amount of experimentation, in order: Qwen3 Coder, DeepSeek, MiniMax, Kimi, and Gemma. Qwen and DeepSeek are consistently good on code; MiniMax and Kimi are worth watching for longer-context tasks; Gemma punches above its weight for its size.
A note for developers still on Claude Pro ($20/month): the plan uses rolling usage windows that reset approximately every five hours. If you plan to code from noon, send Claude a short message at 7 AM. Your window resets right as you sit down, and the next reset lands around 5 PM, which means you can run two back-to-back full sessions from noon through the evening without hitting a cap mid-task. The exact window length has shifted with recent Anthropic updates, so check your own account to calibrate, but the principle holds: prime your session before you need it, not after you have already hit the limit. Since switching to pay-per-usage I no longer need this trick, but it was genuinely useful for the two years I ran on the flat-rate plan.

The four-agent team

Running a general-purpose agent at every task is a bit like asking one person to handle architecture, implementation, code review, and codebase archaeology simultaneously, and expecting them to be equally good at all four. I took inspiration from Gary Tan, CEO of Y Combinator, who described his own layered agent stack (which he calls GStack) as a way of giving each model a specific role rather than asking one model to do everything. I am not using GStack directly, but the framing shaped how I think about agent design: specialize the agents, not the prompts.

In OpenCode, I have defined four named agents, each with a specific model, a specific role, and a specific permission scope.
@architect runs on Claude Sonnet 4.6 and is read-only: no edit or bash access. It asks clarifying questions first, then produces ASCII diagrams and a numbered implementation plan. When it needs to understand the codebase before it can plan, it invokes @scout. The output is written to be clear enough for a junior developer, or for a cheaper model like DeepSeek, to execute without interpretation.
@scout runs on Gemini 2.5 Flash with a one-million token context window and is also read-only. It traces call chains, maps data flow, and produces structured reports with full file paths and line numbers. The large context window makes it the right choice for reading substantial portions of an unfamiliar codebase without losing thread.
@coder runs on DeepSeek V3.1, configured at 40 steps and temperature 0.1. It follows the architect's plan exactly and does not extend scope. Before marking any task complete, it invokes @reviewer.
@reviewer runs on Qwen3 Coder, is read-only, and works through a fixed priority order: security vulnerabilities first, then logic errors, missing error handling, performance bottlenecks, and finally code clarity. It cites exact file paths and line numbers for everything it flags.
The permission constraints are what most people skip, and they are what matter most. A read-only agent cannot accidentally delete files or run shell commands, which limits the blast radius when an agent misreads an instruction. In Claude Code, I apply the same model-tiering logic via the native multi-agent orchestrator: Claude Opus 4.7 for planning, Claude Sonnet 4.6 for execution, and Claude Haiku 4.5 for admin tasks like git summaries and log triage. For the heaviest parallel projects, I use Conductor, which runs multiple Claude Code and Codex instances across separate areas of a codebase without context bleed between them.
A word on scale: I am not trying to run 20 agents across 10 projects simultaneously, and I am not optimizing for that. At any given time I work across two or three projects at most, because beyond that I notice the creative block creeping in and the context-switching cost becoming real. Four concurrent agents is my personal ceiling before things start feeling chaotic rather than productive. Running more agents than you can coherently supervise is not a productivity gain; it is just noise with extra steps. The right number is the one where you still know what each agent is doing and why.

Teaching agents to remember: the CLAUDE.md file

Here is the thing nobody tells you when you start using agentic tools seriously: the agent forgets everything between sessions. Every conversation starts from zero. Your project conventions, the architectural decisions you made three weeks ago, the one gotcha in the authentication middleware that will silently break if you touch it: gone. The agent does not know any of it unless you tell it again.
The fix is a CLAUDE.md file in the root of every project. Claude Code reads this file automatically at the start of each session, which means it walks into the codebase already briefed: how the project is structured, what conventions to follow, what not to touch, and why certain decisions were made. It is the difference between starting a session with a junior developer who has never seen your codebase and starting a session with one who was briefed before they arrived. I include things like folder structure, key design decisions, known gotchas, and any non-obvious constraints. What is not in that file will surface at the worst possible time, usually when the agent is three files deep into something it should not have started. I have learned this lesson more than once.
There are actually two levels of CLAUDE.md worth maintaining separately. The per-project file, described above, carries context specific to that codebase. But there is also a global CLAUDE.md at ~/CLAUDE.md that Claude Code reads across every session, regardless of project. This is where the universal stuff lives: how you like responses formatted, code style preferences that never change, recurring patterns you reach for project after project. The smartest way I have found to populate it: ask Claude directly. Open a session after you have done a few projects and ask it what it has noticed you doing repeatedly across different codebases — preferences, habits, corrections you keep making. The answer is usually more accurate than anything you would write from memory, and it goes straight into the global file. You only have to do that calibration once, and every future session inherits it.
The same logic of "define it once, reuse it forever" applies to custom slash commands. Claude Code lets you create your own /commands by dropping a markdown file into .claude/commands/ inside a project, or into ~/.claude/commands/ for commands that travel with you globally. Anything you find yourself prompting repeatedly is a candidate: I have a /pr command that opens a pull request with a consistent format, a /review command that runs a code review against a checklist I care about, and a /standup command that summarizes what changed since the last commit in plain language. The rule of thumb is the same as for skills: if you have typed the same instruction more than twice, it should be a command. The overhead is a single markdown file; the return is that you never type it again.
In practice, this shapes how I move between tasks. One session handles one specific thing, 95% of the time. When that task is done and I want to start something different, I update the CLAUDE.md first, capturing any decisions made, gotchas discovered, or context the next session will need, and then launch fresh. This is not a workaround; it is the actual workflow. Each session stays focused on exactly one thing, the context window never accumulates unrelated baggage, and the token cost stays predictable because the session ends when the task ends.
One thing I want to push back on slightly: the idea that more persistent memory is always better. The "second brain" framing — building an ever-growing knowledge base that carries everything forward — is appealing in theory, but I have found clean starts genuinely valuable. A fresh session with a tight, well-written CLAUDE.md is often sharper and more focused than a long session carrying the accumulated noise of everything that came before. Starting fresh is not a disadvantage; sometimes it is the whole point. This is especially true when starting a brand new project: no CLAUDE.md, no prior context, no assumptions inherited from a different codebase. The agent approaches it with the same clean slate you do, which means nothing from the last project bleeds into this one. That is not a limitation of the tool; it is the right default. The CLAUDE.md approach hits the balance I actually want: enough persistent context to orient the agent quickly, without the clutter.

When the context window fills up

Agentic coding sessions on complex tasks will, eventually, produce a context window that is full or close to it. The agent's effective memory degrades as the window saturates, and you start getting responses that feel slightly off, repetitive, or strangely overconfident about something it got wrong two thousand tokens ago.
Claude Code handles this with automatic context compaction, which summarizes earlier parts of the conversation to make room. For sessions where I want manual control, I use the /compact command to trigger a summary on demand. When even that is not enough, the bluntest tool available is the right one: start a fresh session, point at the CLAUDE.md file, and re-brief the agent on exactly where the previous session left off. It feels a bit caveman, but a focused, fresh session outperforms a saturated long one almost every time. My experience is that a 10,000-token focused session produces better output than a 100,000-token sprawling one that has lost the thread.

Web scraping: where the Zyte layer comes in

Web data is not optional for serious agentic workflows. Agents that can research, verify facts, monitor changes, track competitors, or enrich datasets with live information are dramatically more useful than agents working from static knowledge alone. The web is the data source.
The problem is that the web does not cooperate equally. Some pages render entirely in JavaScript and return nothing useful to a basic HTTP request. Others sit behind rate limits, bot detection, or login walls. Some block entire cloud IP ranges outright. An agent that tries to fetch a page and gets a 403, a CAPTCHA, or a JavaScript shell with no content is effectively blind, and it will usually not tell you that clearly; it will just work with whatever it got. For the straightforward cases, Claude's built-in browsing is fine. For anything beyond that, you need a layer that actually understands how modern websites are built and how to get through them reliably.
That is where Zyte's tooling earns its place. For web scraping work, the setup picks up a layer specific to Zyte's tooling. Zyte publishes an official set of Claude Code skills at github.com/zyte-ai/claude-skills, installable in two commands:

claude plugin marketplace add zyte-ai/claude-skills
claude plugin install zyte-web-data@zyte-ai

Once installed, the skills slot into Claude Code as slash commands and activate automatically on relevant prompts. The ones I reach for most are:

/scrape: end-to-end workflow from a URL to a working Scrapy spider with web-poet page objects; this is the one you use when you just want to hand Claude a URL and a description of what to extract
/scrape-define: downloads a single detail page, discovers extractable fields, and iterates on the schema in the terminal until you approve it; good for quickly scoping what a site can give you
/scrape-explore-site: crawls from a start URL and saves a diverse set of pages (start, list, and detail) with classified links; useful before committing to a schema
/scrape-codegen: takes an extraction spec and generates the web-poet page object code; the output of /scrape-define feeds directly into this
/scrape-scrapy-cloud: deploys projects, schedules spiders, manages jobs, and surfaces logs and items from Scrapy Cloud, all from the terminal The skills integrate with the Web Scraping Copilot and are designed to pick up scraping prompts automatically, so you do not need to invoke a specific command for routine requests. If you are curious how these fit into a broader workflow, the post on supercharging web scraping with Claude skills covers the combination in detail. Everything is git-tracked, including personal side projects. The ghs alias in my shell switches git identity instantly between my Zyte work email and my personal email, which eliminates the risk of pushing to the wrong remote after a context switch between work and a side project. On MCP servers versus CLI tools: my standing rule is to reach for a CLI tool first and add an MCP server only when there is genuinely no CLI equivalent. MCP servers add indirection between the agent and the tool, and that indirection is not free: it makes the toolchain harder to audit, harder to debug, and slightly more likely to produce ambiguous outputs. If you are weighing your options, the comparison of Claude skills, MCP, and Web Scraping Copilot is worth reading before committing. One area where I have been rethinking the default recently is web search. Most agents fall back to keyword-based search, which is fine for locating a documentation page but falls apart when an agent needs to do actual research. I came across Exa at a local developer meetup, and it is built specifically for AI agents using semantic search rather than keyword matching, which produces noticeably better results when the agent needs to find conceptually related content rather than an exact phrase. The catch, and the reason I have not fully switched over, is that Exa currently only offers an MCP server and not a CLI utility. That puts it in direct conflict with the CLI-first rule: every time the agent invokes an MCP server there is a context switch, a round-trip, and a small but real cost in time and tokens that adds up over a long session. So for now I enable Exa selectively on projects where deep research is a core part of the work, and fall back to Claude's built-in search everywhere else. I am still exploring it, and if a CLI lands I will probably use it much more broadly. The last piece of the scraping layer is what I think of as the objective metric loop. Before running the agent on a scraping task, I define a concrete, measurable target: field fill rate above 95%, zero extraction errors across 100 test URLs, or a specific field-level accuracy requirement. The agent runs, the output is evaluated automatically against that metric, and I re-prompt with the delta. The loop continues until the metric is hit, not until the code looks right on inspection. "Looks right" is not a metric. ## A few principles, for what they are worth These are not best practices from a blog post. They are things I arrived at through repetition, usually after doing the opposite first. Stop obsessing over prompts. Models are meaningfully smarter than they were twelve months ago, and they will be smarter again in twelve more. A clear, complete description of what you want is almost always sufficient today. Intricate prompt engineering made more sense when models were brittle; spending that energy on your workflow instead will compound better. Anything done twice should become a skill. If you have guided an agent through the same process more than once, it belongs in a skill file. A skill is a reusable, well-described prompt with clear inputs and outputs. The overhead of writing one is low; the compounding return is not. Each skill should do exactly one thing. A skill that researches a topic, writes a script, and suggests titles is three skills waiting to be separated. Single-purpose skills are easier to debug, easier to improve, and much easier to reason about when something breaks. Skills can be chained into workflows. Three separate skills (research the next video topic, write a script from the research, suggest titles from the script) can be combined in sequence to produce a full workflow while remaining individually useful and testable. The composition is more flexible than a monolith, and any one skill can be swapped out without rebuilding everything. Bundle custom scripts with the skills that need them. If a skill depends on a helper script (a parser, a formatter, a validator), keep it in the same directory. Skills that rely on tools scattered elsewhere become fragile. Skills that travel with their dependencies stay portable. ## The rest of the bench Not everything in my setup is fully integrated or daily-use. A few tools I keep within reach at different stages of exploration: ChatGPT (GPT 4.5 and above) is where I go for conversational research: thinking through a problem in plain language, getting a second opinion on an approach before committing to it in code, or just having a broad discussion that would clog an agentic workflow. Not everything needs an agent. Perplexity covers manual search and research where I want cited sources rather than a generated answer. I am also currently poking at Perplexity Computer, though it is genuinely early days and I do not have a settled opinion on it yet. Local LLMs via LM Studio and Ollama, used for offline experimentation. I will be honest: my current hardware is the constraint, not the tooling. Running anything genuinely capable locally is a stretch on my machine. If you are in the same position and want to know what you can actually run before committing to a download, LLMFit is a handy utility that evaluates your system specs and tells you which models are feasible, and worth running before you spend an afternoon downloading a 70B model that will not fit in your RAM. ## Pick one thing The setup works because of the discipline behind it, not the tools themselves: plan before executing, give agents only the permissions they actually need, write the CLAUDE.md file before you need it (not after), evaluate against metrics rather than impressions, and restart aggressively when the context window is saturated. Most of what I have described here is free or pay-as-you-go, and none of it requires a large upfront commitment to try. If you are coming to this fresh, pick one piece rather than the whole stack. Enforcing plan mode before every task will return more value more quickly than any new tool installation, and adding a CLAUDE.md to a project you already work in will pay off in the first session. If you work with Scrapy and want to add the web scraping layer that connects this setup to Zyte's toolchain, the Zyte free trial is where to start. There is more to cover — the Karpathy metric loop in more depth, how I use CLAUDE.md across different project types, and how the local LLM setup is evolving as hardware catches up. If any of that sounds worth a Part 2, let me know in the comments. And if you want to stay across what the team at Zyte is building, subscribing to the Zyte newsletter means you will not miss it.

The compiler caught a lot. It didn't catch enough.

John Rooney — Sun, 31 May 2026 18:20:02 +0000

I built a small web scraping framework in Rust, mostly with an AI doing the typing. It's called ferrous — a Colly-style collector: register CSS selector callbacks, queue URLs, write JSONL. About 700 lines. The pitch I kept hearing, and half-believed, was that Rust and LLMs are a good match now: the borrow checker is a correctness oracle the model can lean on, so the class of bugs that plagues AI-written Python just won't compile.

That's true. It's also where the story gets uncomfortable, because the build was green and the code was still wrong.

How I worked

I'm not a Rust native. ferrous was partly an excuse to get fluent — build something real instead of reading about lifetimes — and partly a test of how far an LLM could carry the typing while I drove. The loop was plain: describe the next change in English, let the model write the Rust, read what came back, run cargo, move on. It kept observations.md as a running design journal, one entry per change, each with a short rationale for the decision it made.

That setup has a soft spot, and it's the whole point of this post. When you drive a language you don't fully know, the only reviewer you've got with real authority is the compiler. Everything past that — is this idiomatic, is it the right abstraction, does it actually do what the journal claims — depends on already knowing what correct looks like. Which is exactly the knowledge a learner doesn't have yet. Keep that in mind through the next part, because it's the difference between the one bug I could have caught and the six I couldn't.

The bug that the toolchain told me wasn't there

I ship two fetch backends. The default goes through the Zyte API; an optional one, gated behind a wreq feature, makes direct requests with browser TLS emulation. Each has an example. After a refactor that added URL resolution — ctx.resolve_and_visit(href) so callbacks stop hand-building absolute URLs — I had the model update the examples to use it. It did, and it wrote up the change in a running design journal it kept:

Added ctx.resolve(href) and ctx.resolve_and_visit(href) ... The examples were updated to use ctx.resolve_and_visit(&href).

cargo test passed. cargo check passed. I believed the journal. Here is the actual line in the feature-gated example:

.on_html("li.next a", |el, ctx| {
    if let Some(href) = el.attr("href") {
        ctx.resolve_and_visit(href);   // href is String; the method wants &str
    }
})

resolve_and_visit takes &str. href is a String. That does not compile — String doesn't coerce to &str in argument position, only &String does. The Zyte example got the &; the wreq one didn't. The model fixed one of the two call sites it claimed to have fixed, and reported both as done.

So why was the build green? Because the broken example is behind a feature flag, and cargo check and cargo test don't build it by default. You have to ask:

$ cargo check --all-targets --all-features
error[E0308]: mismatched types
  --> examples/books_direct.rs:20:39
   |
20 |                 ctx.resolve_and_visit(href);
   |                     ----------------- ^^^^ expected `&str`, found `String`

This is the part worth sitting with. The compiler would have caught it — it's a textbook E0308, the friendliest error rustc produces, complete with help: consider borrowing here. The type system did its job perfectly. It just never ran on that file, because the default build target set didn't include it, and nobody — not me, not the model — pointed it at the path where the error lived. The oracle was switched off for exactly the line that needed it, and the green checkmark covered the gap.

What the compiler buys you, honestly

I don't want to undersell the good part, because it's real and it's specific. The most interesting moment in the whole project was the model refactoring the Element type. Originally it stored matched HTML as a string and re-parsed it on every field access — three accessors, three parses of the same fragment. The fix was to parse once and store the scraper::Html. But Html is !Send, and the crawl loop runs callbacks inside spawned tasks. The model reasoned about this explicitly:

Html is !Send but this is safe: Element is only created and consumed inside synchronous callbacks and is never stored in an Arc or sent across threads.

And it structured the crawl loop to hold that invariant — parse the document, run every callback, drain the results into owned Vecs, and drop the Html before the first .await:

let (all_visits, all_items) = {
    let doc = Html::parse_document(&html);
    // ... run callbacks, collect owned results ...
    (all_visits, all_items)
    // doc dropped here, before any await
};

In Python this reasoning is a comment you hope stays true. In Rust, if the model had gotten it wrong — held the Html across the await, stuffed an Element into the task's captured state — it wouldn't compile. The Send bound is load-bearing. That's the version of "the compiler helps the AI" that actually holds up: it converts a class of concurrency mistakes into build failures, and the model can lean on that to attempt refactors it would otherwise have to be timid about.

The same goes for the ordinary stuff. Widening push_item(Value) to push_item<T: Serialize>, swapping a spin-polling semaphore loop for a JoinSet, threading a FetchError enum through the fetch path — these landed cleanly and idiomatically, because they're well-trodden Rust patterns and the types kept the model honest about the seams.

The class of bugs that compiles fine

Then I had a second model do a senior-review pass over the finished code, and it found seven things the green build was hiding. None of them are type errors. All of them compile.

The library panics on normal user mistakes — an invalid selector, a missing API key, a bad output path all unwrap/expect their way to a crash, even though run() returns a ScrapeResult as if failure were representable. concurrency(0) doesn't error; the inner while tasks.len() < 0 loop never spawns anything, tasks.is_empty() is immediately true, and the crawl exits having silently processed nothing. join_next().await discards its Result, so a panicking callback vanishes and the run still reports success. And the stats are quietly miscounted: fetch_errors never increments for HTTP 4xx/5xx — only for network and parse failures — so a crawl that 500s on every page can report fetch_errors: 0 and had_errors() == false. The README, meanwhile, promises that successful status codes are tracked; they aren't recorded at all.

Every one of these is the same shape of mistake: code that satisfies the types and the test suite while doing the wrong thing. The compiler has nothing to say about whether concurrency(0) is meaningful, whether a 500 counts as an error, or whether your docs match your behavior. Those are semantic claims, and semantics is exactly the layer where LLMs are fluent and confident and wrong — and it's the layer Rust doesn't police.

There's a tidy demonstration of this sitting in the repo: the design journal and the README both describe the intended state of the code, not its actual state. The journal says both examples were fixed; one wasn't. The README says status codes are tracked; they're partly not. The model writes the world as it meant to leave it, and prose has no type checker.

You can't review what you can't read

Go back through those seven findings and ask what it would have taken to catch each one by reading. concurrency(0) exiting silently: you'd have to know that while tasks.len() < 0 is vacuously false, then trace what an empty JoinSet does on the next line. The fetch_errors miscount: you'd have to be holding the intended meaning of "error" in your head and notice the HttpError arm bumps a status counter where you expected it to bump the error count. The &href that wouldn't compile: you'd have to know that String doesn't coerce to &str in argument position — the single Rust fact the entire bug turns on.

None of these are exotic. They're the things you internalize after enough hours in the language. But that's precisely the trap when a learner pairs with an AI: the model emits code that looks right, reads fluently, and compiles, and the only way to know it's wrong is to already know the thing you were hoping the model would handle for you. Knowing Rust well wouldn't have stopped the model from writing the concurrency(0) hole — it would have stopped me from nodding past it in review.

So the green build gets promoted past its pay grade. When you can't evaluate the semantics yourself, "it compiles" slides from not obviously broken to works, because compiling is the only check you're actually equipped to read. That promotion is the real hazard of writing an unfamiliar language with a model that writes it confidently. The bugs didn't come from the model being bad at Rust — it handles the syntax better than I do. They came from neither of us being positioned to see the gap between code that satisfies the types and code that does the right thing. The model can't see it because it's a semantic claim about intent; I couldn't see it because I didn't yet know what right looked like.

So, is Rust better for AI now?

For the bugs Rust can see, yes, unambiguously, and more than I expected before I watched the !Send refactor go through. The borrow checker and the trait system catch a real category of AI error at compile time, and that lets a model attempt more aggressive changes without silently corrupting state. The floor is genuinely higher than in a dynamic language.

But the floor isn't the problem. Most of what was wrong with ferrous compiled, passed its tests, and was described accurately by documentation that was itself false. The compiler raised the floor; it did nothing for the ceiling, and feature flags punched a hole in even the floor by hiding a target from the default build. "It compiles, the tests are green, and the model says it's done" turned out to be three independent false comforts stacked on top of each other.

The lesson I'd actually act on has two parts. The mechanical one is cheap: the green checkmark is scoped and the model doesn't know the scope, so run --all-features --all-targets on every check, read the diff instead of the summary of the diff, and treat anything the model asserts about its own output — "both examples updated," "status codes tracked" — as a claim to verify rather than a result to log.

The other part is slower and matters more. If you're using an AI to write a language you're still learning, the AI is not a substitute for learning it — it's the thing that makes the learning feel optional right up until a 500 gets counted as a success. The type system is a fast, narrow oracle that flags the bugs it can see and stays silent on the ones that matter most. Closing that silence is on you, and you can only close it by knowing the language well enough to read what the model wrote and see where it's quietly wrong. I came out of ferrous knowing more Rust than I went in with. That, more than the framework, was the point — and it's the only thing that would have caught the other six.

Why Your Scraper Breaks Without Warning (And How to Fix It)

John Rooney — Sun, 31 May 2026 18:08:26 +0000

Most scraper failures don't raise exceptions. The spider finishes, the pipeline writes a file, the process exits with code 0 — and the output contains 0 items, or 800 instead of 8,000, or fields that are all empty strings. A CSS selector stopped matching after a site redesign. A field that used to be present is now conditional. The "next page" link moved.

None of these produce tracebacks. Without explicit checks, you won't know until someone looks at the data.

This post covers three layers of defence: validating individual items before they reach storage, checking aggregate counts at the end of a run, and setting up logging that's actually useful.

Why selectors fail silently

BeautifulSoup and Scrapy's CSS selectors return None or an empty list when nothing matches — they don't raise. The problem propagates depending on what you do next.

# This raises AttributeError if price_element is None
price_element = book.find("p", class_="price_color")
price = price_element.text.strip()  # AttributeError: 'NoneType' has no 'text'

That's a loud failure, which is fine. But this version is silent:

price = book.css("p.price_color::text").get(default="")

get(default="") is convenient, but it means a changed selector produces empty strings in your output rather than an error. You end up with a file full of {"title": "Some Book", "price": "", "rating": ""} records that look complete until you actually check the values.

The fix is validation — checking that the fields you extracted are actually populated before writing anything.

Item validation

A validation function takes an item dict and raises if any required field is missing or empty:

REQUIRED_FIELDS = ["title", "price", "rating"]

def validate_item(item: dict) -> dict:
    problems = [f for f in REQUIRED_FIELDS if not item.get(f)]
    if problems:
        raise ValueError(f"Missing fields {problems}: {item}")
    return item

not item.get(f) catches None, "", [], and 0 — all values that get() defaults produce when a selector fails. If you want to allow 0 as a valid value, use item.get(f) is None instead.

In a standalone scraper:

import requests
from bs4 import BeautifulSoup
import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)
logger = logging.getLogger(__name__)

REQUIRED_FIELDS = ["title", "price", "rating"]

def validate_item(item: dict) -> dict:
    problems = [f for f in REQUIRED_FIELDS if not item.get(f)]
    if problems:
        raise ValueError(f"Missing fields {problems}: {item}")
    return item

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

resp = session.get("https://books.toscrape.com/", timeout=15)
resp.encoding = "utf-8"
soup = BeautifulSoup(resp.text, "html.parser")

items = []
errors = 0

for book in soup.find_all("article", class_="product_pod"):
    raw = {
        "title":  book.find("h3").find("a")["title"],
        "price":  book.find("p", class_="price_color").text.strip(),
        "rating": book.find("p", class_="star-rating")["class"][1],
    }
    try:
        items.append(validate_item(raw))
    except ValueError as e:
        logger.warning("Skipping invalid item: %s", e)
        errors += 1

logger.info("Scraped %d valid items, %d errors", len(items), errors)

Whether to skip invalid items or abort depends on the use case. For a scrape where partial data is usable, log and skip. For a scrape where every record matters, raise immediately.

Validating in a Scrapy pipeline

In Scrapy, validation belongs in a pipeline that runs before the storage pipelines. The ItemAdapter class normalises access across Scrapy Items, dataclasses, and plain dicts:

# pipelines.py
from itemadapter import ItemAdapter
import logging

logger = logging.getLogger(__name__)

REQUIRED_FIELDS = {"title", "price", "rating"}


class ItemValidatorPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        missing = REQUIRED_FIELDS - {
            k for k, v in adapter.items()
            if v not in (None, "", [])
        }

        if missing:
            raise ValueError(
                f"Item missing required fields: {missing} | {dict(adapter)}"
            )

        return item

Raising ValueError (or Scrapy's DropItem) from a pipeline drops the item and logs the error. It doesn't stop the spider. Enable it at a lower priority number than your storage pipelines so it runs first:

# settings.py
ITEM_PIPELINES = {
    "myproject.pipelines.ItemValidatorPipeline": 100,
    "myproject.pipelines.JsonlPipeline":         300,
    "myproject.pipelines.SqlitePipeline":        400,
}

Count checking

Per-item validation catches bad fields, but not a situation where the spider simply stopped finding items at all. A selector change that breaks the outer find_all("article", class_="product_pod") call produces zero items with zero errors — everything looks fine.

A count check at the end of a run catches this:

MIN_EXPECTED = 10

logger.info("Scraped %d valid items, %d errors", len(items), errors)

if len(items) < MIN_EXPECTED:
    logger.error(
        "Item count %d below threshold %d — check selectors or site structure",
        len(items),
        MIN_EXPECTED,
    )

In Scrapy, this belongs in an extension that connects to the spider_closed signal:

# extensions.py
from scrapy import signals
import logging

logger = logging.getLogger(__name__)


class ItemCountChecker:
    MIN_ITEMS = 10

    @classmethod
    def from_crawler(cls, crawler):
        ext = cls()
        crawler.signals.connect(ext.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
        return ext

    def spider_opened(self, spider):
        self.item_count = 0
        spider.crawler.signals.connect(
            self.item_scraped, signal=signals.item_scraped
        )

    def item_scraped(self, item, spider):
        self.item_count += 1

    def spider_closed(self, spider):
        if self.item_count < self.MIN_ITEMS:
            logger.error(
                "Spider %s closed with only %d items (threshold: %d). "
                "Check selectors or site structure.",
                spider.name,
                self.item_count,
                self.MIN_ITEMS,
            )
        else:
            logger.info(
                "Spider %s: %d items scraped.", spider.name, self.item_count
            )

Enable in settings:

EXTENSIONS = {
    "myproject.extensions.ItemCountChecker": 500,
}

The threshold needs thought. A scraper that usually returns 800 items should probably alert below 700, not below 10. Track a few runs first, then set the threshold relative to your expected baseline.

Logging that's useful

Scrapy's default logging is verbose at INFO level — request counts, middleware messages, and stats flood the output along with the things you actually care about. Two settings improve it:

# settings.py
LOG_LEVEL = "WARNING"   # suppress INFO noise from Scrapy internals
LOG_FILE  = "scrape.log"

Set LOG_LEVEL = "WARNING" for Scrapy internals, but log your own pipeline and extension messages at INFO. Because Scrapy uses Python's standard logging module, you can configure your own loggers separately:

import logging
logging.getLogger("myproject.pipelines").setLevel(logging.INFO)
logging.getLogger("myproject.extensions").setLevel(logging.INFO)

For standalone scrapers, the format matters. The default %(message)s loses context. At minimum include the timestamp and level:

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
)

When something goes wrong at 2am, the timestamp is how you correlate a failure with a site change or a rate-limit event.

The three signals that something is broken

In practice, silent scraper failures show up three ways:

A count drop with no errors. Item count is 0 or far below baseline, error count is 0, the spider exited cleanly. This is a selector change — the outer container no longer matches.

A count that looks right but fields are empty. Item count is normal, but price and rating are empty strings across all records. An inner selector changed while the outer one still matches.

A count drop accompanied by validation errors. Item count is lower than expected, and the log contains Skipping invalid item messages. A field is now conditional — present on some items, absent on others, probably due to a new page layout.

Each failure mode points at a different selector level. Logging both counts and individual errors separately makes it easier to tell which one you're dealing with.

Putting it together

A scraper with validation, count checking, and useful logging doesn't need to be complex. These three additions together cover the most common silent failure modes:

Validate each item before writing — catches changed inner selectors
Check the final count against a threshold — catches broken outer selectors
Log counts and errors at the end of every run — makes failures visible without manual inspection

For a production scraper that runs on a schedule, route the logs somewhere you'll see them. LOG_FILE to a location your monitoring system watches, or send the error-level messages to a webhook. A scraper that runs silently and produces empty output for a week is worse than one that fails loudly on the first run.

Tags: python scrapy webscraping tutorial

How to Store What You Scrape (Without Making a Mess)

John Rooney — Sun, 31 May 2026 18:07:43 +0000

The default for a first scraper is usually printing to stdout or dumping everything into a JSON array. Both work for a single test run. Neither works reliably at scale — a JSON array can't be read incrementally, and stdout isn't a data format.

The format you choose matters more than it seems. The wrong one creates problems that appear later: a 2GB JSON file that has to be fully parsed before you can read a single record; a CSV that corrupts silently when a field contains a comma; a database that slows to a crawl because nobody added an index. This post covers the three formats that cover most scraping use cases, and when to reach for each.

JSONL over JSON arrays

A JSON array written to a file is convenient until the scrape crashes halfway through, at which point you have a truncated, unparseable file. You also can't append to it without reading the whole thing first.

JSONL (JSON Lines) writes one JSON object per line. Each line is independently parseable. A crashed scrape leaves a valid file up to the last complete line. Appending is open("file", "a"). Streaming a large file is a for line in f loop.

import json
import requests
from bs4 import BeautifulSoup
from pathlib import Path

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

output = Path("books.jsonl")

with output.open("w") as f:
    resp = session.get("https://books.toscrape.com/", timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        item = {
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        }
        f.write(json.dumps(item) + "\n")

# Reading back
with output.open() as f:
    records = [json.loads(line) for line in f]

print(f"Read {len(records)} records")
print(records[0])

Output:

Read 20 records
{'title': 'A Light in the Attic', 'price': '£51.77', 'rating': 'Three'}

For a paginated scrape, open the file in append mode ("a") and write after each page. If the scrape is interrupted on page 15 of 50, you still have the first 14 pages intact. You can resume from page 15 without re-scraping the ones you already have.

CSV with DictWriter

CSV is the right format when the output has a fixed, flat schema and you need to open it in a spreadsheet or feed it to a tool that expects CSV. For nested data or variable fields, JSONL is more flexible.

import csv
import requests
from bs4 import BeautifulSoup
from pathlib import Path

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

FIELDS = ["title", "price", "rating"]
output = Path("books.csv")

with output.open("w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=FIELDS)
    writer.writeheader()

    resp = session.get("https://books.toscrape.com/", timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        writer.writerow({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

Two things to get right here: newline="" in the open() call is required on Windows (without it, csv.writer produces double line endings); and DictWriter with explicit fieldnames means extra keys in your dict are ignored rather than causing a crash, which matters when scraped data is messier than expected.

DictWriter also handles quoting automatically — if a field contains a comma or a quote character, it wraps the field correctly. Manual string concatenation doesn't.

SQLite for deduplication and querying

SQLite is worth reaching for when you're doing repeated scrapes of the same site (price monitoring, availability checking) and need to avoid re-inserting data you already have. INSERT OR IGNORE skips any row that would violate a UNIQUE constraint, with no error raised.

import sqlite3
import requests
from bs4 import BeautifulSoup

HEADERS = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
}

session = requests.Session()
session.headers.update(HEADERS)

conn = sqlite3.connect("books.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS books (
        title  TEXT,
        price  TEXT,
        rating TEXT,
        UNIQUE(title)
    )
""")
conn.commit()

resp = session.get("https://books.toscrape.com/", timeout=15)
resp.encoding = "utf-8"
soup = BeautifulSoup(resp.text, "html.parser")

inserted = 0
skipped  = 0

for book in soup.find_all("article", class_="product_pod"):
    try:
        conn.execute(
            "INSERT INTO books (title, price, rating) VALUES (?, ?, ?)",
            (
                book.find("h3").find("a")["title"],
                book.find("p", class_="price_color").text.strip(),
                book.find("p", class_="star-rating")["class"][1],
            ),
        )
        inserted += 1
    except sqlite3.IntegrityError:
        skipped += 1  # already in the DB

conn.commit()
conn.close()

print(f"Inserted: {inserted}, skipped (duplicate): {skipped}")

The parameterised query (? placeholders) is not optional. String formatting SQL with scraped data is a SQL injection vulnerability even when you control the scraper — the scraped site controls the data.

For repeated scrapes where you want to track price history rather than deduplicate, swap the UNIQUE constraint and INSERT OR IGNORE for a timestamp column and a plain INSERT. Each run adds new rows without touching old ones.

Scrapy pipelines

In Scrapy, storage belongs in item pipelines. A pipeline is a class with a process_item method that receives each item as it comes off the spider. Pipelines can be chained — one validates, one writes to JSONL, one writes to SQLite — and each gets called in the order defined in ITEM_PIPELINES in your settings.

# pipelines.py
import json
import sqlite3
from itemadapter import ItemAdapter


class JsonlPipeline:
    def open_spider(self, spider):
        self.file = open("output.jsonl", "w")

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        line = json.dumps(ItemAdapter(item).asdict())
        self.file.write(line + "\n")
        return item  # must return item for the next pipeline to receive it


class SqlitePipeline:
    def open_spider(self, spider):
        self.conn = sqlite3.connect("output.db")
        self.conn.execute("""
            CREATE TABLE IF NOT EXISTS items (
                title  TEXT,
                price  TEXT,
                rating TEXT,
                UNIQUE(title)
            )
        """)
        self.conn.commit()

    def close_spider(self, spider):
        self.conn.close()

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        self.conn.execute(
            "INSERT OR IGNORE INTO items (title, price, rating) VALUES (?, ?, ?)",
            (adapter["title"], adapter["price"], adapter["rating"]),
        )
        self.conn.commit()
        return item

Enable in settings.py:

ITEM_PIPELINES = {
    "myproject.pipelines.JsonlPipeline":   300,
    "myproject.pipelines.SqlitePipeline":  400,
}

The integer values are priority — lower numbers run first. The return item at the end of each process_item is mandatory; if you forget it, the next pipeline in the chain receives None.

open_spider and close_spider are the correct place to open and close file handles and database connections. Opening them in __init__ means they persist for the lifetime of the pipeline object even when no spider is running.

Choosing a format

Format	Use when
JSONL	Default for most scrapes; variable fields; needs to survive interruption
CSV	Fixed flat schema; downstream tools expect CSV; spreadsheet output
SQLite	Repeated scrapes of the same targets; need to query across runs; deduplication
PostgreSQL / MySQL	Multiple scrapers writing concurrently; data volumes above ~10GB

The jump to a full database server is rarely needed for a single-scraper project. SQLite handles hundreds of millions of rows on modern hardware without issue, provided you're not writing from multiple processes simultaneously.

Tags: python scrapy webscraping tutorial

Scraping Paginated Sites Without Getting It Wrong

John Rooney — Sun, 31 May 2026 18:07:12 +0000

Pagination is where a lot of scrapers quietly go wrong — not with errors, but with missing data. A scraper that stops two pages early produces no exceptions. Neither does one that re-fetches the same page in a loop. The output just looks thin, and you may not notice until you check it against the actual record count.

There are three common pagination patterns. Identifying which one you're dealing with takes 30 seconds in DevTools; implementing it correctly takes another ten minutes. This post covers all three.

Pattern 1: Page number in the URL

The simplest pattern. The URL contains a page parameter — either as a query string (?page=2) or as part of the path (/catalogue/page-2.html). Increment it until you get a 404 or an empty result set.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://books.toscrape.com/catalogue/"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_books = []
page = 1

while True:
    url = f"https://books.toscrape.com/catalogue/page-{page}.html"
    resp = session.get(url, timeout=15)

    if resp.status_code == 404:
        break  # past the last page

    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")
    books = soup.find_all("article", class_="product_pod")

    if not books:
        break  # empty page — also done

    for book in books:
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    print(f"Page {page}: {len(books)} books")
    page += 1

print(f"\nTotal: {len(all_books)} books")

Two termination conditions, not one. Some sites return an empty 200 for out-of-range pages rather than a 404. Check both.

Pattern 2: Following the "next" link

A cleaner approach for HTML-paginated sites: let the page tell you where to go next, rather than constructing URLs yourself. Most paginated sites include a "Next" link in the HTML. Follow it until it disappears.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE = "https://books.toscrape.com/catalogue/"

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

url = "https://books.toscrape.com/catalogue/page-1.html"
all_books = []

while url:
    resp = session.get(url, timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

print(f"Scraped {len(all_books)} books")

The urljoin(BASE, next_btn["href"]) call is worth noting. The href in a "next" link is often relative (page-2.html, ../page-2.html). urljoin resolves it against the base URL correctly regardless of what form the relative path takes. Concatenating strings instead will break on unusual relative paths.

Pattern 3: API cursor / continuation token

JSON APIs often paginate differently. Instead of page numbers, they return a token or flag telling you whether more results exist, and sometimes a cursor to pass back on the next request.

The simplest version: a has_next boolean.

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_quotes = []
page = 1

while True:
    resp = session.get(
        "https://quotes.toscrape.com/api/quotes",
        params={"page": page},
        timeout=15,
    )
    resp.raise_for_status()
    data = resp.json()

    all_quotes.extend(data["quotes"])
    print(f"Page {page}: {len(data['quotes'])} quotes")

    if not data["has_next"]:
        break
    page += 1

print(f"\nTotal: {len(all_quotes)} quotes")

Some APIs use a cursor instead — the response includes a next_cursor or next_page_token field that you pass as a parameter on the subsequent request. The structure changes but the loop logic is the same: keep going until the cursor field is null or absent.

# Generic cursor pattern
params = {"limit": 100}

while True:
    resp = session.get("https://example.com/api/items", params=params, timeout=15)
    data = resp.json()

    items.extend(data["results"])

    cursor = data.get("next_cursor")
    if not cursor:
        break
    params["cursor"] = cursor

Rate limiting

Sending requests as fast as the network allows is not scraping — it's a load test. Most sites will rate-limit or block traffic that arrives faster than a human could generate it. A 1-2 second delay between pages is a reasonable starting point; adjust based on the site's response times and any explicit rate-limit headers it sends.

import time

while url:
    resp = session.get(url, timeout=15)
    # ... process page ...

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

    if url:
        time.sleep(1)  # only sleep if there's another request coming

Sleeping after the last page is unnecessary. Put the sleep before the next request or, as above, after confirming there is a next request.

For higher-volume work, time.sleep with a fixed value is blunt. A better approach uses a random delay within a range — time.sleep(random.uniform(0.5, 2.0)) — which avoids the metronomic request timing that fixed delays produce.

Duplicate URL detection

Some sites have inconsistent pagination — "next" links that eventually loop back, or page parameters that wrap around. A simple seen_urls set catches this before it turns into an infinite loop:

from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

BASE = "https://books.toscrape.com/catalogue/"
url = "https://books.toscrape.com/catalogue/page-1.html"
seen_urls = set()
all_books = []

while url:
    if url in seen_urls:
        print(f"Loop detected at {url} — stopping")
        break
    seen_urls.add(url)

    resp = session.get(url, timeout=15)
    resp.encoding = "utf-8"
    soup = BeautifulSoup(resp.text, "html.parser")

    for book in soup.find_all("article", class_="product_pod"):
        all_books.append({
            "title":  book.find("h3").find("a")["title"],
            "price":  book.find("p", class_="price_color").text.strip(),
            "rating": book.find("p", class_="star-rating")["class"][1],
        })

    next_btn = soup.select_one("li.next a")
    url = urljoin(BASE, next_btn["href"]) if next_btn else None

print(f"Scraped {len(all_books)} books across {len(seen_urls)} pages")

Using Scrapy's CrawlSpider

If you're building on Scrapy, CrawlSpider handles link following automatically via rules. This is the idiomatic Scrapy approach for sites where pagination follows a consistent pattern:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor


class BooksSpider(CrawlSpider):
    name = "books"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    rules = (
        # Follow "next page" links
        Rule(
            LinkExtractor(restrict_css="li.next a"),
            callback="parse_page",
            follow=True,
        ),
    )

    def parse_page(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title":  book.css("h3 a::attr(title)").get(),
                "price":  book.css("p.price_color::text").get(default="").strip(),
                "rating": book.css("p.star-rating::attr(class)").get(default="").split()[-1],
            }

CrawlSpider deduplicates URLs by default (Scrapy's built-in duplicate filter handles it), respects DOWNLOAD_DELAY in your settings, and handles retries. For a site with straightforward pagination, it removes most of the boilerplate above.

One thing to know: CrawlSpider calls the rules on every response, including the ones your callback generates. If a page both contains items and a "next" link, the rule fires correctly — but if you override parse() directly on a CrawlSpider, you'll break the rule processing. Use a separate callback method, as above.

Quick decision guide

Situation	Approach
URL has `/page/2` or `?page=2`	Increment, stop on 404 or empty
Page has a "Next" link in HTML	Follow href with `urljoin`, stop when absent
JSON API with `has_next` flag	Loop until flag is false
JSON API with cursor/token	Pass cursor back each request, stop when null
Building on Scrapy	`CrawlSpider` + `LinkExtractor`

Tags: python scrapy webscraping tutorial

How to Handle JavaScript-Rendered Pages Without a Full Browser

John Rooney — Sun, 31 May 2026 18:06:27 +0000

The HTML that requests downloads is what the server sends before any JavaScript runs. For a large and growing number of sites, that document is nearly empty — a shell with <script> tags that populate the content after the browser executes them. BeautifulSoup finds nothing because there is nothing to find in the source.

Two approaches handle this. The first is to skip the HTML entirely and call the underlying API the JavaScript is already talking to. The second is to use a real browser. The first option is faster, more reliable, and available more often than people expect.

Identifying a JS-rendered page

The test is straightforward: compare what you see in "View Source" against what you see in DevTools' Elements panel. View Source shows the raw server response — exactly what requests receives. The Elements panel shows the live DOM after JavaScript has run.

If your target data appears in the Elements panel but not in View Source, it's JS-rendered.

The other common indicator: requests returns a 200 with a short body. A page that renders 50 product cards in the browser but returns 2KB of HTML to a plain HTTP request is loading its content dynamically.

Option 1: Find the underlying API

When a page loads content via JavaScript, that JavaScript has to get the data from somewhere. Usually it makes an XHR or Fetch request to a JSON API. That same API is available to your scraper — with no browser required.

To find it, open DevTools, go to the Network tab, filter by "Fetch/XHR", then reload the page. Watch for requests that return JSON. Click on one, check the "Preview" tab — if it contains your data, you've found the endpoint.

Here's a concrete example. quotes.toscrape.com/js/ renders its quotes via JavaScript. The raw HTML response contains zero quote elements:

import requests
from bs4 import BeautifulSoup

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-GB,en;q=0.9",
})

resp = session.get("https://quotes.toscrape.com/js/", timeout=15)
soup = BeautifulSoup(resp.text, "html.parser")
print(len(soup.find_all("div", class_="quote")))  # 0

But the JavaScript is fetching from /api/quotes. That endpoint returns clean JSON and supports pagination via a page parameter:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
    "Accept": "application/json",
    "Accept-Language": "en-GB,en;q=0.9",
})

all_quotes = []
page = 1

while True:
    resp = session.get(
        "https://quotes.toscrape.com/api/quotes",
        params={"page": page},
        timeout=15,
    )
    resp.raise_for_status()
    data = resp.json()

    all_quotes.extend(data["quotes"])
    print(f"Page {page}: {len(data['quotes'])} quotes")

    if not data["has_next"]:
        break
    page += 1

print(f"\nTotal: {len(all_quotes)} quotes")

Output:

Page 1: 10 quotes
Page 2: 10 quotes
...
Page 10: 10 quotes

Total: 100 quotes

This is faster than any browser-based approach and produces cleaner data. Change the Accept header to application/json when hitting JSON endpoints — some APIs check it.

When inspecting Network requests, look for:

URLs containing /api/, /graphql, /v1/, /data/, or .json
Requests where the Preview tab shows an object or array
Query parameters like page, offset, cursor, limit — these tell you the pagination model upfront

Not every site exposes a clean API. Some assemble their data server-side, some use GraphQL, some obfuscate the endpoints. When the API route isn't viable, use a browser.

Option 2: Playwright

Playwright drives a real browser, so what it sees is identical to what a user sees. It's slower than requests and consumes more memory, but it's the correct tool when JavaScript execution is unavoidable.

Install:

pip install playwright
python -m playwright install chromium

Basic scrape of the same JS-rendered page:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    quotes = page.locator("div.quote").all()
    for q in quotes:
        text   = q.locator("span.text").inner_text()
        author = q.locator("small.author").inner_text()
        print(f"{text[:70]}... — {author}")

    browser.close()

wait_until="networkidle" tells Playwright to wait until there are no ongoing network requests for at least 500ms — enough time for most JS-driven content to load. For pages with long-running background requests (analytics pings, chat widgets), "domcontentloaded" or a specific element wait is more reliable:

# Wait for a specific element rather than network quiet
page.goto("https://quotes.toscrape.com/js/")
page.wait_for_selector("div.quote")

page.locator() is preferable to page.query_selector_all(). Locators are lazy — they don't execute until you call a method on them — and they retry automatically if the element isn't immediately present. This makes them more tolerant of pages that render content in stages.

Blocking unnecessary resources

By default, Playwright fetches everything: stylesheets, images, fonts, analytics scripts. For scraping, none of that matters. Blocking it cuts load time noticeably:

from playwright.sync_api import sync_playwright

def block_non_essential(route):
    if route.request.resource_type in ("image", "font", "stylesheet", "media"):
        route.abort()
    else:
        route.continue_()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.route("**/*", block_non_essential)

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    texts = page.locator("div.quote span.text").all_inner_texts()
    print(f"Found {len(texts)} quotes")

    browser.close()

The page.route() call intercepts every request. For anything that matches image, font, stylesheet, or media, it calls route.abort() instead of letting it through. The data-carrying requests — the HTML document and any XHR calls — continue normally.

Using Playwright to discover hidden APIs

Even when you intend to use Playwright for the actual scraping, it can show you API calls you didn't know existed. Register a response handler before navigation:

from playwright.sync_api import sync_playwright

api_calls = []

def capture_api(response):
    content_type = response.headers.get("content-type", "")
    if "json" in content_type:
        api_calls.append(response.url)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.on("response", capture_api)

    page.goto("https://quotes.toscrape.com/js/", wait_until="networkidle")

    print("JSON responses observed:")
    for url in api_calls:
        print(f"  {url}")

    browser.close()

If any JSON responses appear, check whether they contain your target data. If they do, you can switch from Playwright to a plain requests call against that URL — no browser needed for subsequent runs.

Choosing between the two approaches

API scraping is worth the investigation time. A requests loop that pages through a JSON API runs 10-50x faster than Playwright, uses a fraction of the memory, and produces structured data without parsing HTML. If the endpoint exists and isn't authenticated in a way you can't replicate, use it.

Playwright is the right answer when: the page builds its content from multiple sources with no single API, authentication involves cookies set by JavaScript challenges, or content only appears after user interactions like scroll events or button clicks.

The two approaches also compose. Use Playwright to log in and capture session cookies, then hand those cookies to a requests.Session for the actual data collection:

from playwright.sync_api import sync_playwright
import requests

# Step 1: get session cookies via browser
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/login")
    page.fill("#username", "you@example.com")
    page.fill("#password", "your-password")
    page.click("button[type=submit]")
    page.wait_for_url("**/dashboard")
    cookies = page.context.cookies()
    browser.close()

# Step 2: transfer cookies to requests session
session = requests.Session()
for cookie in cookies:
    session.cookies.set(cookie["name"], cookie["value"], domain=cookie["domain"])

# Step 3: scrape with requests
resp = session.get("https://example.com/api/data", timeout=15)
print(resp.json())

Tags: python webscraping playwright tutorial

DEV Community: Extract by Zyte

Your scraping framework already exists: it's Scrapy

The framework appears one fix at a time

Start with the boring infrastructure

scrapy-poet separates navigation from interpretation

Spidermon defines what a successful crawl means

scrapy-playwright makes the browser an acquisition choice

Write a wreq download handler when the network stack is the variable

Scrapy is a runtime, not the whole platform

A practical migration path

Frequently asked questions

Is Scrapy too old for modern Python and async code?

Can one spider mix normal HTTP, Playwright, and wreq?

When is a plain script still the better choice?

Does Scrapy solve scaling across machines?

Build your scraper, not another crawling framework

So you want to get data from the web

What web scraping becomes once someone depends on it

A small Python scraper should leave evidence behind

JavaScript rendering should be an explicit escalation

Retry transient failures without turning a scrape into a traffic spike

TLS fingerprinting starts before the server sees your headers

A production scraper is a replayable data pipeline

Raw snapshots make parser changes cheap

Idempotent writes make retries safe

Quality gates should fail the run, not merely decorate a dashboard

The shortcuts that fail when the workload grows

How to choose the smallest architecture that will survive

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of requests?

When should I use Playwright for web scraping?

How do I know whether a scraper is returning bad data?

How do you scale web scraping without overwhelming a site?

Start with a replay test

How to use Camoufox with Scrapy (New EASY Method)

Why the stock browsers get caught

The middle ground that was missing

How browser providers actually work

Why this fits the way Scrapy is built

What it changes, and what it doesn't

GPT-5.6, Fable 5, and GLM-5.2 went for a crawl and got hit by The Rate Limit - My new fav foundational model

The setup, kept light

The one table that answers the question

What every dollar tier gets right: extraction

What separates the cheap tier: currency

What the flagship premium buys: nothing I could measure

What the frontier model does: it refuses the job

The one task I judged by eye: a landing-page build-off

The stuff that does not show up in a price table

The five personas

A verdict you can act on

The menu lies, and here is how

Where this falls short

The real takeaway: know your models, then buy the least you need

5 lessons from 5 interviews on AI agents and web scraping

1. Simpler code is usually better scraping code

2. Developers are moving from coding-first to design-first

3. AI multiplies judgment. It does not replace it.

4. Data quality begins before AI enters the workflow

5. The best agent skill does not over-explain

The pattern across all five

Links

Why everyone is talking about loop-engineering and how is it changing agentic ai workflows? Claude Code and Web Scraping examples

What loop engineering actually is

Web scraping already has the hard part

The smallest self-healing spider I could build

Loops I want to see the community build

Self-healing spider fleets

Spider factories with a definition of done

Per-site memory that compounds

Cost-aware escalation loops

Plausibility graders that catch the lies

Schema drift scouts

Coverage loops for discovery

How I plan to use it

Argue with me

My agentic coding setup: Claude Code, multi-agent orchestration, and more

My unfair advantage

The workspace that opens itself

Plan first, always

When should I use a web scraping framework instead of `requests`?