So you want to get data from the web

Getting data from one web page can take 20 lines of Python. Keeping a trustworthy dataset alive while the source changes, rate limits you, returns partial responses, and occasionally serves a consent page with a 200 OK is a different engineering problem, because the selector is rarely as difficult as knowing when the result is wrong.

What web scraping becomes once someone depends on it

Web scraping is automated retrieval and extraction of data from web resources. At the small end, that means an HTTP request followed by an HTML parser. At the other end, the same basic operation sits inside a distributed acquisition pipeline with scheduling, traffic controls, schema validation, raw snapshots, observability, and a recovery path. Both are web scraping, but they should not have the same architecture.

The easiest way to choose that architecture is to classify the job by its failure cost:

Workload	A reasonable starting point	The first problems you will meet
A few known pages, needed once	One process and local output	Missing fields, pagination, duplicate URLs, odd encodings
A scheduled crawl of one site	A repeatable job with persistent state	Timeouts, rate limits, reruns, template changes
A JavaScript application	HTTP plus targeted browser automation where required	Background requests, timing, cookies, session expiry
Many sources feeding another product	A queue-based data pipeline	Source drift, partial runs, backfills, data contracts, cost control

This is the useful complexity ladder. A one-off question does not need a queue, six workers, and an observability stack, while a revenue-critical feed should probably graduate from a laptop before print() becomes its monitoring strategy.

A small Python scraper should leave evidence behind

The first version should already separate fetching from parsing because that boundary lets you test selectors without hitting the site again and distinguish a network failure from an extraction failure. It also means that when the markup changes, you can fix the parser against a saved response instead of repeatedly fetching the same page while you debug it.

Here is a complete example for a static product page. It uses Parsel, expects semantic attributes such as data-product-name and data-price, saves the raw response to a path chosen by the caller, and returns a typed record. Parsel is also the selector layer underneath Scrapy, so the CSS and XPath expressions carry across when the crawler grows into a Scrapy project.

Install the two dependencies first:

python -m pip install requests parsel

from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal, InvalidOperation
from pathlib import Path
from urllib.parse import urldefrag

import requests
from parsel import Selector


@dataclass(frozen=True)
class Product:
    source_url: str
    name: str
    price: Decimal
    fetched_at: str


def parse_product(
    html: str,
    *,
    source_url: str,
    fetched_at: str,
) -> Product:
    selector = Selector(text=html)
    root = selector.css("[data-product]")

    if not root:
        raise ValueError("missing product container")

    name = root.css("[data-product-name]::text").get()
    raw_price = root.css("[data-price]::attr(data-price)").get()

    if name is None or raw_price is None:
        raise ValueError("missing required product field")

    name = name.strip()

    try:
        price = Decimal(raw_price)
    except InvalidOperation as exc:
        raise ValueError("invalid product price") from exc

    if not name or not price.is_finite() or price < 0:
        raise ValueError("product failed validation")

    return Product(
        source_url=source_url,
        name=name,
        price=price,
        fetched_at=fetched_at,
    )


def fetch_product(url: str, snapshot_path: Path) -> Product:
    response = requests.get(
        url,
        headers={"User-Agent": "ExampleDataCollector/1.0 [e-mail]"},
        timeout=(5, 20),
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")
    if "text/html" not in content_type:
        raise ValueError(f"expected HTML, received {content_type!r}")

    fetched_at = datetime.now(timezone.utc).isoformat()
    source_url, _fragment = urldefrag(response.url)

    snapshot_path.parent.mkdir(parents=True, exist_ok=True)
    snapshot_path.write_bytes(response.content)

    return parse_product(
        response.text,
        source_url=source_url,
        fetched_at=fetched_at,
    )

The timeout has separate connection and read limits, raise_for_status() prevents a 404 page from flowing into the parser, and the content-type check catches JSON, images, and some challenge pages before Parsel sees them. The raw body is stored before parsing, so an extraction error does not destroy the evidence needed to reproduce it.

There is no content hash here because this example does not need one. The caller can name snapshots by run ID, URL key, or whatever fits the job. Hashes become useful later, when the system needs to detect unchanged responses, deduplicate stored bodies, or connect a parsed record to the exact bytes that produced it. Introducing that machinery here would explain the production architecture before we have earned it.

The parser rejects invalid records instead of returning a half-populated dictionary. Returning None for a missing price feels convenient until it reaches an aggregate and quietly changes the answer.

The test does not need the network:

from decimal import Decimal


def test_parse_product() -> None:
    html = """
    <article data-product>
      <h1 data-product-name>Mechanical Keyboard</h1>
      <span data-price="89.95">£89.95</span>
    </article>
    """

    product = parse_product(
        html,
        source_url="https://example.com/products/keyboard",
        fetched_at="2026-07-26T10:00:00+00:00",
    )

    assert product.name == "Mechanical Keyboard"
    assert product.price == Decimal("89.95")

This is the first architectural boundary worth keeping: fetchers produce source material and metadata, parsers turn that material into validated records, and storage comes after validation.

JavaScript rendering should be an explicit escalation

Some pages return the data in the initial HTML, while others return an application shell and fetch the useful payload after JavaScript runs. Check the response body and browser network panel before reaching for a headless browser because structured data embedded in the document or a background JSON response is usually easier to operate than a full browser session.

Playwright is excellent when you genuinely need rendering or interaction. It also turns deterministic request code into a small distributed-systems problem living inside Chromium, complete with page lifecycle events, async UI updates, browser crashes, context cleanup, and much higher CPU and memory use.

When a browser is justified, wait for a condition that means the data is ready. networkidle is often a weak signal because analytics, polling, and streaming requests can keep the network busy indefinitely.

Install Playwright and its Chromium build before running the example:

python -m pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright


def fetch_rendered_html(url: str) -> str:
    with sync_playwright() as playwright:
        browser = playwright.chromium.launch(headless=True)
        page = browser.new_page()

        try:
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)
            page.locator("[data-product-loaded='true']").wait_for(
                state="attached",
                timeout=10_000,
            )
            return page.content()
        finally:
            browser.close()

The selector is intentionally site-specific. Replace it with a marker that appears only when the content you need is present, or wait for the relevant response. Playwright's auto-waiting documentation explains what its actions wait for automatically, while its evaluation guide covers the boundary between Python and code running in the page.

Standard Playwright also has a recognisable automation surface. If a site reacts differently to an automated browser than to the corresponding normal browser, stealth-oriented projects are another rung on the complexity ladder. Camoufox is a Firefox fork with Playwright compatibility and browser-level fingerprint changes, while Patchright patches the Chromium side of Playwright and presents itself as a drop-in replacement. Both are moving projects with their own compatibility gaps, release cadence, and operational cost, so treat them as specialised browser runtimes rather than magic flags. We will leave the comparison there because each deserves a separate article.

Retry transient failures without turning a scrape into a traffic spike

Once a job runs on a schedule, failures become normal input: DNS lookups fail, connections reset, upstream services return 503, and rate limits change during the day. Retries help with those transient failures, but unbounded or immediate retries make incidents worse.

requests can use urllib3's retry policy through an adapter:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry


def build_session() -> requests.Session:
    retry = Retry(
        total=4,
        connect=4,
        read=4,
        status=4,
        backoff_factor=0.5,
        status_forcelist={429, 500, 502, 503, 504},
        allowed_methods={"GET", "HEAD"},
        respect_retry_after_header=True,
        raise_on_status=False,
    )

    adapter = HTTPAdapter(max_retries=retry, pool_connections=20, pool_maxsize=20)
    session = requests.Session()
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

With a backoff factor of 0.5, urllib3 applies increasing delays between eligible retries. The final response still needs raise_for_status(). A 403 or 404 is not in the retry list because repeating a permanent failure four times adds load without changing the answer.

HTTP 429 Too Many Requests may include a Retry-After header. RFC 6585 leaves the server free to count requests by credential, cookie, client, resource, or another policy, which means changing one network attribute does not necessarily reset the limit.

At more than one worker, enforce concurrency and delay per host across the whole worker pool. A local sleep(1) inside each worker gives you ten requests per second when you scale to ten workers. Put the host budget in shared state or route each host through its own queue.

TLS fingerprinting starts before the server sees your headers

Changing the User-Agent header does not make an HTTP client behave like a browser. Before the server receives the HTTP request, the client has already sent a TLS ClientHello containing an ordered set of cipher suites, extensions, supported groups, signature algorithms, and ALPN values. Those choices, along with their ordering, can be summarised as JA3 or JA4 fingerprints. HTTP/2 adds another layer through SETTINGS values, connection behaviour, and pseudo-header ordering.

This is why a request can claim to be Firefox in its headers while looking nothing like Firefox on the wire. A conventional Python client is designed to make correct HTTP requests, not to reproduce a browser's network stack. For many sites that distinction does not matter. On sites that classify traffic using several signals, it can explain why copying browser headers changes nothing.

wreq exposes Python bindings over the Rust client, so there is no reason to leave the Python stack just to control the network fingerprint. It uses BoringSSL and supports TLS, JA3/JA4, HTTP/2, and header-order profiles. The current package ships more than 100 browser emulation profiles and offers both async and blocking clients.

Install wreq alongside Parsel:

python -m pip install wreq parsel

import asyncio

from parsel import Selector
from wreq import Client, Emulation


async def fetch_title(url: str) -> str:
    client = Client(emulation=Emulation.Chrome149)
    response = await client.get(url)

    if response.status >= 400:
        raise RuntimeError(f"request failed with status {response.status}")

    selector = Selector(text=await response.text())
    title = selector.css("title::text").get()

    if title is None:
        raise ValueError("response did not contain a title")

    return title.strip()


if __name__ == "__main__":
    print(asyncio.run(fetch_title("https://example.com")))

The profile needs to be internally consistent. If the TLS handshake resembles one browser version while the headers, operating system hints, cookies, or JavaScript-visible properties describe another, the inconsistency becomes a signal of its own. Keep a client alive across requests so connection pooling and cookies behave consistently, pin the package and profile versions you have tested, and expect fingerprints to age as browsers and detection systems change.

wreq solves the network-stack part of the problem rather than JavaScript rendering. If the page needs a browser, use one; if the HTML is already in the response and the mismatch is at TLS or HTTP/2, launching Chromium adds a large amount of machinery to solve a lower-level issue.

A production scraper is a replayable data pipeline

Once the output feeds search, analytics, pricing, or a customer-facing feature, the system needs more than fetch workers. A practical design looks like this:

flowchart LR
    S["Scheduler: cron or Airflow"] --> Q["Host-aware URL queues: SQS or Redis"]
    Q --> F["HTTP fetchers: requests or wreq"]
    Q --> B["Browser fetchers: Playwright, Camoufox, or Patchright"]
    F --> RAW["Raw snapshots and metadata: S3"]
    B --> RAW
    F --> RETRY["Retry and review queue"]
    B --> RETRY
    RAW --> P["Versioned parser workers"]
    P --> V["Schema and run-quality gates"]
    V --> DB["Idempotent records: PostgreSQL"]
    DB --> C["Search, analytics, or product consumers"]
    F -. metrics .-> O["Metrics, logs, traces, and alerts"]
    B -. metrics .-> O
    P -. metrics .-> O
    V -. metrics .-> O

The names are examples, not requirements. A single VM, SQLite, and a filesystem can implement the same boundaries for a smaller workload. The boundaries matter because they let you retry and replace one stage without repeating every other stage.

Raw snapshots make parser changes cheap

Store response bytes with the source URL, final URL after redirects, fetch timestamp, status, content type, relevant response headers, and a content hash. Add the fetcher and parser version to the eventual record. At this point the hash has a clear job: it gives the raw body a stable identity, makes unchanged responses cheap to detect, and links each parsed record back to the exact bytes that produced it.

When a selector breaks, rerun the new parser against stored responses. This is faster, cheaper, and kinder to the source than fetching every URL again. It also lets you compare old and new parser output before a backfill reaches production.

Retention is a policy decision, especially if pages contain personal or sensitive data. An immutable archive is useful engineering infrastructure. It is not a reason to keep everything forever.

Idempotent writes make retries safe

A worker can fail after writing a record but before acknowledging its queue message. The message will be delivered again. If the second write creates another row, your retry policy has become a duplication policy.

Use a stable key, such as (source, external_id), and upsert. This PostgreSQL statement uses SQLAlchemy-style named parameters:

INSERT INTO products (
    source,
    external_id,
    name,
    price,
    fetched_at,
    content_hash,
    parser_version
)
VALUES (
    :source,
    :external_id,
    :name,
    :price,
    :fetched_at,
    :content_hash,
    :parser_version
)
ON CONFLICT (source, external_id) DO UPDATE SET
    name = EXCLUDED.name,
    price = EXCLUDED.price,
    fetched_at = EXCLUDED.fetched_at,
    content_hash = EXCLUDED.content_hash,
    parser_version = EXCLUDED.parser_version;

This gives you the latest record. If you also need history, write changes to an append-only version table or event log. Do not overload one table to be both current state and an audit trail.

Quality gates should fail the run, not merely decorate a dashboard

Transport metrics tell you whether requests completed, while data metrics tell you whether the output is believable.

Track at least:

• attempted, succeeded, retried, and finally failed URLs by host
• status-code distribution and request latency
• records produced, required-field completeness, and uniqueness
• new, changed, and missing records relative to the previous successful run
• parser errors grouped by source and parser version

Then define stop conditions. If yesterday's crawl produced 48,000 products and today's produces 730, publishing the result should require an explanation. The threshold depends on the source, but the decision must exist in code rather than in someone's memory.

Here is a small quality gate with deliberately configurable limits:

from dataclasses import dataclass


@dataclass(frozen=True)
class RunStats:
    attempted: int
    succeeded: int
    records: int
    records_with_required_fields: int


def validate_run(
    current: RunStats,
    *,
    previous_records: int,
    min_success_rate: float = 0.95,
    min_completeness: float = 0.99,
    min_volume_ratio: float = 0.70,
) -> None:
    if current.attempted <= 0 or current.records <= 0:
        raise ValueError("run produced no usable work")

    success_rate = current.succeeded / current.attempted
    completeness = current.records_with_required_fields / current.records
    volume_ratio = current.records / max(previous_records, 1)

    failures = []
    if success_rate < min_success_rate:
        failures.append(f"success rate {success_rate:.1%}")
    if completeness < min_completeness:
        failures.append(f"completeness {completeness:.1%}")
    if volume_ratio < min_volume_ratio:
        failures.append(f"volume ratio {volume_ratio:.1%}")

    if failures:
        raise ValueError("run failed quality gates: " + ", ".join(failures))

Those defaults are examples, not universal scraping constants. A news index and a product catalogue have different expected churn. Start by observing normal variation, then choose thresholds that catch abnormal runs without paging someone every morning.

The shortcuts that fail when the workload grows

Several designs work just long enough to become expensive:

Putting fetch and parse logic in one function. You cannot test a parser without network traffic, and every parser fix requires another crawl.

Using a browser for every URL. It simplifies the first implementation when some pages need JavaScript, then spends far more CPU and memory on pages that do not. Keep HTTP and browser fetchers behind the same response interface, and route only the pages that need rendering.

Retrying every error. A timeout may recover. A malformed selector, an authentication failure, or a deleted page will not. Classify failures and send permanent or suspicious ones to review.

Scaling with global concurrency. One large domain and one tiny domain do not have the same capacity or rules. Global concurrency lets the busiest host consume the whole budget and makes request pressure unpredictable.

Treating row count as the only quality metric. A parser can return the expected number of rows with the wrong field in every row. Validate types, ranges, uniqueness, and changes in field-level completeness.

The common thread is lost evidence. Once the fetch, parser, and output are entangled, you cannot tell which part failed without repeating the entire job.

How to choose the smallest architecture that will survive

Use one script when all of these are true: the page set is small, the source is stable, the work is easy to rerun, and a bad result has limited consequences. Even then, save the raw input and validate required fields.

Add persistent state and scheduling when runs repeat. Add browser automation only for pages that require it. Add queues when one process no longer meets the throughput or isolation you need. Add separate parser workers when you want to replay stored responses without fetching again.

The forcing function is not URL count by itself. Ten authenticated, stateful workflows can be harder to operate than a million static pages from a stable bulk source.

Before increasing concurrency, write down:

1. What is the per-host request budget?
2. Which failures are safe to retry?
3. Which quality signal stops publication?
4. What raw evidence and version metadata will you retain?
5. How will you replay or backfill without duplicating records?

If those answers are vague, more workers will mostly help you fail faster.

Frequently asked questions about scalable web scraping

When should I use a web scraping framework instead of requests?

Use requests or httpx when the URL set is small and you can manage scheduling, retries, deduplication, and persistence directly. A framework such as Scrapy earns its place when you need concurrent crawling, downloader middleware, per-domain controls, request deduplication, item pipelines, and a consistent extension model. A framework removes plumbing, not the need to understand the source.

When should I use Playwright for web scraping?

Use Playwright when data only appears after browser-side execution or requires an interaction that a plain HTTP client cannot reproduce. Do not use it by default for static pages. Inspect the initial HTML and background requests first, then route only the pages that require rendering through a browser pool.

How do I know whether a scraper is returning bad data?

Validate each record against a schema, then compare run-level metrics with the previous successful run. Required-field completeness, type and range checks, uniqueness, record volume, and the number of changed or missing entities catch different classes of failure. Keep raw snapshots so you can reproduce the parser's decision.

How do you scale web scraping without overwhelming a site?

Set concurrency and request rates per host across all workers, respect 429 and Retry-After, cache or fetch incrementally where appropriate, and use bounded retries for transient failures. Scale the scheduler and parser independently from the fetch rate. More parsing capacity does not require more traffic to the source when raw responses are replayable.

Start with a replay test

Take one representative page, save the raw response, and make the parser pass against that fixture. Then change or remove one required field and confirm the test fails for the reason you expect.

That small exercise tells you whether you are building a script that happens to return data today, or a system you can still explain after the source changes tomorrow.