DEV Community: PromptCloud

Competitor Price Tracking: Building Offers, Sellers and Price History From Retailers' Own Pages

PromptCloud — Fri, 23 Oct 2026 10:29:28 +0000

Competitor price tracking is the repeated collection of competitors' publicly displayed prices for the same products, stored as a time series so you can see who is cheaper today, when each price changed and how deep and how long each promotion ran. A single price check tells you where you stand. A tracked history tells you how your market behaves.

This post builds the core of a tracker: what to track, how to collect it responsibly, a data model that keeps history intact, change detection that turns observations into events, and queries that answer the questions a pricing team actually asks. All of the code below was run against sample data.

What the dataset needs

Every competitor price tracker, however large, comes down to three things:

Offers: one seller's listing for one product, identified by GTIN.
Sellers: the retailers making those offers.
Observations: repeated readings of each offer's price, original price, delivery charge and availability over time.

Get these right and everything else (dashboards, alerts, repricing rules) sits on top.

Step 1: decide what to track

Scope is where most trackers go wrong, usually by trying to track everything. Settle three lists before writing any code:

Products: start with the lines that drive revenue or traffic, with a GTIN for each. GTINs let you match the same product across retailers that name it differently.
Competitors: the retailers your customers genuinely compare you with, not every site that stocks the product.
Cadence: daily suits most categories; fast-moving ones may need several readings a day. Match it to how quickly you can act.

Step 2: collect responsibly from retailers' own pages

Each retailer's product page is the original source of its price. Collect from public product pages only, identify your crawler honestly, keep request rates low, respect each site's terms, and check robots.txt before every fetch:

from urllib import robotparser
from urllib.parse import urlsplit

_parsers: dict[str, robotparser.RobotFileParser] = {}
USER_AGENT = "acme-price-monitor/1.0 (+https://example.com/bot)"

def allowed(url: str) -> bool:
    """Check a retailer's robots.txt before fetching one of its product pages."""
    parts = urlsplit(url)
    root = f"{parts.scheme}://{parts.netloc}"
    rp = _parsers.get(root)
    if rp is None:
        rp = robotparser.RobotFileParser(root + "/robots.txt")
        rp.read()
        _parsers[root] = rp
    return rp.can_fetch(USER_AGENT, url)

f a path is disallowed, skip it and log it. A tracker that depends on ignoring a site owner's instructions is not one you want under pricing decisions. For a fuller checklist of polite crawling habits, see these web scraping best practices.

Step 3: a data model that keeps history

Three tables cover offers, sellers and observations. The examples use SQLite so you can run them locally; the same shape works in any SQL warehouse.

CREATE TABLE sellers (
  seller_id   INTEGER PRIMARY KEY,
  name        TEXT NOT NULL,          -- storefront name as displayed
  domain      TEXT NOT NULL UNIQUE
);

CREATE TABLE offers (
  offer_id    INTEGER PRIMARY KEY,
  gtin        TEXT NOT NULL,          -- the join key across sellers
  seller_id   INTEGER NOT NULL REFERENCES sellers(seller_id),
  url         TEXT NOT NULL UNIQUE,
  first_seen  TEXT NOT NULL
);

CREATE TABLE price_observations (
  offer_id     INTEGER NOT NULL REFERENCES offers(offer_id),
  observed_at  TEXT NOT NULL,         -- UTC, 'YYYY-MM-DD HH:MM:SS'
  price        NUMERIC NOT NULL,
  list_price   NUMERIC,               -- original price, if a reduction is shown
  shipping     NUMERIC,
  currency     TEXT NOT NULL,
  availability TEXT NOT NULL,
  PRIMARY KEY (offer_id, observed_at)
);

Two rules matter. price_observations is append-only: never update a price in place, or you destroy the history that makes the data valuable. And GTIN is the join key: titles vary between retailers, GTINs do not.

Step 4: turn observations into events

Raw observations are hard to act on. A pricing team wants to know what changed: a competitor dropped a price, started a promotion or ran out of stock. Compare each new observation with the previous one for the same offer:

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class Observation:
    offer_id: int
    price: Decimal
    list_price: Decimal | None
    availability: str          # in_stock, out_of_stock, ...

def detect_changes(prev: Observation | None, curr: Observation,
                   min_move: Decimal = Decimal("0.01")) -> list[str]:
    """Compare two observations of the same offer and name what changed."""
    if prev is None:
        return ["new_offer"]
    events = []
    move = (curr.price - prev.price) / prev.price
    if move <= -min_move:
        events.append(f"price_drop:{move:.1%}")
    elif move >= min_move:
        events.append(f"price_rise:{move:.1%}")
    if prev.list_price is None and curr.list_price is not None:
        events.append("promotion_started")
    elif prev.list_price is not None and curr.list_price is None:
        events.append("promotion_ended")
    if prev.availability != "in_stock" and curr.availability == "in_stock":
        events.append("back_in_stock")
    elif prev.availability == "in_stock" and curr.availability != "in_stock":
        events.append("went_out_of_stock")
    return events

A drop from 219.00 to 199.00 with a struck-through 229.00 returns ['price_drop:-9.1%', 'promotion_started']. A move of 50p on a 219.00 item returns nothing, because min_move filters out noise. Tune it per category, then route events to alerts, and keep the observations themselves for analysis.

Step 5: queries that answer real questions

A view for the latest observation per offer, including the delivered price shoppers actually pay:

CREATE VIEW latest_prices AS
SELECT o.gtin, s.name AS seller, p.price, p.shipping,
       p.price + COALESCE(p.shipping, 0) AS delivered_price,
       p.currency, p.availability, p.observed_at
FROM price_observations p
JOIN offers o  ON o.offer_id = p.offer_id
JOIN sellers s ON s.seller_id = o.seller_id
WHERE p.observed_at = (SELECT MAX(observed_at)
                       FROM price_observations
                       WHERE offer_id = p.offer_id);

The market spread for each product, counting in-stock offers only:

SELECT gtin,
       COUNT(*)                        AS sellers,
       MIN(delivered_price)            AS cheapest,
       MAX(delivered_price)            AS dearest,
       ROUND(AVG(delivered_price), 2)  AS average
FROM latest_prices
WHERE availability = 'in_stock'
GROUP BY gtin;

The lowest price per offer over the previous 30 days:

SELECT o.gtin, s.name AS seller, MIN(p.price) AS low_30d
FROM price_observations p
JOIN offers o  ON o.offer_id = p.offer_id
JOIN sellers s ON s.seller_id = o.seller_id
WHERE p.observed_at >= datetime('now', '-30 days')
GROUP BY o.offer_id;

That last one is more useful than it looks. A "40% off" label means little until you know what the product sold for over the previous month, and in the EU, retailers announcing a price reduction must show the lowest price applied in at least the 30 days before it. A 30-day low lets you read competitors' promotions accurately and check your own.

Practical notes

Compare delivered prices. A cheaper shelf price plus a delivery charge is often not cheaper.
Filter on availability. An out-of-stock offer at a low price is not competition today.
Store currency on every row. Never infer it from a domain.
Keep raw page captures for a sample of observations, so disputed figures can be checked.
Watch the run, not just the rows. A sudden fall in observations from one retailer usually means its site changed, not that it stopped selling.

Wrapping up

Competitor price tracking is a time-series problem as much as a collection problem. Scope it to the products and competitors that matter, collect from retailers' own public pages within their rules, store every observation in an append-only, GTIN-keyed model, and turn changes into events your pricing team can act on. Build that core well and the dashboards and repricing rules on top become the easy part.

FAQ

What is competitor price tracking?

Competitor price tracking is the repeated collection of competitors' publicly displayed prices for the same products, stored over time so a business can compare price positions, detect price changes and promotions, and understand how its market behaves.

How often should you track competitor prices?

Daily collection suits most retail categories. Fast-moving categories may need several readings a day, while slower categories and strategic reviews can work with weekly data. Match the cadence to how often prices change and how quickly you can act.

How should competitor price history be stored?

In an append-only table with one row per offer per observation, holding price, any original price, delivery charge, currency, availability and a UTC timestamp, linked to products by GTIN so the same item can be compared across retailers.

Scraping E-commerce Product Data: Variants, Prices and Stock Without Losing Accuracy

PromptCloud — Thu, 22 Oct 2026 06:33:36 +0000

E-commerce data scraping is the collection of public product information from online stores (titles, identifiers, variants, prices, availability and pack sizes) into structured records you can compare across sites and over time. Fetching the pages is rarely the hard part. Accuracy is lost in the modelling: a price attached to the wrong variant, a sale price read as the list price, or "Only 2 left" stored as free text.

These are the five fields that break most often:

Variant price: the page shows one price, but each size or colour has its own.
Sale versus list price: two numbers on the page, and the parser picks the wrong one.
Currency and number format: 1.299,00 and 1,299.00 are the same amount.
Stock status: a dozen phrasings for four or five real states.
Pack size: a 6 x 330ml multipack compared with a single can.

This guide works through each one, schema first, with Python you can adapt. Every snippet below was run against sample data.

1. Decide the grain: one record per variant

The most important decision comes before any parsing. If a T-shirt comes in four sizes and three colours, are you storing one product or twelve?

For price and stock work, store one record per variant (SKU), linked to its parent. Prices and availability live at variant level, so a parent-level record will eventually average away the thing you care about.

{
  "parent_id": "TS-100",
  "sku": "TS-100-NAVY-L",
  "title": "Organic Cotton T-Shirt",
  "varies_by": ["colour", "size"],
  "attributes": { "colour": "Navy", "size": "L" },
  "gtin": "05012345678900",
  "price": "19.99",
  "list_price": "24.99",
  "currency": "GBP",
  "availability": "in_stock",
  "unit_price": null,
  "source_url": "https://example-shop.co.uk/p/ts-100?colour=navy&size=l",
  "collected_at": "2026-10-22T06:00:00Z",
  "parser_version": "3.2.0"
}

Three fields here are not product data at all: source_url, collected_at and parser_version. Keep them on every record. When a number looks wrong three weeks later, they tell you where it came from, when, and which code produced it.

2. Read the most structured source first

Most product pages expose the same data in up to three places. In order of preference:

Embedded structured data, usually schema.org Product, ProductGroup and Offer in JSON-LD.
Embedded application state, the JSON a JavaScript front end uses to render the page.
The visible page, parsed from the HTML.

Schema.org's ProductGroup type is built for variants: hasVariant lists each variant, variesBy names the properties that distinguish them, and productGroupID identifies the parent. Google added support for this markup in its product structured data in early 2024, so more stores now publish it.

import json
from bs4 import BeautifulSoup

def iter_nodes(data):
    """Yield every dict in a JSON-LD blob, including @graph members."""
    if isinstance(data, list):
        for item in data:
            yield from iter_nodes(item)
    elif isinstance(data, dict):
        yield data
        for item in data.get("@graph", []):
            yield from iter_nodes(item)

def extract_products(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            blob = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue  # malformed JSON-LD is common; fall back to the DOM
        for node in iter_nodes(blob):
            kind = node.get("@type")
            if kind == "ProductGroup":
                for variant in node.get("hasVariant", []):
                    variant.setdefault("_parent_id", node.get("productGroupID"))
                    products.append(variant)
            elif kind == "Product":
                products.append(node)
    return products

One caution: structured data is not always what the shopper sees. Some stores only mark up the default variant, and some cache it longer than the visible price. Spot-check a sample of records against the rendered page for each source, and treat a mismatch as a parser bug until proven otherwise.

3. Prices: two numbers, one currency, no floats

When a page shows a struck-through price and a lower one, store both: price (what the shopper pays now) and list_price (the original). In schema.org markup, the original is usually an Offer price specification with priceType set to StrikethroughPrice, the convention Google documents for sale pricing.

Take the currency from priceCurrency or the store's locale, never from the symbol: "$" is used by many currencies. Store amounts as Decimal, not floats, and parse with the source's decimal separator:

from decimal import Decimal, InvalidOperation
import re

def parse_price(raw: str, decimal_sep: str = ".") -> Decimal | None:
    """Parse '1.299,00' or '1,299.00' into Decimal('1299.00')."""
    if raw is None:
        return None
    digits = re.sub(r"[^\d,.]", "", str(raw))
    thousands_sep = "," if decimal_sep == "." else "."
    digits = digits.replace(thousands_sep, "").replace(decimal_sep, ".")
    try:
        return Decimal(digits)
    except InvalidOperation:
        return None

Set decimal_sep per source in configuration rather than guessing from the string. 1.299 is ambiguous on its own.

4. Stock: map many phrasings to a few states

Schema.org's ItemAvailability has twelve values, and visible pages add their own wording on top. Downstream users usually need five or six states. Map explicitly, keep the raw value alongside, and treat anything unmapped as unknown so it surfaces rather than hides:

AVAILABILITY = {
    "InStock": "in_stock",
    "LimitedAvailability": "in_stock",
    "OnlineOnly": "in_stock",
    "InStoreOnly": "store_only",
    "OutOfStock": "out_of_stock",
    "SoldOut": "out_of_stock",
    "BackOrder": "backorder",
    "PreOrder": "preorder",
    "PreSale": "preorder",
    "MadeToOrder": "made_to_order",
    "Discontinued": "discontinued",
    "Reserved": "unavailable",
}

def parse_availability(raw: str | None) -> str:
    if not raw:
        return "unknown"
    key = raw.rstrip("/").rsplit("/", 1)[-1]  # handles full schema.org URLs
    return AVAILABILITY.get(key, "unknown")

5. Pack sizes: compare like with like

Comparing a £3.60 six-pack with a £0.75 single can is meaningless until both are a price per litre. Parse the pack size and compute a unit price:

PACK = re.compile(r"(?:(\d+)\s*[x×]\s*)?(\d+(?:[.,]\d+)?)\s*(ml|l|g|kg)\b", re.I)
TO_BASE = {"ml": ("l", Decimal("0.001")), "l": ("l", Decimal(1)),
           "g": ("kg", Decimal("0.001")), "kg": ("kg", Decimal(1))}

def unit_price(title: str, price: Decimal) -> tuple[Decimal, str] | None:
    """'Sparkling Water 6 x 330ml' at 3.60 -> (Decimal('1.82'), 'l')"""
    m = PACK.search(title)
    if not m:
        return None
    count = int(m.group(1) or 1)
    size = Decimal(m.group(2).replace(",", "."))
    base_unit, factor = TO_BASE[m.group(3).lower()]
    quantity = count * size * factor
    return (price / quantity).quantize(Decimal("0.01")), base_unit

Prefer a dedicated size field or structured attributes when the store provides them. The title is a fallback, and a regex like this one should be tested against each source's naming habits.

6. Validate every record, then every run

Parsing produces data. Validation tells you whether to trust it. Check each record against rules:

REQUIRED = ("sku", "source_url", "collected_at", "currency", "price", "availability")

def validate(rec: dict) -> list[str]:
    issues = [f"missing:{f}" for f in REQUIRED if rec.get(f) in (None, "")]
    price, list_price = rec.get("price"), rec.get("list_price")
    if price is not None and price <= 0:
        issues.append("price_not_positive")
    if price is not None and list_price is not None and price > list_price:
        issues.append("sale_above_list")
    if rec.get("currency") and not re.fullmatch(r"[A-Z]{3}", rec["currency"]):
        issues.append("currency_not_iso4217")
    if rec.get("availability") == "unknown":
        issues.append("availability_unmapped")
    for attr in rec.get("varies_by", []):
        if not rec.get("attributes", {}).get(attr):
            issues.append(f"variant_attr_missing:{attr}")
    return issues

Then check each run as a whole, because the worst failures (a site redesign that silently drops half the catalogue) look fine record by record:

def check_run(today: list[dict], yesterday_count: int, tolerance: float = 0.15) -> list[str]:
    alerts = []
    if yesterday_count and abs(len(today) - yesterday_count) / yesterday_count > tolerance:
        alerts.append(f"record_count_shift:{yesterday_count}->{len(today)}")
    null_share = sum(r.get("price") is None for r in today) / max(len(today), 1)
    if null_share > 0.02:
        alerts.append(f"null_price_share:{null_share:.1%}")
    return alerts

Flag large price jumps for review rather than dropping them. A 60% drop is sometimes a parser error and sometimes a genuine clearance sale, and only one of those should be deleted.

Why the effort pays off

Accurate variant-level records are what make the data usable for the work it is collected for: price comparison, assortment analysis and keeping a catalogue complete and consistent. Teams that manage e-commerce product data across many sources find that the cost of a modelling mistake grows with every downstream system that trusts it. Fix it at extraction, where it is cheapest.

Two practical notes to finish. Collect from public product pages only, at a polite request rate, and respect each site's terms. And store the raw extraction next to the parsed record, so that when you fix a parser you can reprocess history instead of recollecting it.

FAQ

What data can you scrape from e-commerce sites?

Typically public product information: titles, brand, identifiers such as GTIN or MPN, variant attributes (size, colour), current and original prices, currency, availability, pack size, category and product URL. Collect from publicly visible product pages only.

How do you handle product variants when scraping?

Store one record per variant (SKU), linked to a parent product ID, with the attributes that distinguish it. Where a store publishes schema.org ProductGroup markup, hasVariant and variesBy provide this structure directly.

Should you scrape JSON-LD or the visible page?

Start with JSON-LD or other embedded structured data because it is more stable and better structured, then spot-check it against the visible page. Some stores only mark up the default variant or update markup less often than the displayed price.

Scraping at Scale: Architecture Patterns for Millions of Pages a Day

PromptCloud — Tue, 29 Sep 2026 06:47:00 +0000

A scraper that works for ten thousand pages does not work for ten million by running a thousand copies of itself. Somewhere on the way up, scale stops being a bigger version of the same problem and becomes a different problem, a distributed-systems problem, dominated by three forces a small scraper never feels: decoupling, politeness versus throughput, and cost. The patterns below are the ones that serve those three, and they are what separate a script that scales to a system that does.

The naive way to scale a scraper is to take the one that works and run more of it. It fails in specific, predictable ways. A thousand parallel copies with no coordination hammer the same domains and get everyone blocked. They re-fetch the same URLs because nothing tracks what has been seen. One misbehaving source backs up the whole fleet because there is no isolation. And the bill explodes, because at ten million pages a day every wasted request and every unnecessary browser render is multiplied by ten million. Scaling a scraper is not a throughput exercise, it is an architecture exercise, and here are the patterns that matter.

Pattern 1: decouple everything with a queue

The foundational move is to stop thinking of a scrape as a linear program and start thinking of it as stages connected by queues. A URL frontier holds work to be done; a pool of fetch workers pulls from it; fetched pages flow to a render stage if they need it, then to parse, validate, and store, each stage its own pool of workers connected by its own queue.

discovery -> [url frontier] -> fetch workers -> [fetched queue] ->
    render workers (only if needed) -> [parsed queue] ->
        extract + validate -> [results queue] -> store / deliver

This decoupling buys you three things at once. Each stage scales independently, so when rendering is your bottleneck you add render workers without touching the fetch pool. Bursts are absorbed by the queues instead of overwhelming a stage. And workers can be stateless, holding no shared state locally, which means you can autoscale them horizontally on queue depth and lose one at any time without losing work. The frontier, the seen-set, and the results all live in external stores; the workers are cattle. This one pattern is the difference between a system you can grow by adding machines and a monolith you can only grow by making it bigger.

Pattern 2: be massively parallel globally, gentle per domain

Here is the tension unique to scraping at scale: you need enormous global throughput, and you need to be polite to every individual site, and those pull in opposite directions. Ten million pages a day is a firehose in aggregate, but pointed at any single domain it must be a trickle, or you take that site down and get blocked. So global concurrency and per-domain concurrency are two different controls, and you need both.

The pattern is per-domain scheduling on top of global parallelism: shard the frontier by domain, and gate each domain with its own rate limit or token bucket and a concurrency cap, so a worker may only pull the next URL for a domain when that domain's budget allows. Thousands of domains each drip politely while the fleet as a whole runs flat out. Without this, raw parallelism concentrates on whatever domains happen to be in the queue and hammers them; with it, throughput and politeness stop being in conflict.

Pattern 3: deduplicate before you spend

At millions of URLs a day, doing the same work twice is a first-order cost, so deduplication becomes infrastructure rather than a nicety. You need a fast, scalable seen-set to answer "have we already queued or fetched this URL" before spending a request on it, and at this volume an exact set can be memory-prohibitive, so a probabilistic structure like a Bloom filter is often the right trade, accepting a tiny false-positive rate to keep the check cheap and bounded. Beyond URL dedup, content hashing lets you skip re-processing pages that have not changed since last time, and conditional requests using validators like ETag or last-modified let the origin tell you a page is unchanged before you download it at all. Every one of these is a way of not spending compute, bandwidth, or a request, and at ten million a day, not spending is the whole game.

Pattern 4: assume failure is the steady state

At small scale a failure is an event. At millions a day, thousands of things fail every hour, blocks, timeouts, malformed pages, dead proxies, and the system has to treat failure as normal operation rather than an exception to be surprised by. That means several habits working together:

def worker(msg, deps):
    domain = domain_of(msg.url)

    if deps.breaker.is_open(domain):          # circuit breaker: stop hammering a burned domain
        return deps.requeue_later(msg, domain)

    try:
        resp = deps.fetch(msg.url, identity=deps.identity_for(domain))
        kind = classify(resp)                 # DATA | BLOCK | CHALLENGE | THROTTLE
        if kind != "DATA":
            deps.breaker.record_failure(domain)
            return deps.retry_or_deadletter(msg, reason=kind)

        record = extract(resp)
        if not contract.valid(record):
            return deps.deadletter(msg, reason="failed_validation")

        deps.store.upsert(record.id, record)  # idempotent: at-least-once delivery is safe
        deps.breaker.record_success(domain)

    except Exception as e:
        deps.retry_or_deadletter(msg, reason=str(e))   # backoff; poison messages end in the DLQ

Retries with exponential backoff handle the transient failures. A dead-letter queue catches the poison messages so one un-parseable page does not block a worker forever. Per-domain circuit breakers stop the fleet from throwing requests at a domain that is returning nothing but blocks, which both saves cost and lets the domain recover. Bulkheads, isolating each source's work so one pathological site cannot starve the workers every other source depends on, keep a local fire from becoming a fleet outage. And because queues deliver at-least-once, writes must be idempotent, keyed on a deterministic ID, so a redelivered message overwrites rather than duplicates. None of these is exotic; together they are what keeps a system upright when failure is constant.

Pattern 5: watch aggregates, not jobs

You cannot supervise ten million individual jobs, so at scale the unit of observability shifts from the job to the aggregate. The control surface is a handful of fleet-level signals: throughput in pages per minute, overall success rate, queue depth and lag per stage, block rate per domain, and validation pass rate. A healthy system is a set of steady lines; an incident is a line moving, queue lag climbing because a stage is starved, block rate spiking on a domain that just tightened its defences, validation rate dropping because a source redesigned. You manage the system by its vital signs, and you alert on deviations in those, because no human is reading logs at this volume.

Pattern 6: cost is an architectural constraint

At millions a day, cost stops being something you optimise later and becomes something the architecture is shaped around, because every inefficiency is multiplied by the volume. The expensive operations are rendering and bandwidth, so the design minimises both structurally: render only the fraction of pages that genuinely need a browser and fetch the rest over plain HTTP, block images, fonts, and media so you are not paying to download pixels no parser reads, reserve expensive residential proxies for the hard targets and serve the bulk on cheap datacenter IPs, and lean on the dedup and conditional-request patterns above so you never pay twice for the same unchanged page. Efficiency here is not tuning; it is the difference between a viable unit cost and one that makes the whole operation uneconomic.

Build, or consume

Assembled, this is a serious distributed system: a durable frontier, staged autoscaling workers, per-domain schedulers, a dedup layer, circuit breakers and dead-letter queues, fleet observability, and a cost-control regime, all maintained as targets and volumes change. It is entirely buildable, and some teams should build it because crawling at scale is their core competency. For many others it is undifferentiated heavy lifting, which is the case for treating large-scale crawling as a consumed data crawling service rather than a platform to operate: the patterns here become someone else's operational burden and you receive the data at the end. Either way, the point stands that at this volume the interesting engineering is the system, not the scraper.

The takeaway

Scaling to millions of pages a day is not about making a scraper faster, it is about making it a distributed system: decouple stages with queues so they scale independently and contain failure, schedule per domain so you are polite and parallel at once, deduplicate and use conditional requests so you never spend twice, treat failure as the steady state with retries and breakers and idempotent writes, observe the fleet by its aggregate vital signs, and shape the whole thing around cost because every waste is multiplied by the volume. Get those patterns right and adding another few million pages a day is a matter of adding workers, which is exactly the property you were trying to buy.

FAQ

How do you scrape millions of pages a day without getting everything blocked?

By separating global throughput from per-domain politeness. The fleet runs massively parallel in aggregate, but the frontier is sharded by domain and each domain is gated by its own rate limit and concurrency cap, so no single site is hammered even as total volume is enormous. This is layered on top of the usual per-request concerns like coherent identity and proxy management, but the scale-specific insight is that raw parallelism must be constrained per domain, or high global throughput simply concentrates into taking individual sites down and getting your fleet banned.

What is the most important architecture pattern for large-scale scraping?

Decoupling the pipeline with queues. Turning discovery, fetching, rendering, parsing, and storage into separate stages connected by durable queues is what lets each stage scale independently, lets workers be stateless and autoscale on queue depth, absorbs bursts, and contains failures rather than letting one bad stage stall everything. Almost every other scale pattern, per-domain scheduling, dead-letter queues, circuit breakers, idempotent writes, depends on that queue-based decoupling being in place first. Without it you have a big monolith; with it you have a system you grow by adding machines.

Scraping for RAG: Keeping Your Retrieval Index Fresh (and Why Staleness Hallucinates)

PromptCloud — Mon, 28 Sep 2026 07:06:58 +0000

When a RAG system confidently states last quarter's price, last year's policy, or a fact that stopped being true in March, the reflex is to blame the model. Often the model did exactly its job: it grounded its answer faithfully in what retrieval handed it, and what retrieval handed it was stale. Your vector index is a cache of the world, and like any cache it goes wrong not by erroring but by quietly serving the past. Here is how to keep an index built from scraped web data fresh, and why freshness is a hallucination-prevention strategy rather than a nice-to-have.

Retrieval-augmented generation was supposed to be the answer to hallucination. Ground the model in retrieved documents and it stops making things up. That works, but it quietly relocates the problem rather than removing it. The model is now only as truthful as the chunks it retrieves, and if those chunks are out of date, the model will generate a fluent, confident, well-grounded answer that happens to be wrong. To the user that is indistinguishable from a hallucination. To you it should be diagnosable as something more specific: a freshness failure in a data pipeline you own.

That reframing is the whole point. A large share of what gets logged as "the model hallucinated" in a production RAG system is really "the retrieval index was stale, and the model told the truth about a world that no longer exists." Once you see hallucination as partly an infrastructure problem, the fix moves from prompt-tuning to pipeline engineering, which is a place you can actually make progress.

Three ways an index goes stale

"Stale" is not one failure, it is three, and they need different handling.

The first is stale content: a document you indexed is still there at the source, but its facts have changed. The price moved, the policy was rewritten, the spec was revised, and your index still holds the old embedding of the old text. Retrieval finds it, the model grounds on it, and the answer is confidently out of date.

The second is missing content: something new exists at the source but was never crawled and indexed, so it is not retrievable at all. The model asks for context, gets back either nothing or a semantically-near-but-wrong chunk, and fills the gap the way models do. This is the classic retrieval miss, and it produces some of the most convincing hallucinations because the surrounding context looks relevant.

The third is orphaned content: a document was removed at the source but still lives in your index, so the model cites something that no longer exists. Rarer, but corrosive to trust when it happens.

All three come from the same root cause: the index and the world have drifted apart, and nothing in the system noticed.

The index is a cache, so treat it like one

The productive mental model is that your retrieval index is a cache of the live web, and cache engineering has known answers: detect what changed, refresh it on a cadence matched to how fast it changes, invalidate what disappeared, and know how old everything is. The mistake teams make is treating the index as a one-time build, embedded once and left, when it is really a cache that has to be continuously reconciled against a moving source. Everything below is cache discipline applied to a corpus.

Detect change, do not re-embed the world

The naive refresh is to re-crawl and re-embed everything on a schedule. It is slow, and embedding compute is expensive, so it does not scale and it tempts you to refresh rarely, which is the opposite of what you want. The efficient approach is to re-embed only what actually changed, and content hashing is how you know:

def refresh_document(url, store):
    html = fetch(url)                      # reliable access is the precondition (see below)
    content = extract_and_normalise(html)  # strip boilerplate; normalise whitespace, dates, prices
    new_hash = sha256(content)

    record = store.get_doc(url)
    if record and record.hash == new_hash:
        store.touch(url, verified_at=now())   # unchanged: just stamp it fresh, skip embedding
        return "unchanged"

    chunks = chunk(content)
    for c in chunks:
        c.id = deterministic_id(url, c.index)   # stable key: re-index REPLACES, never duplicates
        c.embedding = embed(c.text)
        c.source_url = url
        c.verified_at = now()                   # freshness metadata travels with the chunk
    store.upsert(url, new_hash, chunks)          # idempotent upsert keyed by chunk id
    store.remove_missing_chunks(url, keep=[c.id for c in chunks])  # handle shrinkage
    return "reindexed"

Two properties matter here. The hash comparison means an unchanged page costs you one fetch and no embedding, which is what lets you afford to check often. And the deterministic chunk IDs make the upsert idempotent, so re-indexing a page overwrites its chunks rather than accumulating duplicate copies of near-identical text, which is its own quiet cause of bad retrieval.

Refresh on a cadence that matches volatility

Not everything changes at the same rate, so a single global crawl schedule is always wrong in both directions: too fast for your reference docs, too slow for your prices. Assign a freshness policy per source or template based on how volatile its content is. Pricing, inventory, and news get a short TTL and frequent checks; documentation, specifications, and archival material get a long one. You are budgeting a finite crawl-and-embed spend across a corpus, and volatility is how you allocate it. A page's TTL is a promise about the worst-case age of what you will serve from it, so set it against how wrong a stale answer from that page would be.

Make retrieval and generation freshness-aware

Keeping the index fresh is half the job. The other half is letting freshness influence what the model sees, because even a well-maintained index will sometimes hand back something past its useful life. Because every chunk now carries a verified_at and a source, you can act on it at query time:

def retrieve(query, store, k=8):
    hits = store.search(query, k=k * 2)          # over-fetch, then apply freshness

    fresh = []
    for h in hits:
        ttl = ttl_for(h.source_url)              # per-source volatility policy
        if age(h.verified_at) > ttl:
            continue                             # drop chunks past their freshness promise
        fresh.append(h)

    if not fresh or fresh[0].score < MIN_RELEVANCE:
        return NoConfidentContext()              # retrieval miss: let the model decline, not invent

    # Pass the as-of date INTO the context so the answer can be dated and hedged.
    return [f"[source: {h.source_url}, as of {h.verified_at:%Y-%m-%d}]\n{h.text}"
            for h in fresh[:k]]

Three moves do the work. Filtering out chunks older than their source's TTL stops the index serving content it can no longer stand behind. Detecting a weak top result and returning an explicit "no confident context" lets the model say it does not know rather than confabulate from a poor match, which is the single highest-leverage anti-hallucination behaviour you can add. And passing the as-of date into the context lets the model date or hedge its answer instead of stating a stale fact as timeless truth. You are not just retrieving text, you are retrieving text plus its age, and the age changes how it should be used.

Watch staleness as a metric

You cannot manage what you do not measure, so treat freshness as a monitored quantity, not an assumption. Track the age distribution of indexed chunks per source against their TTL, the fraction of known source content that is actually indexed (coverage), and the rate at which change detection is firing (drift). A rising p95 age on a high-volatility source, or coverage slipping below a threshold, is an early warning that your RAG answers are about to start "hallucinating" in the very specific sense this article is about.

Freshness is downstream of access

One honest dependency ties this back to the rest of the stack. Every refresh in the pipeline above begins with successfully fetching the source, and on the modern web that is not guaranteed: pages are defended, they change structure, and access breaks. An index is only as fresh as your ability to reliably re-reach its sources, which makes freshness downstream of access reliability, and access reliability the foundation of the whole web data infrastructure for AI that a RAG system quietly depends on. A beautiful freshness architecture sitting on top of a scraper that silently stopped returning data is just a stale index with good intentions.

The takeaway

If your RAG system hallucinates, check the index before you blame the model. Treat the index as a cache of the world: detect change with hashing so you refresh cheaply, set TTLs by volatility so you spend freshness where staleness would hurt, index idempotently so re-runs replace rather than duplicate, carry a verified-at date on every chunk so retrieval and generation can act on age, and monitor staleness as a first-class metric. Do that and a whole category of "hallucinations" turns out to have been a data-freshness bug all along, which is far better news than a mysterious model failure, because it is a bug you know how to fix.

FAQ

Why does a stale retrieval index cause hallucinations?

Because RAG grounds the model's answer in retrieved chunks, and if those chunks are out of date the model faithfully generates an answer based on old facts. From the user's point of view that is a confident wrong answer, indistinguishable from a hallucination, even though the model did its job correctly. Retrieval misses, where new content was never indexed, are worse still, because the model fills the gap from a semantically near but wrong chunk. In both cases the root cause is the index having drifted from the live source, which makes it a data-freshness problem rather than purely a model problem.

How do you keep a RAG index fresh without re-embedding everything?

Detect change instead of blindly re-embedding. Store a content hash per document, and on re-crawl only re-chunk and re-embed the documents whose hash changed, stamping the rest as freshly verified. Refresh each source on a cadence matched to its volatility rather than one global schedule, use deterministic chunk IDs so re-indexing replaces rather than duplicates, remove chunks whose source content disappeared, and store a verified-at timestamp on every chunk so retrieval can filter or down-rank stale content. This keeps embedding cost proportional to how much actually changed rather than to the size of the corpus.

Proxy Management for Web Scraping: Residential, Datacenter, and When Each Fails

PromptCloud — Thu, 24 Sep 2026 06:33:45 +0000

Most proxy guides are shopping guides: here are the types, here is which to buy. This one is about the part that actually decides whether your scraper works, which is what each type cannot do. A proxy changes exactly one thing about your request, and almost every proxy failure in the wild comes from expecting it to change more.

There is a reflex, when a scraper starts getting blocked, to reach for proxies as though they were the answer to blocking in general. They are not. A proxy changes one signal: the origin and reputation of the IP address your request appears to come from. That is genuinely useful, because IP is one of the first things a defensive system judges. But it is one signal among many, and if you treat the proxy as the whole solution you will buy the most expensive tier, watch it fail anyway, and have no idea why. So before comparing types, hold onto the single most useful fact in this whole topic: a proxy fixes your IP, and nothing else.

With that framing, the types make sense, and so do their failure modes.

Datacenter proxies: fast, cheap, and easy to spot

Datacenter proxies are IPs that belong to hosting providers and cloud infrastructure. They are abundant, fast, and cheap, often billed per IP rather than per gigabyte, which makes them the natural default for volume.

Their weakness is written into their origin. Every IP belongs to an autonomous system, and the ASN of a datacenter IP announces that it belongs to a cloud provider, not a home broadband connection. Real users do not browse consumer retail sites from an AWS range, so a defensive system that checks ASN reputation can flag datacenter traffic before it evaluates anything else about the request. That is exactly when datacenter proxies fail: against targets that score IP reputation by ASN. They will sail through unprotected sites, internal tools, and anything that does not scrutinise where the connection originates, and they will be rejected almost on sight by a well-defended consumer site, no matter how many of them you rotate through, because rotating through a thousand IPs that all say "datacenter" changes the address but not the tell.

Residential proxies: trusted, slow, and not a fingerprint fix

Residential proxies route your request through real consumer devices on real ISP connections, so the IP presents with a consumer ISP's ASN and looks like an ordinary household. That is what you are paying for, and it is why they get past ASN-reputation checks that stop datacenter traffic dead.

They fail in three distinct ways, and it is worth being precise about each. First, cost and speed: residential bandwidth is usually billed per gigabyte and routed through consumer connections, so it is markedly more expensive and slower than datacenter, and using it for jobs that never needed it is how scraping budgets quietly explode. Second, reliability: a residential IP is somebody's actual device, so it can vanish mid-session when they close their laptop, which makes long, stateful sessions fragile. Third, and most important, residential proxies do not fix anything except the IP. This is the failure I see most often: a team switches to residential expecting the blocks to stop, and they do not, because the request still carries a mismatched TLS fingerprint, an incoherent header order, no JavaScript engine, and a robotic request cadence. A good IP wrapped around a bot-shaped request is still a bot, just a more expensive one. Residential proxies raise the ceiling on IP reputation; they do nothing for the other signals a modern anti-bot layer reads.

There is also an ethical dimension worth naming: residential pools are only as clean as the way their IPs were sourced, and consent-based, transparently sourced pools matter both ethically and for reliability. It is a real part of choosing this tier, not a footnote.

Mobile and ISP proxies: the edges of the trade-off

Two more types sit at the extremes. Mobile proxies route through carrier networks, and because carrier-grade NAT means many real users share one mobile IP, sites are highly reluctant to block them, which makes mobile the highest-trust and also the slowest and priciest option, reserved for the hardest targets. ISP or static residential proxies are the hybrid: hosted in datacenters for speed and stability but registered under consumer ISP ASNs, giving you residential-looking reputation with datacenter reliability, at a middle price. Neither changes the underlying rule. They move you along the reputation-versus-cost curve; they do not exempt you from everything else.

The failure they all share

Step back and the pattern is clear: every proxy type fails the same way, by being asked to do a job it structurally cannot. The IP is one component of an identity, and the other components have to agree with it. If your IP says residential Britain but your TLS handshake says a Python HTTP client, your headers are ordered like a script, and you fire requests faster than any human could read, the proxy is not the weak link, the incoherence is. The right mental model is that a proxy is one field in a bundle that has to tell a single consistent story:

# A proxy is not a strategy; it is one field in a coherent identity.
class Identity:
    def __init__(self, proxy, tls_profile, headers, cookies, engine):
        self.proxy      = proxy         # the ONE thing a proxy changes: IP origin/reputation
        self.tls_profile = tls_profile  # handshake fingerprint must match the claimed browser
        self.headers     = headers      # order and values consistent with that browser
        self.cookies     = cookies      # session continuity, tied to THIS proxy
        self.engine      = engine        # JS execution when the target needs it

    # These travel together for the life of a session. You rotate the WHOLE
    # identity, never the proxy alone, because swapping the IP mid-session
    # while keeping the same cookies looks more suspicious, not less.

Rotating the IP on its own, mid-session, is a classic self-inflicted wound: you take a coherent session and make it incoherent, and behavioural systems notice a "user" whose network origin jumped between requests. The proxy has to rotate as part of the identity, not independently of it.

What proxy management actually means

Once you accept that a proxy is one lever, "proxy management" stops meaning "which type do I buy" and starts meaning the operational discipline around the pool. Five things carry most of it.

Match the tier to the target, not the reverse. Default to cheap datacenter, and escalate to residential or mobile only for the specific targets whose defences demand it, decided per target and cached, so you are not paying residential rates for sites that never checked. Keep sessions coherent: bind an IP to its cookies and fingerprint for the life of a logical session, use sticky sessions where a workflow spans pages or a login, and rotate between whole identities rather than within one. Run continuous pool hygiene, because proxies rot: health-check them, quarantine the ones that start failing, retire burned IPs, and track success rate per subnet and ASN rather than per individual IP, since bans often cluster by subnet. Rotate on the right trigger, per session, per N requests, or on a detected block, and back off honouring Retry-After on a 429 rather than hammering through it. And close the loop with ban detection, so a blocked response is recognised as a block rather than parsed as data, the burned identity is retired, and the pool learns which ranges are burning.

async def fetch(target, pool):
    policy = policy_for(target)                 # tier + rotation rule, cached per target
    identity = pool.acquire(tier=policy.tier)

    for attempt in range(policy.max_attempts):
        resp = await send(target, identity)
        kind = classify(resp)                   # DATA | BLOCK | CHALLENGE | THROTTLE

        if kind == "DATA":
            pool.report_success(identity)
            return resp
        if kind == "THROTTLE":
            backoff(resp)                        # respect Retry-After, keep the identity
            continue

        pool.retire(identity, reason=kind)       # burned: pull it, learn the subnet
        if policy.escalate_on_block:
            identity = pool.acquire(tier=policy.next_tier)  # datacenter -> residential only now
        else:
            identity = pool.acquire(tier=policy.tier)

    raise Blocked(target)

Two economic notes make the difference at scale. Because residential is billed by bandwidth, block images, fonts, and media so you are not paying per gigabyte to download pixels no parser reads. And because the tiers differ in cost by an order of magnitude, serving the bulk of your traffic on datacenter and reserving residential for the hard fraction is usually the difference between a viable unit cost and an alarming one.

Build, or consume

Assembled, this is a real system: a multi-tier pool, health checking, per-target policy, identity coherence, rotation logic, ban detection, and bandwidth control, all of it maintained as targets change their defences. Plenty of teams should build it. Others are better served treating it as solved and consuming clean data instead, which is the premise behind a managed approach to using proxies for web scraping rather than operating the pool yourself. Either way, the decision should be made knowing what proxies do and do not do, because the expensive mistakes all come from the same misunderstanding.

The takeaway

Choose datacenter for reach and cost, residential or mobile for reputation against defended targets, and expect each to fail the moment you ask it to do more than change your IP. The proxy is one signal. It works when the rest of your identity is coherent, when the tier matches the target, and when the pool is actively managed, and it fails, every time, when it is treated as a substitute for those things rather than one part of them.

FAQ

Should I use residential or datacenter proxies for web scraping?

Match the tier to the target rather than defaulting to one. Datacenter proxies are fast and cheap and work well against sites that do not scrutinise IP origin, but they are easy to flag because their ASN identifies them as hosting infrastructure. Residential proxies present as ordinary consumer connections and get past ASN-reputation checks, but they are slower, billed by bandwidth, less reliable per IP, and do nothing to fix a bot-like fingerprint or behaviour. The cost-effective pattern is to serve the bulk of traffic on datacenter and escalate to residential or mobile only for the specific targets whose defences require it.

Why am I still getting blocked even with residential proxies?

Because a proxy only changes your IP, and IP is one signal among several. If your TLS handshake, header order, and request cadence still look automated, a residential IP just makes you a bot with a better address. Blocks that persist after switching to residential almost always trace to an incoherent identity (a mismatched fingerprint, a missing JavaScript engine, robotic timing) or to bad rotation, such as changing IP mid-session while keeping the same cookies, which looks more suspicious rather than less. The fix is to make the whole identity coherent and rotate it as a unit, not to buy a more expensive proxy.

Handling the Six Web-Access Barriers in One Pipeline: A Reference Design

PromptCloud — Wed, 23 Sep 2026 07:46:10 +0000

There are six barriers between a request and the data on the modern web: edge firewalls, behavioural bot management, CAPTCHA, JavaScript rendering, rate limiting, and TLS fingerprinting. The common mistake is to solve each one where you hit it, bolting a fix onto a script until you have six brittle point-solutions that fight each other. This is the opposite: one pipeline, two organising principles, and each barrier handled as a composable stage rather than a patch.

If you have read the field-guide version of this, you already know the six barriers and how to tell which tier of infrastructure a site is forcing on you. Diagnosis is half the job. The other half is architecture: how do you build a single data pipeline that handles all six without turning into an unmaintainable pile of special cases? Most teams get here by accretion. A site starts blocking, so they add a proxy. Another needs rendering, so they bolt on a headless browser. A third throws CAPTCHAs, so they wire in a solver. Six fixes later they have a system where the proxy layer does not know what the browser layer is doing, the retry logic double-charges the rate limiter, and nobody can say why one source in ten silently returns empty rows.

The way out is to stop thinking in fixes and start thinking in principles. Two of them carry almost the entire design.

Principle 1: identity coherence

Every one of the defensive barriers is, underneath, trying to answer a single question: is this a real browser driven by a real person, or automation pretending to be one? They just ask it at different layers. TLS fingerprinting asks during the handshake. Edge firewalls and bot management ask by inspecting headers, connection behaviour, and request cadence. CAPTCHA asks the client to do something a script finds hard.

That means your defining requirement is not any single trick, it is coherence. Every signal your client emits must tell the same story. If your TLS handshake says Chrome but your header order says Python, your JavaScript engine is absent, and your connection is HTTP/1.1, you have not presented as a browser, you have presented as automation wearing a Chrome badge, and each barrier catches the mismatch at its own layer. Get identity coherence right and most barriers stop firing, not because you defeated them individually but because you stopped tripping the question they all ask. Get it wrong anywhere in the stack and no amount of proxy rotation saves you, because the incoherence travels with every identity you rotate to.

Principle 2: response classification is the universal detector

The second principle addresses the failure mode that makes all six barriers dangerous rather than merely annoying: they mostly do not return errors. A WAF block, a bot-management interstitial, a CAPTCHA page, a rate-limit throttle, and an empty JavaScript shell can all arrive as HTTP 200. If your pipeline treats a 200 as success and hands the body straight to your parser, every barrier becomes a silent data-quality incident.

So the pipeline needs one component that no naive scraper has: a classifier that decides what a response actually is before anything tries to parse it. Data, or challenge, or block, or empty shell. This single stage is what turns six invisible failure modes into six observable, routable events, and it is the backbone the rest of the design hangs off.

The reference design

Picture the pipeline as a request flowing through composable stages, with a shared identity object and a classifier gate, orchestrated by a per-target policy. In sketch form:

# Each barrier maps to a stage. Stages are composed, not hard-wired,
# and engaged per target by policy rather than always-on.

async def fetch(target, policy):
    identity = identity_pool.acquire(policy)   # coherent TLS + headers + IP + cookies + engine

    for attempt in retry(policy.max_attempts):
        # Rendering stage: cheapest path that works (Barrier 4)
        resp = await transport.get(
            target.url,
            identity=identity,                 # coherent handshake defeats TLS fingerprinting (Barrier 6)
            render=policy.render_mode,         # none | http | headless, escalated by policy
            pacing=policy.pacing,              # per-identity cadence under the cap (Barrier 5)
        )

        # The universal detector: what IS this response? (turns 200s into events)
        kind = classify(resp)                  # DATA | CHALLENGE | BLOCK | EMPTY

        if kind == "DATA":
            record = extract(resp)
            if contract.valid(record):         # validation gate: real data, not a disguised block
                return record
            kind = "EMPTY"                      # passed HTTP, failed the contract -> treat as failure

        if kind == "CHALLENGE":                # CAPTCHA / interactive check (Barriers 2, 3)
            token = await challenge.solve(resp, identity)
            identity.attach(token)             # cache clearance for the session, solve once not per request
            continue

        if kind in ("BLOCK", "EMPTY"):         # WAF / rate-limit / broken extraction (Barriers 1, 4, 5)
            identity = identity_pool.rotate(identity, reason=kind)
            observe(target, barrier=kind)      # feed the control plane
            backoff(attempt, resp)             # honour Retry-After on 429
            continue

    raise AccessFailed(target, last=kind)

Read it as five request-path stages, each mapping to barriers, plus the classifier that ties them together.

The identity stage assembles a coherent client: a TLS profile, header set, HTTP version, and, where rendering is needed, a browser engine that all agree on one story. This is what neutralises TLS fingerprinting (barrier six) and takes the edge off the firewall and bot-management layers (barriers one and two), because it stops the mismatch they look for.

The rendering stage is tiered, cheapest first. A plain HTTP fetch or structured-data extraction handles most pages; a pooled, resource-blocking headless browser is engaged only when a target genuinely assembles its content client-side (barrier four). Rendering is a cost, so the policy decides when to pay it rather than paying it by default.

The pacing and rotation stage manages identity over time. It keeps an IP, its cookies, and its fingerprint together as one coherent session rather than swapping IPs mid-conversation, paces requests per identity under the rate cap, and honours Retry-After on a 429 with real backoff (barrier five). Rotation is a response to a detected block, not a thing you do blindly on every request.

The challenge stage exists because a CAPTCHA or interactive check is not data and must never be parsed as such (barriers two and three). When the classifier flags a challenge, this stage resolves it and, crucially, caches the resulting clearance on the identity so you clear once per session instead of fighting the same wall on every request.

The validation stage is the last line against silent success. Even a DATA response gets checked against a contract, required fields, types, plausible ranges, expected volume, so a page that passed HTTP but returns half-empty because a redesign broke extraction is caught and demoted to a failure rather than shipped.

The control plane

Stages handle a single request. What makes it a system rather than a clever function is the control plane around it.

# Policy is resolved per target TEMPLATE, cached, and updated by feedback.
policy = policy_store.get(target.template) or discover(target)   # escalate tier only as needed

Three concerns live here. Routing policy decides, per site template, which stages to engage, because most targets need only a coherent identity and cheap rendering, and paying for headless browsers and rotation everywhere is how costs balloon. The expensive discovery of what a template needs happens once and is cached against the template, not rediscovered per URL. Observability records which barrier bit, block rate, challenge rate, 429 rate, and contract-failure rate per target, so a rising wall is visible as a trend rather than a mystery gap in your data. And idempotent retry makes the whole escalation loop safe: because a retry writes the same record by a deterministic key rather than appending, the pipeline can retry, rotate, and re-run freely without ever duplicating or half-writing state. Safe retries are what let the classifier-driven loop above recover on its own.

Wire those together and you get the last property worth having: a feedback loop. When observability sees a target's block or contract-failure rate climb, it bumps that template up a tier and updates the routing policy, so the pipeline adapts to a newly-defended site instead of quietly failing against it.

Build, or don't

None of this is exotic, but notice what it actually is: an identity system, a tiered renderer, a proxy and pacing manager, a challenge handler, a response classifier, a validation layer, and a control plane that ties them together with observability and a feedback loop. That is a real platform, and every one of its parts is a standing maintenance commitment that grows as the barriers escalate. Plenty of teams should build it, because access is core to what they do. Plenty of others are better served treating resilient access as a solved layer they consume, which is the entire premise of a managed web crawling service: the six barriers become someone else's reference design to maintain, and you consume clean data at the end of it. The right answer depends on whether web access is your product or your dependency.

The takeaway

Six barriers do not need six solutions. They need one pipeline built on two ideas: present a coherent identity so you stop tripping the question every barrier asks, and classify every response before you parse it so no barrier can fail silently. Map each barrier to a composable stage, engage the stages by policy rather than by default, and wrap the whole thing in observability, idempotent retry, and a feedback loop. Do that and adding the seventh barrier, when it arrives, is a new stage in a clean architecture rather than the sixth patch on a script that was never designed to carry them.

FAQ

Why not just handle each web-access barrier separately as I hit it?

Because point-solutions accumulate into a system whose parts do not cooperate, and the barriers interact. Your proxy rotation undermines the session coherence your fingerprint needs; your retry logic burns your rate-limit budget; your parser chokes on a CAPTCHA page the transport layer never flagged. Handling the barriers in one pipeline lets them share an identity model and a response classifier, which is what stops them fighting each other. The barriers all ask variants of the same question, so a coherent architecture answers it once rather than six inconsistent times.

What single component matters most in a resilient scraping pipeline?

The response classifier that decides whether a response is data, a challenge, a block, or an empty shell before anything tries to parse it. Almost every barrier can return HTTP 200 while withholding the actual content, so without classification a pipeline treats blocks and CAPTCHA pages as successful data and corrupts itself silently. With it, every barrier becomes a visible, routable event you can retry, rotate, or escalate against, and a validation contract on the extracted record catches the cases where extraction broke without any barrier firing at all.

The Six Barriers Between Your Crawler and the Data: A 2026 Field Guide

PromptCloud — Tue, 22 Sep 2026 08:36:02 +0000

The web still returns a 200 for almost everything. That is not the same as being reachable reliably, on a schedule, at scale. Here are the six barriers that sit between a request and the data in 2026, why they are multiplicative rather than additive, and how to work out which tier of infrastructure a given site is actually forcing on you.

If you have shipped a crawler this year, you already know the feeling: the code that worked in a notebook against ten URLs falls over the moment it runs continuously against ten thousand. Nothing in your parser changed. What changed is that the modern web is built and defended for a world where more than half of traffic is automated, and the defences do not distinguish your legitimate data pipeline from the bad bots they were bought to stop.

That "more than half" is not a figure of speech. PromptCloud's State of Web Access 2026 report puts automated traffic at 53% of the total, with bad bots alone accounting for 40% (source: Thales's 2026 Bad Bot Report). Roughly a fifth of all websites sit behind a single vendor's infrastructure, which means access policy for a large slice of the web is now set in a handful of places. For anyone building data collection, the practical question has shifted. Reach is mostly solved. Reliable, repeatable reach is the engineering problem, and it is defined by six barriers.

The six barriers, from most to least common

Web application firewalls (WAF). The most common barrier by far, and the one you rarely chose to fight, because it is bundled into the CDN or hosting the site already uses. A WAF inspects requests at the edge and drops the ones that look wrong before your code ever touches application logic. It is always-on and it is everywhere.
Anti-bot / bot management. A layer above the WAF that scores behaviour rather than individual requests. It watches request cadence, header consistency, session shape, and the small tells that separate a browser driven by a human from one driven by a script. Its defining property is that it learns: it builds a signature for a recurring scrape and gets better at recognising yours specifically over time. A pattern that works today can degrade on its own next week with no change on your side.
CAPTCHA. Selective rather than blanket, concentrated on logins, checkouts, and forms, which is to say on exactly the high-value actions. It is best thought of as a continuous line item rather than a one-off obstacle, because it reappears wherever the value is highest.
JavaScript rendering. Not a defence at all, just how a large share of the web is now built. The initial HTML is a shell and the content assembles in the browser, which forces you to run browser infrastructure to see what a user sees. It costs you regardless of anyone's intent to block you, and it is the barrier most likely to quietly dominate your compute bill.
Rate limiting. The bluntest instrument in the set. Cross a request threshold and you get a 403 or a 429, often on the very next request. It is trivial to implement, which is exactly why it is common, and it turns "fetch everything quickly" into a paced, scheduled problem.
TLS fingerprinting. The rarest and the hardest to diagnose, because it rejects you during the TLS handshake, before a single HTTP request is logged. Your client's TLS signature does not match a real browser's, and the connection dies before you get a status code to debug. When a target "works in curl but not in your stack, and there is no error you can read," this is the usual suspect.

Why they stack, and why that is the whole problem

The trap is treating these as a checklist you clear once. In practice they combine, and the effect is multiplicative, not additive. The report frames access difficulty as tiers of required infrastructure, and the model is worth internalising because it tells you what a site is actually costing you:

Tier 1 is a plain HTTP request. A requests.get and a parser is enough.
Tier 2 is a headless browser behind a firewall: you need to render JavaScript and present as a real client to get past edge filtering.
Tier 3 adds rotating IPs, paced scheduling, and TLS parity: you are now managing identity and timing, not just fetching.
Tier 4 is specialist territory, where dedicated anti-bot systems require dedicated countermeasures and constant upkeep.

Two things make this hard to plan around. First, each barrier you add is an independent way for the pipeline to break and a separate thing to monitor, so a Tier 3 target is not "a bit harder" than Tier 1, it is a qualitatively different operational commitment. Second, any site can move up a tier at any time, without warning, because the barrier is usually a configuration change on infrastructure the site owner does not even operate themselves. You can go to bed at Tier 2 and wake up at Tier 3.

And there is a pattern to which sites sit where. The data that matters most and the data that is hardest to reach reliably tend to be the same data. Competitor pricing, live inventory, high-demand listings: these are precisely the pages their owners have the most commercial incentive to defend, so difficulty tracks value almost by definition. The easy targets are easy because nobody minds.

Diagnosing your tier

Before you reach for heavier infrastructure, work out what you are actually up against, because the wrong diagnosis is expensive in both directions. A rough field procedure:

1. Plain GET with an honest client.
   200 with your data in the body?          -> Tier 1. Stop. Do not add machinery.
   200 but the body is an empty shell?      -> JS rendering. Move toward Tier 2.
   403 / 429 immediately?                   -> rate limiting or WAF.
   Connection fails before any HTTP status? -> suspect TLS fingerprinting (Tier 3+).

2. Add a real headless browser (proper TLS, real headers, JS execution).
   Works now?                               -> Tier 2. Budget for browser infra.
   Still blocked after N clean requests,
   then blocked on a pattern?               -> behavioural bot management is learning you. Tier 3+.

3. Watch it over days, not minutes.
   Success rate decays on a fixed schedule/volume? -> you are being profiled. Plan for IP rotation,
                                                       pacing, and continuous monitoring, not a one-off fix.

The point of the exercise is restraint as much as escalation. Plenty of teams run Tier 3 machinery against Tier 1 sites because a single 403 spooked them, and plenty run Tier 1 code against Tier 3 sites and call the source "flaky." Match the infrastructure to the tier, and re-check the tier periodically, because it moves.

What this means for how you build

The honest takeaway is that a crawler is not a script you finish, it is a system you keep alive. The status code lies: a 200 tells you the request completed, not that the data behind it is complete, current, or reachable again tomorrow at volume. Budget for the barriers you will actually hit, instrument every one of them so a silent tier change surfaces as an alert rather than a gap in your data, and be ruthless about not over-building against sites that never needed it. The web has not closed. It has just made reliable access an engineering discipline with a running cost, and the first step to managing that cost is naming the six things that create it.

FAQ

Why does my scraper work locally but fail at scale?

Because the barriers that matter are triggered by volume and pattern, not by a single request. A handful of requests from your laptop looks like a human; thousands of paced, structurally identical requests from the same identity looks like exactly what it is, and trips rate limits and behavioural bot management that a small test never reaches. Anti-bot systems specifically learn the signature of a recurring scrape over time, so a pattern that passes today can degrade on its own later with no change to your code.

What is TLS fingerprinting and why is it so hard to debug?

TLS fingerprinting identifies your client from the characteristics of its TLS handshake, the way it negotiates the encrypted connection, and rejects it if that signature does not match a real browser. It is hard to diagnose because the rejection happens before any HTTP request is processed, so you never get a status code or an error body to inspect: the connection simply fails. The classic tell is a target that behaves in one client but silently dies in another with no readable error, which usually means you are being blocked at the handshake rather than at the application layer.

Rendering JavaScript at Scale Without a Browser Farm: An Architecture Walkthrough

PromptCloud — Mon, 21 Sep 2026 09:19:22 +0000

The reflex when a site is JavaScript-heavy is to spin up headless Chrome and scale it horizontally. That works, and it is also the most expensive, slowest, most fragile way to solve the problem. Here is an architecture that renders only the pages that genuinely need it, and renders those lean, so a small pool does the work of a farm.

If you have ever built a crawler for a modern site, you know the moment: the HTML comes back and the content you wanted is not in it. It is a single-page app, the DOM is assembled in the browser, and requests plus a parser gets you an empty shell. The obvious fix is to render the page in a real browser, and the obvious way to do that at scale is to run a lot of browsers. That is the browser farm, and it is where a lot of scraping infrastructure quietly goes to die: headless Chrome is memory-hungry, slow relative to an HTTP call, and every version bump or anti-automation change means babysitting fleets of instances.

The good news is that most of the time you do not need it. A JavaScript-heavy page is heavy in the browser, but the data underneath it almost always arrives in a form you can get to without rendering anything. The architecture below is built around one principle: full rendering is a last resort, not a default, and the job of the system is to route each URL to the cheapest tier that actually works.

The principle: rendering is a cost, not a step

Treat a full browser render as the most expensive operation in your pipeline, because it is. An HTTP request costs milliseconds and a few kilobytes of memory. A rendered page costs hundreds of milliseconds to seconds, tens to hundreds of megabytes, and a browser process you have to manage. If you render every JS page by default, you have priced your entire crawl at the rate of your single most expensive operation. The whole design goal is to push as much traffic as possible into cheaper tiers and reserve the browser for the residue that has no other route.

Tier 0: find the API the page is already calling

This is the single biggest lever, and it is the one most often skipped. A single-page app does not conjure content from nothing. It renders by calling a backend, almost always over fetch or XHR, and almost always getting back clean JSON. The browser assembles that JSON into a DOM. You do not have to. You can call the same endpoint the page calls and parse the JSON directly, which is faster, lighter, and usually more stable than scraping rendered HTML.

Finding it is a one-time investigation per site: open the network panel, filter to XHR/fetch, and watch what the page requests as it loads the data you want. You are looking for the request whose response contains your fields. Once you have it, you replicate its method, headers, and parameters in a plain HTTP client. This is the core of most modern crawling techniques for JavaScript-heavy websites: the render is a distraction, and the real target is the data feed behind it.

Two caveats keep this honest. Some endpoints are protected by tokens or signatures the page generates in JavaScript, in which case you may need a light render once to obtain a token, then reuse it. And some are paginated or shaped awkwardly. Neither changes the principle: an API call, even an occasionally-primed one, is an order of magnitude cheaper than rendering every page.

Tier 1: read the state the page shipped in its HTML

When there is no live endpoint to call, the data is very often sitting in the initial HTML already, embedded as JSON for the framework to hydrate from. This is server-side rendering and hydration, and it leaves fingerprints you can parse without a browser:

import json
from bs4 import BeautifulSoup

def extract_embedded_state(html: str):
    soup = BeautifulSoup(html, "lxml")

    # Next.js ships page props here
    nextjs = soup.find("script", id="__NEXT_DATA__")
    if nextjs:
        return json.loads(nextjs.string)

    # Common hand-rolled and Nuxt/Vue patterns live in inline scripts
    for script in soup.find_all("script"):
        text = script.string or ""
        for marker in ("__INITIAL_STATE__", "__NUXT__", "__APOLLO_STATE__"):
            if marker in text:
                start = text.index("{")
                return json.loads(text[start:text.rindex("}") + 1])

    # Structured data is a gift when it is present
    ld = soup.find("script", type="application/ld+json")
    if ld:
        return json.loads(ld.string)

    return None

NEXT_DATA, window.INITIAL_STATE, NUXT, APOLLO_STATE, and application/ld+json blocks cover a large share of the framework-built web. When one of them holds your fields, you have the fully-formed data with a single HTTP request and a JSON parse, and you never opened a browser.

Tier 2: render, but only what is left, and only leanly

Some pages defeat both tiers above: the data is fetched by a request you cannot easily replay, assembled across several calls, or gated behind interaction. These genuinely need a browser. The mistake is to let that small residue dictate the cost of the whole system by running heavyweight, general-purpose browser instances for it.

Render as a shared, stateless service behind a queue, and make each render as cheap as you can. Three things do most of the work: reuse one browser across many pages via separate contexts rather than launching a process per URL, block every resource that does not contribute to the data you need, and put a hard budget on every render so a slow page cannot stall the pool.

from playwright.async_api import async_playwright

BLOCK = {"image", "media", "font", "stylesheet"}

async def render(url: str, wait_for: str, browser):
    context = await browser.new_context()  # cheap; isolate, then discard
    page = await context.new_page()

    # Drop everything that does not produce data: images, fonts, CSS,
    # analytics and ad calls. This alone cuts render time and memory hard.
    async def gate(route):
        req = route.request
        if req.resource_type in BLOCK or "analytics" in req.url:
            await route.abort()
        else:
            await route.continue_()
    await page.route("**/*", gate)

    try:
        await page.goto(url, wait_until="commit", timeout=15000)
        await page.wait_for_selector(wait_for, timeout=8000)
        return await page.content()
    finally:
        await context.close()  # hard budget: never leak a context

Blocking images, fonts, stylesheets, media, and third-party analytics or ad requests routinely removes the majority of a page's bytes and a large slice of its render time, because you are no longer downloading and painting things no parser will ever read. Reusing one browser across isolated contexts avoids the per-process launch cost that makes naive farms so heavy. And a strict wait_for_selector with timeouts means you wait for the specific element that signals your data has arrived, not for some arbitrary sleep, and you fail fast when it does not. The residue still costs more than an HTTP call, but a lean render on a shared pool is a different order of expense from a farm of full browsers.

The routing layer: decide once per template, not once per URL

The tiers only save money if something cheap decides which tier a URL needs, and the key insight is that you almost never decide per URL. Sites are template-driven. Every product page on a site is the same page with different data, so if one product page exposes a JSON endpoint at Tier 0, they all do. The expensive discovery, finding the endpoint or the embedded-state key, happens once per template, and the result is cached against a URL pattern.

So the router works like this: classify a URL to its template (by path shape or a learned pattern), look up the known route for that template, and apply it directly, HTTP-only for Tier 0 and 1, queued render for Tier 2. Only when a template is unseen, or its cached route starts failing, do you run the one-off investigation and update the cached decision. In steady state almost every request is served by a cached routing decision, and only a trickle triggers rediscovery. This is also where a self-healing loop lives: a route that begins returning empty or malformed data flags the template for re-investigation rather than silently shipping bad rows.

Caching, dedup, and the numbers that result

Two more cheap wins sit on top. Deduplicate in-flight requests so you never render the same URL twice concurrently, and cache tier outputs with a sensible freshness window so repeat crawls of slow-changing pages skip work entirely.

Put it together and the economics invert. Instead of rendering 100% of JS pages, a mature pipeline of this shape typically serves the large majority of traffic from Tier 0 and Tier 1 at HTTP cost, and renders only the remainder, leanly, on a small shared pool. You have replaced a browser farm with an HTTP-first pipeline that happens to keep a modest rendering service on call. That is cheaper to run, faster end to end, and far less fragile, because most of your crawl no longer depends on the most breakable component you own.

The takeaway

A JavaScript-heavy site is not a rendering problem, it is a routing problem. The data is nearly always reachable without a browser, through the API the page calls or the state it shipped in its HTML, and the small share that truly needs rendering can be rendered lean on a shared pool rather than a farm. Build the tiers, cache the decision at the template level, and reserve the browser for the residue. You will run a fraction of the infrastructure and break a fraction as often.

FAQ

How do I find the hidden API a JavaScript site uses?

Open your browser's developer tools, go to the Network panel, and filter to XHR/fetch requests. Reload the page and watch which request returns a response containing the data you want, usually as JSON. That is the endpoint the page itself calls to render. Note its URL, method, headers, and query parameters, then replicate the call in a plain HTTP client. If the endpoint is protected by a token the page generates, you may need to render once to capture that token and then reuse it across many direct API calls.

Do I ever actually need a headless browser for scraping?

Yes, but far less often than the default reflex suggests. You need one when the data is produced by requests you cannot easily replay, is assembled across multiple interactions, or is gated behind clicks and scrolls. The goal is not to eliminate rendering but to minimise it: route everything you can to direct API calls and embedded-state parsing, and reserve a lean, shared rendering service for the genuine residue rather than rendering every page by default.

Can LLMs Solve CAPTCHAs Now? What It Means for Scraping Architecture

PromptCloud — Fri, 18 Sep 2026 07:22:27 +0000

Short answer: yes for the easy visual puzzles, no for the hard interactive ones, and it mostly does not matter either way. If you are designing a scraping pipeline around a CAPTCHA-solving step, you are probably solving the wrong problem. Here is why, and what to design instead.

"Can an LLM just solve the CAPTCHA now?" is the question every engineer asks the first time a scraper hits a challenge page. It feels like the whole problem, because the CAPTCHA is the visible wall in front of the data. The honest answer is more interesting than yes or no, and the architectural conclusion is the opposite of what most people expect: the better modern CAPTCHA systems get, the less a solver matters, because by the time you are staring at a puzzle you have usually already lost.

What LLMs can and cannot do on CAPTCHAs

Multimodal LLMs are genuinely good at the classic visual challenges. A "select all the squares with a traffic light" grid, or distorted text, is well within reach of current vision-capable models, and there is a steady stream of research demonstrating it. If the challenge is a static image-recognition puzzle, treating it as a vision problem now works a lot of the time.

The interactive, multi-step challenges are a different story. A recent benchmark, Open CaptchaWorld, tested current multimodal agents on the kind of CAPTCHAs that involve several reasoning steps and interaction rather than a single classification. Humans scored 93.3%. The best agent tested, an o3-based browser agent, managed 40.0%. That is a gap of over fifty percentage points on exactly the challenges the modern systems are moving towards. So "LLMs solve CAPTCHAs" is true for the yesterday version of CAPTCHA and shaky for the tomorrow version.

But even the 40% figure, and even a hypothetical 90% one, is answering a question that the modern architecture has already made secondary.

The puzzle is not the defence any more

Here is the shift that changes everything about how you should design for this. On a modern challenge system, the visible puzzle is the last resort, not the first line. Before any image is ever shown, the system computes a risk score from a pile of signals: your IP reputation, your TLS and browser fingerprint, your request patterns, cookies and session history, and behavioural cues like mouse movement and timing. reCAPTCHA v3, hCaptcha's risk-based flow, and Cloudflare Turnstile all work on this principle. The score decides your fate. A request that looks human sails through invisibly. A request that looks automated is challenged, throttled, or blocked.

This inverts the whole problem. You only see a visual puzzle because the system already decided you look suspicious. So the ability to solve that puzzle is not a way past the gate; it is a coin flip you are forced into after you have already been flagged. Worse, on risk-scored systems, solving the puzzle does not reset the judgement that produced it. If everything else about your session says "bot," a correct answer to the image challenge does not necessarily buy you trust, and repeatedly hitting and solving challenges is itself a pattern that confirms automation.

Some newer systems add proof-of-work, forcing the client to burn CPU before proceeding. There, an LLM solving an image is beside the point entirely, because the gate is compute cost and behaviour, not visual recognition.

The uncomfortable conclusion: if your architecture depends on solving CAPTCHAs, you have built your pipeline around the symptom instead of the cause. The CAPTCHA appearing is the alarm. Silencing the alarm does not fix what set it off.

What this means for scraping architecture

Reframing CAPTCHA as a risk-score symptom rather than a visual puzzle changes what you build.

Treat a CAPTCHA as a signal, not a step. The most useful thing your pipeline can do with a challenge is log it as a health metric. A rising challenge rate on a source means your risk profile on that source is deteriorating, and that is diagnostic information you want on a dashboard, not something to silently paper over with a solver and forget.

Design to not trigger the score. The durable work is upstream of the puzzle: presenting a consistent, real browser fingerprint at every layer, so nothing contradicts anything else; maintaining good IP reputation rather than hammering from flagged ranges; reusing sessions and cookies the way a real user would; and keeping request patterns and pacing plausible. This is the same "every layer must tell the same story" principle that governs the rest of anti-bot detection. Get it right and the challenge mostly never appears, which is worth far more than being able to solve it.

Make challenge-handling a fallback, not the plan. There will still be sources and moments where a challenge is unavoidable, and having a way to handle it has its place. But it belongs at the edge of the architecture as a fallback for the residual cases, not at the centre as the mechanism the whole pipeline relies on. A design whose happy path runs through a solver is a design that is already flagged on every request.

Budget for the arms race. Both sides move. Detection adds signals; models get better at puzzles; detection shifts to signals models cannot fake. A one-time configuration that works today degrades, silently, as the systems on the other side change. Whatever you build needs someone watching the challenge rate over time, because a slow climb is the early warning that your approach is ageing out.

# Wrong mental model: CAPTCHA as a step to solve
request -> [CAPTCHA?] -> solve_captcha() -> data      # brittle: you're already flagged

# Better mental model: CAPTCHA as a symptom to avoid
request (consistent fingerprint, good IP, human-like session)
   -> risk score stays low
   -> no challenge shown
   -> data
# and: challenge_rate is a monitored health metric, not a silent retry

When to stop building this yourself

The reason many teams route serious collection through a managed layer is not that CAPTCHA-solving is hard in isolation. It is that keeping the risk score low across many sources, as detection keeps evolving, is continuous engineering rather than a feature you finish. Understanding how AI solves Captcha and, more importantly, how modern systems decide whether to show a challenge at all, is what separates a pipeline that quietly keeps delivering from one that spends its life fighting puzzle pages. If your team's core product is not web data infrastructure, this is often the part worth not owning.

The takeaway

Can LLMs solve CAPTCHAs now? Well enough on the old visual puzzles, not well on the new interactive ones (40% against a human 93%), and it is the wrong question regardless. Modern challenge systems decide with a risk score computed before any puzzle appears, so a solver operates only after you have already been flagged and does not repair the judgement that flagged you. The architecture that lasts does not centre on solving challenges. It centres on not triggering them: consistent fingerprints, clean IP reputation, plausible sessions, and treating a rising challenge rate as the health signal it is. Solve the cause, and the puzzle stops showing up.

FAQ

Can AI or LLMs solve CAPTCHAs?

Partially. Current multimodal LLMs handle classic static visual challenges, such as image grids and distorted text, fairly reliably. They are much weaker on modern interactive, multi-step challenges: on the Open CaptchaWorld benchmark, humans scored 93.3% while the best agent reached 40.0%. So the answer depends heavily on which generation of CAPTCHA you mean.

If LLMs can solve CAPTCHAs, does that defeat CAPTCHA-based protection?

Not really, because modern systems do not rely on the puzzle. reCAPTCHA v3, hCaptcha, and Cloudflare Turnstile compute a risk score from signals like IP reputation, browser fingerprint, session history, and behaviour before deciding whether to show any challenge. Solving a puzzle you were only shown because you looked suspicious does not reset that judgement, so puzzle-solving alone is a weak strategy.

How should a scraping pipeline handle CAPTCHAs then?

Treat them as a symptom, not a step. Design upstream so you rarely trigger a challenge: present a consistent real-browser fingerprint, keep IP reputation clean, reuse sessions plausibly, and pace requests realistically. Log your challenge rate as a health metric so a rising trend warns you early. Keep any challenge-handling as a fallback for residual cases rather than the core mechanism your happy path depends on.

Why Your Playwright Scraper Gets Blocked: TLS Fingerprinting (JA3/JA4) Explained

PromptCloud — Thu, 17 Sep 2026 04:53:25 +0000

You launched a real Chromium, set a real User-Agent, and still got blocked on the first request, before your script even touched the page. The likely culprit sits below HTTP, in the TLS handshake. Here is what JA3 and JA4 actually see, and why the fix is usually not the one people reach for.

You did everything the tutorials said. Real browser via Playwright, a genuine Chrome User-Agent, sensible delays. And the site still served you a challenge page or a 403 on the very first request, before a single line of your scraping logic ran. That timing is the tell. A block that lands that early is not about your behaviour on the page, because you have not done anything yet. It is about how your client looked the instant it opened the connection, and a large part of that first impression is your TLS fingerprint.

What TLS fingerprinting actually is

Before any HTTP request is sent over HTTPS, the client and server complete a TLS handshake. The very first message the client sends is the ClientHello, and it is surprisingly revealing. It advertises the TLS version, the ordered list of cipher suites the client supports, the extensions it offers, the elliptic curves, the signature algorithms, and the ALPN protocols. None of this is your data. It is the shape of your networking stack, decided by whichever library or browser opened the socket.

TLS fingerprinting takes those fields and reduces them to a compact identifier. JA3, the original method, concatenates the key ClientHello fields into a string and hashes it with MD5 to produce a 32-character fingerprint. A real Chrome build produces one recognisable JA3. Python's requests produces a completely different one. Go's net/http produces another. The server can compute your fingerprint before it answers, compare it to a list of known-good browser fingerprints, and decide whether you look like a browser or like a script wearing a browser's User-Agent.

That mismatch is the classic failure. A request whose header says "I am Chrome" but whose TLS handshake says "I am a Python HTTP library" is trivially flagged, because the two halves of the same request disagree about what the client is.

# A JA3 string is built from ordered ClientHello fields:
#   TLSVersion,Ciphers,Extensions,EllipticCurves,ECPointFormats
# e.g.
771,4865-4866-4867-49195-49199-...,0-23-65281-10-11-...,29-23-24,0
# MD5(that) -> e7d705a3286e19ea42f587b344ee6865   <- your fingerprint

# The problem is not that the hash exists.
# It is that a browser User-Agent + a non-browser JA3 = a contradiction.

So why does a real Playwright browser get caught

Here is the part most articles skip. If you are driving a genuine, unmodified Chromium through Playwright, your TLS handshake is Chromium's handshake. The JA3 or JA4 you present is a real browser fingerprint, because a real browser is making the connection. Pure TLS fingerprinting, on its own, does not usually catch vanilla Playwright plus bundled Chromium. So when a real browser still gets blocked early, one of a few specific things is usually going on.

The most common is that somewhere in the stack, the request stopped being made by the browser. Teams reach for Playwright's HTTP request API, or drop down to an HTTP client for the "simple" endpoints to save resources, and those requests carry the library's fingerprint, not the browser's. You now have a session that is part real browser and part script, and the scripted parts are the ones that get flagged.

The second is a TLS-terminating proxy in the path. If your traffic is routed through a proxy that terminates and renegotiates TLS, the ClientHello the destination sees is the proxy's, not Chromium's. Your carefully-real browser fingerprint is replaced by the proxy's fingerprint before it ever reaches the site. This one is easy to miss because everything on your side still looks like Chrome.

The third is that TLS is only the first layer. Modern detection does not stop at JA3. There is an HTTP/2 fingerprint derived from how the client sets up frames, header ordering, and settings; there is the ordering and casing of your HTTP headers; there is IP reputation; and there is the actual browser environment tested with JavaScript once the page loads. A real Chromium can pass the TLS check and still fail one of these, and the block can arrive early enough that TLS gets the blame.

JA3 is fading, and JA4 is why

There is a live reason not to over-index on JA3 specifically. Chrome began randomising the order of its TLS extensions in the ClientHello (the change shipped in Chrome 110, in early 2023). Because JA3 hashes the extensions in the order they appear, that order changing on every connection means a real Chrome now produces a different JA3 each time. JA3 as a stable identifier for Chrome effectively broke.

The industry response is JA4, part of the JA4+ suite developed by John Althouse at FoxIO, the same person behind the original JA3. JA4 is built to survive the randomisation: it sorts the extension list before hashing, so a permuted ClientHello still maps to the same fingerprint, and it is more structured and human-readable than a single MD5 blob. JA4+ extends the idea beyond TLS into a family of fingerprints, including one for HTTP and one for the server side. The practical upshot for a scraper: detection is moving to fingerprints that are harder to accidentally spoof and harder to destabilise, so the days of matching a single JA3 string and calling it solved are ending.

What actually fixes it

The instinct is to hunt for a magic fingerprint to paste in. That is the wrong level to solve this at, because you are fighting a whole family of signals that all have to agree, and they have to keep agreeing as browsers and detection both change.

The durable principle is consistency. Every layer of a request should tell the same story: if the User-Agent says Chrome on Windows, then the TLS handshake, the HTTP/2 fingerprint, the header order, and the JavaScript environment should all be that same Chrome on Windows. The moment two layers disagree, you are detectable, no matter how real any single layer is.

In practice that means making all your requests through the real browser stack rather than mixing in HTTP clients for convenience; being careful that proxies pass TLS through rather than terminating it; and keeping the browser build current so its fingerprint matches the real population of browsers, not a version nobody runs any more. Understanding how a headless browser for web scraping actually presents itself at every layer is what separates a scraper that quietly keeps working from one that gets blocked and leaves you guessing which of a dozen signals gave it away.

This is also, frankly, why a lot of teams stop maintaining this themselves at scale. Keeping every layer consistent across many sources, as both browsers and anti-bot systems keep moving, is ongoing work, not a one-time configuration. It is solvable, but it is a job, not a snippet.

The takeaway

If your Playwright scraper gets blocked before it does anything, look below HTTP. TLS fingerprinting reads the shape of your ClientHello and flags clients whose handshake contradicts their User-Agent. A genuine Chromium usually passes that specific check, so an early block on a real browser normally means either a non-browser request slipped into your session, a proxy rewrote your handshake, or a different layer such as the HTTP/2 fingerprint caught you. JA3 is fading as Chrome randomises its extensions, and JA4 is replacing it. The fix is not a magic string. It is making every layer of the request consistent, and keeping it that way.

FAQ

What is a TLS or JA3/JA4 fingerprint?

It is an identifier computed from the fields in the TLS ClientHello a client sends at the start of an HTTPS handshake: the TLS version, cipher suites, extensions, elliptic curves, and more. JA3 hashes those fields with MD5; JA4 is a newer, more robust successor. A server can read your fingerprint before it answers and tell whether you look like a real browser or like a script, independently of your User-Agent.

Why does my scraper get blocked even with a real browser and a real User-Agent?

Because detection compares layers. A genuine Chromium usually presents a legitimate TLS fingerprint, so an early block normally means something else: a request made by an HTTP client rather than the browser, a proxy that terminated and rewrote your TLS handshake, or a non-TLS signal such as the HTTP/2 fingerprint, header ordering, or IP reputation. The block lands early, so TLS gets blamed, but the real cause is often a mismatch elsewhere.

Is JA3 still used, or has JA4 replaced it?

Both are in use, but JA3 has weakened. Since Chrome began randomising its TLS extension order in early 2023, a real Chrome produces a different JA3 on each connection, which makes JA3 unreliable as a stable browser identifier. JA4, part of the JA4+ suite from the creator of JA3, sorts the extensions before hashing so it survives that randomisation, and detection is steadily moving towards it and its relatives.

The EU AI Act Is Now Enforceable. If You Train AI on Web Data, Read This.

PromptCloud — Tue, 15 Sep 2026 08:21:07 +0000

As of August 2026, the AI Act's rules for general-purpose AI have teeth. If your training or fine-tuning data comes from the web, a few of those rules land directly on your engineering team, not just legal.

For a couple of years the EU AI Act was a thing legal teams talked about and engineers ignored, because nothing was being enforced yet. That changed on August 2, 2026, when the enforcement regime for general-purpose AI (GPAI) obligations came into effect. The provisions are now live, with real penalties attached, and a few of them are not abstract policy. They translate into concrete requirements for how you collect, document, and filter the web data you train on. This is a practical rundown for engineers, not lawyers. It is not legal advice, but it will tell you which parts of your data pipeline the regulation now touches.

What actually changed

The AI Act phases in over several years. The milestone that matters here is that the obligations for providers of general-purpose AI models, and crucially the ability to enforce them, are now in effect. Non-compliance with the GPAI-provider obligations can draw penalties of up to 15 million euros or 3% of global annual turnover, whichever is higher. The separate, higher tier for outright prohibited AI practices runs up to 35 million euros or 7%. Either way, "we'll deal with it later" stopped being a viable position in August.

The Act reaches beyond the EU, too. If your model is placed on the EU market or its output is used in the EU, the obligations can apply regardless of where your team sits. For a lot of teams building on web data, that means it applies whether or not you think of yourselves as a European company.

The three requirements that land on engineering

Most of the AI Act is not about data collection. But three of its requirements turn directly into engineering work if your training data comes from the web.

The first is training-data transparency. GPAI providers must publish a sufficiently detailed public summary of the content used to train the model, using a template the European Commission has now finalized and made mandatory. In practice this means you need provenance for your training data. If your dataset is a pile of scraped content with no record of where each part came from, producing that summary is going to be painful. The engineering implication is that data provenance, tracking which sources contributed to a training set, stops being nice-to-have documentation and becomes a compliance artifact you have to be able to generate.

The second is respecting machine-readable opt-outs. The Act ties into the EU's text-and-data-mining rules, under which rights holders can reserve their content from being mined, including for AI training, through machine-readable means. That means the signals sites publish to opt out, expressed through robots.txt directives and emerging conventions, are not just etiquette anymore; honoring them is part of the compliance posture for training data. The engineering implication is that your crawler needs to actually read and respect those opt-out signals at collection time, and you need to be able to show that it did.

The third is the GDPR overlap, which the AI Act does not replace. European data-protection guidance has made clear that scraping publicly accessible data does not exempt you from GDPR when that data includes personal information. A page being public does not make the personal data on it free to train on without a lawful basis. The engineering implication is that personal data in a training set needs handling, minimization, filtering, or a defensible legal basis, rather than being swept in because it was reachable.

A practical checklist for teams training AI on web data

If your pipeline feeds scraped web data into model training or fine-tuning, here is where to start.

Track provenance from the point of collection. Record, per record or per source, where the data came from and when. You cannot document a training set after the fact if you never captured its origins.

Read and honor opt-out signals in your crawler. Make robots.txt and machine-readable TDM reservations a real gate in collection, not something checked manually and inconsistently. Log the decision so you can demonstrate compliance.

Filter or flag personal data. Build the ability to identify and handle personal data in what you collect, so a training set is not carrying personal information you have no basis to use.

Be able to produce a training-data summary. Assume you will have to describe your training content in the Commission's format, and structure your data catalog so that is a query, not an archaeology project.

Keep an audit trail. Across all of the above, the recurring theme is that you may need to show what you did, not just assert it. Logging collection decisions and data lineage is the connective tissue that makes the rest defensible.

None of this is exotic engineering. It is mostly provenance, filtering, and logging applied to a pipeline that historically had none of them, because collection was optimized for volume rather than accountability. That gap is exactly where the compliance risk now sits, and it is why treating collection as a governed process, with real data governance around provenance and opt-out handling, is becoming part of building AI responsibly rather than a separate legal exercise bolted on at the end.

Who this actually applies to

The obligations fall most directly on providers of general-purpose AI models. But the practical reach is wider. If you fine-tune or build on top of models, the provenance and lawful-basis questions still touch your data. If your product reaches the EU market, the extraterritorial scope can pull you in even from outside Europe. The safe assumption for most teams training on web data at meaningful scale is that some of this applies to you, and the specifics are worth a real conversation with counsel rather than a guess.

The takeaway

The EU AI Act moved from theoretical to enforceable in August 2026, and for teams that train AI on web data, the parts that matter are not buried in legal abstraction. They are provenance, opt-out handling, and personal-data filtering, three things that live in your data pipeline. The teams that built collection for volume and nothing else now have engineering work to do. The good news is that it is ordinary, tractable work, provenance tracking, honoring opt-out signals, filtering personal data, logging decisions, and doing it turns a compliance exposure into a governed pipeline. The risk is real, but so is the fix.

This article is general information for engineers, not legal advice. The AI Act is complex and its application depends on your specific situation. Consult qualified counsel, especially before making decisions about training data, personal data, or EU market placement.

FAQ

Does the EU AI Act apply to scraping data for AI training?

The AI Act does not ban scraping, but it imposes obligations on how training data is documented and sourced for general-purpose AI models. Providers must publish a summary of training content using the Commission's mandatory template, and must respect machine-readable text-and-data-mining opt-outs. Separately, the GDPR still applies to any personal data in scraped content.

What are the penalties for non-compliance?

For breaches of the general-purpose AI provider obligations, penalties can reach 15 million euros or 3% of global annual turnover, whichever is higher. The higher tier, for prohibited AI practices, can reach 35 million euros or 7%. Enforcement of the GPAI obligations took effect in August 2026.

What do engineers actually need to change?

The practical work is provenance (recording where training data came from), honoring machine-readable opt-out signals in the crawler, filtering or lawfully handling personal data, being able to generate a training-data summary in the required format, and logging these decisions for an audit trail. Most of it is provenance, filtering, and logging added to a collection pipeline that historically lacked them.

Build vs Buy Web Scraping in 2026: An Honest TCO Reality Check

PromptCloud — Mon, 07 Sep 2026 09:56:08 +0000

Most build-vs-buy arguments compare the wrong numbers. Here's a cost model that counts what a build actually costs over its life, not what it costs to demo.

Every engineering team that needs web data has this argument eventually. One side says "we can just build this, it's a scraper." The other says "we should buy it." Both sides usually argue from the sticker price: a vendor quote on one side, a rough guess at engineering days on the other. That comparison is almost always wrong, because the expensive part of building a scraper is not building it. It is keeping it working.

This is a total-cost-of-ownership model you can actually use. All the numbers below are illustrative placeholders, clearly marked, so you can drop in your own. The point is not the exact figures. It is the line items most build estimates leave out entirely.

What a build actually costs

A scraper build has one honest one-time cost and a stack of recurring costs that the initial estimate usually ignores. Here is the full shape of it, for an illustrative mid-size operation collecting from ~50 sources of mixed difficulty.

One-time

Initial build (~4-8 eng-weeks): writing the crawlers, parsers, and delivery for the first set of sources.

Recurring, every year

Maintenance engineering (~0.3-0.7 FTE): fixing scrapers when sources change layout, add defenses, or break silently.
Proxy / IP infrastructure (usage-based): residential and datacenter IPs, priced per GB or per request; scales with volume and source difficulty.
Rendering compute (usage-based): headless browser capacity for JS-heavy sources, often the single biggest infra line at scale.
Data QA / validation (~0.1-0.3 FTE): catching silent failures, partial data, stale caches, and challenge pages returned as 200s.
On-call / incident response (overhead on all of the above): someone owns the pager when a feed breaks the night before a report is due.

The one-time build is the number that goes in the slide. Everything below it is the number that actually determines cost, and it recurs every year for as long as you need the data.

A rough annualized model looks like this:

Annual TCO (build) =
    (maintenance_FTE + qa_FTE) * fully_loaded_eng_cost
  + proxy_spend
  + rendering_compute
  + (one_time_build_cost / amortization_years)

# Illustrative, US-loaded eng cost ~ $150k/FTE, substitute your own:
#   (0.5 + 0.2) * 150,000            = 105,000   people
#   proxy_spend                      =  20,000   (varies wildly by volume)
#   rendering_compute                =  25,000   (JS-heavy sources dominate this)
#   build 6 wks @ 150k / 3-yr amort  =   5,800
#   -------------------------------------------------
#   Annual TCO (build)               ~ 155,800 / year

The exact total is yours to compute. What matters is that the honest number is dominated by the recurring people cost, not the build, and the build estimate is the part everyone anchors on.

The multiplier nobody budgets

Here is the line that most quietly wrecks a build estimate: maintenance does not scale linearly with the number of sources. It scales with sources times the sophistication of their defenses, and both are trending up.

The reason is that failures are correlated, not independent. A large share of the web sits behind a small number of anti-bot and CDN vendors, so when one of them ships a change, every source behind it tends to break in the same window rather than on its own schedule. Fifty sources you could handle one at a time become five simultaneous fires in a bad week. Shared infrastructure couples them further: one source moving to heavier client-side rendering quietly eats your headless pool's capacity and slows sources that never changed. So the maintenance FTE you budgeted at "0.3 of an engineer" has a way of becoming 0.7, then a full headcount, as your source list grows and the web's defenses get more active.

That is the number a demo can never show you, because a demo runs for an afternoon, and the maintenance cost only exists over months.

What "buy" actually costs, and what it doesn't

The buy side has an obvious cost: the invoice. It also has a real hidden cost that the vendor pitch skips: integration effort, some lock-in, and less control over exactly how collection happens.

But the buy side removes the entire recurring column above from your team's plate. No maintenance FTE, no proxy management, no rendering infra to size, no on-call for broken feeds. You are converting a variable, growing, people-heavy cost into a fixed line item. That trade is the actual decision, and it is a very different decision than "invoice vs a few engineering days."

The honest framing is not "building is cheaper" or "buying is cheaper." It is: building trades a predictable vendor cost for an unpredictable internal one that grows with your source count and the web's hostility, and buying does the reverse.

The honest decision framework

Strip away the vendor marketing and the not-invented-here instinct, and it comes down to a few real questions.

Build when web data collection is core to your product, your source list is small and stable, and you have engineers who want to own this as a specialty rather than a chore. If the data is the product, owning the collection is often right.

Buy when web data is an input to your product rather than the product itself, your source list is large or growing, your sources are defended and changing, and your engineers' time is better spent on the thing the data feeds. That is most teams, most of the time, which is exactly why this argument recurs.

Working through it honestly, with your own numbers rather than a gut estimate, is the whole point. This interactive breakdown of the build vs buy decision is a good way to run the model against your actual source count and volume, because the answer genuinely changes with scale, and a framework beats an argument.

The takeaway

The build-vs-buy question is not really about capability. Almost any competent team can build a scraper. It is about the total cost of keeping dozens of them alive against a web that is actively getting harder to collect from, and whether that recurring, correlated, growing maintenance load is a cost you want on your own roadmap. Count the whole column, not just the build, and the decision usually makes itself.

FAQ

Is it cheaper to build or buy web scraping?
It depends on scale and stability. Building is often cheaper for a small, stable set of sources where data collection is core to the product. Buying is usually cheaper in total cost of ownership once the source list is large, growing, or defended, because the recurring maintenance, proxy, and rendering costs dominate the one-time build cost.

What costs do build estimates usually miss?
The recurring ones: ongoing maintenance engineering to fix scrapers when sources change, proxy and IP infrastructure, headless-rendering compute for JS-heavy sites, data QA to catch silent failures, and on-call. These typically exceed the one-time build cost within the first year.

Why does scraper maintenance grow faster than the number of sources? Because failures are correlated. Many sites share the same anti-bot and CDN vendors, so a single change upstream can break many sources at once, and shared rendering infrastructure couples sources together. Maintenance tracks sources times their shared dependencies, not source count alone.