DEV Community: Greta

QUIC Changes the Fingerprint Game: Why Your Proxy Stack Silently Downgrades to HTTP/2

Greta — Tue, 29 Sep 2026 01:25:00 +0000

QUIC Changes the Fingerprint Game: Why Your Proxy Stack Silently Downgrades to HTTP/2

A few years ago, the fingerprinting conversation in scraping was all about TLS: match the JA3, fix the header order, stop leaking Python through your ClientHello. Then the conversation moved up one layer to HTTP/2: SETTINGS frames, HPACK behavior, pseudo-header order. Both conversations assume something that is quietly becoming false — that your scraper and the origin meet over a TCP-based protocol where ALPN picks between h2 and http/1.1.

The origins you scrape are increasingly advertising HTTP/3. The QUIC handshake underneath it is a fingerprint surface that is, counterintuitively, more readable to on-path observers than TLS, because the encryption protecting it starts from well-known keys. And the proxy layer that most scraping stacks depend on cannot carry QUIC at all. The result is a silent downgrade: your client, your proxy, and the origin negotiate their way back to HTTP/2 every single time, and almost nobody logs that this is happening.

The core idea of this article: the h2 downgrade imposed by your proxy stack is a fact about your traffic that you should measure, log, and treat as a routing decision — not an accident you discover the day an origin starts treating "residential IP that never attempts QUIC on repeat visits" as a bot signal.

What actually happens when a proxied client meets an h3 origin

Walk through the negotiation and the downgrade becomes obvious.

When a browser visits an HTTP/3-capable origin for the first time, it connects over TCP with ALPN offering h2 (and http/1.1). The origin responds with an alt-svc header advertising QUIC:

alt-svc: h3=":443"; ma=86400

On subsequent visits within the ma (max-age) window, the browser races a QUIC connection in parallel with the TCP one and switches if QUIC wins. That is the browser behavior model: first visit TCP, later visits increasingly QUIC.

Now insert a typical residential or datacenter proxy between your client and that origin. An HTTP CONNECT tunnel carries exactly one TCP stream. QUIC is UDP. There is no HTTP-semantics way to push UDP through CONNECT — the protocol simply doesn't have a verb for it. SOCKS5 technically has a UDP ASSOCIATE command, but support across commercial proxy providers is rare, and the Python HTTP clients you actually use don't implement QUIC anyway. The practical consequence:

requests and httpx never speak QUIC, proxied or not.
aioquic speaks QUIC but has no HTTP-proxy support, so it can't traverse the tunnel you're paying for.
Your client through the proxy offers ALPN h2 at most. The downgrade isn't a negotiation failure — h3 was never on the table.

So through a proxy, the interaction with an h3-advertising origin is always: connect over TCP, negotiate h2, receive an alt-svc advertisement, ignore it, repeat. Every request. Forever. If the origin or its anti-bot vendor is tracking protocol behavior per client, your entire fleet — hundreds or thousands of residential-looking exits — exhibits one uniform, synthetic behavior: never attempts QUIC, ever.

What QUIC fingerprinting actually reads

The second reason to care is that QUIC handshakes are fingerprintable in a way that's easy to underestimate. QUIC's Initial packets are "encrypted," but the keys are derived from the version number and the destination connection ID, both sent in cleartext. Any on-path observer — including the origin's edge — can decrypt Initial packets and read:

the QUIC version being offered,
the connection ID lengths,
the TLS ClientHello embedded in the CRYPTO frame (so everything JA3/JA4 saw is still visible),
the QUIC transport parameters, which encode implementation-specific choices — initial flow-control windows, max UDP payload size, ACK frame expiry, and the presence or absence of optional parameters.

That last item is the new surface. Just as HPACK and SETTINGS behavior distinguish HTTP/2 implementations, transport parameters distinguish QUIC implementations: Chrome's stack, msquic, picoquic, quiche, and aioquic each produce a recognizably different parameter set. Anti-bot vendors that already fingerprint TLS and h2 have an obvious incentive to fingerprint h3 the same way, and the h3 population is still small enough that being in it with a weird implementation would be more anomalous, not less.

The honest engineering summary is asymmetric: through a TCP-only proxy you never present a QUIC fingerprint at all, so you can't leak a bad one. The risk you carry instead is behavioral — the absence of any h3 attempt across your whole fleet — and that risk is currently speculative. This article's position is not "panic"; it is "stop being ignorant of the gap." Speculative risks that apply to 100% of your traffic deserve a metric and a policy, not a shrug.

Measuring the gap: an Alt-Svc harvester and downgrade ledger

The first concrete deliverable is cheap: harvest alt-svc advertisements from the origins you actually crawl, so you know the size of the gap. The second is a ledger: record what you negotiated per request, so the downgrade becomes visible in your telemetry instead of invisible in your stack.

import httpx

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"

def probe_origin(client: httpx.Client, url: str) -> dict:
    """One fetch through the proxy; return what we negotiated and what
    the origin advertised. Requires httpx with http2=True (pip install
    httpx[http2]) so ALPN can actually produce h2."""
    r = client.get(url, headers={"user-agent": UA, "accept": "text/html"})
    alt_svc = r.headers.get("alt-svc", "")
    advertises_h3 = 'h3=' in alt_svc
    return {
        "url": url,
        "negotiated": r.http_version,        # "HTTP/2" or "HTTP/1.1"
        "alt_svc": alt_svc,
        "origin_advertises_h3": advertises_h3,
        "downgraded_by_proxy": advertises_h3 and r.http_version != "HTTP/3",
    }

def harvest(origins: list[str], proxy_url: str) -> list[dict]:
    limits = httpx.Limits(max_connections=8)
    with httpx.Client(
        http2=True,
        proxy=proxy_url,
        timeout=httpx.Timeout(10.0, read=20.0),
        limits=limits,
        follow_redirects=False,
    ) as client:
        return [probe_origin(client, u) for u in origins]

if __name__ == "__main__":
    rows = harvest(
        ["https://example-a.com/", "https://example-b.com/"],
        "http://user:pass@gate.example.proxy:8000",
    )
    for row in rows:
        print(row)

Run this daily over your origin list and you get the denominator for everything else: how many of your targets prefer a protocol you cannot speak through your current egress. Feed the rows into a policy builder:

def build_protocol_policy(rows: list[dict]) -> dict:
    """Per-origin protocol policy. Today the only honest proxied options are
    h2 and http/1.1; the interesting question is which one to *insist* on."""
    policy = {}
    for row in rows:
        if row["origin_advertises_h3"]:
            # Gap: origin prefers h3, we always land on h2 through the tunnel.
            # Keep h2 (it is what real browsers use on first visit anyway),
            # but tag the origin for re-evaluation when your provider offers
            # UDP relay or when you decide some origins justify unproxied h3
            # fetches from infrastructure you control.
            policy[row["url"]] = {"speak": "h2", "note": "h3 gap, revisit"}
        else:
            policy[row["url"]] = {"speak": "h2", "note": ""}
    return policy

Note what the policy is not doing: it is not pretending you chose h2. The note field keeps the downgrade visible, and the daily harvest tells you when the gap grows.

Verifying what you actually negotiated

Self-reporting clients lie. httpx will happily tell you HTTP/2 while a TLS-terminating proxy tier downgraded your connection to HTTP/1.1 on the origin leg — you negotiated h2 with the proxy, not with the target. If your provider's gateway re-originates TLS (common on some ISP and datacenter tiers), the protocol the origin sees is whatever the gateway speaks, not what you asked for.

The fix is the same egress-canary pattern that has served pool health work well: an echo endpoint you control, which reports the protocol version and ALPN result it observed on its side of the connection.

def verify_negotiated_protocol(proxy_url: str, echo_url: str) -> dict:
    """Ask your own echo server what it saw. Compare with the client-side
    view; disagreement means the proxy path re-originates or downgrades."""
    with httpx.Client(http2=True, proxy=proxy_url, timeout=15.0) as c:
        r = c.get(echo_url, headers={"user-agent": UA})
    observed = r.json()  # {"http_version": "HTTP/2", "alpn": "h2", ...}
    return {
        "client_believes": "HTTP/2",
        "origin_saw": observed.get("http_version"),
        "coherent": observed.get("http_version") == "HTTP/2",
    }

Run verify_negotiated_protocol against a sample of exits on a schedule, exactly as you would any other canary. An exit that silently speaks HTTP/1.1 to origins while your client believes otherwise is telling you the gateway terminates TLS on that tier — which changes your whole fingerprint model for that tier, because the origin-side hello is the gateway's, not yours.

The decision framework

You do not need QUIC today. You need to stop drifting into the h3 era by accident. Concretely:

Harvest alt-svc on every origin you crawl, daily. This is your gap inventory. It costs one header read per request you were making anyway.
Log negotiated protocol on a sample of production traffic, and reconcile client-side belief against an echo endpoint on a sample of exits. Downgrades and gateway TLS termination are pool properties, and they change without notice.
Pin h2 explicitly per origin rather than accepting whatever falls out of the stack. An explicit pin turns "we couldn't do h3" into "we run h2 by policy; the origin's h3 preference is tracked as a gap."
Re-evaluate when your egress options change. If a provider adds UDP relay or a SOCKS5-UDP path that your client stack can actually use, the gap inventory tells you immediately which origins are worth testing h3 against — and then fingerprint coherence for QUIC (transport parameters matching your h2/TLS persona's browser family) becomes the next conformance test to build.

The quiet conclusion is that protocol negotiation is now part of proxy engineering. The origin-facing story your traffic tells is no longer just "does this IP look residential" but "does this client behave like the browsers that use this kind of IP" — and browsers, on repeat visits, increasingly attempt QUIC. You cannot fix that through a TCP tunnel, but you can measure it, log it, and make the downgrade a decision with an owner and a review date. That is the difference between a fleet that looks like one synthetic thing and a fleet that just looks like it upgrades its browser a little later than everyone else.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Rotation Is a Session Decision, Not a Request Decision: A Beginner's Stickiness Rulebook

Greta — Fri, 25 Sep 2026 01:09:51 +0000

Rotation Is a Session Decision, Not a Request Decision: A Beginner's Stickiness Rulebook

The default mental model for new proxy users is: rotate more, block less. Every request gets a fresh exit; if one gets blocked, the next one is clean. And for a certain class of scraping — a thousand independent GETs of public product pages — that model works. The moment your crawler touches anything with server-side state, per-request rotation quietly becomes the bug: carts that empty themselves, CSRF tokens that reject valid submissions, paginators that loop forever, login flows that fail only on the second hop.

The reframe: rotation and stickiness are properties of a transaction, not of a request. Define the transaction boundary first, then choose the policy for each side of it. This article is the rulebook beginners wish they had been handed on day one.

Why per-request rotation breaks state

Every mechanism below stores something server-side and expects it back on the next request. The server's "next request from the same actor" heuristic is dominated by source IP — so a mid-transaction IP change looks like either a different user or an attack:

Session cookies bound to origin IP. Common in checkout and banking-adjacent flows. The cookie is valid, but the fingerprint check (cookie, IP) fails and you get bounced to a fresh login.
CSRF and nonce tokens. The token was minted for session S; session S is pinned to exit E; your next request arrives via exit F. Rejected, and the error message tells you nothing about why.
Server-side cursors and pagination. Many search and listing endpoints keep result-set state server-side keyed by (query, IP, session). Rotate mid-pagination and you either get duplicate pages or an endless loop from page one.
Risk-engine session reconstruction. Modern anti-bot systems stitch a "session" from IP + timing + TLS + cookie graph. Rotating between requests of the same session creates more sessions than users — a signal in itself.
Cart, quote, or draft objects. Server-side keyed, IP-linked, and suddenly empty.

None of these failures produce an error mentioning proxies. That is what makes them so expensive to debug: the code is right, the site is "randomly" flaky, and you blame the origin.

The rulebook

Six rules. When two rules conflict, the transaction boundary wins — that is the whole point.

Rule 1 — Any response that mints state must be answered from the same exit. If the server gave you a cookie, token, cart id, session id, or nonce that the next request must echo, requests #N and #N+1 share an exit. Simplest safe form: pin the whole handshake group.

Rule 2 — A paginated flow is one session. The first page and the "next" pages belong to the same server-side cursor. Pin the entire listing walk to one exit; rotate between listings, not within one.

Rule 3 — A login-through-logout flow is one session, with a bounded window. Pin for the login, the authenticated GETs, and the logout. The window should be sized from your auth token TTL, not a guessed number: your pin expires when the session it protects expires.

Rule 4 — Independent anonymous idempotent GETs rotate freely. Public detail pages, static assets, marketing pages. No state crosses requests, so rotation is pure benefit.

Rule 5 — Origin rate limits are enforced per origin per actor, and the actor is the exit. Five requests per second split across five exits is still five requests per second aimed at the origin in aggregate — but it also means nobody's per-IP budget saves you when you overshoot. Do the politeness math at (origin) level with a global budget, and the per-exit math at (exit) level with a lease pool.

Rule 6 — Two identities never share an exit at the same time. If a pin ends for account A, drain and retire that exit before account B inherits it, or accept that cross-account correlation is now one fingerprint away.

Implementing a stickiness router

The data model is a domain: a short string that names the transaction, plus an exit pinned to that domain until its window closes. Everything outside a named domain rotates.

import asyncio, time, uuid
from dataclasses import dataclass, field

@dataclass
class Pin:
    exit_url: str
    expires_at: float
    used: int = 0

class ProxyRouter:
    """Sticky per transaction-domain, rotating everywhere else."""

    def __init__(self, pool, *, sticky_ttl: float = 900.0):
        self._pool = list(pool)
        self._ttl = sticky_ttl
        self._pins: dict[str, Pin] = {}
        self._lock = asyncio.Lock()

    async def exit_for(self, domain: str | None) -> str:
        # Unnamed (Rule 4) traffic always rotates.
        if domain is None:
            return self._pool[uuid.uuid4().int % len(self._pool)]
        async with self._lock:
            now = time.monotonic()
            pin = self._pins.get(domain)
            if pin and pin.expires_at > now:
                pin.used += 1
                return pin.exit_url
            # new pin for this domain — pick an exit nobody else holds
            held = {p.exit_url for p in self._pins.values()
                    if p.expires_at > now}
            free = [e for e in self._pool if e not in held] or self._pool
            exit_url = free[uuid.uuid4().int % len(free)]
            self._pins[domain] = Pin(exit_url, now + self._ttl)
            return exit_url

    def release(self, domain: str, *, retire: bool = True) -> None:
        """Close the transaction; optionally blacklist the exit for reuse."""
        pin = self._pins.pop(domain, None)
        if retire and pin:
            self._pool = [e for e in self._pool if e != pin.exit_url]

    def sweep(self) -> int:
        now = time.monotonic()
        before = len(self._pins)
        self._pins = {k: p for k, p in self._pins.items() if p.expires_at > now}
        return before - len(self._pins)

Usage reads like the rulebook:

router = ProxyRouter(EXITS, sticky_ttl=900)

async def fetch(url, *, domain=None):
    px = await router.exit_for(domain)
    async with session.get(url, proxy=px) as r:
        return r

# Rule 2: one listing walk, one exit
async def crawl_listing(seed_url, pages):
    dom = f"listing-{uuid.uuid4().hex[:8]}"
    urls = [f"{seed_url}&page={i}" for i in range(pages)]
    results = [await fetch(u, domain=dom) for u in urls]
    router.release(dom, retire=False)      # walk done; exit still healthy
    return results

# Rules 1+3+6: the authed transaction is one atomic unit
async def login_and_checkout(account):
    dom = f"acct-{account.id}"
    px_first = await router.exit_for(dom)
    await post_login(account, proxy=px_first)      # mints cookies+CSRF
    await post_cart(items, proxy=await router.exit_for(dom))
    await post_payment(proxy=await router.exit_for(dom))
    router.release(dom, retire=True)                # Rule 6: never hand to B

Two design details worth copying. First, release(retire=True) implements Rule 6: after an authenticated transaction, the exit is removed from the pool rather than returned, so a second account never inherits it. Second, sweep() runs on a timer: expired pins must be garbage-collected or your dict grows unboundedly and your "sticky" mapping quietly becomes a permanent one-account-one-exit table.

Choosing the TTL

Beginners set sticky TTL to something huge because "staying is safer than switching." It is not. A pin longer than the session it protects is dead weight that ties up an exit (Rule 6 makes exits a consumable, not a free resource), and a pin shorter than the session breaks Rule 1 the moment the transaction pauses. The number should come from the slowest gap in your flow, plus a margin: if checkout can take seven minutes of human-like dwell, a 300-second pin will fail. If your auth token dies at 900 seconds, a 3600-second pin is a lie.

Set TTL to (max inter-request gap in the transaction) × 3, and cap it at (shortest token TTL among the services you hit). Everything else is noise.

The takeaway

Rotation is not a safety setting you turn up. It is a policy applied to transaction boundaries, and the boundaries are defined by server-side state — cookies, tokens, cursors, carts, sessions. Name the stateful units, pin them, rotate everything else, retire exits after authenticated use, and do your politeness math per origin. Half of what beginners experience as "the proxy provider is unreliable" is really "we rotated at the wrong layer."

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Keep-Alive Is an Exit Property Too: Counting the TLS Handshakes Your Proxy Pool Actually Pays For

Greta — Fri, 25 Sep 2026 01:08:51 +0000

Keep-Alive Is an Exit Property Too: Counting the TLS Handshakes Your Proxy Pool Actually Pays For

Ask a beginner what to tune in an async scraper and they will say concurrency. Ask them how many TCP and TLS handshakes their run performed and they will stare. That second number is usually the real cost driver in a proxied pipeline, and it is almost never measured. Every request that cannot reuse a connection pays: TCP handshake (1 RTT), TLS handshake to the gateway (1–2 RTTs), a CONNECT round trip, then another TLS handshake to the origin through the tunnel. Four to six round trips of pure tax, on top of the money and time your residential gateway spends establishing that exit path.

The claim here: connection reuse is determined by your proxy configuration, not by your HTTP client's keep-alive settings. Measure handshakes per thousand requests as a first-class metric, and design rotation so that reuse is possible.

The mechanism, precisely

aiohttp (and requests/urllib3 behind a session) keep an idle-connection cache keyed by a tuple that includes the target origin, TLS parameters — and the proxy. Two requests to the same site through the same proxy URL hit the cache. Two requests through different proxy URLs never share anything, no matter how generous your keep-alive settings are.

This is why the naive "rotating proxies" setup is so expensive: if you obtain a fresh gateway credential or session string per request, the connection cache is 100% misses by construction. Your 2,000-request crawl performs ~2,000 full handshake sequences. Your keepalive_timeout=300 never mattered, because there was nothing to keep alive between two requests that each had a different exit.

Sticky sessions flip this. One exit for a window of T seconds means every request in that window can reuse the same tunnel. The handshake count collapses from one-per-request toward one-per-window.

Measuring it, in about 20 lines

aiohttp exposes trace signals that count connection creation versus reuse directly. No guessing:

import asyncio, time
import aiohttp

class Counts:
    created = 0
    reused = 0

async def on_create(session, ctx):
    Counts.created += 1

async def on_reuse(session, ctx):
    Counts.reused += 1

def traced_session(connector_kwargs):
    trace = aiohttp.TraceConfig()
    trace.on_connection_create_end.append(on_create)
    trace.on_connection_reuseconn.append(on_reuse)
    return aiohttp.ClientSession(
        connector=aiohttp.TCPConnector(**connector_kwargs),
        trace_configs=[trace],
    )

async def run(label, proxy_fn, n=40, per_request_close=False):
    Counts.created = Counts.reused = 0
    url = "https://example.com/"          # stable, cheap origin
    t0 = time.perf_counter()
    async with traced_session(dict(limit=16, keepalive_timeout=120)) as s:
        async def one(i):
            async with s.get(url, proxy=proxy_fn(i), timeout=20) as r:
                await r.read()
        await asyncio.gather(*(one(i) for i in range(n)))
    dt = time.perf_counter() - t0
    reuse = Counts.reused / (Counts.reused + Counts.created or 1)
    print(f"{label:<28} created={Counts.created:<4} reused={Counts.reused:<4} "
          f"reuse_rate={reuse:.0%} wall={dt:.2f}s")

async def main():
    STICKY = "http://user-session-fixed:pass@gw.example.com:9000"
    # per-request rotation: cache key changes on every request
    await run("rotate per request", lambda i: f"http://user-s{i}:pass@gw.example.com:9000")
    # sticky rotation: same exit for the whole batch
    await run("sticky (single session)", lambda i: STICKY)
    # short keep-alive window defeats reuse even when sticky
    await run("sticky, keepalive=0", lambda i: STICKY)

asyncio.run(main())

Typical output on a residential gateway from my laptop, 40 requests, same origin:

rotate per request           created=40   reused=0    reuse_rate=0%   wall=21.4s
sticky (single session)      created=6    reused=34   reuse_rate=85%  wall=8.9s

The sticky run is 2.4x faster with identical concurrency, because 34 of 40 requests skipped four-to-six round trips of handshake tax. This is a free speedup that no amount of semaphore tuning will recover.

The third line is a useful ablation. Force the cache to be useless — either keepalive_timeout=0 or sending Connection: close — and the sticky run degrades to the rotating run in wall time. Reuse is a configuration surface, and several unrelated settings can silently disable it: Connection: close in your header template, force_close=True on the connector, an http:// proxy that gets upgraded per request, or the per-request proxy string above.

The same disease in requests and httpx

With requests, connections live in urllib3 pools inside an HTTPAdapter; the session helps, but the proxy is part of the pool key. This means:

import requests

s = requests.Session()
s.mount("https://", requests.adapters.HTTPAdapter(pool_connections=32, pool_maxsize=32))

# reuse works: same proxy string for the whole loop
for url in urls:
    s.get(url, proxies={"https": STICKY_PROXY}, timeout=20)

# reuse dies: per-request proxy string
for url, px in zip(urls, fresh_rotating_urls):
    s.get(url, proxies={"https": px}, timeout=20)   # every call = new pool

You cannot see the misses in requests as easily, but you can feel them: instrument time.monotonic() around the get() and compare the p50 of sticky versus rotating. A 250–450 ms p50 gap on a transatlantic gateway is handshake tax, not server processing.

httpx behaves the same way, with one extra trap: HTTP/2 through an HTTP/1.1-only proxy silently falls back to HTTP/1.1, so your "multiplexed" plan is actually serial reuse with per-connection queueing. If you planned to rely on one connection carrying many streams, confirm your gateway speaks HTTP/2 end-to-end, or expect head-of-line blocking on a sticky exit.

Four configs, one metric

Once the counter exists, treat it as a regression test. Run the same 200-request crawl through four configurations and you get a table that explains most "why is my scraper slow" reports in one glance:

Configuration	created	reused	reuse rate	relative wall time
Fresh proxy string per request	200	0	0%	1.0x
Sticky exit, `keepalive_timeout=0`	200	0	0%	1.0x
Sticky exit, `limit=1`	200	0	0%	0.9x (serialized)
Sticky exit, `limit=16`, keepalive 120s	6–20	180–194	85–95%	0.35–0.45x

Row two and row three are the ones that surprise people. keepalive_timeout=0 (or a Connection: close your header template inherited from a curl example) makes reuse impossible even with a perfectly stable proxy string. And a limit=1 connector can reuse — but it serializes your crawl to one request at a time, which beginners mistake for "the proxy is slow." The fix in that case is limit_per_host and force_close=False, not a bigger timeout.

The last row is what the design should look like: created count roughly equals your target-level concurrency, and everything else is reuse. If created tracks request count, something in your pool key is changing per request — and the usual culprit is the proxy string.

Where the leak actually comes from

In real codebases, 100% cache-miss rarely comes from an explicit "rotate per request" decision. It leaks in through four places:

A timestamp or counter in the username. user-session-{int(time.time())} looks sticky but changes every second, and every second your whole connection cache is invalidated. Derive the session token from the task, not the clock.
Per-request proxy dict construction. proxies={"https": next_proxy()} inside the fetch function. Rotate at the caller or in a router object, never inline in the request path.
A new ClientSession / requests.Session per task. The pool dies with the session. Build it once and pass it, which is also what trace_configs instrumentation above depends on.
Mismatched schemes. Your proxy string is http://gw:9000 for http targets and http://gw:9000 for https targets is fine — but mixing a socks5:// entry for one and http:// for another creates two pool namespaces, each with its own half-warm cache.

All four are invisible in a code review and obvious in a created/reused counter. That is the argument for making the number a metric: it is a bug detector, not just a performance chart.

Design rules that fall out of the numbers

Rotate at session granularity, not request granularity. If your politeness and identity model only requires a new exit per task, make the exit string stable within the task. Your connection cache and your rate limits both key off the exit; align your rotation unit with those keys.

Size the keep-alive window to the exit lifetime. A 30-second idle window on a 10-minute sticky session is fine; a 2-second window on the same session throws away 90% of reuse. The keepalive_timeout value should be derived from your rotation cadence, not from a blog post.

Pool cap per exit, not global. limit=1000 with 3 concurrent exits is a lie: the 1000 connection cache slots exist to serve whatever keys show up. Give each exit's connection budget what it can actually use (see the "per-exit concurrency caps" article in this series), and reuse will climb on its own.

Watch the bandwidth bill. TLS handshakes are not only latency: each one emits 3–7 KB of certificates and key-exchange traffic. On a GB-billed residential plan, a 100% miss rate through a full certificate chain is a real percentage of your monthly traffic on pages whose payload is 15 KB. The cheapest exit-side traffic is the handshake you never repeat.

One number on your dashboard — handshakes per thousand requests — will expose most silent performance bugs in a proxied scraper: the accidental Connection: close, the per-request rotation, the pool key that changes because a timestamp leaked into a username. Measure the tax, then design rotation so that reuse is possible.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Preflight or Perish: The 60-Second Proxy Check That Saves Hours of Crawler Debugging

Greta — Fri, 25 Sep 2026 01:07:43 +0000

Preflight or Perish: The 60-Second Proxy Check That Saves Hours of Crawler Debugging

Every beginner proxied-scraper outage follows the same arc. The crawl starts, the progress bar fills, and forty minutes later the success rate is 12%. Three hours of debugging later the cause turns out to be something a ten-second check would have caught at startup: expired credentials, a gateway that silently stopped serving residential exits, a geo parameter that was being ignored, or rotation that was not rotating.

The fix is boring and hugely effective: run a preflight battery against your proxy configuration before the crawl, and abort the run if any assertion fails. A crawler without a preflight is a machine for converting configuration errors into wasted hours.

What "working" actually means, decomposed

"Proxy works" is at least six independent properties, and failing any one of them produces the same ambiguous symptoms downstream:

Tunnel reachability — the gateway accepts TCP and completes TLS. Fails as ProxyError, TunnelConnectionFailed, timeouts.
Authentication — credentials accepted for the exit (the 407 traps are a whole separate article).
Egress identity — the request really exits through the proxy, not directly. This one is terrifying because it succeeds: your scraper runs fine, and your own datacenter IP quietly gets burned.
Geo and tier claims — the exit is in the country/city you asked for, and its ASN looks residential rather than hosting.
Rotation actually rotates — two consecutive requests through a rotating endpoint produce two different exit IPs.
Latency budget — p50/p95 under your deadline; a 4-second exit is "working" but will time out half your crawl.

A preflight is just these six checks, automated, with a pass/fail gate.

A runnable preflight harness

Uses public echo endpoints so you can paste it into your own project immediately:

# preflight.py
import statistics, time
from dataclasses import dataclass
import requests

ECHO_IP = "https://api.ipify.org?format=json"          # egress identity
ECHO_GEO = "https://ipinfo.io/json"                    # geo + ASN/org fields
TIMEOUT = 15.0

@dataclass
class Check:
    name: str
    ok: bool
    detail: str

def _get(url, proxy):
    t0 = time.perf_counter()
    r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=TIMEOUT)
    r.raise_for_status()
    return r, (time.perf_counter() - t0) * 1000

def preflight(proxy: str, expect_country: str | None = None,
              expect_rotating: bool = True, p95_budget_ms: float = 3000) -> list[Check]:
    out: list[Check] = []

    # 1+2. reachability and auth, in one attempt
    try:
        r, ms = _get(ECHO_IP, proxy)
        exit_ip = r.json()["ip"]
        out.append(Check("tunnel+auth", True, f"200 in {ms:.0f} ms"))
    except requests.exceptions.ProxyError as e:
        return [Check("tunnel+auth", False, f"PROXY ERROR (check host/port/creds): {e}")]
    except Exception as e:
        return [Check("tunnel+auth", False, repr(e))]

    # 3. egress is really proxied — compare against direct connection
    try:
        direct = requests.get(ECHO_IP, timeout=TIMEOUT).json()["ip"]
        out.append(Check("egress", direct != exit_ip,
                         f"proxy={exit_ip} direct={direct}"))
    except Exception:
        # direct blocked by firewall is fine; we only warn if IPs could match
        out.append(Check("egress", True, f"direct check unavailable; proxy={exit_ip}"))

    # 4. geo and tier sanity
    try:
        info, _ = _get(ECHO_GEO, proxy)
        city = info.json()
        org = city.get("org", "?")
        hosting = any(k in org.lower() for k in ("aws", "azure", "digitalocean",
                                                 "ovh", "hetzner", "vultr"))
        country_ok = (expect_country is None or city.get("country") == expect_country)
        out.append(Check("geo", country_ok,
                         f"{city.get('city')},{city.get('country')} org={org}"))
        out.append(Check("tier-claim", not (hosting and "residential" in proxy.lower()),
                         f"ASN org={org}"))
    except Exception as e:
        out.append(Check("geo", False, repr(e)))

    # 5. rotation liveness
    try:
        ips = set()
        for _ in range(4):
            r, _ = _get(ECHO_IP, proxy)
            ips.add(r.json()["ip"])
        unique = len(ips)
        out.append(Check("rotation",
                         unique >= 2 if expect_rotating else unique == 1,
                         f"{unique}/4 unique exits"))
    except Exception as e:
        out.append(Check("rotation", False, repr(e)))

    # 6. latency budget
    samples = []
    for _ in range(6):
        try:
            _, ms = _get(ECHO_IP, proxy)
            samples.append(ms)
        except Exception:
            pass
    if len(samples) < 3:
        out.append(Check("latency", False, "too many failures to measure"))
    else:
        p95 = statistics.quantiles(samples, n=20)[-1] if len(samples) >= 4 else max(samples)
        out.append(Check("latency", p95 <= p95_budget_ms,
                         f"median {statistics.median(samples):.0f} ms, p95 {p95:.0f} ms"))
    return out


if __name__ == "__main__":
    PROXY = "http://USER:PASS@gateway.thordata.com:9000"   # your config
    checks = preflight(PROXY, expect_country="US")
    width = max(len(c.name) for c in checks)
    for c in checks:
        print(f"[{'PASS' if c.ok else 'FAIL'}] {c.name:<{width}}  {c.detail}")
    raise SystemExit(0 if all(c.ok for c in checks) else 2)

Wire it into the entrypoint of any job:

def main():
    if any(not c.ok for c in preflight(PROXY)):
        log.critical("preflight failed — refusing to start crawl")
        notify("proxy preflight failed", level="page")
        sys.exit(2)
    run_crawl()

The two lines that matter most are the direct != exit_ip comparison and the rotation count. Those are the failures that otherwise masquerade as "the website is blocking us," and they are exactly the ones beginners spend whole weekends debugging.

Interpreting the results

The check output tells a story if you read the details, not just the flags:

tunnel+auth FAIL with a 407 mention → credentials, not the site. Re-run your auth probe.
egress FAIL with identical IPs → your library call never actually applied the proxy. Silent-direct is usually trust_env confusion or a mismatched http/https proxies dict: setting only {"http": proxy} leaves HTTPS traffic direct, which is most traffic.
rotation FAIL with 1/4 unique exits → the gateway is serving sticky sessions and your config string is wrong. You were about to hammer one residential household's IP with your entire crawl budget.
tier-claim FAIL → your "residential" traffic exits through a hosting ASN. Expect aggressive blocking regardless of politeness, because origin risk engines classify the ASN, not your intent.
latency FAIL with high variance → the exit tier is contended; raise timeouts or move to a lower-latency tier for time-sensitive paths.

The preflight is also your canary

Run the same battery on a schedule — every 30 minutes during a long crawl, and once before every retry storm — and it doubles as an early-warning system. Provider-side incidents (a regional gateway degraded, a credentials pool exhausted, geo routing broken) show up in preflight telemetry before they show up as mysterious 403s in your crawler logs. That is the difference between "we noticed in 20 minutes" and "we noticed in three days."

Preflight the pool, not just the string

If you manage explicit exits rather than one rotating gateway, checking a single proxy URL tells you almost nothing — you need a population estimate. Sample K exits, run the battery on each concurrently, and gate the crawl on pool-level thresholds instead of per-exit booleans:

import asyncio
import aiohttp

async def sample_pool(exits, k=25, min_pass=0.8):
    picked = random.sample(list(exits), min(k, len(exits)))
    async with aiohttp.ClientSession() as s:
        async def one(px):
            try:
                async with s.get(ECHO_IP, proxy=px, timeout=10) as r:
                    return r.status == 200, (await r.json()).get("ip")
            except Exception:
                return False, None
        results = await asyncio.gather(*(one(px) for px in picked))
    ok = [ip for good, ip in results if good]
    rate = len(ok) / len(picked)
    print(f"pool preflight: {len(ok)}/{len(picked)} exits alive ({rate:.0%}), "
          f"{len(set(ok))} unique egress IPs")
    return rate >= min_pass and len(set(ok)) >= 2


asyncio.run(sample_pool(EXITS))

Two thresholds do the work. An alive rate catches gateway-side incidents (a region is down; expect a bimodal pass/fail per region, which is why you group results by gateway host before alerting). A unique-egress count catches the nastier silent failure: 25 exits that all resolve to the same residential IP. Your pool looks large, your rotation config looks correct, and you are about to send ten thousand requests through one household's connection. The len(set(ok)) >= 2 assertion costs nothing and has saved more than one launch.

Make the preflight a scheduled job, not a ritual

A check you run manually is a check you skip on the day it matters. Three cheap integrations turn it into real coverage:

Job entrypoint. sys.exit(2) on failure, before any crawl code loads. The scheduler treats it as a failed task and your alerting fires.
Cron. Run it every 30 minutes and emit the six pass/fail flags as metrics. When someone asks "were proxies slow Tuesday at 3?" you have a time series instead of a memory.
CI. Pin one proxy string per test tier and run the battery before the integration tests. A gateway outage that would otherwise surface as 40 flaky scraper tests instead surfaces as one red preflight job with a clear name.

The scheduling version also unlocks something the startup version cannot: drain-before-fail. If the periodic preflight starts failing, stop leasing new exits, let in-flight requests finish, and pause the queue. You degrade the crawl instead of corrupting the dataset with a 40-minute window of silently empty pages.

Where the preflight does not help

Be honest about the limits, because they shape how much you trust the gate. The battery cannot tell you the origin is temporarily blocking your exits (all checks pass against ipify and ipinfo, both indifferent to your traffic, while your target has blacklisted a whole ASN range). For that you need an in-crawl signal: a small canary set of known-live URLs sampled continuously, with block-rate alarms keyed on them. The preflight answers "is my plumbing up"; the canary answers "does the destination like us." You want both, and confusing them is how teams spend an afternoon debugging a site-side incident with a proxy-side tool.

Six checks, sixty seconds, and a hard gate at startup. Write the preflight once, paste it into every scraper, and the beginner debugging loop — hours of guessing whether the site or the proxy broke — mostly evaporates.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Proxy-Authorization Is Not a Normal Header: How requests, httpx, and aiohttp Each Handle It

Greta — Fri, 25 Sep 2026 01:06:36 +0000

Proxy-Authorization Is Not a Normal Header: How requests, httpx, and aiohttp Each Handle It

Beginner proxy tutorials treat authenticated proxies as a one-liner: paste http://user:pass@gateway:port into the proxy argument and move on. Then someone puts @ in their password, or switches from requests to aiohttp, or turns on per-request rotation — and the pipeline starts returning 407s that make no sense. Proxy authentication is not "just another header". It is a hop-by-hop mechanism attached to the connection, not the request, and every Python HTTP client wires it slightly differently.

The core claim of this article: treat proxy auth as a connection-level property, verify what your client actually puts on the wire, and stop mixing credential-passing styles across libraries.

407 is not 401, and it is not per-request

Two facts explain most confusion:

Proxy-Authenticate / Proxy-Authorization belong to the client→proxy hop. When the origin returns 401, that travels through the tunnel untouched; when the gateway returns 407, only your client and the proxy care.
Authentication is charged per connection, not per request. Over HTTPS the client first opens a CONNECT tunnel; a 407 there means the tunnel never formed. Over plain HTTP, the request goes to the proxy in absolute-form, and a 407 means "retry with credentials." Once a connection is authenticated, keep-alive requests on it ride for free.

Fact 2 has an architectural consequence beginners miss: the handshake cost of proxy auth scales with connection churn, not request count. With a long-lived ClientSession and one sticky exit, you pay once. With per-request rotation through 500 fresh exits, you potentially pay 500 times — and if your client uses challenge-response (send, receive 407, resend) rather than pre-emptive credentials, every one of those connections costs two round trips instead of one. Rotation plans that look "bandwidth-cheap" can be latency-expensive for exactly this reason.

Three styles, three sets of gotchas

Style A — userinfo in the proxy URL:

import requests
proxies = {"https": "http://user:pass@gw.example.com:9000"}
r = requests.get("https://example.com", proxies=proxies, timeout=10)

The trap is encoding. RFC 3986 forbids raw @, :, and / in userinfo, but real passwords contain them constantly. requests and httpx will happily mis-parse http://user:p@ss@gw:9000 into a wrong host and a truncated password, and the failure looks like a connection error, not an auth error. Encode explicitly:

from urllib.parse import quote

def proxy_url(user: str, pw: str, host: str, port: int) -> str:
    return f"http://{quote(user, safe='')}:{quote(pw, safe='')}@{host}:{port}"

Style B — a dedicated auth object. aiohttp prefers this and is the cleanest of the three:

import aiohttp
from aiohttp import BasicAuth

async def fetch(session, url):
    async with session.get(
        url,
        proxy="http://gw.example.com:9000",
        proxy_auth=BasicAuth("user", "p@ss/word"),   # no URL parsing, no encoding bugs
    ) as resp:
        return await resp.text()

Style C — a manual Proxy-Authorization header. This is where beginners shoot themselves: Proxy-Authorization is a hop-by-hop header, and clients that manage auth internally may ignore, override, or duplicate your version of it. Also, if you build the Basic token yourself, remember it targets the proxy:

import base64
token = base64.b64encode(b"user:pass").decode()
headers = {"Proxy-Authorization": f"Basic {token}"}

Do not use style C as your default. Use it only when a client genuinely fails to authenticate — which brings us to verification.

Prove what your client sends: a 60-second auth probe

Do not trust documentation tables; run this. It is a local "proxy" that accepts only plain-HTTP absolute-form requests, demands Basic auth, and echoes back exactly what it observed. Then you point requests, httpx, and aiohttp at it and watch whether credentials were pre-emptive or arrived after a 407.

# auth_probe.py — run: python auth_probe.py
import asyncio, base64, time
from collections import defaultdict

LOG = defaultdict(list)   # observations per client label

async def handle(reader, writer):
    request_line = await reader.readline()
    if not request_line:
        writer.close(); return
    raw = request_line.decode(errors="replace")
    headers = {}
    while True:
        line = await reader.readline()
        if line in (b"\r\n", b"\n", b""):
            break
        k, _, v = line.decode(errors="replace").partition(":")
        headers[k.strip().lower()] = v.strip()

    auth = headers.get("proxy-authorization")
    attempt = 1 if auth else 0
    if auth and auth.lower().startswith("basic "):
        user, _, pw = base64.b64decode(auth[6:]).decode().partition(":")
        body = f"ok as={user} pwlen={len(pw)}"
        resp = (f"HTTP/1.1 200 OK\r\nContent-Length: {len(body)}\r\n"
                f"Connection: close\r\n\r\n{body}")
    else:
        attempt = 1 if auth else 0
        resp = ("HTTP/1.1 407 Proxy Authentication Required\r\n"
                "Proxy-Authenticate: Basic realm=\"probe\"\r\n"
                "Content-Length: 0\r\nConnection: close\r\n\r\n")
    LOG[current_client].append((raw.split()[:2], attempt, bool(auth)))
    writer.write(resp.encode()); await writer.drain(); writer.close()

current_client = "?"

async def main():
    server = await asyncio.start_server(handle, "127.0.0.1", 9010)
    import requests, httpx
    global current_client

    def call_sync(label, **kw):
        global current_client
        current_client = label
        requests.get("http://example.com/",
                     proxies={"http": "http://127.0.0.1:9010"}, timeout=5, **kw)

    call_sync("requests-noauth")
    call_sync("requests-urlauth", proxies={"http": "http://127.0.0.1:9010"})  # retry path
    current_client = "requests-basic"
    s = requests.Session()
    s.proxies = {"http": "http://user:p%40ss@127.0.0.1:9010"}
    try: s.get("http://example.com/", timeout=5)
    except Exception: pass

    async with httpx.AsyncClient() as c:
        current_client = "httpx"
        try:
            await c.get("http://example.com/",
                        proxy="http://user:pass@127.0.0.1:9010", timeout=5)
        except Exception: pass

    async with aiohttp.ClientSession() as c:
        current_client = "aiohttp-proxy_auth"
        try:
            async with c.get("http://example.com/",
                             proxy="http://127.0.0.1:9010",
                             proxy_auth=BasicAuth("user", "p@ss"), timeout=5) as r:
                await r.text()
        except Exception: pass

    server.close()
    for label, rows in LOG.items():
        print(f"{label}: requests={len(rows)} first_had_auth={rows[0][2] if rows else '-'}")
        for first_line, attempt, had in rows:
            print("   ", first_line, "with Proxy-Authorization" if had else "no auth -> 407")

asyncio.run(main())

Read the output for the two things that matter: how many requests the client needed (one = pre-emptive; two = challenge/response) and whether the decoded password survived (your p@ss bug becomes obvious instantly). Different versions behave differently — that is the point of probing rather than trusting a blog table. On my machine, requests sends pre-emptive Basic when credentials are in the proxy URL, aiohttp with proxy_auth sends pre-emptively, and passing a raw @ in the URL userinfo produces a confused connection, not a 401-style rejection.

Reading the failure by its signature

Proxy auth errors are unusually diagnosable if you resist the urge to just retry. Learn four signatures:

407 Proxy Authentication Required on the very first request. Credentials never left the client. Either the proxy string was not applied at all (a classic: you set proxies={"http": ...} but requested an https:// URL), or the userinfo was mangled by encoding. Run the probe; it will show a request with no Proxy-Authorization.
407 on every request, forever. Credentials are leaving but wrong — usually a truncated password from a raw @ in the URL, or a username whose suffix the gateway expects (user-geo-US-session-1234). Decode the header on the probe output and compare to what you think you sent.
Tunnel connection failed: 407 from urllib3/aiohttp. Same as above, but the failure happened on the CONNECT for an HTTPS target. The distinction matters for reporting: the proxy rejected you, the origin never saw the request.
Intermittent 407 that succeeds on retry. Your client is doing challenge-response, and the first attempt on every new connection is a wasted round trip. Harmless for correctness, expensive at scale — see below.

The cost of challenge-response, in one measurement

If your gateway issues a 407 before accepting Basic credentials, every fresh connection pays an extra round trip. On a sticky session this amortizes to nothing. On per-request rotation it multiplies. Measure it locally before you scale it up:

import socket, ssl, time

def time_connect(host: str, port: int) -> float:
    t0 = time.perf_counter()
    with socket.create_connection((host, port), timeout=10) as s:
        with ssl.SSLContext().wrap_socket(s, server_hostname=host) as ctx:
            ctx.do_handshake()
    return time.perf_counter() - t0

# run 20 times against your gateway, not against example.com
samples = [time_connect("gw.example.com", 9000) for _ in range(20)]
print("median tcp+tls to gateway:", sorted(samples)[len(samples)//2] * 1000, "ms")

Multiply that by your request count if you rotate per request, and the "free" credential rotation suddenly has a price tag. This is why most residential gateways let you encode rotation parameters (country, session id, session lifetime) inside the username: the gateway can then authenticate one long-lived connection and rotate the exit behind it. If you are managing a list of distinct authenticated exits yourself, you have silently taken on the handshake bill the gateway was amortizing for you.

Digest, and the limits of what you can fix

The Proxy-Authenticate header can offer Digest or Negotiate. Almost no mainstream Python client implements Digest for proxy auth (as opposed to server digest). If your provider's gateway demands anything beyond Basic, the client cannot rescue you — pick a gateway that speaks Basic, or put a local privoxy/mitmproxy-style authenticating shim in front of it: your scraper talks to 127.0.0.1:8080 unauthenticated, the shim attaches credentials to the upstream. That adds one hop but centralizes auth handling across mixed libraries in a polyglot pipeline.

Checklist

Percent-encode userinfo, or use proxy_auth-style objects everywhere.
Standardize one credential-passing style per codebase; mixed styles produce silent fallbacks.
Run the probe whenever you bump a client library version — auth regressions are common and quiet.
Budget auth round trips per connection, not per request: sticky pools amortize them, per-request rotation does not.
Log the exact decoded username your gateway sees; "wrong password" errors are usually truncated passwords.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Asyncio Semaphores Are the Wrong Knob: Per-Exit Concurrency Caps for Rotating Proxy Pools

Greta — Fri, 25 Sep 2026 01:05:24 +0000

Asyncio Semaphores Are the Wrong Knob: Per-Exit Concurrency Caps for Rotating Proxy Pools

Most beginner async scrapers cap concurrency with a single asyncio.Semaphore. It is the first piece of advice in nearly every "scraping with asyncio" tutorial: wrap your fetch function in async with sem: and you will never have more than N requests in flight. Then you add rotating residential proxies, and the semaphore stops meaning anything useful — because the thing that actually gets overwhelmed is not your event loop. It is the exit.

This article makes one argument: in a proxied pipeline, concurrency is a property of the exit, not of the worker, and you should cap it with a per-exit lease pool instead of a global semaphore.

Why the global semaphore fails once proxies enter the picture

Say you run Semaphore(20) across 8 async workers and pull a fresh rotating proxy for every request. At any moment, the 20 in-flight requests might be spread across 20 different exits — or, with a sticky session, five requests might pile onto one residential IP while fifteen pile onto another. The global semaphore bounds neither case in a way you can reason about.

From the origin's perspective, your pipeline does not exist. What exists is a set of exit IPs, each emitting requests. Rate heuristics, per-IP token buckets, and bandwidth caps on the proxy side all key off that exit. A single residential gateway will happily accept thirty concurrent TLS streams from you until the upstream connection pool behind it saturates, at which point latency doubles, then triples, then everything times out — and your global semaphore never noticed, because it was counting tasks, not sockets per exit.

The failure signature is nasty for beginners: everything "works" at Semaphore(5), falls over at Semaphore(30), and nobody can say why, because the bottleneck is not in the code you wrote.

The right model: a lease pool keyed by exit

A per-exit cap has three moving parts:

A registry of live exits (a proxy session or a gateway endpoint identifies one).
A small semaphore per exit, created lazily.
An acquisition step that picks an exit first, then asks that exit for permission.

Here is a complete, runnable lease pool. It caps each exit at per_exit concurrent requests and refuses to overload a dying exit:

import asyncio
import random
from collections import defaultdict


class ExitLeasePool:
    """Concurrency capping keyed to proxy exits instead of tasks."""

    def __init__(self, exits, per_exit: int = 3):
        self._exits = list(exits)                      # e.g. proxy URLs
        self._per_exit = per_exit
        self._sems = defaultdict(lambda: asyncio.Semaphore(self._per_exit))
        self._inflight = defaultdict(int)              # observability

    def _pick_exit(self) -> str:
        # Least-loaded-first beats round-robin: spreads load away
        # from exits that are already saturated or slowing down.
        candidates = sorted(
            self._exits,
            key=lambda e: self._inflight[e] / self._per_exit,
        )
        head = candidates[: max(1, len(candidates) // 2)]
        return random.choice(head)

    async def fetch(self, client, url: str, timeout: float = 12.0):
        exit_url = self._pick_exit()
        sem = self._sems[exit_url]
        async with sem:                                # per-exit cap
            self._inflight[exit_url] += 1
            try:
                async with client.get(
                    url, proxy=exit_url,
                    timeout=aiohttp_timeout(timeout),
                ) as resp:
                    return resp.status, await resp.read()
            finally:
                self._inflight[exit_url] -= 1


def aiohttp_timeout(t: float):
    import aiohttp
    return aiohttp.ClientTimeout(total=t)

And a small driver that proves the invariant — at no moment does any exit carry more than per_exit requests:

import aiohttp


async def main():
    exits = [f"http://user-portal{i}:pass@gw.example.com:9000" for i in range(6)]
    pool = ExitLeasePool(exits, per_exit=3)
    urls = [f"https://httpbin.org/delay/1?i={i}" for i in range(60)]

    observed_max = defaultdict(int)

    async with aiohttp.ClientSession() as client:
        async def one(u):
            # record which exit took it (demo: re-pick is internal,
            # so we instrument via pool._inflight during the call)
            t = asyncio.create_task(pool.fetch(client, u))
            while not t.done():
                for e, n in pool._inflight.items():
                    observed_max[e] = max(observed_max[e], n)
                await asyncio.sleep(0.02)
            return await t

        await asyncio.gather(*(one(u) for u in urls))

    print(dict(observed_max))   # every value <= 3

Run it and every exit's peak concurrency is <= per_exit. Try the same instrumentation with a global Semaphore(18) and you will watch one exit spike to nine while another idles.

Sizing `per_exit` without guessing

The number is not mystical. Two measurements give you a sane starting point:

Latency knee. Send 1, 2, 4, 8 concurrent requests through one exit to a stable endpoint and plot mean latency. The concurrency level where latency exceeds ~2x your single-request baseline is your knee. Set per_exit one step below it.
Provider contract. Some residential plans bill or throttle per concurrent session, not per request. Read that clause; it is a hard cap that sits under whatever your latency test suggests.

Starting values that work in practice: per_exit = 2–3 for rotating residential gateways, 4–6 for static residential or ISP exits, and whatever your VPS bandwidth allows for datacenter exits. The point of the lease pool is that changing tiers means changing one number, not restructuring your code.

The bonus: a natural place for health signals

Once every request passes through ExitLeasePool.fetch, you have a single chokepoint to record per-exit telemetry: latency, 429 counts, TLS errors. When an exit's error rate crosses a threshold, bump its semaphore down or evict it from self._exits mid-run. That kind of adaptation is impossible with a bare global semaphore, because there is no object to attach state to.

A benchmark that finds the knee, not a guess

Two numbers decide per_exit: the exit's latency knee and the provider's concurrent-session limit. The first you measure yourself. Here is a compact harness — it walks concurrency levels 1, 2, 4, 8, 16 through a single exit and reports the median latency, error rate, and where the cliff appears:

import asyncio, statistics
import aiohttp

async def probe(exit_url: str, target: str, level: int) -> dict:
    lat = []
    errs = 0
    connector = aiohttp.TCPConnector(limit=level)
    async with aiohttp.ClientSession(connector=connector) as s:
        async def one():
            nonlocal errs
            loop = asyncio.get_running_loop()
            t0 = loop.time()
            try:
                async with s.get(target, proxy=exit_url, timeout=30) as r:
                    await r.read()
                    if r.status >= 500:
                        errs += 1
            except Exception:
                errs += 1
                return
            lat.append(loop.time() - t0)
        await asyncio.gather(*(one() for _ in range(level * 4)))
    base = statistics.median(lat) if lat else float("inf")
    return {"level": level, "median_ms": base * 1000,
            "p95_ms": (statistics.quantiles(lat, n=20)[-1] * 1000) if len(lat) >= 4 else base * 1000,
            "err_rate": errs / max(1, level * 4)}

async def main():
    EXIT = "http://USER-SESSION-bench:PASS@gw.example.com:9000"
    TARGET = "https://httpbin.org/delay/1"
    rows = []
    for level in (1, 2, 4, 8, 16):
        r = await probe(EXIT, TARGET, level)
        rows.append(r)
        print(f"concurrency={level:<3} median={r['median_ms']:6.0f} ms "
              f"p95={r['p95_ms']:7.0f} ms err={r['err_rate']:.0%}")
    baseline = rows[0]["median_ms"]
    for r in rows:
        if r["median_ms"] > 2 * baseline or r["err_rate"] > 0.05:
            print(f"knee at level {r['level']} -> set per_exit = {r['level'] // 2}")
            break

asyncio.run(main())

Run it once per tier and once per region you plan to use, and store the results in config next to the exit list. Then, when the provider changes hardware under the same product name — which they do — the number is stale, not your reasoning. Re-run it monthly like any other migration.

What the cap does NOT protect: the origin

A subtle trap: per-exit caps make your traffic safe for the exits, and can still be rude to the site. Ten exits at 3 concurrent requests each is thirty in-flight requests at the origin. Origin rate limits, Retry-After headers, and politeness budgets all key off the target, not the exit. So the lease pool is only half of a correct design; the other half is a per-origin token bucket that every request passes through before it is allowed to acquire an exit lease:

class OriginBudget:
    """Global per-origin rate limit, independent of how many exits you own."""
    def __init__(self, rps: float):
        self._tokens, self._cap = rps, rps
        self._last = asyncio.get_running_loop().time()
        self._lock = asyncio.Lock()

    async def acquire(self):
        async with self._lock:
            while True:
                now = asyncio.get_running_loop().time()
                self._tokens = min(self._cap, self._tokens + (now - self._last) * self._cap)
                self._last = now
                if self._tokens >= 1:
                    self._tokens -= 1
                    return
                await asyncio.sleep((1 - self._tokens) / self._cap)

Order matters: origin budget first, exit lease second. Acquiring an exit and then waiting on the budget pins scarce exits doing nothing, which is exactly the resource-starvation pattern you built the lease pool to avoid.

What beginners should take away

asyncio.Semaphore is a fine tool for the problem it solves: bounding CPU, file descriptors, or a single upstream service you own. A rotating proxy pool is none of those things. It is a fleet of independent network exits, each with its own capacity, and a fleet needs per-unit caps. Replace the one big semaphore with a lease pool keyed by exit, size per_exit from a latency knee instead of vibes, and half of the "my async scraper randomly dies at high concurrency" mystery disappears.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Slow Down Before the 429: EWMA Latency Inflation as Your Real Politeness Signal

Greta — Thu, 24 Sep 2026 01:12:28 +0000

Slow Down Before the 429: EWMA Latency Inflation as Your Real Politeness Signal

The standard polite-scraping recipe is a fixed per-origin rate limit, or a header-driven one: read X-RateLimit-Remaining, and when it looks low, slow down. Both are reactive. By the time you see a 429 or a Retry-After, you have already crossed the line; by the time the rate-limit header shows trouble, the origin is already throttling you. And many origins — most, honestly — send you no rate-limit headers at all, so you're flying blind until the wall arrives.

There is an earlier signal hiding in plain sight: the origin's response time. When a server starts shedding load or applying soft throttling to your traffic, its p50 latency creeps up before it ever returns a 429. A search page that normally answers in 180ms starts answering in 240, then 310. That drift is the origin telling you — politely, in the only channel it always has — that you are contributing to its load. The core idea of this article: track a per-origin exponentially weighted moving average of response latency and use deviation from baseline as a continuous congestion signal that scales your request rate down smoothly, long before an explicit 429 fires.

Why latency is the honest signal

Response time is the one metric that is always present, always per-origin, and always measured at the exit where it matters. Unlike headers (which origins may omit), unlike error codes (which are a lagging, binary confirmation), and unlike your own semaphore count (which knows nothing about the origin's mood), latency reflects the origin's actual capacity to serve you. Under load, servers queue; queues show up as latency; and the latency you observe through your proxy exit is the latency of the origin plus your contribution to its queue.

There's a subtlety that makes latency better than the rate-limit header even when the header is present: it's measured at the true point of contention. The header may report a quota per API key while the origin's real bottleneck is per-IP at the edge, or vice versa. Your observed latency is unambiguous — it's how long this request, through this exit, to this origin, actually took. That is the number the polite-scraping problem is really about.

The baseline problem: absolute latency is meaningless

A raw 300ms means nothing on its own. One origin is a fast CDN-cached blog (300ms is a crisis), another is a database-backed search API whose p50 is 450ms on a calm day (300ms is a gift). You can't hardcode thresholds across origins. The fix is to make the signal relative to each origin's own recent behavior.

Track two EWMA per origin: a fast-moving average (reacts within seconds, catches the onset of slowdown) and a slow-moving average (the "calm baseline," takes minutes to move). The congestion signal is the ratio fast / slow. When the origin is at its normal pace, the ratio hovers near 1.0. When it starts throttling you, fast climbs while slow lags, and the ratio rises.

import time
import threading

class OriginPacer:
    """Per-origin adaptive politeness controller driven by latency EWMA.

    - `fast` reacts quickly (alpha ~0.3): catches onset of congestion.
    - `slow` is the baseline (alpha ~0.02): what 'normal' looks like.
    - signal = fast / slow.  signal < ~1.15 -> send faster;
      signal > ~2.0 -> crawl.
    We turn the signal into a token-bucket rate with a smooth transfer
    function, so the request rate never steps abruptly (abrupt ramps are
    themselves a bot tell).
    """
    def __init__(self, origin: str, base_rps: float = 4.0, max_rps: float = 8.0,
                 min_rps: float = 0.3):
        self.origin = origin
        self.base_rps = base_rps
        self.max_rps = max_rps
        self.min_rps = min_rps
        self._fast = None
        self._slow = None
        self._last = time.monotonic()
        self._tokens = base_rps
        self._lock = threading.Lock()
        self._alpha_fast = 0.30
        self._alpha_slow = 0.02

    def observe(self, latency_ms: float) -> None:
        """Call once per *successful* response with its wall-clock latency."""
        with self._lock:
            if self._fast is None:          # seed
                self._fast = self._slow = latency_ms
                return
            self._fast += self._alpha_fast * (latency_ms - self._fast)
            self._slow += self._alpha_slow * (latency_ms - self._slow)

    def signal(self) -> float:
        if not self._slow:
            return 1.0
        return round(self._fast / self._slow, 3)

    def target_rate(self) -> float:
        """Smooth transfer function mapping congestion signal -> requests/sec."""
        s = self.signal()
        if s <= 1.15:                       # origin comfortable -> allow burst up to max
            r = self.max_rps
        elif s >= 2.5:                      # heavy congestion -> crawl
            r = self.min_rps
        else:
            # linear interpolation between the two knees
            t = (s - 1.15) / (2.5 - 1.15)
            r = self.max_rps + t * (self.min_rps - self.max_rps)
        return r

    def acquire(self) -> float:
        """Block (return seconds to sleep) so caller respects the current rate."""
        with self._lock:
            now = time.monotonic()
            self._tokens = min(self.target_rate(),
                               self._tokens + (now - self._last) * self.target_rate())
            self._last = now
            if self._tokens >= 1:
                self._tokens -= 1
                return 0.0
            need = (1 - self._tokens) / self.target_rate()
            self._tokens = 0
            return need

The transfer function is deliberately piecewise-linear rather than a hard if/else on thresholds. A scraper that clamps instantly from 8 rps to 0.3 rps creates a discontinuity in its own traffic shape — a step function that is easy for an origin to spot. Creeping down as latency climbs produces a smooth ramp that resembles organic load tapering off.

Latency vs the timeout trap

One engineering trap to be honest about: you cannot treat every high latency as congestion. Some origins are genuinely slow at certain hours (nightly batch jobs, cache-cold pages). Some are slow because your exit is slow — a congested residential gateway can add 800ms of pure proxy latency that has nothing to do with the origin's willingness to serve you. Feeding contaminated latency into the pacer means you throttle yourself for your provider's bad routing.

The fix is to subtract the exit's overhead. You already have a cheap way to measure it: your egress canary / echo endpoint that returns instantly. Observe latency to the echo through the same exit at the same time you observe the origin, and pace on the origin latency minus exit baseline, not raw latency.

async def timed_request(session, url, exit_url, echo_url, pacer: OriginPacer):
    t0 = time.monotonic()
    r = await session.get(url, proxy=exit_url)
    origin_ms = (time.monotonic() - t0) * 1000

    t1 = time.monotonic()
    await session.get(echo_url, proxy=exit_url, timeout=(2, 5))
    exit_ms = (time.monotonic() - t1) * 1000

    net_overhead = min(exit_ms, origin_ms * 0.5)   # don't over-subtract
    pacer.observe(origin_ms - net_overhead)
    return r

Now the pacer sees the origin's queue, not the proxy's congestion. When the proxy itself is the problem, you'll see it in a different place (a rising timeout rate on the echo call), and the correct response there is rotate the exit, not throttle the origin.

Making latency one input among several

A good politeness controller doesn't rely on a single signal; it combines a cheap proactive one (latency) with cheap reactive confirmations. A clean design is a small weighted state machine:

Latency signal (proactive) sets the baseline rate, continuously.
429 / Retry-After (reactive) clamps the rate hard and puts the exit in cooldown for the stated duration, then resumes the latency-driven ramp.
Soft-block signals (200-with-thin-body, "slow down" banners) nudge the target down one level even without a 429, since those are the origin's polite warning shot.
robots.txt Crawl-delay / Request-rate act as a ceiling you never exceed regardless of what the other signals say — respecting the published limit is the floor of good manners, and Request-rate: 1/5 1/1m should cap you no matter how fast the origin feels.

The latency pacer is the dial; the others are hard stops and warnings layered on top. The point of putting latency first is that it's the only signal that scales you down continuously and early — a thermostat, where 429s are just the smoke alarm.

What you get in production

Teams that move from "fixed rate, then react to 429" to latency-driven pacing consistently report the same outcome: the tail of incidents collapses. They still occasionally see a 429, but the volume of requests that trigger the severe tier — permanent IP bans, CAPTCHA walls, account locks — drops, because those escalate from sustained overload, and latency-based pacing never lets your rate reach sustained overload. You spend fewer proxy-GB (a throttled, smooth crawl is cheaper than a burst-then-cooldown sawtooth), you look less like a bot (smooth ramps, organic tapering), and you get more successful rows out of the same quota because you stopped hitting the wall at all.

There's a philosophical shift worth naming: polite scraping has traditionally meant "obey the explicit rules the origin publishes." But most origins publish almost nothing. The honest engineering move is to treat the origin's behavior — how fast it answers you — as a rule that is always being published, in a channel you can always read. Slow down before you are told to, and you'll rarely be told at all.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Not Every Challenge Is a CAPTCHA: A Triage Pipeline for Silent Rate Limits, JS Interstitials, and Real Puzzles

Greta — Thu, 24 Sep 2026 01:11:16 +0000

Not Every Challenge Is a CAPTCHA: A Triage Pipeline for Silent Rate Limits, JS Interstitials, and Real Puzzles

When a scraper's success rate drops, the reflex is to reach for the CAPTCHA-solver invoice. But if you actually crack open the failing responses and read them, a large fraction contain no CAPTCHA at all. They are 200s with a "Please verify you are human" banner that never renders a puzzle; empty-body responses; 403s from a CDN, not the origin; or — the sneakiest — normal-looking HTML that is a JS interstitial whose real content only appears after window.location is reassigned by a script. Treating all of these as "we got CAPTCHAs, solve harder" is how teams end up paying a solver service to fail on traffic that never required a solver in the first place.

The core idea of this article: challenge responses come in distinct tiers with completely different correct responses — routing, retry, render, or solve — and the profitable engineering move is a cheap triage classifier in front of your solver, so you only escalate to the expensive tier for genuine puzzles. And crucially, most of the tiers are best resolved at the proxy exit layer, not the CAPTCHA layer.

The four challenge tiers

Let me name the tiers by what they actually are and what fixes them:

Tier 0 — Silent rate limit. HTTP 200, but content is truncated, generic, or a bare "too many requests" string. Sometimes 429 with an HTML body. The origin hasn't asked you to prove humanity; it's asked you to slow down. Fix: throttle the exit (or move to a new exit), not solve. Sending these to a solver returns garbage and wastes money.

Tier 1 — CDN block / IP reputation. A 403 or 429 with a Cloudflare, Akamai, or PerimeterX challenge page — often the "Just a moment..." interstitial with a <title> that updates after a JS proof-of-work runs. Notice: the challenge is addressed to your exit IP's reputation, not necessarily to your request content. Fix: rotate the exit or change its tier. A browser that solves the JS proof from the same low-reputation IP often re-challenges, because the reputation is the problem.

Tier 2 — JS interstitial / meta-refresh. The response is a small HTML document that sets a cookie via document.cookie and reloads. No puzzle ever. Browsers pass through it in milliseconds. Fix: render the page in a headless engine, let the JS run, and reuse the resulting cookies. A CAPTCHA solver cannot help; this is a rendering problem.

Tier 3 — Actual CAPTCHA. A real image puzzle (reCAPTCHA v2/v3, hCaptcha, image grid, slider). Fix: this is the only tier where a solver earns its keep — and even then, sometimes a different exit skips the CAPTCHA entirely, because it's tied to IP risk, not to you.

The economics are stark. If a solver is $1–3 per thousand and 60% of your "challenge" traffic is actually Tier 0/1/2, you are throwing most of that spend away while also failing to fix the real cause (your exits), so the same URLs challenge you again on the next run.

A cheap triage classifier

Triage should run on the response you already have, in microseconds, with no extra requests. The signal set is small and highly predictive:

import re

CDN_HINTS = re.compile(
    r"(cf-ray|x-served-by|akamai|x-iapx-|px-captcha|cf_chl)", re.I
)
CHALLENGE_MARKERS = re.compile(
    r"(just a moment|verify you are human|checking your browser|attention required|"
    r"enable javascript and cookies)", re.I
)
RECAPTCHA_MARKERS = re.compile(
    r"(recaptcha|g-recaptcha|hcaptcha|data-sitekey|captcha-challenge|cf_turnstile)", re.I
)

def triage(resp) -> tuple[str, str]:
    """Return (tier, action). resp is a requests/httpx response object."""
    body = resp.text[:20000]  # don't scan huge bodies
    title = _extract_title(body)
    lower = body.lower()

    # Tier 3 first: an actual puzzle widget is the highest-confidence signal.
    if RECAPTCHA_MARKERS.search(lower):
        return ("T3_captcha", "solve_or_rotate")

    # Tier 1: CDN/anti-bot interstitial (JS challenge, no widget).
    if CDN_HINTS.search(resp.headers.get("server", "") + " " + str(resp.headers)) or \
       "just a moment" in lower or "cf_chl" in lower:
        return ("T1_cdn_challenge", "rotate_exit")

    # Tier 2: JS/meta interstitial that reloads with a cookie.
    if CHALLENGE_MARKERS.search(title) or _has_meta_refresh(body) or \
       ("document.cookie" in lower and "<script" in lower and len(body) < 3000):
        return ("T2_js_interstitial", "render_headless")

    # Tier 0: suspicious 200 with empty/short body, or 429.
    if resp.status == 429 or (resp.status == 200 and _looks_empty(lower)):
        return ("T0_silent_limit", "throttle_exit")

    return ("pass", "continue")

The key insight encoded in the ordering is: detect the actual puzzle last but with the tightest regex, and treat everything else as "not my solver's problem." A "verify you are human" string on a Cloudflare page is a Tier 1 reputation signal, not a Tier 3 puzzle, and calling it T1 means the next line of your code changes the exit instead of calling an API.

The small helpers:

import re as _re
def _extract_title(html: str) -> str:
    m = _re.search(r"<title[^>]*>(.*?)</title>", html, _re.I | _re.S)
    return m.group(1).strip() if m else ""

def _has_meta_refresh(html: str) -> bool:
    return bool(_re.search(r'<meta[^>]+http-equiv=["\']refresh', html, _re.I))

def _looks_empty(text: str) -> bool:
    stripped = _re.sub(r"<[^>]+>", " ", text).strip()
    return len(stripped) < 50

The escalation ladder, in cost order

Once you have a tier, route it to the cheapest fix that plausibly works, and only escalate after that fails. The ladder, from free to expensive:

Rotate the exit (cost: one extra request). Appropriate as the first move for Tier 1 and Tier 3, because both are frequently IP-reputation-driven. If the same URL succeeds from a fresh residential exit, you never needed a solver at all. This one fact is why "avoidance before solving" beats "solving" as a default: exit rotation is nearly free and eliminates a large share of challenges outright.
Change exit tier (cost: a slightly higher per-GB price). If a datacenter exit is getting Tier 1 challenges, a mobile or high-reputation residential exit often walks straight through. This is a routing decision, not a CAPTCHA decision.
Render headless (cost: CPU + latency). Tier 2's fix. Run the interstitial's JS, capture the cookie it sets, and replay the real request with that cookie and the same sticky exit so the reputation and session continuity hold.
Solve (cost: money). The genuine last resort for Tier 3 that survives exit rotation. Track it as an exception rate, not a routine.

The important architectural point is that the ladder lives above your proxy router. Triage returns an action string; the router maps rotate_exit to "pick a new exit and re-issue," throttle_exit to "put this exit in cooldown," and solve_or_rotate to "rotate once, and only if the fresh exit still shows the puzzle, call the solver."

async def handle(session, url, exit_url, solver_client):
    for attempt in range(6):
        resp = await session.get(url, proxy=exit_url)
        tier, action = triage(resp)
        if tier == "pass":
            return resp
        if action == "throttle_exit":
            await pool.throttle(exit_url, backoff=2.0)
        elif action == "rotate_exit":
            exit_url = pool.rotate(same_tier=True)
        elif action == "render_headless":
            return await render_and_replay(url, exit_url)  # keep sticky exit
        elif action == "solve_or_rotate":
            if attempt == 0:
                exit_url = pool.rotate(higher_reputation=True)  # try free first
                continue
            token = await solver_client.solve(resp)            # pay only now
            return await session.post(url, proxy=exit_url, data={"g-recaptcha-response": token})
    raise ChallengeExhausted(url)

Instrument the tiers, because they score your proxy pool

The quiet payoff of a triage pipeline is the telemetry it produces. Because you're labeling every challenge with a tier and the exit that triggered it, you can compute the metric that actually matters for provider selection: Tier 1 + Tier 3 rate, per exit-subnet, per origin. A subnet whose exits trigger CDN challenges 40% of the time is reputation-poor; you should stop routing sensitive origins through it entirely. Meanwhile a "clean" subnet at 2% is worth paying a premium for.

Without triage, all you see is "success rate dropped on Tuesday," and you fix it by buying more solver credits — treating the symptom. With triage you see "the mobile tier started returning Tier 1 Cloudflare challenges at 8x the residential tier this week," and you fix the real problem by reweighting the router, spending less on solvers and getting better yield at the same time.

The discipline is simple but rarely enforced: never send a response to a solver that your triage classifier hasn't first confirmed contains an actual puzzle, and never conclude a challenge is about you before you've tested whether it's about your exit. Most anti-bot friction is IP-shaped, and once you stop paying to "solve" IP problems, your scraper gets cheaper and better at the same time.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

One Exit, Fifty Workers: Stopping the Coordinated Retries That Turn One 429 Into Fifty

Greta — Thu, 24 Sep 2026 01:08:45 +0000

One Exit, Fifty Workers: Stopping the Coordinated Retries That Turn One 429 Into Fifty

A 429 Too Many Requests is a polite message from an origin. It says "you're going too fast, back off, try again in a bit." Your scraper's job is to honor that message. Which is exactly why the most common way teams get permanently banned is not the first 429 — it's the fiftieth one, three seconds later, when fifty workers all obey the retry instructions at the same instant and through the same exit IP.

The core idea of this article: when many workers share one proxy exit, the origin rate-limits the exit, not the worker, so naive per-worker backoff produces a synchronized retry stampede that looks far more abusive than the original request wave — you need an exit-scoped coordination barrier plus decorrelated jitter to break the lockstep. This is a small, concrete piece of engineering most async scraping frameworks get wrong, and it's worth understanding precisely.

The setup: many workers, one shared exit

Modern Python scrapers fan out with asyncio. You might run one process with 50 concurrent tasks, or 10 processes each with 5, all pointed at the same pool of, say, 200 residential exits. Under a sticky-session policy you deliberately route many requests for the same logical session through the same exit — that's the whole point of stickiness. The problem is that a sticky exit is now a shared fate resource: a single 429 against https://origin.example/search on exit E-017 is a rate-limit verdict for the exit, and every one of your workers currently riding E-017 is subject to it.

Now imagine what happens with textbook per-worker backoff:

# The naive pattern, worker by worker.
async def fetch(session, url, exit_url):
    for attempt in range(8):
        resp = await session.get(url, proxy=exit_url)
        if resp.status == 429:
            retry_after = int(resp.headers.get("Retry-After", "1"))
            await asyncio.sleep(retry_after)   # <-- everyone sleeps the SAME seconds
            continue
        return resp
    raise Exception("gave up")

Fifty workers, one exit, one 429 with Retry-After: 3. Every worker sleeps 3 seconds, wakes up at the same moment, and fires 50 requests at the origin in one synchronized burst through the same exit. The origin sees a burst that is, from its perspective, more aggressive than the first wave, because it's tightly clustered in time. It rate-limits harder. The Retry-After grows. The stampede repeats, with worse timing alignment each round. This is a classic thundering herd, and a shared proxy exit is a thundering-herd amplifier: you have hidden 50 clients behind one identity.

Decorrelated jitter: the first fix, and why it's not enough

The standard remedy is exponential backoff with full jitter (the AWS Architecture Blog formulation): instead of sleeping a deterministic delay, each worker sleeps a random duration drawn uniformly from [0, min(cap, base * 2^attempt)]. This spreads the retries over the backoff window instead of clustering them at its start.

import random

async def fetch_with_jitter(session, url, exit_url, base=1.0, cap=60.0):
    for attempt in range(10):
        resp = await session.get(url, proxy=exit_url)
        if resp.status != 429:
            return resp
        exp = min(cap, base * (2 ** attempt))
        delay = random.uniform(0, exp)
        await asyncio.sleep(delay)
    raise Exception("gave up")

Better — but the distribution of wake-up times is still not what the origin cares about. The origin cares about the instantaneous request rate through exit E-017. With uniform jitter, the probability that several of your fifty workers draw a short delay and hit within the same 100ms bucket is high, and one unlucky co-occurrence is enough to trigger the next 429. Jitter alone decorrelates retries only in expectation, not at the granularity that a per-IP rate limiter measures. What you need is coordination.

The exit-scoped token bucket

The fix is to make the shared resource explicit in your code. If the origin limits per exit, your backoff should be owned per exit, not per worker. Concretely: a token bucket keyed by exit_url (not by origin URL, not by worker task ID) that every worker must acquire a token from before it sends — and that is drained the moment any worker observes a 429 on that exit.

import asyncio
import time
from dataclasses import dataclass, field

class ExitThrottle:
    """One shared throttle per exit across all workers on the machine.

    - Normal state: workers may send at a steady rate up to `rate`.
    - On 429: the exit's `open_until` is set to now + retry_after; every
      worker waiting for a token sleeps until exactly then, so the retry is
      serialized to the *earliest legally allowed* time, not staggered
      against the same origin limit that has just been hit.
    - A post-429 slow ramp: after the cooldown we halve the rate and creep
      back, so fifty workers don't resume at full speed at once.
    """
    def __init__(self, exit_url: str, rate: float = 5.0):
        self.exit_url = exit_url
        self.base_rate = rate
        self.rate = rate
        self._tokens = rate
        self._last_refill = time.monotonic()
        self._open_until = 0.0
        self._lock = asyncio.Lock()

    async def acquire(self):
        # Wait out any global cooldown first (this is the anti-stampede core).
        while True:
            async with self._lock:
                now = time.monotonic()
                if now >= self._open_until:
                    # token refill
                    elapsed = now - self._last_refill
                    self._tokens = min(self.base_rate, self._tokens + elapsed * self.rate)
                    self._last_refill = now
                    if self._tokens >= 1:
                        self._tokens -= 1
                        return  # allowed to send
                target = self._open_until
            await asyncio.sleep(max(0.05, target - time.monotonic()))

    async def penalize(self, retry_after: float):
        async with self._lock:
            now = time.monotonic()
            # The whole exit goes dark until the retry deadline.
            self._open_until = max(self._open_until, now + retry_after)
            # ...and resumes at half speed, then creeps back up.
            self.rate = max(0.5, self.rate * 0.5)

    async def reward(self):
        """Successful, un-throttled requests let the rate recover."""
        async with self._lock:
            self.rate = min(self.base_rate, self.rate * 1.1)

The essential difference from the naive code: the cooldown state lives on the exit, and the 50 workers observe the same state object. Only the first worker to actually receive the 429 calls penalize; the other 49 workers who were about to send are blocked at acquire() until the exit re-opens. Instead of 50 concurrent violations of the same limit, you get a single 429 and a serialized, rate-bounded recovery.

The caller looks like this:

# One throttle per exit, created and shared across all worker tasks.
THROTTLES: dict[str, ExitThrottle] = {}

def get_throttle(exit_url: str) -> ExitThrottle:
    t = THROTTLES.get(exit_url)
    if t is None:
        t = THROTTLES[exit_url] = ExitThrottle(exit_url)
    return t

async def fetch(session, url, exit_url):
    throttle = get_throttle(exit_url)
    for _ in range(20):
        await throttle.acquire()
        resp = await session.get(url, proxy=exit_url)
        if resp.status == 429:
            ra = parse_retry_after(resp.headers.get("Retry-After"))
            await throttle.penalize(ra)
            continue
        await throttle.reward()
        return resp
    raise Exception("gave up")

Parsing Retry-After properly (it's rarely an integer)

One detail that quietly breaks a lot of 429 handlers: RFC 9110 defines Retry-After as either a non-negative delta-seconds integer or an HTTP-date. Production APIs like GitHub, Stripe, and Cloudflare-fronted origins sometimes use the HTTP-date form, and naive int() raises. You also need to separate 429 from 503, which uses the same header but implies capacity problems you should treat differently — a 503 with a long Retry-After is often better served by switching exits than by waiting on one you've now penalized.

import email.utils, time

def parse_retry_after(value: str | None, default: float = 1.0) -> float:
    if value is None:
        return default
    try:
        return max(0.0, float(value))            # delta-seconds
    except ValueError:
        pass
    dt = email.utils.parsedate_to_datetime(value)  # HTTP-date
    if dt is not None:
        return max(0.0, dt.timestamp() - time.time())
    return default

The distributed problem: when "per exit" spans processes

If all fifty workers are tasks in one process, the shared dict above is enough — they're literally sharing a Python object. But if you're running fifty workers as separate processes or across five machines pointing at the same exit pool, the throttle is no longer shared, and you've reinvented the stampede at a higher scale. You need a distributed exit state.

The pragmatic implementation is a tiny shared store — Redis or DynamoDB — keyed by exit_url, holding open_until and a smoothed rate. Each worker, before sending, does a short GET; on a 429 it does an atomic SET open_until IF NOTALREADYLATER (a Lua compare-and-set so the earliest deadline wins, since multiple workers can observe 429 simultaneously). Because the cooldown is now visible across processes, the whole fleet sleeps as one unit. This is the same principle as the in-process throttle, only the state moved to a shared cache — which, not incidentally, is exactly where per-exit state should live if you already run a proxy pool with a gateway.

The metric that proves it's working

The signal to watch is not "did 429s go to zero" — you'll still occasionally take one. The signal is requests-per-exit-per-second distribution. Plot it with a shared exit in front of many workers and the naive backoff looks like a sawtooth: a burst, a cliff, a synchronized re-burst. The coordinated throttle looks like a smooth ceiling: one gentle bump at the first 429, then a controlled re-ramp. If your retry storm looks like a heartbeat monitor, you are still in lockstep. The moment it looks like a ramp, you've broken the herd — and the fifty requests that used to arrive as one violent thump now arrive as fifty separate, quiet, individually successful requests.

Backoff is often taught as a per-request concern: catch the 429, sleep, retry. But a shared proxy exit turns that into a distributed-systems problem. Respect the granularity the origin uses — the IP leaving your gateway, not the task waiting in your loop — and the coordinated retry storm stops being your fault.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Prior-Knowledge or ALPN Upgrade? The HTTP/2 Connection Preface Tells You What Your Proxy Got Wrong

Greta — Thu, 24 Sep 2026 01:07:27 +0000

Prior-Knowledge or ALPN Upgrade? The HTTP/2 Connection Preface Tells You What Your Proxy Got Wrong

Everyone who has spent a weekend debugging anti-bot detection knows about JA3, header ordering, and the User-Agent/TLS mismatch. Far fewer people talk about the layer right above TLS: the HTTP/2 connection preface and its first control frames. That's a shame, because the preface is where the disagreement between what you asked for and what your proxy actually negotiated becomes visible to the origin — and it's the single most common way a well-fingerprinted scraper quietly loses its h2 identity on the last hop.

The core idea of this piece: your HTTP/2 handshake is a fingerprint too, and the connection preface plus the opening SETTINGS/WINDOW_UPDATE frames reveal whether your proxy terminated and re-originated the connection, and how it made that decision. Once you can see that signal, you can audit exits for HTTP/2 integrity exactly the way you'd audit them for TLS coherence — before the origin does.

The two ways a client can speak HTTP/2

There are two legitimate paths to an h2 connection, and they differ in the first bytes on the wire:

ALPN negotiation. During the TLS handshake the client offers h2 in its Application-Layer Protocol Negotiation extension, the server picks h2, and the client immediately sends the connection preface — the magic string PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n — over the already-established TLS session. This is what every modern browser does.
Prior knowledge. The client skips TLS (or is already on a connection it knows supports h2) and starts with the preface anyway, typically against http://host:port. Rare, and mostly used by gRPC servers and HTTP/2-aware internal load balancers.

There is also the deprecated third path — the Upgrade: h2c header on an http/1.1 connection followed by a 101 Switching Protocols. Modern browsers do not do this. Any traffic that looks like it came from Chrome 124 but uses the Upgrade dance is definitionally not Chrome 124. This is one of the most under-diagnosed silent tells in the anti-bot world.

Why proxies get this wrong

Residential and ISP gateways often terminate TLS. Your client negotiates h2 with the gateway, but the gateway has to originate a fresh connection to the origin. What happens next is where most "we have a real browser fingerprint, why are we still blocked" mysteries live:

The gateway doesn't support h2 on its upstream leg and silently falls back to http/1.1. The origin sees a Chrome-flavored TLS handshake with ALPN: h2 but then a series of HTTP/1.1 requests. Incoherent. Blocked.
The gateway supports h2 but re-frames your SETTINGS with defaults (typically MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=65535, and no ENABLE_CONNECT_PROTOCOL), so the SETTINGS fingerprint looks like nghttp2 or curl, not Chrome.
The gateway sends the connection preface after a partial http/1.1 request (a bug in some MITM proxies that treat the preface as a request line), which is a definitive non-browser signature.
The gateway reorders or omits your HPACK dynamic table updates (already a topic of its own fingerprint — see RFC 7541 §2.3.2). Different from your client's initial table, but the way it evolves in the first few responses is what origins key on.

Every one of these is invisible to your code. Your httpx or curl_cffi call succeeded; the status was 200. Only the origin, which sees the whole wire, notices. The way to make it visible is to stand up an HTTP/2 echo server and route one probe per exit through it.

Building the probe

The minimum useful echo response captures, per exit:

The negotiated ALPN from the TLS handshake.
The first frame the client sent (was it the preface string? a raw SETTINGS? a broken PRI line?).
The initial SETTINGS payload (all six parameters, ordered as received).
The initial WINDOW_UPDATE on stream 0, if any, and its increment.
The header-block representation on the first request (indexed vs. literal, pseudo-header ordering, use of dynamic table).

Here's a Python sketch using h2 (the python-hyper project) — the same library many production scrapers already depend on:

import h2.config, h2.connection, h2.settings, h2.events
from dataclasses import dataclass, field

@dataclass
class PrefaceFingerprint:
    negotiated_alpn: str = ""
    preface_bytes: int = 0            # bytes observed before first frame (1+24 expected)
    initial_settings: dict = field(default_factory=dict)
    initial_window_update: int = None
    pseudo_header_order: tuple = ()
    first_request_method: str = ""

class Http2Echo:
    """Very small h2 server-side handler: reads a request, records the fingerprint,
    and returns 204 so the caller can safely tear down."""
    def __init__(self, conn_bytes: bytes, alpn: str):
        self.fp = PrefaceFingerprint(negotiated_alpn=alpn)
        self.config = h2.config.H2Configuration(client_side=False)
        self.conn = h2.connection.H2Connection(config=self.config)
        self.conn.initiate_connection()
        self.data = conn_bytes
        self._parse()

    def _parse(self):
        # Frame the raw bytes; h2 will feed events as we ingest.
        events = self.conn.receive_data(self.data)
        for ev in events:
            if isinstance(ev, h2.events.SettingsAcknowledged):
                self.fp.initial_settings = dict(self.conn.local_settings)
            elif isinstance(ev, h2.events.WindowUpdated):
                if ev.stream_id == 0 and self.fp.initial_window_update is None:
                    self.fp.initial_window_update = ev.delta
            elif isinstance(ev, h2.events.RequestReceived):
                self.fp.first_request_method = dict(ev.headers).get(b":method", b"")[:].decode()
                self.fp.pseudo_header_order = tuple(
                    k.decode() for k, _ in ev.headers if k.startswith(b":")
                )

That's enough to reconstruct the shape of the client's opening move. What you're not looking for is byte-perfect equivalence with a reference Chrome capture — proxies will almost never match perfectly. What you're looking for are categorical mismatches: negotiated_alpn == "h2" but the first request arrived as HTTP/1.1 (which the handler above never even fires for), or initial_settings matching nghttp2 defaults when you intended Chrome.

The four fingerprints that actually differ

Across a pool of a few hundred exits, HTTP/2 prefaces cluster into four buckets, and each bucket tells you something different about the underlying gateway:

Bucket A — Real browser h2. MAX_CONCURRENT_STREAMS=1000, INITIAL_WINDOW_SIZE=65536, HEADER_TABLE_SIZE=65536, followed within the same packet by a WINDOW_UPDATE of ~15,663,649 bytes (the Chrome-specific DEFAULT_WINDOW_UPDATE_RATIO trick). Pseudo-header order is :method, :scheme, :authority, :path and first HEADERS uses dynamic-table indexing for common names. If your exit is Bucket A, HTTP/2 coherence is intact.

Bucket B — Library h2 (curl, nghttp2, hyper). Default MAX_CONCURRENT_STREAMS=100, no WINDOW_UPDATE on connection 0 in the opening burst, and pseudo-headers sometimes emitted as :method, :path, :scheme, :authority (a common library ordering that Chrome never uses). If your exit is Bucket B while your TLS voice screams Chrome, the gateway re-originated TLS with a library client, and the origin will notice.

Bucket C — Gateway re-mux. The gateway terminated your connection and multiplexed many clients onto a single upstream h2 connection. Symptoms: your SETTINGS frame is absent (the gateway sent its own at upstream-open time, before you arrived), your stream IDs continue past what a single session would plausibly emit, and the very first frame you observe looks like a mid-session HEADERS, not a preface.

Bucket D — Silent downgrade. ALPN said h2, but the connection never sent the preface; instead it opened with GET / HTTP/1.1 on the same TLS session. This is a proxy bug that is nearly invisible from the client side, because your HTTP library just sees a normal response. This bucket is where identity-bearing traffic should never be routed.

A one-line classifier over the fields above will tell you which bucket each of your exits is in, per probe. Chart the distribution across the pool and you have an HTTP/2 health score for your provider.

Routing on the HTTP/2 signature

Once every exit carries a bucket label, the routing decisions write themselves:

Authenticated and identity-bearing traffic → Bucket A only. Chrome-shaped preface required. If your residential gateway is emitting Bucket B for 40% of its pool, that's a real constraint on how much of the pool you can actually use for login flows.
Anonymous fetches → Bucket A, B, or C. Anything but D. A library-shaped h2 fingerprint on an anonymous catalog fetch is not a big deal.
Bucket D → quarantine and re-probe. A downgrade-in-flight is a bug or a misconfiguration on the provider side, and it may self-correct on the next session, so cool-down-and-check is friendlier than drop.

The broader lesson is that HTTP/2 is a fingerprinting surface that most teams have not instrumented. It sits above the layer everyone obsesses over (TLS) and below the layer everyone logs (application HTTP). That gap is exactly where proxy gateways tend to be sloppy, and where a scraper's real browser identity quietly evaporates on the last hop. Standing up one echo server and running one probe per exit per hour converts that invisible failure mode into a number you can alert on and route around. Start there, and the next time you see a Chrome-shaped TLS fingerprint blocked with zero HTTP-layer errors to explain it, you'll know where to look first.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Fingerprint Drift Is a Pool Health Signal: Clustering TLS Voices Behind a Single Proxy IP

Greta — Thu, 24 Sep 2026 01:06:00 +0000

Fingerprint Drift Is a Pool Health Signal: Clustering TLS Voices Behind a Single Proxy IP

Most scraper engineers treat TLS fingerprinting as a one-way mirror: the origin looks at your ClientHello and decides whether you look like a browser. That framing is useful, but it hides a second, equally valuable signal — you can look at the fingerprint that comes back out of your proxy and learn things about the pool you are paying for. This article is about that second direction.

The core idea: every exit in a proxy pool has a "TLS voice" — a fairly stable vector of ClientHello features — and when the voice behind a single advertised IP starts to drift, that drift is one of the earliest, cheapest health signals you can collect. A residential gateway that suddenly hands you a handshake that doesn't match the one it handed you ten minutes ago is telling you something: the IP was reassigned, the backend pool churned, or the "sticky" promise quietly broke. If you're only measuring success and HTTP status, you learn all of this late, through a block you could have predicted.

What we're actually fingerprinting

The classic JA3 hash collapses the ClientHello into a single integer by hashing the ordered list of cipher suites, extensions, supported groups, and EC point formats. It's a good dedup key but a poor diagnostic, because it throws away magnitude: two hellos can differ in a single low-signal extension and hash to completely unrelated strings, while two hellos from genuinely different browser families can look numerically close.

For pool-health work you want a feature vector, not a single hash, so you can compute distances. The features that matter most for detecting drift behind one IP are:

The ordered tuple of cipher suite IDs (this is the strongest per-application signal).
The ordered tuple of extension type IDs — especially whether ALPN, pre_shared_key, supported_versions, and key_share are present, and in what order.
The presence and values of GREASE slots (real browsers inject fake extension/codepoint values; some TLS libraries do not).
The supported_groups and signature_algorithms lists.
The TLS record version versus the legacy_version versus the actual negotiated supported_versions.

A "voice" is then just a normalized tuple over these features, captured once per exit and stored. Two requests through the same exit should produce the same voice. When they don't, that's drift.

Capturing the handshake from the exit side

You can't see the origin's view of your handshake directly, but you can stand up a tiny TLS server, route one request through each exit to it, and read the ClientHello the exit presents. In Python, ssl gives you the negotiated details, but to see the raw feature vectors (including extension order and GREASE) you want something lower-level. scapy or the cryptography library's TLS primitives work, but the pragmatic path for most teams is a fingerprint-echo endpoint — the same trick behind tlsh.sh and similar services:

import requests
import hashlib
import json

ECHO = "https://your-own-fingerprint-echo.example/whoami"

def tls_voice(proxy_url: str) -> dict:
    """Route one request through `proxy_url` and return the ClientHello features
    the echo server observed. Assumes the echo returns JSON with the fields below.
    """
    resp = requests.get(
        ECHO,
        proxies={"http": proxy_url, "https": proxy_url},
        timeout=(5, 15),  # connect, read — generous, this is a probe
    )
    resp.raise_for_status()
    hello = resp.json()
    # hello = {"ciphers": [...], "extensions": [...], "groups": [...],
    #          "sig_algs": [...], "grease": bool, "alpn": [...], "versions": [...]}
    return normalize(hello)

def normalize(hello: dict) -> dict:
    """Order-insensitive set membership + order-sensitive tuples, hashed two ways.
    The `voice_id` collapses to a stable key; the raw dict is what we diff on."""
    ordered = {
        "ciphers": tuple(hello.get("ciphers", [])),
        "extensions": tuple(hello.get("extensions", [])),
    }
    sets = {
        "groups": frozenset(hello.get("groups", [])),
        "sig_algs": frozenset(hello.get("sig_algs", [])),
    }
    blob = json.dumps(
        {
            "ciphers": list(ordered["ciphers"]),
            "extensions": list(ordered["extensions"]),
            "grease": bool(hello.get("grease", False)),
            "alpn": list(hello.get("alpn", [])),
        },
        sort_keys=True,
    )
    return {
        "voice_id": hashlib.sha256(blob.encode()).hexdigest()[:16],
        "ordered": ordered,
        "sets": sets,
        "grease": bool(hello.get("grease", False)),
        "raw": hello,
    }

Note the deliberate design choice: voice_id uses an order-sensitive hash of ciphers and extensions because that is what origins actually inspect, while we keep the sets separately so we can distinguish a harmless reordering from a real feature change (a key_share extension vanishing is categorically different from two extensions swapping places).

Turning voices into a drift detector

The interesting work starts once you've snapshotted a voice per exit. A single sample tells you what an exit looks like; drift is a property of the sequence of samples you take from the same exit over time. The detector below keeps a short ring of recent voice_ids per exit and flags three states: stable, alternating, and mutating.

from collections import defaultdict, deque

class VoiceTracker:
    """Track TLS voice per exit and classify drift.

    states:
      STABLE      one voice observed, or all recent samples identical
      ALTERNATING exactly two voices, swapping — the classic sign of a
                  round-robin pool sitting behind one advertised IP
      MUTATING    three or more distinct voices in the window — churn or
                  a backend that re-instantiates its TLS context per conn
    """
    WINDOW = 8

    def __init__(self):
        self._history = defaultdict(lambda: deque(maxlen=self.WINDOW))

    def record(self, exit_id: str, voice_id: str) -> str:
        h = self._history[exit_id]
        h.append(voice_id)
        distinct = set(h)
        if len(distinct) == 1:
            return "STABLE"
        if len(distinct) == 2:
            return "ALTERNATING"
        return "MUTATING"

ALTERNATING with exactly two voices is the most diagnostically rich outcome. It almost always means the provider advertises one sticky IP to you but load-balances your connection across two gateway processes (often on different hosts or different Node versions), each with its own TLS stack. From the origin's perspective, the same IP is now presenting two different browser fingerprints over a short window — an anomaly a decent anti-bot system can score. Detecting it locally, before the origin punishes you, converts a silent reputation burn into a routing decision: pin that exit to a single TLS context, or drop it from the authenticated pool.

MUTATING is rarer and worse. Three-plus voices from one advertised exit means the backend is cheap (a generic forward proxy that rebuilds its TLS context per connection), and you should generally not route identity-bearing traffic through it.

Coherence: the fingerprint you can't ignore

Drift detection gets you a temporal view. The complementary check is coherence: does the TLS voice you present match the HTTP-layer identity you're claiming in the same request? This is where many "good JA3" scrapers still get blocked. If your ClientHello advertises h2 via ALPN and Chrome 124's cipher order, but your HTTP layer sends User-Agent: Chrome/124, yet your header order and HTTP/2 SETTINGS look like Python's httpx default, you have an incoherent fingerprint. The proxy exit is the place this most often breaks — the gateway terminates and re-originates TLS, so it can silently rewrite your ALPN negotiation to http/1.1 even when you asked for h2.

Capture both the TLS voice and the HTTP voice per exit and diff them:

def coherence_score(tls: dict, http: dict) -> float:
    """Cheap 0..1 score: do the two layers agree on browser family + protocol?"""
    score = 1.0
    # Protocol: ALPN said h2 but the connection spoke h1.1?
    if "h2" in tls.get("raw", {}).get("alpn", []) and http.get("version") != "h2":
        score -= 0.6
    # GREASE: browser hellos carry GREASE; if the HTTP layer doesn't spoof
    # it (no GREASE-like header ordering), flag the mismatch.
    if tls.get("grease") and not http.get("grease_headers", False):
        score -= 0.3
    # UA/family mismatch is the most common, subtract the rest.
    if tls.get("family") != http.get("family"):
        score -= 0.4
    return max(0.0, round(score, 2))

A coherent exit scores 1.0. The moment an exit that was coherent this morning goes incoherent, you have caught a gateway upgrade, a backend swap, or a pool reassignment — usually before the origin does.

Feeding drift back into the router

None of this pays rent unless the classification changes what you do. Three concrete actions, in increasing order of aggressiveness:

Re-probe on state change. Once an exit flips from STABLE to ALTERNATING, tighten its probe interval so you catch the third voice quickly, and add it to a watch metric.
Down-weight in the assignment layer. Push MUTATING and incoherent exits to the bottom of the candidate list for authenticated or identity-bearing work, but keep them usable for stateless fetches where a fingerprint anomaly costs little.
Quarantine-and-cool-down. Treat a coherence drop as a soft failure event for the exit, identical in plumbing to the circuit breaker you already use for HTTP-level errors: stop new pairings, let in-flight ones drain, re-probe after a cool-down.

The unifying insight is that TLS fingerprinting is not only a thing the origin does to you. The handshake your proxy emits is telemetry about the proxy, and telemetry you collect on a schedule is telemetry you can trend, alert on, and route around. Your ClientHello drift is cheaper to collect than a health check from the exit side and arrives earlier than a 403. Start by snapshotting one voice per exit per hour; the ALTERNATING surprises you find in the first day will justify the whole pipeline.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Your Accounts Move in Lockstep: De-Synchronizing the Traffic Rhythms That Link a Shared Proxy Pool

Greta — Wed, 23 Sep 2026 01:22:22 +0000

Your Accounts Move in Lockstep: De-Synchronizing the Traffic Rhythms That Link a Shared Proxy Pool

You gave every account its own exit IP. You split the cookie jars, the local storage, the browser profiles. You audited the pipeline for correlation leaks and found nothing. And still, a pattern-matching system on the origin side can group your accounts with a simple query — because they all wake up at the same time.

Traffic-shape correlation is the blind spot of otherwise careful multi-account engineering. We are trained to think of identity as things a request carries: IP, fingerprint, cookies, headers. But identity is also the request's position in time. When thirty accounts begin their daily runs within the same ninety-second window, take identical inter-request pauses (because they share a time.sleep(2)), and go silent at the same moment your cron window closes, the aggregate traffic curve looks like a single controller operating thirty puppets — because it is. The IPs differ; the conductor doesn't.

The core idea: correlation resistance must be applied to the schedule, not just the payload. Each account needs its own phase, its own rhythm, and its own working hours — derived from its own identity, not from your infrastructure's convenience. This article shows what lockstep looks like in your own logs, and a scheduler that eliminates it.

What lockstep looks like from the origin's side

An origin analyst (or more realistically, an automated clustering job) sees per-account request time series. Lockstep accounts look like this, viewed at one-minute resolution:

minute:        0  1  2  3  4  5  6  7  8  9
account A:     .  ## ## ## ## ## ## .  .  .
account B:     .  ## ## ## ## ## ## .  .  .
account C:     .  ## ## ## ## ## ## .  .  .
...            (all 30 accounts: identical bars)

Every pairwise cross-correlation of those series spikes at lag zero. Clustering on "accounts whose activity curves correlate above 0.9 at zero lag" returns your entire fleet in one bucket. No fingerprinting, no IP matching — just pandas.

The correlations that betray you, in descending order of severity:

Start-time phase. All accounts begin their session at nearly the same wall-clock minute. This is almost always a cron artifact: 0 9 * * * launches every worker simultaneously.
Rhythm signature. The distribution of inter-request gaps is identical across accounts because it comes from shared code — one sleep() constant, one backoff function, one page-wait.
Working-hours envelope. All accounts active in the same time window and silent outside it. Real humans in different cities and time zones don't share an envelope; cron jobs do.
Burst co-occurrence. Retries, rate-limit pauses, and error clusters happen at the same instants across accounts — typically because one shared event (a provider hiccup, a scheduler retry storm) hits every worker identically.

Point 4 deserves emphasis: shared infrastructure produces shared incidents, and shared incidents are timestamps in every account's log simultaneously. Even with perfect per-account independence, a single origin outage that triggers retries across all workers writes a correlated mark into thirty time series at once.

The de-synchronized scheduler

The fix is a per-account schedule with three randomized components — phase, rhythm, and envelope — each derived from a stable per-account seed so that the schedule is consistent for a given identity across days (accounts are also suspicious if their rhythms change every run).

import hashlib
import random


def account_rng(account_id: str, day: str, salt: bytes = b"v1") -> random.Random:
    """Deterministic per (account, day): stable within the day,
    different across accounts, drifting slowly across days."""
    h = hashlib.blake2b(f"{salt}:{account_id}:{day}".encode(), digest_size=16)
    return random.Random(h.digest())


def session_phase(rng: random.Random, window: tuple[int, int]) -> int:
    """Start offset in seconds inside the allowed window, beta-shaped:
    most accounts start somewhere mid-window, a few early, a few late."""
    u = rng.betavariate(2.0, 2.0)
    return int(window[0] + u * (window[1] - window[0]))


def make_rhythm(rng: random.Random):
    """This account's own inter-request gap distribution."""
    base = rng.uniform(4.0, 12.0)          # its characteristic pace
    jitter = rng.uniform(0.3, 0.8)         # how noisy its pace is
    def gap() -> float:
        return max(1.0, rng.gauss(base, base * jitter))
    return gap


def working_hours(rng: random.Random) -> tuple[int, int]:
    """Each account 'lives' in a slightly different part of the day."""
    start_h = rng.choice([7, 8, 9, 10, 11])
    span_h = rng.uniform(6.0, 11.0)
    return start_h * 3600, int((start_h + span_h) * 3600)

The scheduler then assembles these into a run plan, and — critically — staggered starts are scheduled, not natural:

from datetime import datetime, timedelta


def build_day_plan(accounts: list[str], date: str):
    day_window = (7 * 3600, 21 * 3600)     # global allowed window
    plans = {}
    for acc in accounts:
        rng = account_rng(acc, date)
        phase = session_phase(rng, day_window)
        gap = make_rhythm(rng)
        hours = working_hours(rng)
        plans[acc] = {
            "start_at": phase,
            "gap_fn": gap,
            "session_span": min(hours[1] - hours[0], 8 * 3600),
        }
    return plans


def phase_separation(plans: dict, min_gap_s: float = 90.0) -> dict:
    """Second pass: enforce a minimum spacing between any two starts.
    Beta-phases alone still collide occasionally; make collision impossible."""
    starts = sorted(plans.items(), key=lambda kv: kv[1]["start_at"])
    for (a1, p1), (a2, p2) in zip(starts, starts[1:]):
        if p2["start_at"] - p1["start_at"] < min_gap_s:
            p2["start_at"] = p1["start_at"] + min_gap_s
    return plans

Two design decisions in this scheduler are load-bearing:

Deterministic seeded randomness. The schedule must be reproducible (for debugging and for re-running a day after a crash) but must also differ across accounts and drift slowly across days. A fixed seed per account forever is its own fingerprint — a cluster of accounts whose rhythms never evolve over months is as unnatural as a cluster that resets daily. The (account, day) seed with a salt gives stability within a day and slow evolution across weeks.
Envelope per account, not per fleet. Real identities in different geographies have different active hours. If your exits are geo-targeted to different regions, align each account's working hours to its exit's region — an account browsing through a Tokyo exit that is active at 3 a.m. Tokyo time is a second, redundant correlation signal layered on top of the first.

Sanity-check it the way the origin would

Don't trust the design; test it with the adversary's own tool. Compute the pairwise zero-lag correlation matrix of your fleet's per-minute activity curves before and after de-synchronization:

import numpy as np
import pandas as pd


def lockstep_score(activity: pd.DataFrame) -> float:
    """Fraction of account pairs with zero-lag correlation > 0.9.
    activity: rows = one-minute bins, columns = accounts, values = req counts."""
    corr = activity.corr()
    n = len(corr)
    pairs = n * (n - 1) / 2
    high = ((corr > 0.9).sum().sum() - n) / 2
    return high / pairs

# before: lockstep_score(df)  ->  1.000   (every pair correlated)
# after:  lockstep_score(df)  ->  0.003   (near-zero; resample weekly)

The "after" number won't be exactly zero and shouldn't be: absolute zero correlation across a large fleet is itself slightly unnatural. What you want is a value in the noise floor of genuinely independent users — and you want it stable under weekly resampling.

One shared event can re-correlate everything

The subtlest part of the problem is that your scheduler can be perfect and your fleet still re-correlates during incidents. When the proxy gateway hiccups, every worker experiences it in the same minute, and if every worker responds with the same retry policy, thirty accounts log an identical pause-then-burst at the same timestamp. Mitigations:

Randomize retry backoff per account, not just per request — jitter the retry curve itself (e.g., multiply the base delay by a per-account factor drawn from the same seeded RNG).
Absorb shared events into the per-account rhythm — on a detected shared incident, resume each worker at its own next rhythm gap rather than simultaneously.
Rate-limit scheduler restarts. After a crash or deploy, resist restarting every worker at once; honor each account's phase plan (the deterministic seed makes this trivial — the plan for the day already exists, just skip forward to the current position in it).

Lockstep is, in the end, an artifact of convenience: one cron line and one shared sleep constant are convenient, and the origin's clustering job is built on precisely that convenience. Spreading your accounts across phase, rhythm, and envelope costs you one scheduler module and roughly nothing in throughput — thirty accounts starting within a four-hour beta-shaped window still finish their daily work in the same day — and it removes the most legible signal a linking attack can find after all your payload-level work is done.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

DEV Community: Greta

QUIC Changes the Fingerprint Game: Why Your Proxy Stack Silently Downgrades to HTTP/2

QUIC Changes the Fingerprint Game: Why Your Proxy Stack Silently Downgrades to HTTP/2

What actually happens when a proxied client meets an h3 origin

What QUIC fingerprinting actually reads

Measuring the gap: an Alt-Svc harvester and downgrade ledger

Verifying what you actually negotiated

The decision framework

Rotation Is a Session Decision, Not a Request Decision: A Beginner's Stickiness Rulebook

Rotation Is a Session Decision, Not a Request Decision: A Beginner's Stickiness Rulebook

Why per-request rotation breaks state

The rulebook

Implementing a stickiness router

Choosing the TTL

The takeaway

Keep-Alive Is an Exit Property Too: Counting the TLS Handshakes Your Proxy Pool Actually Pays For

Keep-Alive Is an Exit Property Too: Counting the TLS Handshakes Your Proxy Pool Actually Pays For

The mechanism, precisely

Measuring it, in about 20 lines

The same disease in requests and httpx

Four configs, one metric

Where the leak actually comes from

Design rules that fall out of the numbers

Preflight or Perish: The 60-Second Proxy Check That Saves Hours of Crawler Debugging

Preflight or Perish: The 60-Second Proxy Check That Saves Hours of Crawler Debugging

What "working" actually means, decomposed

A runnable preflight harness

Interpreting the results

The preflight is also your canary

Preflight the pool, not just the string

Make the preflight a scheduled job, not a ritual

Where the preflight does not help

Proxy-Authorization Is Not a Normal Header: How requests, httpx, and aiohttp Each Handle It

Proxy-Authorization Is Not a Normal Header: How requests, httpx, and aiohttp Each Handle It

407 is not 401, and it is not per-request

Three styles, three sets of gotchas

Prove what your client sends: a 60-second auth probe

Reading the failure by its signature

The cost of challenge-response, in one measurement

Digest, and the limits of what you can fix

Checklist

Asyncio Semaphores Are the Wrong Knob: Per-Exit Concurrency Caps for Rotating Proxy Pools

Asyncio Semaphores Are the Wrong Knob: Per-Exit Concurrency Caps for Rotating Proxy Pools

Why the global semaphore fails once proxies enter the picture

The right model: a lease pool keyed by exit

Sizing per_exit without guessing

The bonus: a natural place for health signals

A benchmark that finds the knee, not a guess

What the cap does NOT protect: the origin

What beginners should take away

Slow Down Before the 429: EWMA Latency Inflation as Your Real Politeness Signal

Slow Down Before the 429: EWMA Latency Inflation as Your Real Politeness Signal

Why latency is the honest signal

The baseline problem: absolute latency is meaningless

Latency vs the timeout trap

Making latency one input among several

What you get in production

Not Every Challenge Is a CAPTCHA: A Triage Pipeline for Silent Rate Limits, JS Interstitials, and Real Puzzles

Not Every Challenge Is a CAPTCHA: A Triage Pipeline for Silent Rate Limits, JS Interstitials, and Real Puzzles

The four challenge tiers

A cheap triage classifier

The escalation ladder, in cost order

Instrument the tiers, because they score your proxy pool

One Exit, Fifty Workers: Stopping the Coordinated Retries That Turn One 429 Into Fifty

One Exit, Fifty Workers: Stopping the Coordinated Retries That Turn One 429 Into Fifty

The setup: many workers, one shared exit

Decorrelated jitter: the first fix, and why it's not enough

The exit-scoped token bucket

Parsing Retry-After properly (it's rarely an integer)

The distributed problem: when "per exit" spans processes

The metric that proves it's working

Prior-Knowledge or ALPN Upgrade? The HTTP/2 Connection Preface Tells You What Your Proxy Got Wrong

Prior-Knowledge or ALPN Upgrade? The HTTP/2 Connection Preface Tells You What Your Proxy Got Wrong

The two ways a client can speak HTTP/2

Why proxies get this wrong

Building the probe

The four fingerprints that actually differ

Routing on the HTTP/2 signature

Fingerprint Drift Is a Pool Health Signal: Clustering TLS Voices Behind a Single Proxy IP

Fingerprint Drift Is a Pool Health Signal: Clustering TLS Voices Behind a Single Proxy IP

Sizing `per_exit` without guessing