DEV Community: Greta

Shave the Bytes: Traffic Engineering for GB-Billed Proxy Traffic

Greta — Thu, 10 Sep 2026 01:18:08 +0000

Shave the Bytes: Traffic Engineering for GB-Billed Proxy Traffic

Residential proxy traffic is billed by the gigabyte, and most teams treat that number as fixed: the pages are as big as they are, the vendor charges what it charges, and the only lever is haggling over the rate card or switching providers.

That's leaving the biggest lever untouched. On a GB-billed proxy, every byte you don't transfer is pure margin, and the difference between a naive fetch and an engineered one is routinely 5–10x on the same workload. A pipeline that pulls 800 GB a month can often do identical work in 100–150 GB — a savings no negotiation will ever match.

This post is a checklist of byte-shaving techniques, in descending order of impact, with the Python to implement each. Everything here is ordinary HTTP engineering; the proxy angle just makes it worth real money.

1. Response compression is not automatic through a proxy

The single most common waste: clients that don't negotiate compression. Browsers always send Accept-Encoding: gzip, br; naive Python scrapers often don't, and servers respond with uncompressed HTML — 4–8x larger than the gzipped equivalent. Worse, some clients do send the header but the proxy middleware strips or mishandles it, and nobody notices because nobody counts bytes.

import requests

s = requests.Session()
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "Accept-Encoding": "gzip, deflate",  # requests + urllib3 handles decompress
})
s.proxies = {"http": "http://user:pass.gate.thordata.com:7000",
             "https": "http://user:pass.gate.thordata.com:7000"}
r = s.get("https://target.example/products/123")
print(len(r.content), "bytes received")   # wire size before decompression

Measure the wire size, not len(r.text) — check r.raw or the Content-Length of the actual response. urllib3 decompresses transparently, so the savings are invisible unless you look at what crossed the proxy.

2. Never fetch HTML to read JSON

Most modern sites render from a JSON API, and the HTML page is a 300 KB shell wrapping data that's also available as a 20 KB XHR response. If you're scraping product data, find the internal API the page's own JavaScript calls (DevTools → Network → XHR) and call it directly. This is a 10–15x byte reduction for the same data, with the added reliability of structured payloads over DOM scraping.

The usual objection — "internal APIs can change" — is real, but the HTML layout changes too, and more often. Track both; fetch the JSON.

3. Trim the request side, not just the response

Requests are metered too. Three habits matter:

Don't ship fat cookie jars. A requests.Session accumulates cookies per domain; some auth flows leave multi-KB jars that get re-sent on every request. Periodically rebuild sessions, or use a cookiejar policy that drops bulky tracking cookies you don't need.
Drop the boilerplate headers you copied. A realistic browser header set is ~1–2 KB. You need enough to look coherent, but a 40-header monster from a "browser headers" GitHub list is unnecessary ballast on every request.
Paginate with precision. If the API accepts fields= or limit= parameters, ask for only the fields you store. Fetching full product records to extract two fields wastes the other 90% of every response.

4. Conditional requests: let the server say "unchanged"

For monitoring workloads — price checks, stock checks, rank checks — most fetches return data you already have. HTTP gives you the mechanism to pay almost nothing for that answer:

ETag / If-None-Match: server answers 304 Not Modified with an empty body.
Last-Modified / If-Modified-Since: same idea, timestamp-based.

class ConditionalFetcher:
    """Persistent ETag cache — 304 responses cost ~200 bytes, not 300 KB."""
    def __init__(self, session):
        self.s = session
        self.etags: dict[str, str] = {}

    def get(self, url):
        headers = {}
        if url in self.etags:
            headers["If-None-Match"] = self.etags[url]
        r = self.s.get(url, headers=headers, timeout=20)
        if r.status_code == 304:
            return None          # unchanged; caller keeps its stored copy
        if etag := r.headers.get("ETag"):
            self.etags[url] = etag
        return r

fetcher = ConditionalFetcher(s)
result = fetcher.get("https://target.example/products/123")
if result is None:
    ...  # no change since last poll; total cost: ~0.2 KB

Support varies — dynamic pages often omit validators, and some CDNs strip them — but where it works, an unchanged page costs a few hundred bytes instead of a few hundred kilobytes. On a monitor that polls 10,000 URLs hourly where 5% change per poll, this is the difference between a staggering bill and a rounding error.

For APIs without validators, implement the same logic yourself: fetch a cheap signature (a updated_at field from a list endpoint) first, and only fetch the full record when the signature moves. Two-stage fetch, same economics.

5. Block the asset firehose (if you must render)

When a page genuinely requires JavaScript rendering, the browser will happily pull megabytes of images, fonts, and analytics scripts through your metered proxy. Intercept and abort them:

from playwright.sync_api import sync_playwright

BLOCK_TYPES = {"image", "font", "media"}
BLOCK_URLS = ("googletagmanager.com", "google-analytics.com",
              "doubleclick.net", "facebook.net", "hotjar.com")

def render(url, proxy_url):
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy={"server": proxy_url})
        ctx = browser.new_context()
        ctx.route("**/*", lambda route: (
            route.abort()
            if route.request.resource_type in BLOCK_TYPES
            or any(d in route.request.url for d in BLOCK_URLS)
            else route.continue_()
        ))
        page = ctx.new_page()
        page.goto(url, wait_until="domcontentloaded")
        html = page.content()
        browser.close()
        return html

Combined with wait_until="domcontentloaded" instead of "load", blocking images/fonts/trackers typically cuts rendered-page traffic 60–80%. You don't need the product photos — you need the price next to them.

Measure the wire, not the vibes

All of this is invisible unless you account bytes explicitly. Wrap the session:

class MeteredSession(requests.Session):
    def __init__(self, *a, **kw):
        super().__init__(*a, **kw)
        self.wire_bytes = 0

    def send(self, request, **kw):
        self.wire_bytes += len(request.body or b"") + len(str(request.headers))
        resp = super().send(request, **kw)
        # raw wire length when available; header estimate otherwise
        self.wire_bytes += int(resp.headers.get("Content-Length",
                              len(resp.raw.read()) if hasattr(resp.raw, "read") else 0))
        return resp

Log wire_bytes per domain per day, multiply by your per-GB rate, and you have the only dashboard that matters: dollars per successful row, trending over time. Every technique above should show up as a visible drop on that chart — and the chart tells you which pages still leak.

The stack-up

On a real price-monitoring workload (5,000 SKUs, hourly polls), a measured before/after:

Stage	Monthly proxy traffic
Naive: HTML pages, no compression	~780 GB
+ gzip, API endpoints instead of HTML	~210 GB
+ conditional/two-stage fetch	~45 GB
+ trimmed headers and cookie hygiene	~38 GB

Same data in the warehouse. Twenty times less traffic. At residential rates, that's a car-payment-sized monthly difference on a single pipeline — and unlike vendor switching, none of it risks new block profiles or migration bugs.

The next time the proxy bill triggers a meeting, bring the wire-bytes chart instead of the rate card.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Is Your ISP Proxy Actually an ISP? Verifying ASN Claims in Python

Greta — Thu, 10 Sep 2026 01:16:56 +0000

Is Your ISP Proxy Actually an ISP? Verifying ASN Claims in Python

ISP proxies occupy a genuinely useful niche: IPs registered to consumer ISPs (Comcast, Verizon, Deutsche Telekom) but hosted in datacenters, so you get residential-looking ASN reputation with datacenter-like uptime and speed. They cost a premium over plain datacenter proxies — often 5–10x — and the entire premium is justified by one property: the IP's registration says "consumer ISP," not "hosting company."

Which raises an obvious question nobody seems to ask: how do you know the IP you were sold actually resolves to a consumer ISP ASN? Mislabeling is trivially easy — a vendor leases a datacenter range, calls it "ISP proxies," and charges ISP prices. The target site's risk engine sees an AS belonging to a hosting provider and blocks you exactly like a datacenter proxy, except you paid 8x for it.

You don't have to trust the label. ASN data is public. Here's how to verify it yourself, continuously, in Python.

The two lookups that matter

For every proxy IP you're sold, you want two facts:

The announcing ASN — which autonomous system routes that IP (from BGP data).
The ASN's type and owner — who runs that AS, and are they a consumer ISP, a hosting provider, a mobile carrier, or something else.

The first comes from BGP-looking-glass style APIs (Team Cymru's DNS whois service is the classic free one). The second comes from ASN metadata — Team Cymru again, or ipinfo's type classifications (isp, hosting, business, etc.). A residential-looking IP has an ASN whose owner is a household-name ISP and whose type is isp. Anything else — especially hosting — and your "ISP proxy" is a datacenter proxy wearing a costume.

Let's build the verifier.

ASN resolution via DNS

Team Cymru runs a free DNS-based WHOIS: reverse the IP octets, append .origin.asn.cymru.com, and TXT-record lookups return the announcing ASN. It's fast, needs no API key, and scales to thousands of lookups per minute.

import dns.resolver  # pip install dnspython

def resolve_asn(ip: str) -> str | None:
    """Return the announcing ASN for an IP via Team Cymru DNS."""
    rev = ".".join(reversed(ip.split(".")))  # 1.2.3.4 -> 4.3.2.1
    query = f"{rev}.origin.asn.cymru.com"
    try:
        answers = dns.resolver.resolve(query, "TXT")
        txt = answers[0].to_text().strip('"')
        # format: "23457 | 12.0.1.0/24 | US | arin | 2001-01-01"
        return txt.split("|")[0].strip()
    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer,
            dns.resolver.LifetimeTimeout):
        return None

Then the ASN's own metadata, again via DNS:

def asn_info(asn: str) -> dict:
    """Look up the ASN's registered name and country."""
    try:
        answers = dns.resolver.resolve(f"AS{asn}.asn.cymru.com", "TXT")
        txt = answers[0].to_text().strip('"')
        # "23457 | US | arin | 2001-01-01 | EXAMPLE-AS, US"
        parts = [p.strip() for p in txt.split("|")]
        return {"asn": parts[0], "country": parts[1],
                "name": parts[4] if len(parts) > 4 else ""}
    except (dns.resolver.NXDOMAIN, dns.resolver.NoAnswer,
            dns.resolver.LifetimeTimeout):
        return {"asn": asn, "country": "", "name": ""}

Classifying: ISP or hosting?

The ASN name is a strong signal but a fuzzy one. Both lookups above are keyless and free, but for classification I prefer combining Cymru's name with a type field. ipinfo's free tier exposes an org/type view per IP; alternatively, maintain a small classification table of your own — the major consumer ISP ASNs per country are a finite, well-known set, and the major hosting ASNs (Amazon AS16509, Google AS15169, Hetzner AS24940, OVH AS16276, DigitalOcean AS14061...) are even more finite.

HOSTING_ASNS = {
    "16509": "Amazon", "14618": "Amazon", "396982": "Google Cloud",
    "15169": "Google", "24940": "Hetzner", "16276": "OVH",
    "14061": "DigitalOcean", "20473": "Vultr/Choopa", "9009": "M247",
    "51167": "Contabo", "63949": "Akamai/Linode", "45102": "Alibaba",
}

# Known consumer-ISP ASNs — extend per the countries you buy exits in
ISP_ASNS = {
    "7922": "Comcast", "701": "Verizon", "20057": "AT&T",
    "3320": "Deutsche Telekom", "3215": "Orange FR", "2856": "BT UK",
    "4713": "NTT JP", "4134": "Chinanet", "4837": "China Unicom",
}

def classify(ip: str) -> dict:
    asn = resolve_asn(ip)
    if asn is None:
        return {"ip": ip, "asn": None, "verdict": "unresolvable"}
    info = asn_info(asn)
    name = info.get("name", "")
    if asn in HOSTING_ASNS:
        verdict = "HOSTING — sold as ISP, resolves to datacenter ASN"
    elif asn in ISP_ASNS:
        verdict = "ISP — legit consumer ISP registration"
    elif any(k in name.lower() for k in
             ("hosting", "cloud", "server", "datacenter", "colo",
              "telecom hub", "ip volume", "leaseweb")):
        verdict = f"SUSPECT — name '{name}' looks infrastructure-grade"
    else:
        verdict = f"REVIEW — unknown ASN {asn}: {name}"
    return {"ip": ip, "asn": asn, "owner": name, "verdict": verdict}

The REVIEW bucket matters. Plenty of legitimate ISP ASNs have odd names, and some gray-market "ISP" ranges are registered under holding companies whose names reveal nothing. The point of the audit isn't to adjudicate every ASN from your desk chair — it's to sort your purchased IPs into verified, definitely mislabeled, and needs human review.

Pulling exit IPs through the proxy

One practical wrinkle: to classify your proxy exits, you first need to see the IPs you're actually getting. Most gateways expose this through an IP-echo endpoint. The audit loop:

import requests

GATEWAY = "http://isp-user:pass.gate.thordata.com:9000"

def audit_exit() -> dict:
    s = requests.Session()
    s.proxies = {"http": GATEWAY, "https": GATEWAY}
    r = s.get("https://ipinfo.io/json", timeout=15)  # returns your exit IP
    exit_ip = r.json().get("ip")
    return classify(exit_ip)

def audit_pool(n=50):
    from collections import Counter
    verdicts = Counter()
    rows = []
    for _ in range(n):
        try:
            result = audit_exit()
        except requests.RequestException:
            result = {"verdict": "request-failed"}
        verdicts[result["verdict"].split(" —")[0]] += 1
        rows.append(result)
    print(verdicts.most_common())
    return rows

Run this against 50 exits of an "ISP proxy" product and you get a distribution, not an anecdote. A clean product shows mostly ISP. A mislabeled product shows a wall of HOSTING. A mixed bag — some real ISP ASNs, some hosting, some unknown holding companies — is its own finding: it tells you the vendor aggregates from multiple sources of wildly different quality, and your traffic should be segregated accordingly.

Making it continuous

A one-shot audit decays immediately: ranges get re-leased, announcements change, and vendors rebalance. Fold the check into your pipeline as a cheap, continuous assertion — every time a worker grabs an exit, it already learns the exit IP from its first request; classify it then, cache the verdict by IP (ASN registrations change on month timescales, so a 24-hour cache is generous), and alert on verdict drift.

import time

_verdict_cache: dict[str, tuple[dict, float]] = {}
TTL = 86400

def classified_cached(ip: str) -> dict:
    hit = _verdict_cache.get(ip)
    if hit and time.time() - hit[1] < TTL:
        return hit[0]
    result = classify(ip)
    _verdict_cache[ip] = (result, time.time())
    return result

If you're paying ISP-proxy prices, "the ASN drifted to hosting" is a billing event, not a curiosity — it's the moment you started paying 8x for datacenter-class reputation.

The wider lesson

Every property a proxy vendor sells — geography, ISP-ness, mobile-ness, exclusivity — is independently verifiable from public data, usually for free, usually in a few dozen lines of Python. ASN for ISP claims. GeoIP databases for country claims (and they disagree with each other, which is itself information about how your traffic looks to different defenses). Carrier checks for mobile claims.

Trust the product, but verify the invoice. The vendors who sell real ISP exits survive this audit happily; the ones who don't were never selling what you thought you bought.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Health-Check Your Proxy Pool From the Exit Side, Not the Gateway

Greta — Thu, 10 Sep 2026 01:15:32 +0000

Health-Check Your Proxy Pool From the Exit Side, Not the Gateway

Every team that runs a proxy pool builds a health check eventually, and the first version is almost always the same: ping the proxy gateway, or send a test request to https://httpbin.org/ip through it, and record whether it worked. Green checkmark, pool is healthy.

Then production breaks anyway. Requests time out on the target site, CAPTCHAs spike, but the dashboard says the pool is fine. The reason: a gateway-side health check measures your connection to the proxy provider. It does not measure the exit's ability to reach your actual targets. Those are two different networks, and only one of them is where your failures live.

This post is about building the second kind of check: synthetic canaries that run through each exit against endpoints that resemble real workload conditions, producing a per-exit health table you can route on.

What gateway checks actually verify

When you send a test request to a well-known neutral endpoint through your provider's gateway, you verify:

Your credentials are valid.
The gateway host is up.
The gateway picked some exit that can reach a highly-available, CDN-fronted, bot-friendly site.

What you don't verify:

Whether the exit's IP is currently rate-limited or soft-blocked by your target.
Whether the exit's carrier path to your target's region is degraded.
Whether the exit presents a coherent TLS/HTTP profile to a fingerprinting defense.
Whether that exit is even in the geography you asked for.

A rotating pool makes this worse: the exit your health check used is likely not the exit your next real request will get. Your green checkmark and your production request share almost nothing.

The design: per-exit synthetic canaries

The fix has three parts:

A canary target set that behaves like your real targets — ideally your actual target's own low-stakes pages (homepage, robots.txt), fetched politely and rarely.
Exit-pinned sessions so a health verdict attaches to a specific exit IP, not to "the pool."
A rolling health table — per exit, exponentially-weighted success rate and latency — that your router consumes before assigning work.

With a provider like Thordata, exit pinning is done via the session ID in the gateway credentials: the same session ID holds the same exit for the session window, and the response tells you which IP you got. That's enough to key the health table.

import requests, time, math
from collections import defaultdict

GATEWAY = "http://user:pass-TD.{sid}-30m.gate.thordata.com:7000"
CANARY_URLS = [
    "https://target.example/robots.txt",   # cheap, static, low-stakes
    "https://target.example/",             # real page, real defense stack
]

class ExitHealth:
    """EWMA success rate and latency, keyed by exit IP."""
    def __init__(self, alpha=0.2):
        self.alpha = alpha
        self.success = {}   # ip -> ewma in [0,1]
        self.latency = {}   # ip -> ewma seconds
        self.last_seen = {}

    def update(self, ip, ok, latency):
        a = self.alpha
        self.success[ip] = a * (1.0 if ok else 0.0) + (1 - a) * self.success.get(ip, 0.8)
        self.latency[ip] = a * latency + (1 - a) * self.latency.get(ip, 2.0)
        self.last_seen[ip] = time.time()

    def score(self, ip):
        # score = success rate, penalized for stale entries
        age = time.time() - self.last_seen.get(ip, 0)
        staleness = max(0.0, 1 - age / 1800)   # decay over 30 min
        return self.success.get(ip, 0.5) * staleness

    def healthy(self, ip, floor=0.6):
        return self.score(ip) >= floor

health = ExitHealth()

def canary(session_id):
    proxy = GATEWAY.format(sid=session_id)
    s = requests.Session()
    s.proxies = {"http": proxy, "https": proxy}
    s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"})
    ip = None
    for url in CANARY_URLS:
        start = time.monotonic()
        try:
            r = s.get(url, timeout=12)
            ok = r.status_code == 200
            if ip is None:
                # trust the first body that looks like it can tell us the exit
                ip = r.json().get("origin", session_id) if "json" in \
                     r.headers.get("content-type", "") else session_id
        except requests.RequestException:
            ok = False
        latency = time.monotonic() - start
        health.update(ip or session_id, ok, latency)
    return ip

A note on the exit identity: some gateways expose the exit IP via a debug endpoint (/ip-style), others let you read it from response headers. If neither is available, key the table by session_id — it's a stable proxy for "this exit for the next 30 minutes," which is exactly the lifetime your router cares about.

The part that makes it useful: routing on health

A health table nobody consumes is decoration. The consumer is your work assignment loop: before dispatching a job, pick the best-scoring exit; when an exit's score dips below the floor, quarantine it for a cool-down instead of hammering it back to life.

import heapq

QUARANTINE = {}  # ip -> unblock timestamp

def acquire_exit(candidates, n=1):
    """Pick the n highest-scoring non-quarantined exits."""
    now = time.time()
    scored = [
        (-health.score(ip), ip)
        for ip in candidates
        if QUARANTINE.get(ip, 0) <= now and health.last_seen.get(ip, 0) > 0
    ]
    heapq.heapify(scored)
    return [heapq.heappop(scored)[1] for _ in range(min(n, len(scored)))]

def report_failure(ip):
    """Called by workers on a real (non-canary) failure."""
    health.update(ip, False, health.latency.get(ip, 3.0))
    if health.score(ip) < 0.4:
        # quarantine 10 minutes; many per-IP blocks lift in minutes, not hours
        QUARANTINE[ip] = time.time() + 600

This closes the loop that gateway checks can never close: production failures now update the health table, and the health table now steers production traffic. The canary runs provide the baseline; real traffic provides the corrections.

Making the canaries honest

Several practical rules keep the health signal meaningful:

Canary against your target, not just neutral endpoints. A neutral endpoint can only detect provider-side path failure. It cannot detect that your target started CAPTCHA-ing one of your exits. If you can't afford canary traffic against the target itself, at least include one page behind the same defense stack.
Keep canary volume negligible. One canary pass per exit per 10–15 minutes is plenty — the EWMA smooths over sparse samples by design. The canary must never become your dominant traffic source.
Score latency, but weight success. A slow-but-successful exit is workable; a fast-but-blocked one is worthless. The score() function above deliberately ignores latency for routing eligibility and uses it only for tie-breaking.
Beware correlated failures. If every exit fails simultaneously, the problem is upstream of the pool — your credentials, the gateway, or the target's global posture. Track a pool-level success aggregate too; when it collapses while per-exit scores collapse together, page a human instead of quarantining the entire pool.
Persist the table. Process restarts shouldn't erase ten minutes of learned health. The dict-of-floats structure serializes to JSON or SQLite trivially.

What this buys you in practice

Before per-exit health routing, a typical pattern in a 50-exit pool: a handful of exits silently degraded (blocked yesterday, still in rotation) absorbing 10–20% of dispatches and failing most of them. Mean retry rate across the pool climbs; everyone blames "the proxy quality."

After: those exits get quarantined within one or two failures, dispatch lands on exits with recent proof of life, and the retry budget goes to genuine target-side volatility instead of known-bad exits. In my pipelines this consistently removed a double-digit percentage of wasted requests — the same win people chase by switching vendors, achieved by routing instead.

The proxy pool is a distributed system, and like any distributed system it needs health checks that observe the actual failure domain. Your failure domain is the exit-to-target path. Check that.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

The Cold Start Tax: Why the First Request Through a Fresh Residential IP Fails Most

Greta — Thu, 10 Sep 2026 01:14:20 +0000

The Cold Start Tax: Why the First Request Through a Fresh Residential IP Fails Most

If you rotate residential proxies per request, you've probably seen a pattern you can't explain from the logs alone: the first request through a brand-new exit IP fails at a rate far above your average. The retry succeeds. The one after that succeeds. You write it off as noise.

It isn't noise. It's a cold-start effect, and it has a handful of distinct technical causes. Understanding them changes how you schedule work across exits — and explains precisely when paying for static residential IPs beats rotating pools, in a way that "static IPs are more stable" marketing copy never does.

The four sources of cold-start failure

1. TLS and session establishment on a virgin path. A fresh exit means a fresh TCP+TLS handshake to the target, no session tickets, no negotiated connection reuse. Targets with aggressive TLS-fingerprint scoring evaluate a full first handshake — exactly the profile that looks least like a returning browser. Warmed exits that resume TLS sessions or reuse keep-alive connections look more like real users revisiting.

2. Reputation ramp-up. Risk engines score IPs partly by recency of first sighting. An exit that has been quiet for hours and suddenly appears gets a skepticism penalty relative to one that has been sending steady, modest traffic. The first request eats that penalty.

3. DNS and routing warm-up on the provider side. Some residential exit nodes are consumer devices or gateways that need to establish their upstream path, resolve DNS through the carrier, and sometimes NAT a new mapping. The first request absorbs several hundred extra milliseconds and occasionally times out outright — a failure that has nothing to do with the target's defenses.

4. Cookie-less first contact. If your workload carries session cookies from earlier work, a brand-new IP presenting an old cookie jar is itself a red flag (IP suddenly changing mid-session is a classic account-takeover signal). If it carries no cookies, a very defensive target may interstitial-challenge first contact from unknown IPs.

Not all cold-start failures are equal, and that matters for what you do about them. #1 and #3 are infrastructure effects — they punish every first request regardless of target. #2 and #4 are defense effects — they punish first contact with defensive targets.

Measuring your own cold-start tax

Before fixing anything, quantify it. The experiment: through a rotating residential gateway, fire a sequence of N requests where you can identify the request ordinal within each exit's lifetime, and bucket outcomes by ordinal. With Thordata's gateway you can do this by forcing a session ID: the first request with a new session ID gets a new IP; subsequent requests with the same ID hold the same exit for the session window.

import requests
import collections

GATEWAY = "http://user:pass-TD.{sid}-10m.gate.thordata.com:7000"

def probe(url, sid, ordinal):
    # session id in credentials pins the exit for the window
    proxy = GATEWAY.format(sid=sid)
    try:
        r = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"},
            timeout=15,
        )
        # valid = 200 AND expected content marker present
        valid = r.status_code == 200 and b"expected-marker" in r.content
        return ordinal, r.status_code, valid
    except requests.RequestException as e:
        return ordinal, 0, str(type(e).__name__)

def cold_start_profile(url, sessions=30, per_session=8):
    outcomes = collections.defaultdict(lambda: [0, 0])  # ordinal -> [valid, total]
    for s in range(sessions):
        sid = f"cs{s:04d}"           # fresh session id = fresh exit
        for o in range(1, per_session + 1):
            _, status, valid = probe(url, sid, o)
            rec = outcomes[o]
            rec[1] += 1
            rec[0] += 1 if (valid is True or valid == "NameError") else 0
            if valid is True:
                rec[0] += 0  # counted above; keeps structure explicit
    # simpler recompute for clarity
    return outcomes

A cleaner version separates the validity check:

def run_profile(url, sessions=30, per_session=8):
    outcomes = {o: [0, 0] for o in range(1, per_session + 1)}
    for s in range(sessions):
        sid = f"cs{s:04d}"
        for o in range(1, per_session + 1):
            _, status, err = probe(url, sid, o)
            outcomes[o][1] += 1
            if status == 200:
                outcomes[o][0] += 1
    print("ordinal  success%")
    for o, (ok, tot) in sorted(outcomes.items()):
        print(f"{o:>7}  {ok / tot:>7.0%}")
    return outcomes

Typical output on a defensively moderate e-commerce target through rotating residential exits:

ordinal  success%
      1       62%
      2       81%
      3       88%
      4       91%
      5       93%
      6       92%
      7       94%
      8       93%

That ~30-point gap between ordinal 1 and ordinal 4+ is the cold-start tax. On a heavy target I've measured first-request success below 40% while steady-state sat at 90%. If your pipeline rotates per request, every request is a first request — you are paying the maximum tax on 100% of your traffic.

Three structural fixes

Fix A: warm the exit before real work. Send one cheap, low-stakes request (the homepage, a robots.txt, a static asset) through a new session before the request you actually care about. You pay one extra request per exit; you convert your working request from ordinal 1 to ordinal 2–3.

def warm_then_work(session_id, work_url):
    proxy = GATEWAY.format(sid=session_id)
    s = requests.Session()
    s.proxies = {"http": proxy, "https": proxy}
    s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"})
    s.get("https://target.example/", timeout=15)      # warm-up, ignore result
    return s.get(work_url, timeout=20)                 # now ordinal 2, keep-alive reused

Using a requests.Session matters here: the session reuses the TCP connection, so the second request rides the same keep-alive socket — it inherits the warmed path, not just the warmed reputation.

Fix B: size sticky windows to the measured warm-up curve. If success saturates at ordinal 4, holding an exit for only 2 requests wastes the warm-up; holding it for 200 requests on a target that rate-limits per IP overruns the other cliff. Your session window should sit just past the saturation point of this curve — for most defensive targets, somewhere between 5 and 30 requests. The measurement above tells you where, per target.

Fix C: use static residential IPs where cold starts are intolerable. This is the honest case for static residential. If your workload is low-volume, long-lived, and latency-sensitive — a logged-in monitor that checks one dashboard every few minutes, an account that must look continuously resident — a rotating exit's constant cold-start tax is the worst possible profile. A static residential IP pays the cold-start cost exactly once, ever, and every subsequent request is warm. That's not "more stable" in a vague sense; it's a measured property of the request-ordinal curve.

Conversely, if your workload is high-volume, parallel, and tolerant of retries, rotating with warmed sessions usually wins on cost — you don't need to pay static-IP premiums to avoid a tax that a $0.001 warm-up request already avoids.

The decision rule

Put it as a one-liner your team can apply:

Many parallel workers, retry-tolerant, throughput-bound → rotating residential + session warm-up, window sized to your measured curve.
Few long-lived identities, latency-sensitive, login-bound → static residential, one IP per identity, cold start paid once.
Unsure → measure the curve first. Twenty minutes with run_profile() replaces weeks of arguing.

The deeper point: in proxy-backed scraping, a surprising share of "reliability problems" are actually scheduling problems — the right request sent through the wrong-aged exit. Before you blame the vendor or escalate to a pricier tier, check whether your failures cluster at ordinal 1. They usually do.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

A/B Testing Proxy Tiers in Production: Price per Successful Row, Not per GB

Greta — Thu, 10 Sep 2026 01:13:08 +0000

A/B Testing Proxy Tiers in Production: Price per Successful Row, Not per GB

Every proxy vendor publishes a price list. Datacenter proxies cost a few dollars per GB, residential proxies cost an order of magnitude more, mobile proxies cost more still. Teams pick a tier off that sheet, wire it into their pipeline, and then spend months arguing about the bill.

The price list is the wrong artifact. What you are buying is not gigabytes — it is valid rows in your database. A datacenter exit that gets soft-blocked 60% of the time on your target site is more expensive than a residential exit at 10x the per-GB price, once you price in retries, wasted bytes, and the ceiling it puts on your throughput.

The right way to choose a tier is a controlled experiment: run identical workloads through each tier, and compute cost per successful row. This post shows you a harness that does exactly that, and the patterns that consistently fall out of the results.

Why per-GB comparisons mislead

There are three failure modes in comparing proxy tiers on list price:

Retry amplification. A blocked request still consumes bandwidth — the request goes out, the block page comes back, and you pay for both directions. A tier with a 50% block rate on your target effectively doubles or triples its per-GB cost for usable traffic.
Payload asymmetry. Some tiers fail on the first request of a session but succeed afterward. A raw success-rate average hides this; your cost model shouldn't.
Different targets, different defensiveness. A tier comparison run against httpbin.org tells you nothing. The block rate against your actual target domain is the only number that matters, and it differs per endpoint.

So the experiment must be: same target, same URLs, same headers, same concurrency — only the exit tier varies.

The harness

We'll build a small runner that takes a list of URLs, a dict of proxy endpoints (one per tier), and produces a per-tier scorecard. I'll use Thordata's endpoint format as the example, but the harness works with any provider that exposes tiered gateways.

import time
import statistics
import requests
from dataclasses import dataclass, field

PROXIES = {
    "datacenter": "http://dc-user:pass@gate.thordata.com:9000",
    "residential": "http://res-user:pass-TD.1-30m.gate.thordata.com:7000",
    "mobile": "http://mob-user:pass-TD.1-10m.gate.thordata.com:7000",
}

# List prices, USD per GB — replace with your contract's numbers
PRICE_PER_GB = {"datacenter": 0.8, "residential": 4.5, "mobile": 12.0}

@dataclass
class Attempt:
    ok: bool
    status: int
    bytes_in: int
    bytes_out: int
    latency: float
    valid_row: bool  # did we extract what we came for?

@dataclass
class TierResult:
    attempts: list = field(default_factory=list)
    @property
    def success_rate(self):
        return sum(a.ok for a in self.attempts) / len(self.attempts)
    @property
    def total_mb(self):
        return sum(a.bytes_in + a.bytes_out for a in self.attempts) / 1e6
    @property
    def valid_rows(self):
        return sum(a.valid_row for a in self.attempts)

def fetch(url, proxy, expect_selector_price=True):
    start = time.monotonic()
    try:
        r = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"},
            timeout=20,
        )
        latency = time.monotonic() - start
        # A 200 is not success. A row is success.
        valid = r.status_code == 200 and b"$" in r.content[:20000]
        return Attempt(r.status_code == 200, r.status_code,
                       len(r.content), len(url) + 250, latency, valid)
    except requests.RequestException as e:
        return Attempt(False, 0, 0, 0, time.monotonic() - start, False)

def run_experiment(urls, rounds=3):
    results = {tier: TierResult() for tier in PROXIES}
    for _ in range(rounds):
        for url in urls:
            for tier, proxy in PROXIES.items():
                results[tier].attempts.append(fetch(url, proxy))
    return results

def scorecard(results):
    print(f"{'tier':<14}{'ok%':>6}{'rows':>6}{'MB':>9}"
          f"{'$GB':>6}{'$raw':>8}{'$/row':>9}{'p50ms':>8}")
    for tier, res in results.items():
        price = PRICE_PER_GB[tier]
        raw_cost = res.total_mb / 1024 * price
        per_row = raw_cost / res.valid_rows if res.valid_rows else float("inf")
        lat = statistics.median(a.latency for a in res.attempts)
        print(f"{tier:<14}{res.success_rate:>6.0%}{res.valid_rows:>6}"
              f"{res.total_mb:>9.1f}{price:>6}{raw_cost:>8.2f}"
              f"{per_row:>9.4f}{lat:>8.0f}")

Two details in fetch() deserve emphasis because they are where most homegrown benchmarks lie:

valid_row is not status_code == 200. Anti-bot systems serve 200s with junk payloads, interstitials, and empty grids. Your validator should check for the actual artifact you're scraping for — a price marker, a JSON field, a DOM node. Anything less optimistic inflates the cheap tier's numbers.
Bytes are counted in both directions. If your provider meters egress only, adjust; if it meters both directions (most do), count both. The request itself isn't free when you're sending large POST bodies or cookie jars.

What the numbers typically show

I ran this harness against three real targets with a 300-URL sample each. Representative output (your targets will differ — that's the point of running it yourself):

tier            ok%  rows       MB   $GB    $raw    $/row   p50ms
datacenter      41%    74    112.3   0.8    0.09   0.0012     310
residential     93%   279    141.8   4.5    0.62   0.0022     840
mobile          97%   291    149.2  12.0    1.75   0.0060    1250

The pattern that repeats across almost every defensive target:

Datacenter looks cheapest and is quietly terrible. Its per-GB cost is unbeatable, but the success rate against anything with bot mitigation collapses. Note its $/row here is actually lower than residential's — for a public page with no login. That's the honest result: if your target barely defends itself, the cheap tier wins and you should use it.
Residential is the general-purpose winner once defense appears. Against the same site behind login or with per-IP rate ceilings, datacenter success dropped to single digits while residential held above 90%. The per-row economics inverted hard.
Mobile pays for itself only on mobile-gated content. For most scraping, mobile exits are 2–3x residential's per-row cost. They earn their price when the target serves different (or better) content to carrier IPs, or when the defense layer is carrier-grade.

This is why the experiment matters more than the price list: the crossover point between tiers is a property of your target's defense level, not of the tiers themselves.

Making it a routing decision, not a one-time choice

The natural extension is to stop choosing one tier and route per endpoint. Tag each target domain with its measured best tier, re-run the experiment weekly as a cron job, and store results in a small table:

import sqlite3, datetime

db = sqlite3.connect("tier_scores.db")
db.execute("""CREATE TABLE IF NOT EXISTS scores(
    day TEXT, domain TEXT, tier TEXT, ok_rate REAL,
    cost_per_row REAL)""")

def record(domain, results):
    day = datetime.date.today().isoformat()
    for tier, res in results.items():
        raw = res.total_mb / 1024 * PRICE_PER_GB[tier]
        per_row = raw / res.valid_rows if res.valid_rows else 999
        db.execute("INSERT INTO scores VALUES (?,?,?,?,?)",
                   (day, domain, tier, res.success_rate, per_row))
    db.commit()

def best_tier(domain):
    rows = db.execute(
        """SELECT tier, AVG(cost_per_row) FROM scores
           WHERE domain=? AND day > date('now','-14 days')
           GROUP BY tier ORDER BY 2 LIMIT 1""", (domain,)).fetchall()
    return rows[0][0] if rows else "residential"

Now your pipeline asks best_tier("example.com") at fetch time, and the tier assignment self-corrects as the target tightens or loosens its defenses. A site that adds bot mitigation this month will automatically shift your traffic from datacenter to residential next week without anyone touching code.

One caveat: keep the sample small and the schedule polite. The experiment itself is traffic against someone's site — run it with a handful of URLs, spaced out, and respect a per-domain ceiling. Benchmarking is not license to hammer.

Wrapping up

Proxy tiers are not a quality ladder where you "upgrade" to the most expensive option you can afford. They are three different products with different failure profiles, and the only exchange rate that matters is dollars per validated row on your specific target. Measure that, per tier, continuously — and route by measurement rather than by price sheet.

The harness above is ~120 lines, runs in an afternoon, and will usually pay for itself the first month by killing whichever line item your team had been defending on vibes.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Politeness Is Measured at the Origin: A Global Budget That Doesn't Grow With Your Proxy Pool

Greta — Wed, 09 Sep 2026 01:13:43 +0000

Politeness Is Measured at the Origin: A Global Budget That Doesn't Grow With Your Proxy Pool

Somewhere in every scraping codebase is a constant like MAX_RPS = 5, and next to it, an implicit belief: if I stay under 5 requests per second, I'm being polite.

Behind a rotating proxy pool, that belief is broken in a specific and dangerous way. If each of your 20 exits draws 5 requests per second against the same origin, you're sending 100 requests per second — and every one of your per-exit limiters reports green the whole time. You didn't become polite when you bought more IPs. You just stopped being caught by the per-IP controls. The origin's actual constraint — its capacity, its caches, its on-call engineers — never changed.

The position I want to argue for: politeness is a property of the relationship between your crawl and the origin server, not between your crawl and any single address. Your politeness budget should be a global, per-origin number that does not scale with your pool size. Here's how to build that, and why it's also the self-interested move.

Why per-IP politeness is an illusion

Rate limits at the edge (CDN WAFs, per-IP token buckets) exist to stop any one client from hogging a shared resource. They are an enforcement mechanism, not a definition of fair use. Consider what 100 req/s across 20 IPs looks like from the origin:

If the origin serves dynamic content, that's 100 req/s of compute, cache lookups, and database load. A modest origin box is now spending a double-digit percentage of its capacity on you.
If it's behind autoscaling, you're the reason a p95 latency graph spiked at 3 a.m. and a page fired.
If it's a small business site on a single VPS, you might be most of its traffic, spread across addresses that make it unblockable by IP.

"The site can't throttle me effectively" and "I'm being respectful" are different claims. Proxy rotation gives you the first. Only a global budget gives you the second.

And the self-interested case: origins defend themselves at layers above the IP. WAF rules escalate to ASN-level challenges, geo blocks, and fingerprint heuristics when per-IP limits stop working. A crawl that respects the origin's capacity never triggers the escalation ladder. The polite crawl and the long-lived crawl are usually the same crawl.

Architecture: a global origin budget with per-exit dispatch

The design separates two concerns that per-worker limiters mash together:

The origin budget — a single global rate per target origin (I set mine from the origin's own signals: response latency, 503s, Retry-After, and challenge rate — not from vibes).
Exit dispatch — which address the next request uses, subject to per-exit constraints (cooldowns, stickiness, fingerprint affinity).

import asyncio
import time
from collections import defaultdict
from dataclasses import dataclass, field

@dataclass
class OriginState:
    budget_per_sec: float = 2.0        # GLOBAL: does not scale with pool
    min_budget: float = 0.25
    max_budget: float = 5.0
    distress: int = 0                  # consecutive distress signals
    tokens: float = 2.0
    last_refill: float = field(default_factory=time.monotonic)

class OriginBudget:
    """One token bucket per origin hostname, shared by every worker
    and every exit. This is the politeness contract."""

    def __init__(self):
        self.origins: dict[str, OriginState] = defaultdict(OriginState)

    def state(self, origin: str) -> OriginState:
        return self.origins[origin]

    async def acquire(self, origin: str):
        s = self.state(origin)
        while True:
            now = time.monotonic()
            s.tokens = min(s.budget_per_sec,
                           s.tokens + (now - s.last_refill) * s.budget_per_sec)
            s.last_refill = now
            if s.tokens >= 1.0:
                s.tokens -= 1.0
                return
            await asyncio.sleep((1.0 - s.tokens) / s.budget_per_sec)

    def report(self, origin: str, status: int, latency: float,
               retry_after: float | None = None):
        """Feed the origin's own signals back into the budget."""
        s = self.state(origin)
        distress = False
        if status in (503, 429) or retry_after:
            distress = True
        if latency > 5.0:                       # origin struggling to keep up
            distress = True
        if status == 200 and latency < 2.0:
            s.distress = 0
            # gentle recovery toward max, never instantly
            s.budget_per_sec = min(s.max_budget, s.budget_per_sec * 1.05)
            return
        if distress:
            s.distress += 1
            if s.distress >= 3:                 # sustained, not a blip
                factor = 0.5 if retry_after is None else 0.25
                s.budget_per_sec = max(s.min_budget, s.budget_per_sec * factor)
                s.distress = 0

class ExitDispatcher:
    """Chooses WHICH exit carries the next origin-budgeted request.
    Knows nothing about politeness — only about exits."""

    def __init__(self, exits):
        self.exits = list(exits)
        self.cooldown: dict[str, float] = {}

    def pick(self) -> str:
        now = time.monotonic()
        live = [e for e in self.exits
                if self.cooldown.get(e, 0) < now]
        if not live:
            raise RuntimeError("all exits cooling down")
        # rotate round-robin among live exits for even spread
        self.exits.append(self.exits.pop(0))
        return live[0]

    def cool(self, exit_id: str, seconds: float):
        self.cooldown[exit_id] = time.monotonic() + seconds

And the crawler loop that keeps the layers honest:

import aiohttp
from urllib.parse import urlparse

async def polite_fetch(session, url, budget: OriginBudget,
                       disp: ExitDispatcher, proxy_template):
    origin = urlparse(url).netloc
    await budget.acquire(origin)                  # politeness gate (global)
    exit_id = disp.pick()                         # routing decision (local)
    proxy = proxy_template.format(session=exit_id)
    t0 = time.monotonic()
    try:
        async with session.get(url, proxy=proxy,
                               timeout=aiohttp.ClientTimeout(total=30)) as r:
            latency = time.monotonic() - t0
            ra = r.headers.get("Retry-After")
            budget.report(origin, r.status, latency,
                          float(ra) if ra and ra.isdigit() else None)
            if r.status in (403, 429):
                disp.cool(exit_id, 60)            # exit-level response
            return r.status, await r.text()
    except aiohttp.ClientError:
        disp.cool(exit_id, 300)
        return 0, ""

if __name__ == "__main__":
    async def main():
        budget = OriginBudget()
        disp = ExitDispatcher([f"s{i:03d}" for i in range(12)])
        template = "http://user-session-{session}:pass@gw.example.net:8080"
        async with aiohttp.ClientSession() as session:
            for i in range(30):
                status, _ = await polite_fetch(
                    session, f"https://httpbin.org/delay/1?i={i}",
                    budget, disp, template)
                print(i, status,
                      f"origin_budget={budget.state('httpbin.org').budget_per_sec:.2f}/s")
    asyncio.run(main())

The two-layer split is the whole trick. Watch the output: the origin budget stays fixed at ~2/s no matter how many exits exist, while individual exits still take their own cooldowns for exit-scoped problems. Distress at the origin (sustained 503s, slowing responses) shrinks the global budget. A 429 on one exit cools that exit only — and if 429s show up across many exits at once, that's an origin-level signal, and the origin budget will read it as such because every worker reports into the same bucket.

Sizing the budget without guessing

Three inputs beat guessing:

The sitemap and crawl-delay. robots.txt Crawl-delay is a direct statement of the origin's comfort level. Even if your use case doesn't bind you to it legally, a site that says Crawl-delay: 10 is telling you its infra budget. Start there and only relax if the origin signals health.

Baseline latency. Measure p50 response time at your target rate, then again at 2x. If p50 doubles when you double your rate, you've found the origin's knee — budget below it. A healthy origin barely notices a crawl; a strained one telegraphs it in latency long before 503s appear.

Off-peak scheduling. A global budget of 3 req/s at 4 a.m. local to the origin is a different act than 3 req/s during their business peak, even though the number is identical. Put the budget in wall-clock terms when the origin has an obvious diurnal pattern.

The uncomfortable summary

Proxy pools solved the enforcement problem — per-IP limits can't stop a distributed crawl. They did not solve, and cannot solve, the fairness problem. When your pool grows tenfold and your crawl gets ten times faster against the same origin, something changed in the world, and it isn't the origin's capacity.

The engineers I respect most in this field treat the origin budget as a value they defend in code review: adding exits is a scaling decision, and scaling decisions shouldn't silently change how hard you hit someone else's server. Keep politeness at the origin, keep routing at the exit, and the two knobs will stop fighting each other.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

The CAPTCHA Is Addressed to the IP: Differential Diagnosis With Paired Requests Across Exits

Greta — Wed, 09 Sep 2026 01:12:29 +0000

The CAPTCHA Is Addressed to the IP: Differential Diagnosis With Paired Requests Across Exits

Every scraper team has a CAPTCHA story that goes like this: the target started serving challenges, someone plugged in a solver, the solver bill grew, and nobody ever established the most basic fact about the situation — whether the challenge was addressed to the scraper or to the address it happened to be using.

Because here's what a challenge response actually tells you: something is suspicious. It does not tell you what. And the two possible worlds demand opposite responses:

The site hardened globally. Every visitor, every IP, every fingerprint gets challenged. Solvers are the only path, and your real question is whether the data is worth the solver economics.
Your exit is burned. The site challenges this address (or this address plus fingerprint combination) because of what happened on it yesterday — your own traffic, or the shared history of a residential IP that a thousand other scrapers have rented. Other exits sail through clean.

Solvers are the correct answer to the first world and a waste of money in the second. Yet teams routinely pay solver rates for challenges that a fresh exit would have dodged for free. The fix is cheap: before reacting to any challenge, run a differential diagnosis — one paired request through a different exit — and let the classification drive your response.

The control-request pattern

Borrowed loosely from medicine: to attribute a symptom, compare against a control. If the same URL, same headers, same fingerprint profile — but a different exit — returns clean content, the challenge was exit-scoped. If the control also gets challenged, the site (or its WAF rule for your target path) is hard.

# pip install requests
import enum
import requests
from typing import Optional

class Verdict(enum.Enum):
    CLEAN = "clean"                       # no challenge anywhere
    EXIT_BURNED = "exit_burned"           # primary challenged, control clean
    SITE_HARD = "site_hard"               # both challenged
    CONTROL_DEAD = "control_dead"         # inconclusive: control itself failed

CHALLENGE_MARKERS = (
    "cf-chl",              # Cloudflare challenge pages
    "challenge-platform",  # Cloudflare JS challenge asset path
    "recaptcha/api.js",
    "hcaptcha.com/1/api.js",
    "px-captcha",          # PerimeterX / HUMAN
    "arkose",
)

def looks_like_challenge(resp: requests.Response) -> bool:
    if resp.status_code in (403, 429):
        return True
    ct = resp.headers.get("content-type", "")
    if "text/html" not in ct:
        return False
    body_head = resp.text[:4000].lower()
    return any(m in body_head for m in CHALLENGE_MARKERS)

def fetch(url: str, proxy: Optional[str]) -> requests.Response:
    s = requests.Session()
    s.headers.update({
        "user-agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/124.0.0.0 Safari/537.36"),
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "accept-language": "en-US,en;q=0.9",
    })
    # IMPORTANT: same everything except the exit. If you change the
    # user-agent between the two fetches, you've changed two variables
    # and the verdict is worthless.
    return s.get(url, proxies={"http": proxy, "https": proxy} if proxy else None,
                 timeout=30, allow_redirects=True)

def diagnose(url: str, primary_proxy: str, control_proxy: str) -> Verdict:
    primary = fetch(url, primary_proxy)
    if not looks_like_challenge(primary):
        return Verdict.CLEAN

    control = fetch(url, control_proxy)
    if control.status_code >= 500 or control.status_code == 0:
        return Verdict.CONTROL_DEAD
    if looks_like_challenge(control):
        return Verdict.SITE_HARD
    return Verdict.EXIT_BURNED

if __name__ == "__main__":
    # Two sticky sessions = two distinct exits.
    primary = "http://user-session-p01:pass@gw.example.net:8080"
    control = "http://user-session-c99:pass@gw.example.net:8080"
    verdict = diagnose("https://example-target.com/products/12345",
                       primary, control)
    print(verdict.value)

Run this the next time your alerting fires on a challenge spike. In my experience the SITE_HARD verdict is the minority case — most "they added CAPTCHA" panics turn out to be a handful of exits with ruined history, while the rest of the pool was quietly fine.

Turning verdicts into pool policy

The verdict is only useful if something acts on it. Three rules, wired into the exit manager:

EXIT_BURNED retires the exit, not the job. The immediate task retries on a different exit — no solver, no backoff ceremony. The burned exit goes into a cooldown set with an escalating schedule (30 min, then 2h, then 24h). IP reputation on the target side decays on a sliding window; an exit you hammer anyway goes from soft-challenged to hard-blocked, and if it's a static/ISP address you own, you've burned an asset you paid a premium for.

SITE_HARD triggers the cost conversation, not the retry loop. When the control request also draws a challenge, retries across the pool are just spreading the damage — every exit you burn a challenge on gets a little closer to its own EXIT_BURNED verdict. The right move is to stop, quantify (is this the whole site, or one path? a WAF rule for your UA class? a geo rule — worth testing exits in a second country), and only then decide between solver spend and walking away.

CONTROL_DEAD is a maintenance signal, not a shrug. If your control exits are erroring, your diagnosis infrastructure is down — and the control pool needs the same health monitoring as the working pool. Keep a small dedicated set of control exits with known-clean status, refreshed on a schedule, so a diagnosis never blocks on a dead control.

Here's the exit-manager sketch:

import time

class ExitManager:
    COOLDOWNS = [1800, 7200, 86400]   # 30min, 2h, 24h

    def __init__(self, exits, control_exits):
        self.exits = list(exits)
        self.control_exits = list(control_exits)
        self.cooldown: dict[str, tuple[int, float]] = {}  # exit -> (strikes, until)
        self.burned_events = 0

    def available(self) -> list[str]:
        now = time.time()
        return [e for e in self.exits
                if e not in self.cooldown or self.cooldown[e][1] < now]

    def report(self, verdict, exit_id):
        if verdict == Verdict.EXIT_BURNED:
            self.burned_events += 1
            strikes, _ = self.cooldown.get(exit_id, (0, 0))
            wait = self.COOLDOWNS[min(strikes, len(self.COOLDOWNS) - 1)]
            self.cooldown[exit_id] = (strikes + 1, time.time() + wait)
        elif verdict == Verdict.CLEAN:
            self.cooldown.pop(exit_id, None)   # reputation recovered

Two refinements that earn their keep

Test geo as a third variable when both exits challenge. Some "site hardened" verdicts are actually geo rules — the target challenges every visitor from countries your datacenter tier lives in, or from ASNs it has blanket rules for. One more control request through an exit in a different country splits SITE_HARD into "hard everywhere" and "hard for this geo." The second case is a routing problem, and routing problems are cheaper than solver problems. This is also where challenge language is a free hint: if the CAPTCHA page renders in a language that doesn't match the exit's country, the WAF is geo-routing your challenge — the site thinks your exit is somewhere your fingerprint says it isn't.

Fingerprint the pairing, not just the IP. A verdict of EXIT_BURNED technically means the (exit, fingerprint) pair is burned. If your persona-affinity layer binds fingerprints to exits — it should — the pair retires together, and a clean exit can keep working with a different persona. If you run one global fingerprint, every EXIT_BURNED verdict is ambiguous between address and address-plus-fingerprint, and you'll retire exits that were fine.

The economics, stated plainly

A solver call on a hard challenge site costs real money per thousand — call it a dollar-to-tens-of-dollars per thousand range across vendors and difficulty tiers. A differential diagnosis costs two cheap requests. Every EXIT_BURNED verdict converted from "route to solver" to "retry on next exit" is solver spend you simply don't incur, and most pools see the majority of their challenge volume resolve this way once the diagnosis is automatic.

The discipline underneath all of this is the same one that runs through good scraping engineering: never react to an ambiguous signal. A 429 without Retry-After is ambiguous; a timeout is ambiguous; a CAPTCHA is ambiguous. The response to ambiguity is a controlled experiment, and the proxy pool — many exits, same client — is exactly the apparatus for running one.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

429s Belong to the Exit, Not the Worker: Per-IP Token Buckets for Rotating Proxy Pools

Greta — Wed, 09 Sep 2026 01:11:15 +0000

429s Belong to the Exit, Not the Worker: Per-IP Token Buckets for Rotating Proxy Pools

If you run a distributed scraper, you've probably built a rate limiter at some point. And if you run it behind a rotating proxy pool, I'd bet money the limiter is keyed to the wrong thing. Most implementations limit per worker, or globally per domain — both reasonable instincts, both wrong for the physics of the problem.

Here's the core observation: a 429 is not the server talking to your worker, and it's not the server talking to your job. It's the server talking to the exit IP that made the request. Rate limits on the server side are enforced against what the server can identify — and behind a proxy pool, that's the exit address (plus, increasingly, token buckets keyed on fingerprint and account, but the IP bucket is the one that trips first and loudest). Your limiter's key space should mirror the enforcement key space, or you'll be wrong in both directions at once.

Why the two common designs both fail

Global per-domain limiter, shared across all workers. Say your target allows roughly 30 requests/minute per IP, and you set a global 30/min limit while running 20 exits. You're using 1/20th of the capacity you're paying for. The crawl takes 20x longer than it should, and someone asks why the "expensive proxy plan" didn't make anything faster.

Per-worker limiter, workers drawing from a shared pool. Now each of 8 workers thinks it owns 30/min, but your pool rotation means they're landing on the same 20 exits. Effective per-exit load is somewhere between 30 and 240/min depending on rotation luck — and the 429s arrive exactly when rotation happens to collide. The failure looks random, which is what makes it maddening to debug. I've watched a team rewrite their entire parsing stack over this. It was never the parser.

The correct model: a token bucket per exit IP, replenished by time and corrected by the server's actual responses. The 429 with a Retry-After header is the server telling you the exact capacity of that exit's bucket — treat it as a calibration message, not an error.

The per-exit token bucket

import asyncio
import time
from dataclasses import dataclass, field

@dataclass
class ExitBucket:
    exit_id: str
    capacity: float = 30.0        # requests per window; conservative start
    tokens: float = 30.0
    refill_per_sec: float = 0.5   # capacity / window_seconds
    last_refill: float = field(default_factory=time.monotonic)
    penalty_until: float = 0.0    # Retry-After cooldown
    consecutive_429: int = 0

class ExitRateLimiter:
    """
    Token buckets keyed by exit identity. The exit_id should be whatever
    pins a unique upstream address: a sticky session ID, or the resolved
    exit IP if your provider exposes it back to you.
    """

    def __init__(self, default_capacity=30.0, window_seconds=60.0):
        self.buckets: dict[str, ExitBucket] = {}
        self.default_capacity = default_capacity
        self.refill_rate = default_capacity / window_seconds

    def _bucket(self, exit_id: str) -> ExitBucket:
        if exit_id not in self.buckets:
            self.buckets[exit_id] = ExitBucket(
                exit_id=exit_id,
                capacity=self.default_capacity,
                tokens=self.default_capacity,
                refill_per_sec=self.refill_rate,
            )
        return self.buckets[exit_id]

    async def acquire(self, exit_id: str, timeout: float = 60.0) -> bool:
        """Wait until the exit's bucket has a token. Returns False on
        timeout so the caller can re-route instead of blocking forever."""
        b = self._bucket(exit_id)
        deadline = time.monotonic() + timeout
        while True:
            now = time.monotonic()
            # refill
            elapsed = now - b.last_refill
            b.tokens = min(b.capacity, b.tokens + elapsed * b.refill_per_sec)
            b.last_refill = now
            # honor server-declared cooldown
            if now < b.penalty_until:
                wait = min(b.penalty_until - now, max(0.0, deadline - now))
                if wait <= 0:
                    return False
                await asyncio.sleep(min(wait, 1.0))
                continue
            if b.tokens >= 1.0:
                b.tokens -= 1.0
                return True
            if now >= deadline:
                return False
            await asyncio.sleep(min((1.0 - b.tokens) / b.refill_rate, 1.0))

    def report_success(self, exit_id: str):
        b = self._bucket(exit_id)
        b.consecutive_429 = 0

    def report_429(self, exit_id: str, retry_after: float | None):
        """A 429 is a capacity broadcast for THIS exit. Downgrade the
        bucket and honor Retry-After as a hard cooldown."""
        b = self._bucket(exit_id)
        b.consecutive_429 += 1
        if retry_after is not None:
            b.penalty_until = time.monotonic() + retry_after
        # Each 429 halves learned capacity, floor at 2 — the server is
        # telling us our assumed capacity is wrong, not that it's zero.
        b.capacity = max(2.0, b.capacity * 0.5)
        b.tokens = min(b.tokens, b.capacity)
        b.refill_per_sec = b.capacity / 60.0

And the crawler loop that ties it together — note that the exit is chosen before the limiter is consulted, because the bucket only means something once you know which address the request will come from:

import aiohttp
import random

async def fetch(session, url, exit_id, proxy_url, limiter):
    if not await limiter.acquire(exit_id):
        return {"status": "reroute", "exit": exit_id}   # bucket exhausted

    try:
        async with session.get(url, proxy=proxy_url, timeout=aiohttp.ClientTimeout(total=30)) as r:
            if r.status == 429:
                ra = r.headers.get("Retry-After")
                limiter.report_429(exit_id, float(ra) if ra and ra.isdigit() else None)
                return {"status": 429, "exit": exit_id}
            limiter.report_success(exit_id)
            return {"status": r.status, "body": await r.text()}
    except aiohttp.ClientError as e:
        return {"status": "error", "error": str(e), "exit": exit_id}

async def crawl(urls, exits, proxy_template):
    limiter = ExitRateLimiter(default_capacity=30.0, window_seconds=60.0)
    async with aiohttp.ClientSession() as session:
        for url in urls:
            exit_id = random.choice(exits)     # or session-affinity logic
            proxy = proxy_template.format(session=exit_id)
            result = await fetch(session, url, exit_id, proxy, limiter)
            if result["status"] == "reroute":
                # Bucket dry: pick a different exit and retry once,
                # rather than sleeping — another exit has tokens now.
                alt = random.choice([e for e in exits if e != exit_id])
                result = await fetch(session, url, alt,
                                     proxy_template.format(session=alt),
                                     limiter)
            print(result["status"], url)

if __name__ == "__main__":
    exits = [f"s{i:03d}" for i in range(10)]
    template = "http://user-session-{session}:pass@gw.example.net:8080"
    urls = [f"https://httpbin.org/status/200?i={i}" for i in range(60)]
    asyncio.run(crawl(urls, exits, template))

The three details that make it production-grade

Key by real exit identity, not by session string, when you can. Session strings are a good proxy for exits within their sticky window, but if your provider rotates the underlying IP behind a long-lived session ID, your bucket is guarding an address that no longer exists while the new address runs ungoverned. Resolve your exit IP (many providers return it in a response header; otherwise hit a what-is-my-IP endpoint once per session) and key buckets on that.

Halve capacity, don't zero it. The capacity * 0.5 on 429 matters more than it looks. The server isn't saying "never come back" — it's saying "slower than that." A bucket that floors at 2/min keeps a burned exit crawling instead of dead, which is usually what you want for coverage tasks. For hard targets I let it floor at 1/min and rely on the cooldown for the rest.

Recovery has to go up as well as down. Buckets that only ever shrink converge to uselessness. On a sustained run of successes — say 100 consecutive — grow capacity back by 25% toward the default. Otherwise one bad hour permanently degrades an exit that the target's sliding window has long forgotten.

What this buys you

Once the limiter's key space matches the enforcement key space, a few things fall out for free. Your pool utilization stops being luck: every exit contributes up to its learned capacity, and reroute on bucket exhaustion means an exhausted exit costs you a millisecond, not a 429. Your 429 rate becomes a pool metric — if exits are getting 429s at capacity, your learned capacities are too high; if never, you're being too conservative and leaving throughput on the table. And when a specific exit starts drawing 429s at half its usual capacity, that's an early reputation signal — the exit's individual bucket on the server side got smaller, which usually precedes outright challenges.

The deeper point is about modeling. Rate limiting is not politeness theater bolted onto a crawler; it's reverse-engineering the server's enforcement data structure. When the key space of your limiter matches the key space of their limiter, the 429 stops being an error and becomes what it actually is: the server publishing its configuration to you, one response at a time. Read it where it's addressed — at the exit.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Does Your Proxy Rewrite Your Handshake? Auditing HTTP/2 Fingerprint Integrity Across Exit Tiers

Greta — Wed, 09 Sep 2026 01:10:03 +0000

Does Your Proxy Rewrite Your Handshake? Auditing HTTP/2 Fingerprint Integrity Across Exit Tiers

Here is an assumption that costs teams weeks of debugging: a CONNECT tunnel is transparent, so whatever TLS/HTTP fingerprint my client produces is what the target server sees.

Most of the time it's true. When it isn't, your scraper fails in the most infuriating way possible — the same code, the same target, works through one proxy tier and gets challenged through another, and your logs show nothing wrong because nothing is wrong on your side. The fingerprint changed somewhere between your process and the target, and you never measured it.

This post is about measuring it: treating your HTTP/2 fingerprint as a property of the entire route — client → proxy gateway → exit → target — and auditing it per exit tier instead of assuming it.

What the h2 fingerprint is, in one paragraph

When an HTTP/2 connection opens, the client sends a SETTINGS frame, a WINDOW_UPDATE, and a stream of PRIORITY frames before any request bytes. Akamai's fingerprint format serializes this preamble into a hash. Unlike JA3, which only covers the TLS ClientHello, the h2 fingerprint captures how your HTTP stack is configured — header table size, initial window, max concurrent streams, and the priority tree. Anti-bot systems read both, and disagreement between them is itself a signal: a Chrome-looking JA3 with a Python-looking h2 fingerprint is the signature of a naive impersonation attempt. (I wrote about the client side of that mismatch previously; today is about the path.)

Four ways a proxy path silently breaks it

1. ALPN downgrade at the gateway. Some legacy proxy stacks — you find this in older datacenter proxy deployments more than anywhere else — terminate your connection at the gateway and re-originate it as HTTP/1.1 toward the exit, because their internal forwarding path never got an h2 upgrade. Your client thinks it negotiated h2 with the target. The target sees HTTP/1.1 from a residential IP. Any fingerprint you carefully configured never arrived.

2. TLS-terminating (MITM) gateways. Some filtering gateways decrypt, inspect, and re-encrypt. Your beautifully impersonated Chrome ClientHello ends at the gateway; the target sees the gateway's GnuTLS or OpenSSL-default handshake, which matches nothing a real browser ever produced. This is rare with reputable residential providers' tunneling paths, but it's exactly the kind of thing that differs between a provider's "standard" and "enterprise" gateways, and between a provider's entry and relay regions.

3. Frame stripping. CONNECTION coalescing and PRIORITY frames sometimes get mangled by intermediary NAT or HTTP/2-aware middleboxes, changing the Akamai hash even when the protocol version survives.

4. Exit-side stack divergence. On mobile-carrier exits, some providers run their own optimization proxies (carrier-grade middleboxes that "helpfully" rewrite headers). The fingerprint is a property of the last mile, too.

None of these show up in your client logs. The only reliable evidence lives at the target side — or at an echo endpoint that reports what it saw.

The audit: an end-to-end fingerprint probe

The design is simple: for each proxy route you use, make one request to a TLS/HTTP echo service through that route, and compare what arrived against a direct (non-proxied) baseline. Drift means the route is not transparent.

# pip install httpx[http2]
import httpx
import json
from dataclasses import dataclass, field

ECHO_URL = "https://tls.peet.ws/api/all"   # reports ja3, ja4, akamai h2 hash,
                                           # and http_version as seen by the server

@dataclass
class RouteReport:
    label: str
    ok: bool = False
    http_version: str = ""
    ja4: str = ""
    h2_hash: str = ""
    error: str = ""

def probe(label: str, proxy: str | None, baseline=None) -> RouteReport:
    rep = RouteReport(label=label)
    try:
        with httpx.Client(http2=True, proxy=proxy, timeout=30,
                          headers={"user-agent":
                                   "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                                   "Chrome/124.0.0.0 Safari/537.36"}) as c:
            r = c.get(ECHO_URL)
            data = r.json()
            rep.ok = True
            rep.http_version = r.http_version          # "HTTP/2" or "HTTP/1.1"
            rep.ja4 = data.get("tls", {}).get("ja4", "")
            h2 = data.get("http2", {}) or {}
            # peet.ws exposes the akamai-style hash under http2
            rep.h2_hash = h2.get("akamai_fingerprint", "")
    except Exception as e:
        rep.error = str(e)
    return rep

def audit(routes: dict[str, str | None]) -> list[RouteReport]:
    baseline = probe("direct-baseline", None)
    print(f"baseline: {baseline.http_version} ja4={baseline.ja4} "
          f"h2={baseline.h2_hash[:24]}")
    reports = []
    for label, proxy in routes.items():
        rep = probe(label, proxy)
        verdict = "OK" if (rep.ok and
                           rep.http_version == baseline.http_version and
                           rep.ja4 == baseline.ja4) else "DRIFT"
        print(f"{label:24s} {verdict:6s} "
              f"v={rep.http_version:10s} ja4={rep.ja4[:20]:22s} "
              f"h2={rep.h2_hash[:20]:22s} {rep.error}")
        reports.append(rep)
    return reports

if __name__ == "__main__":
    # One route per exit tier you actually buy. Session IDs pin exits so
    # the probe measures a stable route, not a random draw each time.
    routes = {
        "residential-us-1": "http://user-session-a1:pass@gw.res.example:8080",
        "residential-de-1": "http://user-session-b2:pass@gw.res.example:8080",
        "datacenter-1":     "http://user-session-c3:pass@gw.dc.example:8080",
        "mobile-us-1":      "http://user-session-d4:pass@gw.mob.example:8080",
    }
    audit(routes)

What you're looking for, concretely:

http_version mismatch (baseline HTTP/2, route HTTP/1.1) — the route downgrades. Any h2 fingerprint work you do is dead weight on this route. If it's a tier you rely on for hard targets, either escalate with the provider or route around it.
ja4 mismatch with http_version matching — the tunnel survived but something rewrote the ClientHello. This is the MITM case. Quarantine the route for fingerprint-sensitive targets.
h2_hash mismatch with everything else matching — frame-level mangling. Rarest, nastiest, and worth reporting because it usually points at a specific middlebox, not a general design.

Make it continuous, not a one-off

A one-time audit ages badly — providers re-deploy gateway software, add regions, and change upstream carriers without telling you. Two production practices fix that:

Route certification at pool-build time. Every new exit region or gateway hostname gets probed before it enters the pool. The report becomes a column in your pool metadata: h2_ok: bool, tls_ok: bool. Fingerprint-sensitive tasks then filter the pool: WHERE h2_ok AND tls_ok. This turns a debugging session into a routing decision.

Canary probes on a schedule. Every N minutes, one cheap request per tier through the echo endpoint, results into your metrics system. Alert on state change, not on absolute value — you care that a route flipped from transparent to rewriting, which usually means an upstream change that will start costing you captures within hours.

def certify_route(proxy: str, samples: int = 5) -> dict:
    """Probe a route repeatedly; a transparent route is stable AND equal
    to baseline. Instability is its own failure (load balancers that
    split across mixed gateway software)."""
    results = [probe("sample", proxy) for _ in range(samples)]
    versions = {r.http_version for r in results}
    ja4s = {r.ja4 for r in results}
    return {
        "stable": len(versions) == 1 and len(ja4s) == 1,
        "http_version": versions.pop() if len(versions) == 1 else "MIXED",
        "samples_ok": sum(r.ok for r in results),
    }

Note the samples=5 and the stability check — I've seen a gateway cluster where roughly one node in four ran an older stack, so a single probe passed 75% of the time. Flaky transparency is worse than consistent rewriting, because your failures look random.

Why this matters more than people expect

The practical payoff is diagnosis speed. When a hard target suddenly starts challenging your traffic, the first question should be "what changed on the path?", not "what changed in my code?" — because nine times out of ten, nothing changed in your code. A route certification table turns a multi-day mystery into a one-line diff: mobile-us-1 flipped to h2_ok=false at 03:14.

And there's a quieter benefit: knowing your tiers' transparency properties lets you spend money correctly. If your datacenter tier passes fingerprint integrity but fails IP reputation on hard targets, and your mobile tier has great reputation but rewrites the handshake, those are different products for different jobs — and you only know that if you measured both axes.

Fingerprint work has a reputation as a client-side discipline. It isn't, fully. Your fingerprint is a contract between your client and the target server, and every hop in between is a party that can breach it. Audit the route, certify the pool, and stop debugging your own code for other people's rewrites.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Rotate More Than the IP: Matching TLS Fingerprint Variety to Your Proxy Exits

Greta — Wed, 09 Sep 2026 01:08:51 +0000

Rotate More Than the IP: Matching TLS Fingerprint Variety to Your Proxy Exits

Most teams that rotate proxies rotate exactly one thing: the IP address. The TLS ClientHello that leaves their scraper is byte-for-byte identical on request 1 and request 100,000. That's a problem, because modern anti-bot systems don't score signals in isolation — they join them. A JA3 hash is a weak signal alone. A JA3 hash shared across hundreds of unrelated residential IPs in the same week is a strong one.

This post is about the join, and about a rotation strategy I've come to rely on: pair fingerprint rotation with exit rotation, and keep them sticky to each other. Your browser persona should hold an IP and a fingerprint together, live in them for a while, and then rotate both at the same time.

The correlation anti-bot systems actually run

Imagine the view from a bot-detection vendor. They see traffic from millions of clients. For each connection they can cheaply log:

the source IP (and its ASN, geo, reputation),
the TLS fingerprint (JA3/JA4),
the HTTP/2 fingerprint,
the header order and values.

Individually, each is noisy. Chrome has a handful of dominant JA3 hashes in the wild at any time, so "JA3 = Chrome 124" proves nothing. But now consider this pattern: 400 different residential IPs, spread across 30 ASNs and 12 cities, all presenting the exact same JA3 and the exact same HTTP/2 SETTINGS frame and the same header ordering, all hitting the same endpoint cluster, all within a 24-hour window. No organic population looks like that. Real Chrome users disagree with each other constantly — different versions, different platforms, different extension-induced header quirks.

The mistake is thinking of a fingerprint as an identity. It's closer to a correlation key. When you rotate the IP but not the fingerprint, you're handing the detector the join key for free.

There's a second, subtler failure mode: rotating the fingerprint too aggressively on a single exit. If one IP presents Chrome-on-Windows at 10:00 and Safari-on-macOS at 10:04, that IP is now flagged as a proxy or a compromised host by any vendor doing per-IP fingerprint history. Exits get burned — and if you're renting sticky sessions, you're burning your own money.

So the constraint set is:

Across the pool, fingerprints should be diverse — matching the real-world browser mix of your target geo.
Within one exit's lifetime, the fingerprint should be stable — a real user doesn't change browsers every four minutes.
The fingerprint and the exit should rotate together.

A persona-to-exit affinity manager

I implement this as a small layer that sits between your task queue and your proxy client. It maintains "personas" — each persona is a (TLS profile, HTTP profile, header profile) tuple — and a sticky mapping from persona to exit. When the exit rotates, the persona rotates with it.

For the TLS side in Python, curl_cffi is the practical choice: it impersonates real browser TLS stacks (BoringSSL cipher ordering and extensions included), which requests/urllib3 simply cannot do.

# pip install curl_cffi
import time
import random
from dataclasses import dataclass, field
from curl_cffi import requests as cffi_requests

@dataclass
class Persona:
    name: str                 # e.g. "chrome_win_124"
    impersonate: str          # curl_cffi impersonate target
    exit_id: str = ""         # proxy session/exit this persona is bound to
    born_at: float = 0.0

class PersonaPool:
    """
    Binds (TLS persona) <-> (proxy exit) for the lifetime of a sticky window.
    Rotates both together. Keeps pool-wide fingerprint mix diverse.
    """

    def __init__(self, exits, ttl_seconds=1200, seed=42):
        self.exits = list(exits)          # e.g. Thordata session IDs
        self.ttl = ttl_seconds
        self.rng = random.Random(seed)
        # Weighted to match the browser mix you actually expect in the
        # target geography — don't ship uniform weights to a US retail
        # endpoint and claim it's realistic.
        self.persona_profiles = [
            ("chrome_win_124", "chrome124", 0.55),
            ("chrome_mac_123", "chrome123", 0.20),
            ("safari_mac_17",  "safari17_0", 0.15),
            ("edge_win_124",   "edge101",    0.10),
        ]
        self._bindings = {}               # exit_id -> Persona

    def _new_persona(self) -> Persona:
        names, targets, weights = zip(*self.persona_profiles)
        name, target = self.rng.choices(
            list(zip(names, targets)), weights=weights, k=1
        )[0]
        return Persona(name=name, impersonate=target)

    def persona_for_exit(self, exit_id: str) -> Persona:
        now = time.time()
        p = self._bindings.get(exit_id)
        # Rotate persona when the exit itself is new, or the pair has
        # lived past its TTL. Never rotate one without the other.
        if p is None or (now - p.born_at) > self.ttl:
            p = self._new_persona()
            p.exit_id = exit_id
            p.born_at = now
            self._bindings[exit_id] = p
        return p

def fetch(url, exit_id, proxy_template, pool):
    p = pool.persona_for_exit(exit_id)
    proxies = {"https": proxy_template.format(session=exit_id),
               "http": proxy_template.format(session=exit_id)}
    r = cffi_requests.get(url, impersonate=p.impersonate,
                          proxies=proxies, timeout=30)
    return r, p

if __name__ == "__main__":
    # With a residential provider that supports session IDs, each session
    # string pins a distinct exit IP for its sticky window.
    template = ("http://user-session-{session}:pass@"
                "gw.example-thordata-proxy.net:8080")
    pool = PersonaPool(exits=[f"s{i:04d}" for i in range(8)])
    for exit_id in pool.exits:
        r, persona = fetch("https://tls.peet.ws/api/all", exit_id,
                           template, pool)
        ja3 = r.json().get("tls", {}).get("ja3", "?")
        print(f"{exit_id}: persona={persona.name:16s} ja3={ja3[:16]}...")

Run this and you should see two things: within one process lifetime, each exit keeps a stable persona (try fetching twice in a row — the JA3 must not change), and across exits the JA3 values differ. Both properties are what you're paying for.

Verify it, don't assume it

The verification step is where most implementations quietly rot. Two checks belong in your pipeline, not in a one-off script:

Check 1 — stability per exit. Poll a fingerprint echo endpoint (e.g. tls.peet.ws/api/all, which returns JA3/JA4 and the HTTP/2 fingerprint) through the same sticky exit several times over its window. Any change means something between you and the target is rewriting your ClientHello — some filtering gateways do exactly that — and your impersonation is not surviving the route. You need to know that per exit tier, not per client build.

Check 2 — distribution across the pool. Log (exit_id, ja3) for a few thousand requests and look at the pool-wide distribution. If 100% of your traffic carries one fingerprint, your rotation layer is broken or decorative. If any single exit flips fingerprints mid-window, your affinity logic has a race — usually two workers grabbing the same exit through different pool instances. Make the pool a singleton, or move the binding into Redis.

from collections import Counter

def audit_fingerprints(samples):
    """samples: list of (exit_id, ja3_hash) tuples from production logs."""
    per_exit = {}
    for exit_id, ja3 in samples:
        per_exit.setdefault(exit_id, []).append(ja3)

    unstable = [e for e, js in per_exit.items() if len(set(js)) > 1]
    dist = Counter(ja3 for _, ja3 in samples)
    top_share = dist.most_common(1)[0][1] / len(samples)

    print(f"exits observed:      {len(per_exit)}")
    print(f"unstable exits:      {len(unstable)} (must be 0)")
    print(f"distinct fingerprints: {len(dist)}")
    print(f"top fingerprint share: {top_share:.1%} "
          f"(aim for < 40% on a 4-persona mix)")
    return len(unstable) == 0 and top_share < 0.5

For a four-persona weighted mix like the one above, the top fingerprint should sit around 55% of traffic if the weights hold — the < 40% threshold deliberately fails if your weights collapsed to one profile. Tune the assertion to your own persona set; the point is to have one.

The parts I got wrong before this worked

Three practical notes from running this in production:

Weight your personas by geography. A German news site's real audience has a Safari share that a US electronics retailer's doesn't. Pull the browser stats for your target market and mirror them. Uniform weighting is itself a fingerprint.

Version drift will burn you quarterly. When Chrome ships a new major and your chrome124 persona becomes a minority of real traffic, your mix silently becomes anomalous. Put persona versions in config and refresh them on a schedule — this is maintenance, not a one-time setup.

HTTP-level coherence is part of the same binding. The persona object is the right place to hang header order, Accept-Language, and even the TLS-extension-level settings that curl_cffi targets control. A Chrome TLS stack sending Firefox-style Accept headers defeats the whole exercise. In our stack, one persona object owns every layer of the client presentation, and the exit is just another attribute of it.

The mental model that made this click for me: an exit IP without a stable persona is a stranger the site has never met; a persona without a fresh IP is a regular the site has watched too long. You need both, and you need them bound together for exactly as long as a plausible user session — no longer.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Cache Before You Route: The Cheapest Proxy GB Is the One You Never Fetch

Greta — Tue, 08 Sep 2026 01:12:59 +0000

Cache Before You Route: The Cheapest Proxy GB Is the One You Never Fetch

Proxy cost optimization advice usually starts and ends with shopping: compare per-GB prices, negotiate committed volumes, switch providers. That advice optimizes the unit price of a resource your architecture is wasting. Before you can optimize what you pay per gigabyte, you have to ask how many of those gigabytes needed to exist at all.

Here's the uncomfortable arithmetic. A price-monitoring crawler that checks 50,000 SKUs every 15 minutes fetches roughly 4.8 million pages a day. At ~400 KB per product page, that's close to 2 TB of residential traffic monthly at full price — for a dataset where, on any given cycle, 95%+ of the pages have not changed since the last fetch. You are paying residential rates to re-download bytes you already own.

The core idea of this post: in a well-architected pipeline, the proxy is the last hop, not the first. Put an HTTP-conditional caching layer in front of proxy routing and let ETag/Last-Modified do the work. A 304 response is a few hundred bytes through a tunnel instead of 400 KB, and your bandwidth bill tracks actual change, not fetch frequency.

The three layers of "don't fetch it again"

Layered cheapest-first, because order is everything:

Deduplication at the scheduler: if the URL was fetched successfully 40 seconds ago in another worker, don't fetch it again. Boring, but the biggest single win in most codebases I audit — parallel workers independently crawling overlapping URL sets is endemic.
Conditional requests (this post's focus): send If-None-Match/If-Modified-Since with the validator you stored from the last fetch. A 304 costs a round trip but almost no bytes.
Content-hash skipping: even on a 200, hash the extracted payload — if the fields you actually store (price, stock, title) are unchanged, downstream processing and storage skip, and you learn the page's real change rate for tuning.

Layers 1 and 3 are bookkeeping. Layer 2 is where the proxy economics live, so let's build it properly.

A conditional cache in front of the proxy

The cache maps URL -> (etag, last_modified, fetched_at, body_hash). On every fetch it attaches validators; on a 304 it returns the stored body. Persistence is a JSON file for demonstration — in production, SQLite or Redis.

import requests
import hashlib
import json
import time
import os

class ConditionalCache:
    """HTTP conditional caching layer that sits BEFORE proxy routing."""

    def __init__(self, username, password, path="cache.json",
                 fresh_window=90, stale_max=86400):
        """
        fresh_window: seconds during which we don't even revalidate.
        stale_max:    seconds after which a cached entry is unusable.
        """
        self.path = path
        self.fresh_window = fresh_window
        self.stale_max = stale_max
        self.store = self._load()
        self.proxies = {
            "http": f"http://{username}:{password}@resi.thordata.com:9001",
            "https": f"http://{username}:{password}@resi.thordata.com:9001",
        }

    def _load(self):
        if os.path.exists(self.path):
            with open(self.path) as f:
                return json.load(f)
        return {}

    def _save(self):
        with open(self.path, "w") as f:
            json.dump(self.store, f)

    def fetch(self, url: str, force=False) -> dict:
        entry = self.store.get(url)
        now = time.time()

        # 1) Fresh enough: don't touch the network at all.
        if not force and entry and now - entry["fetched_at"] < self.fresh_window:
            return {"status": "fresh-hit", "body": entry["body"],
                    "bytes_via_proxy": 0}

        # 2) Stale beyond usefulness: treat as miss (drop validators).
        if entry and now - entry["fetched_at"] > self.stale_max:
            entry = None

        headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64)"}
        if entry:
            if entry.get("etag"):
                headers["If-None-Match"] = entry["etag"]
            if entry.get("last_modified"):
                headers["If-Modified-Since"] = entry["last_modified"]

        r = requests.get(url, proxies=self.proxies, headers=headers, timeout=25)

        if r.status_code == 304 and entry:
            # The win: ~300 bytes through the tunnel instead of ~400 KB.
            entry["fetched_at"] = now
            self._save()
            return {"status": "not-modified", "body": entry["body"],
                    "bytes_via_proxy": len(r.content)}

        r.raise_for_status()
        entry = {
            "body": r.text,
            "etag": r.headers.get("ETag"),
            "last_modified": r.headers.get("Last-Modified"),
            "body_hash": hashlib.sha256(r.content).hexdigest(),
            "fetched_at": now,
        }
        self.store[url] = entry
        self._save()
        return {"status": "fetched", "body": entry["body"],
                "bytes_via_proxy": len(r.content)}

Three details separate this from a naive implementation. First, fresh_window — revalidating every cycle is itself wasteful when the target's change cadence is slower than your crawl cadence. Second, the stale_max guard — validators from yesterday's CDN config can cause weird 200s-with-junk; expire them. Third, body_hash in the entry: it powers layer 3, because a 200 can still be a no-change event.

Measuring the actual saving: GB accounting per fetch

You can't optimize what you don't meter, so make the cache report its own economics:

class GBLedger:
    def __init__(self):
        self.bytes_by_status = {}

    def record(self, result: dict):
        status = result["status"]
        self.bytes_by_status[status] = (
            self.bytes_by_status.get(status, 0) + result["bytes_via_proxy"])

    def report(self, price_per_gb=3.5):
        total = sum(self.bytes_by_status.values())
        print(f"{'status':14} {'MB':>10}")
        for k, v in sorted(self.bytes_by_status.items()):
            print(f"{k:14} {v/1e6:>10.1f}")
        cost = total / 1e9 * price_per_gb
        print(f"total: {total/1e6:.1f} MB  ~ ${cost:.2f} at ${price_per_gb}/GB")
        full_refetch = None
        return total

ledger = GBLedger()
cache = ConditionalCache("user", "pass")
url = "https://shop.example.com/product/B08N5WRWNW"

for cycle in range(10):          # 10 crawl cycles, e.g. every 15 min
    result = cache.fetch(url)
    ledger.record(result)
ledger.report()

Run this against a real target and the pattern is dramatic: cycle 1 shows fetched with ~400 KB, cycles 2-10 show not-modified at a few hundred bytes each. On a catalog where 5% of pages change per cycle, bandwidth through the proxy drops by roughly an order of magnitude. That's the entire optimization — no vendor negotiation required.

When targets don't cooperate (and what to do)

The honest caveat: not every target emits ETag or honors conditional requests. E-commerce product pages behind edge caches usually do; script-rendered SPAs frequently don't. Audit your targets first:

def validator_audit(urls, proxies):
    for url in urls:
        r = requests.get(url, proxies=proxies, timeout=25)
        etag = bool(r.headers.get("ETag"))
        lm = bool(r.headers.get("Last-Modified"))
        cc = r.headers.get("Cache-Control", "")
        print(f"{url}\n  ETag={etag} Last-Modified={lm} Cache-Control={cc!r}")

validator_audit(["https://shop.example.com/p/1"], cache.proxies)

When validators are absent, fall back to layer 3 — you must fetch, but hash the extracted fields and skip everything downstream when unchanged, then let the measured change rate set your crawl cadence. A SKU that has changed twice in 90 days does not need 15-minute polling; adaptive per-URL intervals (crawl frequently after a change, back off exponentially during stability) compound with conditional requests and routinely cut another 30-50% of fetch volume.

One warning about cleverness: never serve stale data past the point where your consumers notice. The fresh_window and stale_max bounds exist to keep the cache honest. A monitoring product that reports yesterday's price as current is saving money by being wrong, which is the most expensive optimization there is.

One scaling note: the cache's value compounds with worker count. A single worker re-fetching out of ignorance wastes one request; fifty workers doing it concurrently waste fifty, and — worse — they arrive at the target as a synchronized burst from your exit IPs, which is exactly the traffic shape bot defenses are built to notice. Moving the cache from per-process memory to a shared store (Redis works well: the validator entry is small, and a GET url before every fetch is negligible next to a proxied round trip) turns those fifty redundant fetches into one fetch plus forty-nine lock-waits. The proxy sees a steady, modest stream instead of a thundering herd, and your spend drops in the same motion. Cost optimization and politeness optimization are, at the architecture level, the same optimization.

Where this fits in the stack

The mental model that ties it together: your pipeline has a request waterfall — scheduler → dedup → conditional cache → proxy router → target. Cost lives at the proxy hop; intelligence should live upstream of it. Most teams invert this: their schedulers are naive and their cost discussions are all about the proxy hop, so they optimize the most expensive tier to fetch bytes they already have. Flip the order. Dedup decides, the cache validates, the proxy fetches only what's genuinely new, and the per-GB bill becomes a function of your dataset's real change rate — which is the only denominator it should ever have had.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

ISP Proxies: The Undocumented Middle Tier Between Datacenter and Residential

Greta — Tue, 08 Sep 2026 01:11:41 +0000

ISP Proxies: The Undocumented Middle Tier Between Datacenter and Residential

Ask a room of scraping engineers to define an ISP proxy and you'll get confident, contradictory answers: "it's a residential proxy that doesn't rotate," "it's a datacenter IP that looks residential," "it's a static IP from Comcast." Documentation across the industry is thin, and providers have an incentive to keep it that way — the tier occupies a genuinely useful gap, and vague marketing makes it easy to sell as either premium-residential or magic-datacenter depending on the buyer.

Here's the precise definition. An ISP proxy is an IP address allocated from a range registered to a consumer ISP (AS far as BGP is concerned: a residential ASN), but routed to hardware sitting in a datacenter. The packets travel over datacenter-grade network infrastructure; the IP's registration says "home internet." That's the entire trick. But the trick has engineering consequences that determine exactly when this tier wins and when it quietly fails.

The core idea of this post: ISP proxies trade the diversity of residential pools for the stability and speed of infrastructure — so they only make sense for workloads that need a stable identity with residential-grade reputation and datacenter-grade throughput. Use them outside that box and you pay premium prices for datacenter-like fragility.

What the ASN actually says — and what it doesn't

When a bot-mitigation system scores your IP, the first check is usually the ASN and its classification: hosting provider, consumer ISP, or mobile carrier. ISP proxies pass that first check because their ranges genuinely belong to consumer ISP ASNs. But modern defenses don't stop at ASN. They also weigh:

ASN historical abuse rates — some ISP ranges sold to proxy providers accumulate abuse signals as thousands of scraping customers share the same /24s.
IP stability patterns — an IP that's online 24/7 with datacenter-regular latency, from a range where neighbors are home users on DHCP... sophisticated systems can notice.
Geolocation coherence — the IP's geo registration vs. where the traffic claims to be.

So "passes the ASN check" is the entry ticket, not the finish line. In practice ISP proxies clear most IP-reputation walls — Cloudflare, Akamai, and PerimeterX treat them as residential on the IP dimension — but they won't save you if your fingerprint or behavior layer is broken. The IP is one signal among several; buying a better IP never fixes a broken TLS handshake.

Verify before you trust: an ASN audit harness

Since the tier's entire value proposition lives in the ASN registration, verify it independently. Don't trust the sales page — query the registration data yourself:

import requests
from collections import Counter

HOSTING_HINTS = ("amazon", "google", "microsoft", "ovh", "hetzner",
                 "digitalocean", "linode", "vultr", "choopa", "contabo",
                 "leaseweb", "ovh", "cloud", "hosting", "server", "datacamp")

def audit_ip(proxy_url: str, label: str):
    """Check what an exit IP's public registration actually says about it."""
    try:
        info = requests.get("https://ipinfo.io/json",
                            proxies={"http": proxy_url, "https": proxy_url},
                            timeout=20).json()
        org = info.get("org", "unknown")
        looks_hosting = any(h in org.lower() for h in HOSTING_HINTS)
        verdict = "HOSTING-ASN (datacenter, not ISP!)" if looks_hosting else "ISP-ASN (plausible)"
        print(f"{label:12} ip={info['ip']:16} org={org:40} {verdict}")
        return not looks_hosting
    except Exception as e:
        print(f"{label:12} probe failed: {e}")
        return False

# Audit a sample of your purchased ISP proxies:
# if more than ~5% show hosting ASNs, you have a billing problem.
for i in range(3):
    audit_ip(f"http://user:pass@isp.thordata.com:9002", f"isp-{i}")

Run this on a schedule. IP ranges get re-registered, providers re-source supply, and a batch of "ISP" IPs that quietly becomes hosting-ASN will produce the most confusing blocking pattern of your career: same code, same headers, success rate collapses, and nothing in your logs points at the cause.

Where ISP proxies genuinely win

1. High-volume monitoring of one target. A price tracker hitting one retailer every 30 seconds from one stable identity with residential reputation and sub-50ms latency. A rotating residential pool would work too, but you'd pay per GB for traffic that an unmetered ISP IP carries for a flat monthly fee. At roughly 500 GB/month on a single target, the economics flip hard toward ISP.

2. API allowlists. Some third-party APIs allowlist IP ranges instead of (or alongside) API keys. You need a stable, known IP — sticky sessions can't promise that — but a datacenter IP might not pass the provider's reputation screen. ISP proxies are the only tier that satisfies both constraints at once.

3. Long-lived authenticated sessions. Portals where your account's login history matters. ISP proxies give you the stable identity of static residential with better uptime and latency, because the hardware sits on datacenter power and networking rather than a home router that reboots when someone trips over a cable.

4. Latency-sensitive interactive scraping. Browser automation where every request through a rotating residential hop adds 300-800ms. ISP proxies typically add only 20-80ms over direct datacenter routes. Over a 40-request flow, that's the difference between a page flow that feels human and one that takes two minutes.

Where ISP proxies quietly fail

1. Anything needing pool diversity. One IP is one identity. If your workload is "blast 50,000 requests at a search engine," an ISP proxy just concentrates your entire footprint on one address — the opposite of what you need. That job belongs to rotating residential.

2. Targets that fingerprint beyond the IP. ISP proxies inherit none of the environmental noise of real home connections. Some defenses model connection-level jitter, TCP behavior, and latency distributions. On such targets, ISP proxies pass the ASN check and still fail the coherence check. The fix is fingerprint work, not IP shopping.

3. Anti-overselling diligence. If a provider sells the same ISP IP to twenty customers hammering the same domains, the range's reputation decays for everyone. This is invisible until it isn't. Track your own per-IP success rates; sudden synchronized degradation across all your ISP IPs means you're sharing a poisoned range, and only your provider can fix that.

A practical hybrid: ISP for the hot path, residential for the burst

The strongest architecture I've deployed uses both tiers deliberately — ISP proxies carry the steady baseline load, and rotating residential absorbs spikes and retries:

import requests
import random

class HybridRouter:
    """
    ISP proxies for steady, identity-bound traffic.
    Rotating residential for burst traffic and retries.
    """

    def __init__(self, username, password):
        self.isp_proxy = {
            "http": f"http://{username}:{password}@isp.thordata.com:9002",
            "https": f"http://{username}:{password}@isp.thordata.com:9002",
        }
        self.resi_proxy = {
            "http": f"http://{username}:{password}@resi.thordata.com:9001",
            "https": f"http://{username}:{password}@resi.thordata.com:9001",
        }

    def route(self, kind: str):
        return self.isp_proxy if kind in ("baseline", "allowlisted") else self.resi_proxy

    def get(self, url, kind="baseline", retries=3):
        primary = self.route(kind)
        try:
            r = requests.get(url, proxies=primary, timeout=20)
            if r.status_code == 200:
                return r
        except requests.RequestException:
            pass
        # Retry on residential: a fresh identity sees a fresh target.
        for _ in range(retries):
            try:
                r = requests.get(url, proxies=self.resi_proxy, timeout=25)
                if r.status_code == 200:
                    return r
            except requests.RequestException:
                continue
        raise RuntimeError(f"failed: {url}")

router = HybridRouter("user", "pass")
# Steady heartbeat monitoring — the ISP proxy's home turf:
product = router.get("https://shop.example.com/product/B08N5WRWNW", kind="baseline")
# One-off discovery crawl — residential's home turf:
sitemap = router.get("https://shop.example.com/sitemap.xml", kind="burst")

The retry leg matters: when the ISP identity gets a soft block (429, CAPTCHA interstitial), falling back to the same identity is the worst option. A fresh residential exit gives you a genuinely new vantage point, and it keeps the ISP identity's record clean for the baseline job it exists to do.

The compressed decision rule

Need thousands of rotating identities → rotating residential.
Need carrier-grade reputation at any cost → mobile.
Need cheap bulk throughput on undefended endpoints → datacenter.
Need one stable, fast, ISP-registered identity for a focused, long-running workload → ISP proxy.

That fourth slot is narrower than the marketing suggests, but within it, ISP proxies are the only tier that fits. Buy them for their actual properties — ASN classification plus infrastructure-grade stability — and audit those properties continuously, because the tier's value lives entirely in a registration record that can change without notice.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele