DEV Community

Greta
Greta

Posted on

The Cold Start Tax: Why the First Request Through a Fresh Residential IP Fails Most

The Cold Start Tax: Why the First Request Through a Fresh Residential IP Fails Most

If you rotate residential proxies per request, you've probably seen a pattern you can't explain from the logs alone: the first request through a brand-new exit IP fails at a rate far above your average. The retry succeeds. The one after that succeeds. You write it off as noise.

It isn't noise. It's a cold-start effect, and it has a handful of distinct technical causes. Understanding them changes how you schedule work across exits — and explains precisely when paying for static residential IPs beats rotating pools, in a way that "static IPs are more stable" marketing copy never does.

The four sources of cold-start failure

1. TLS and session establishment on a virgin path. A fresh exit means a fresh TCP+TLS handshake to the target, no session tickets, no negotiated connection reuse. Targets with aggressive TLS-fingerprint scoring evaluate a full first handshake — exactly the profile that looks least like a returning browser. Warmed exits that resume TLS sessions or reuse keep-alive connections look more like real users revisiting.

2. Reputation ramp-up. Risk engines score IPs partly by recency of first sighting. An exit that has been quiet for hours and suddenly appears gets a skepticism penalty relative to one that has been sending steady, modest traffic. The first request eats that penalty.

3. DNS and routing warm-up on the provider side. Some residential exit nodes are consumer devices or gateways that need to establish their upstream path, resolve DNS through the carrier, and sometimes NAT a new mapping. The first request absorbs several hundred extra milliseconds and occasionally times out outright — a failure that has nothing to do with the target's defenses.

4. Cookie-less first contact. If your workload carries session cookies from earlier work, a brand-new IP presenting an old cookie jar is itself a red flag (IP suddenly changing mid-session is a classic account-takeover signal). If it carries no cookies, a very defensive target may interstitial-challenge first contact from unknown IPs.

Not all cold-start failures are equal, and that matters for what you do about them. #1 and #3 are infrastructure effects — they punish every first request regardless of target. #2 and #4 are defense effects — they punish first contact with defensive targets.

Measuring your own cold-start tax

Before fixing anything, quantify it. The experiment: through a rotating residential gateway, fire a sequence of N requests where you can identify the request ordinal within each exit's lifetime, and bucket outcomes by ordinal. With Thordata's gateway you can do this by forcing a session ID: the first request with a new session ID gets a new IP; subsequent requests with the same ID hold the same exit for the session window.

import requests
import collections

GATEWAY = "http://user:pass-TD.{sid}-10m.gate.thordata.com:7000"

def probe(url, sid, ordinal):
    # session id in credentials pins the exit for the window
    proxy = GATEWAY.format(sid=sid)
    try:
        r = requests.get(
            url,
            proxies={"http": proxy, "https": proxy},
            headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"},
            timeout=15,
        )
        # valid = 200 AND expected content marker present
        valid = r.status_code == 200 and b"expected-marker" in r.content
        return ordinal, r.status_code, valid
    except requests.RequestException as e:
        return ordinal, 0, str(type(e).__name__)

def cold_start_profile(url, sessions=30, per_session=8):
    outcomes = collections.defaultdict(lambda: [0, 0])  # ordinal -> [valid, total]
    for s in range(sessions):
        sid = f"cs{s:04d}"           # fresh session id = fresh exit
        for o in range(1, per_session + 1):
            _, status, valid = probe(url, sid, o)
            rec = outcomes[o]
            rec[1] += 1
            rec[0] += 1 if (valid is True or valid == "NameError") else 0
            if valid is True:
                rec[0] += 0  # counted above; keeps structure explicit
    # simpler recompute for clarity
    return outcomes
Enter fullscreen mode Exit fullscreen mode

A cleaner version separates the validity check:

def run_profile(url, sessions=30, per_session=8):
    outcomes = {o: [0, 0] for o in range(1, per_session + 1)}
    for s in range(sessions):
        sid = f"cs{s:04d}"
        for o in range(1, per_session + 1):
            _, status, err = probe(url, sid, o)
            outcomes[o][1] += 1
            if status == 200:
                outcomes[o][0] += 1
    print("ordinal  success%")
    for o, (ok, tot) in sorted(outcomes.items()):
        print(f"{o:>7}  {ok / tot:>7.0%}")
    return outcomes
Enter fullscreen mode Exit fullscreen mode

Typical output on a defensively moderate e-commerce target through rotating residential exits:

ordinal  success%
      1       62%
      2       81%
      3       88%
      4       91%
      5       93%
      6       92%
      7       94%
      8       93%
Enter fullscreen mode Exit fullscreen mode

That ~30-point gap between ordinal 1 and ordinal 4+ is the cold-start tax. On a heavy target I've measured first-request success below 40% while steady-state sat at 90%. If your pipeline rotates per request, every request is a first request — you are paying the maximum tax on 100% of your traffic.

Three structural fixes

Fix A: warm the exit before real work. Send one cheap, low-stakes request (the homepage, a robots.txt, a static asset) through a new session before the request you actually care about. You pay one extra request per exit; you convert your working request from ordinal 1 to ordinal 2–3.

def warm_then_work(session_id, work_url):
    proxy = GATEWAY.format(sid=session_id)
    s = requests.Session()
    s.proxies = {"http": proxy, "https": proxy}
    s.headers.update({"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"})
    s.get("https://target.example/", timeout=15)      # warm-up, ignore result
    return s.get(work_url, timeout=20)                 # now ordinal 2, keep-alive reused
Enter fullscreen mode Exit fullscreen mode

Using a requests.Session matters here: the session reuses the TCP connection, so the second request rides the same keep-alive socket — it inherits the warmed path, not just the warmed reputation.

Fix B: size sticky windows to the measured warm-up curve. If success saturates at ordinal 4, holding an exit for only 2 requests wastes the warm-up; holding it for 200 requests on a target that rate-limits per IP overruns the other cliff. Your session window should sit just past the saturation point of this curve — for most defensive targets, somewhere between 5 and 30 requests. The measurement above tells you where, per target.

Fix C: use static residential IPs where cold starts are intolerable. This is the honest case for static residential. If your workload is low-volume, long-lived, and latency-sensitive — a logged-in monitor that checks one dashboard every few minutes, an account that must look continuously resident — a rotating exit's constant cold-start tax is the worst possible profile. A static residential IP pays the cold-start cost exactly once, ever, and every subsequent request is warm. That's not "more stable" in a vague sense; it's a measured property of the request-ordinal curve.

Conversely, if your workload is high-volume, parallel, and tolerant of retries, rotating with warmed sessions usually wins on cost — you don't need to pay static-IP premiums to avoid a tax that a $0.001 warm-up request already avoids.

The decision rule

Put it as a one-liner your team can apply:

  • Many parallel workers, retry-tolerant, throughput-bound → rotating residential + session warm-up, window sized to your measured curve.
  • Few long-lived identities, latency-sensitive, login-bound → static residential, one IP per identity, cold start paid once.
  • Unsure → measure the curve first. Twenty minutes with run_profile() replaces weeks of arguing.

The deeper point: in proxy-backed scraping, a surprising share of "reliability problems" are actually scheduling problems — the right request sent through the wrong-aged exit. Before you blame the vendor or escalate to a pricier tier, check whether your failures cluster at ordinal 1. They usually do.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Top comments (0)