The CAPTCHA Is Addressed to the IP: Differential Diagnosis With Paired Requests Across Exits

Every scraper team has a CAPTCHA story that goes like this: the target started serving challenges, someone plugged in a solver, the solver bill grew, and nobody ever established the most basic fact about the situation — whether the challenge was addressed to the scraper or to the address it happened to be using.

Because here's what a challenge response actually tells you: something is suspicious. It does not tell you what. And the two possible worlds demand opposite responses:

• The site hardened globally. Every visitor, every IP, every fingerprint gets challenged. Solvers are the only path, and your real question is whether the data is worth the solver economics.
• Your exit is burned. The site challenges this address (or this address plus fingerprint combination) because of what happened on it yesterday — your own traffic, or the shared history of a residential IP that a thousand other scrapers have rented. Other exits sail through clean.

Solvers are the correct answer to the first world and a waste of money in the second. Yet teams routinely pay solver rates for challenges that a fresh exit would have dodged for free. The fix is cheap: before reacting to any challenge, run a differential diagnosis — one paired request through a different exit — and let the classification drive your response.

The control-request pattern

Borrowed loosely from medicine: to attribute a symptom, compare against a control. If the same URL, same headers, same fingerprint profile — but a different exit — returns clean content, the challenge was exit-scoped. If the control also gets challenged, the site (or its WAF rule for your target path) is hard.

# pip install requests
import enum
import requests
from typing import Optional

class Verdict(enum.Enum):
    CLEAN = "clean"                       # no challenge anywhere
    EXIT_BURNED = "exit_burned"           # primary challenged, control clean
    SITE_HARD = "site_hard"               # both challenged
    CONTROL_DEAD = "control_dead"         # inconclusive: control itself failed

CHALLENGE_MARKERS = (
    "cf-chl",              # Cloudflare challenge pages
    "challenge-platform",  # Cloudflare JS challenge asset path
    "recaptcha/api.js",
    "hcaptcha.com/1/api.js",
    "px-captcha",          # PerimeterX / HUMAN
    "arkose",
)

def looks_like_challenge(resp: requests.Response) -> bool:
    if resp.status_code in (403, 429):
        return True
    ct = resp.headers.get("content-type", "")
    if "text/html" not in ct:
        return False
    body_head = resp.text[:4000].lower()
    return any(m in body_head for m in CHALLENGE_MARKERS)

def fetch(url: str, proxy: Optional[str]) -> requests.Response:
    s = requests.Session()
    s.headers.update({
        "user-agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                       "AppleWebKit/537.36 (KHTML, like Gecko) "
                       "Chrome/124.0.0.0 Safari/537.36"),
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "accept-language": "en-US,en;q=0.9",
    })
    # IMPORTANT: same everything except the exit. If you change the
    # user-agent between the two fetches, you've changed two variables
    # and the verdict is worthless.
    return s.get(url, proxies={"http": proxy, "https": proxy} if proxy else None,
                 timeout=30, allow_redirects=True)

def diagnose(url: str, primary_proxy: str, control_proxy: str) -> Verdict:
    primary = fetch(url, primary_proxy)
    if not looks_like_challenge(primary):
        return Verdict.CLEAN

    control = fetch(url, control_proxy)
    if control.status_code >= 500 or control.status_code == 0:
        return Verdict.CONTROL_DEAD
    if looks_like_challenge(control):
        return Verdict.SITE_HARD
    return Verdict.EXIT_BURNED

if __name__ == "__main__":
    # Two sticky sessions = two distinct exits.
    primary = "http://user-session-p01:[e-mail]:8080"
    control = "http://user-session-c99:[e-mail]:8080"
    verdict = diagnose("https://example-target.com/products/12345",
                       primary, control)
    print(verdict.value)

Run this the next time your alerting fires on a challenge spike. In my experience the SITE_HARD verdict is the minority case — most "they added CAPTCHA" panics turn out to be a handful of exits with ruined history, while the rest of the pool was quietly fine.

Turning verdicts into pool policy

The verdict is only useful if something acts on it. Three rules, wired into the exit manager:

EXIT_BURNED retires the exit, not the job. The immediate task retries on a different exit — no solver, no backoff ceremony. The burned exit goes into a cooldown set with an escalating schedule (30 min, then 2h, then 24h). IP reputation on the target side decays on a sliding window; an exit you hammer anyway goes from soft-challenged to hard-blocked, and if it's a static/ISP address you own, you've burned an asset you paid a premium for.

SITE_HARD triggers the cost conversation, not the retry loop. When the control request also draws a challenge, retries across the pool are just spreading the damage — every exit you burn a challenge on gets a little closer to its own EXIT_BURNED verdict. The right move is to stop, quantify (is this the whole site, or one path? a WAF rule for your UA class? a geo rule — worth testing exits in a second country), and only then decide between solver spend and walking away.

CONTROL_DEAD is a maintenance signal, not a shrug. If your control exits are erroring, your diagnosis infrastructure is down — and the control pool needs the same health monitoring as the working pool. Keep a small dedicated set of control exits with known-clean status, refreshed on a schedule, so a diagnosis never blocks on a dead control.

Here's the exit-manager sketch:

import time

class ExitManager:
    COOLDOWNS = [1800, 7200, 86400]   # 30min, 2h, 24h

    def __init__(self, exits, control_exits):
        self.exits = list(exits)
        self.control_exits = list(control_exits)
        self.cooldown: dict[str, tuple[int, float]] = {}  # exit -> (strikes, until)
        self.burned_events = 0

    def available(self) -> list[str]:
        now = time.time()
        return [e for e in self.exits
                if e not in self.cooldown or self.cooldown[e][1] < now]

    def report(self, verdict, exit_id):
        if verdict == Verdict.EXIT_BURNED:
            self.burned_events += 1
            strikes, _ = self.cooldown.get(exit_id, (0, 0))
            wait = self.COOLDOWNS[min(strikes, len(self.COOLDOWNS) - 1)]
            self.cooldown[exit_id] = (strikes + 1, time.time() + wait)
        elif verdict == Verdict.CLEAN:
            self.cooldown.pop(exit_id, None)   # reputation recovered

Two refinements that earn their keep

Test geo as a third variable when both exits challenge. Some "site hardened" verdicts are actually geo rules — the target challenges every visitor from countries your datacenter tier lives in, or from ASNs it has blanket rules for. One more control request through an exit in a different country splits SITE_HARD into "hard everywhere" and "hard for this geo." The second case is a routing problem, and routing problems are cheaper than solver problems. This is also where challenge language is a free hint: if the CAPTCHA page renders in a language that doesn't match the exit's country, the WAF is geo-routing your challenge — the site thinks your exit is somewhere your fingerprint says it isn't.

Fingerprint the pairing, not just the IP. A verdict of EXIT_BURNED technically means the (exit, fingerprint) pair is burned. If your persona-affinity layer binds fingerprints to exits — it should — the pair retires together, and a clean exit can keep working with a different persona. If you run one global fingerprint, every EXIT_BURNED verdict is ambiguous between address and address-plus-fingerprint, and you'll retire exits that were fine.

The economics, stated plainly

A solver call on a hard challenge site costs real money per thousand — call it a dollar-to-tens-of-dollars per thousand range across vendors and difficulty tiers. A differential diagnosis costs two cheap requests. Every EXIT_BURNED verdict converted from "route to solver" to "retry on next exit" is solver spend you simply don't incur, and most pools see the majority of their challenge volume resolve this way once the diagnosis is automatic.

The discipline underneath all of this is the same one that runs through good scraping engineering: never react to an ambiguous signal. A 429 without Retry-After is ambiguous; a timeout is ambiguous; a CAPTCHA is ambiguous. The response to ambiguity is a controlled experiment, and the proxy pool — many exits, same client — is exactly the apparatus for running one.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele