Not Every Challenge Is a CAPTCHA: A Triage Pipeline for Silent Rate Limits, JS Interstitials, and Real Puzzles

When a scraper's success rate drops, the reflex is to reach for the CAPTCHA-solver invoice. But if you actually crack open the failing responses and read them, a large fraction contain no CAPTCHA at all. They are 200s with a "Please verify you are human" banner that never renders a puzzle; empty-body responses; 403s from a CDN, not the origin; or — the sneakiest — normal-looking HTML that is a JS interstitial whose real content only appears after window.location is reassigned by a script. Treating all of these as "we got CAPTCHAs, solve harder" is how teams end up paying a solver service to fail on traffic that never required a solver in the first place.

The core idea of this article: challenge responses come in distinct tiers with completely different correct responses — routing, retry, render, or solve — and the profitable engineering move is a cheap triage classifier in front of your solver, so you only escalate to the expensive tier for genuine puzzles. And crucially, most of the tiers are best resolved at the proxy exit layer, not the CAPTCHA layer.

The four challenge tiers

Let me name the tiers by what they actually are and what fixes them:

Tier 0 — Silent rate limit. HTTP 200, but content is truncated, generic, or a bare "too many requests" string. Sometimes 429 with an HTML body. The origin hasn't asked you to prove humanity; it's asked you to slow down. Fix: throttle the exit (or move to a new exit), not solve. Sending these to a solver returns garbage and wastes money.

Tier 1 — CDN block / IP reputation. A 403 or 429 with a Cloudflare, Akamai, or PerimeterX challenge page — often the "Just a moment..." interstitial with a <title> that updates after a JS proof-of-work runs. Notice: the challenge is addressed to your exit IP's reputation, not necessarily to your request content. Fix: rotate the exit or change its tier. A browser that solves the JS proof from the same low-reputation IP often re-challenges, because the reputation is the problem.

Tier 2 — JS interstitial / meta-refresh. The response is a small HTML document that sets a cookie via document.cookie and reloads. No puzzle ever. Browsers pass through it in milliseconds. Fix: render the page in a headless engine, let the JS run, and reuse the resulting cookies. A CAPTCHA solver cannot help; this is a rendering problem.

Tier 3 — Actual CAPTCHA. A real image puzzle (reCAPTCHA v2/v3, hCaptcha, image grid, slider). Fix: this is the only tier where a solver earns its keep — and even then, sometimes a different exit skips the CAPTCHA entirely, because it's tied to IP risk, not to you.

The economics are stark. If a solver is $1–3 per thousand and 60% of your "challenge" traffic is actually Tier 0/1/2, you are throwing most of that spend away while also failing to fix the real cause (your exits), so the same URLs challenge you again on the next run.

A cheap triage classifier

Triage should run on the response you already have, in microseconds, with no extra requests. The signal set is small and highly predictive:

import re

CDN_HINTS = re.compile(
    r"(cf-ray|x-served-by|akamai|x-iapx-|px-captcha|cf_chl)", re.I
)
CHALLENGE_MARKERS = re.compile(
    r"(just a moment|verify you are human|checking your browser|attention required|"
    r"enable javascript and cookies)", re.I
)
RECAPTCHA_MARKERS = re.compile(
    r"(recaptcha|g-recaptcha|hcaptcha|data-sitekey|captcha-challenge|cf_turnstile)", re.I
)

def triage(resp) -> tuple[str, str]:
    """Return (tier, action). resp is a requests/httpx response object."""
    body = resp.text[:20000]  # don't scan huge bodies
    title = _extract_title(body)
    lower = body.lower()

    # Tier 3 first: an actual puzzle widget is the highest-confidence signal.
    if RECAPTCHA_MARKERS.search(lower):
        return ("T3_captcha", "solve_or_rotate")

    # Tier 1: CDN/anti-bot interstitial (JS challenge, no widget).
    if CDN_HINTS.search(resp.headers.get("server", "") + " " + str(resp.headers)) or \
       "just a moment" in lower or "cf_chl" in lower:
        return ("T1_cdn_challenge", "rotate_exit")

    # Tier 2: JS/meta interstitial that reloads with a cookie.
    if CHALLENGE_MARKERS.search(title) or _has_meta_refresh(body) or \
       ("document.cookie" in lower and "<script" in lower and len(body) < 3000):
        return ("T2_js_interstitial", "render_headless")

    # Tier 0: suspicious 200 with empty/short body, or 429.
    if resp.status == 429 or (resp.status == 200 and _looks_empty(lower)):
        return ("T0_silent_limit", "throttle_exit")

    return ("pass", "continue")

The key insight encoded in the ordering is: detect the actual puzzle last but with the tightest regex, and treat everything else as "not my solver's problem." A "verify you are human" string on a Cloudflare page is a Tier 1 reputation signal, not a Tier 3 puzzle, and calling it T1 means the next line of your code changes the exit instead of calling an API.

The small helpers:

import re as _re
def _extract_title(html: str) -> str:
    m = _re.search(r"<title[^>]*>(.*?)</title>", html, _re.I | _re.S)
    return m.group(1).strip() if m else ""

def _has_meta_refresh(html: str) -> bool:
    return bool(_re.search(r'<meta[^>]+http-equiv=["\']refresh', html, _re.I))

def _looks_empty(text: str) -> bool:
    stripped = _re.sub(r"<[^>]+>", " ", text).strip()
    return len(stripped) < 50

The escalation ladder, in cost order

Once you have a tier, route it to the cheapest fix that plausibly works, and only escalate after that fails. The ladder, from free to expensive:

1. Rotate the exit (cost: one extra request). Appropriate as the first move for Tier 1 and Tier 3, because both are frequently IP-reputation-driven. If the same URL succeeds from a fresh residential exit, you never needed a solver at all. This one fact is why "avoidance before solving" beats "solving" as a default: exit rotation is nearly free and eliminates a large share of challenges outright.
2. Change exit tier (cost: a slightly higher per-GB price). If a datacenter exit is getting Tier 1 challenges, a mobile or high-reputation residential exit often walks straight through. This is a routing decision, not a CAPTCHA decision.
3. Render headless (cost: CPU + latency). Tier 2's fix. Run the interstitial's JS, capture the cookie it sets, and replay the real request with that cookie and the same sticky exit so the reputation and session continuity hold.
4. Solve (cost: money). The genuine last resort for Tier 3 that survives exit rotation. Track it as an exception rate, not a routine.

The important architectural point is that the ladder lives above your proxy router. Triage returns an action string; the router maps rotate_exit to "pick a new exit and re-issue," throttle_exit to "put this exit in cooldown," and solve_or_rotate to "rotate once, and only if the fresh exit still shows the puzzle, call the solver."

async def handle(session, url, exit_url, solver_client):
    for attempt in range(6):
        resp = await session.get(url, proxy=exit_url)
        tier, action = triage(resp)
        if tier == "pass":
            return resp
        if action == "throttle_exit":
            await pool.throttle(exit_url, backoff=2.0)
        elif action == "rotate_exit":
            exit_url = pool.rotate(same_tier=True)
        elif action == "render_headless":
            return await render_and_replay(url, exit_url)  # keep sticky exit
        elif action == "solve_or_rotate":
            if attempt == 0:
                exit_url = pool.rotate(higher_reputation=True)  # try free first
                continue
            token = await solver_client.solve(resp)            # pay only now
            return await session.post(url, proxy=exit_url, data={"g-recaptcha-response": token})
    raise ChallengeExhausted(url)

Instrument the tiers, because they score your proxy pool

The quiet payoff of a triage pipeline is the telemetry it produces. Because you're labeling every challenge with a tier and the exit that triggered it, you can compute the metric that actually matters for provider selection: Tier 1 + Tier 3 rate, per exit-subnet, per origin. A subnet whose exits trigger CDN challenges 40% of the time is reputation-poor; you should stop routing sensitive origins through it entirely. Meanwhile a "clean" subnet at 2% is worth paying a premium for.

Without triage, all you see is "success rate dropped on Tuesday," and you fix it by buying more solver credits — treating the symptom. With triage you see "the mobile tier started returning Tier 1 Cloudflare challenges at 8x the residential tier this week," and you fix the real problem by reweighting the router, spending less on solvers and getting better yield at the same time.

The discipline is simple but rarely enforced: never send a response to a solver that your triage classifier hasn't first confirmed contains an actual puzzle, and never conclude a challenge is about you before you've tested whether it's about your exit. Most anti-bot friction is IP-shaped, and once you stop paying to "solve" IP problems, your scraper gets cheaper and better at the same time.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele