Preflight or Perish: The 60-Second Proxy Check That Saves Hours of Crawler Debugging

Every beginner proxied-scraper outage follows the same arc. The crawl starts, the progress bar fills, and forty minutes later the success rate is 12%. Three hours of debugging later the cause turns out to be something a ten-second check would have caught at startup: expired credentials, a gateway that silently stopped serving residential exits, a geo parameter that was being ignored, or rotation that was not rotating.

The fix is boring and hugely effective: run a preflight battery against your proxy configuration before the crawl, and abort the run if any assertion fails. A crawler without a preflight is a machine for converting configuration errors into wasted hours.

What "working" actually means, decomposed

"Proxy works" is at least six independent properties, and failing any one of them produces the same ambiguous symptoms downstream:

1. Tunnel reachability — the gateway accepts TCP and completes TLS. Fails as ProxyError, TunnelConnectionFailed, timeouts.
2. Authentication — credentials accepted for the exit (the 407 traps are a whole separate article).
3. Egress identity — the request really exits through the proxy, not directly. This one is terrifying because it succeeds: your scraper runs fine, and your own datacenter IP quietly gets burned.
4. Geo and tier claims — the exit is in the country/city you asked for, and its ASN looks residential rather than hosting.
5. Rotation actually rotates — two consecutive requests through a rotating endpoint produce two different exit IPs.
6. Latency budget — p50/p95 under your deadline; a 4-second exit is "working" but will time out half your crawl.

A preflight is just these six checks, automated, with a pass/fail gate.

A runnable preflight harness

Uses public echo endpoints so you can paste it into your own project immediately:

# preflight.py
import statistics, time
from dataclasses import dataclass
import requests

ECHO_IP = "https://api.ipify.org?format=json"          # egress identity
ECHO_GEO = "https://ipinfo.io/json"                    # geo + ASN/org fields
TIMEOUT = 15.0

@dataclass
class Check:
    name: str
    ok: bool
    detail: str

def _get(url, proxy):
    t0 = time.perf_counter()
    r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=TIMEOUT)
    r.raise_for_status()
    return r, (time.perf_counter() - t0) * 1000

def preflight(proxy: str, expect_country: str | None = None,
              expect_rotating: bool = True, p95_budget_ms: float = 3000) -> list[Check]:
    out: list[Check] = []

    # 1+2. reachability and auth, in one attempt
    try:
        r, ms = _get(ECHO_IP, proxy)
        exit_ip = r.json()["ip"]
        out.append(Check("tunnel+auth", True, f"200 in {ms:.0f} ms"))
    except requests.exceptions.ProxyError as e:
        return [Check("tunnel+auth", False, f"PROXY ERROR (check host/port/creds): {e}")]
    except Exception as e:
        return [Check("tunnel+auth", False, repr(e))]

    # 3. egress is really proxied — compare against direct connection
    try:
        direct = requests.get(ECHO_IP, timeout=TIMEOUT).json()["ip"]
        out.append(Check("egress", direct != exit_ip,
                         f"proxy={exit_ip} direct={direct}"))
    except Exception:
        # direct blocked by firewall is fine; we only warn if IPs could match
        out.append(Check("egress", True, f"direct check unavailable; proxy={exit_ip}"))

    # 4. geo and tier sanity
    try:
        info, _ = _get(ECHO_GEO, proxy)
        city = info.json()
        org = city.get("org", "?")
        hosting = any(k in org.lower() for k in ("aws", "azure", "digitalocean",
                                                 "ovh", "hetzner", "vultr"))
        country_ok = (expect_country is None or city.get("country") == expect_country)
        out.append(Check("geo", country_ok,
                         f"{city.get('city')},{city.get('country')} org={org}"))
        out.append(Check("tier-claim", not (hosting and "residential" in proxy.lower()),
                         f"ASN org={org}"))
    except Exception as e:
        out.append(Check("geo", False, repr(e)))

    # 5. rotation liveness
    try:
        ips = set()
        for _ in range(4):
            r, _ = _get(ECHO_IP, proxy)
            ips.add(r.json()["ip"])
        unique = len(ips)
        out.append(Check("rotation",
                         unique >= 2 if expect_rotating else unique == 1,
                         f"{unique}/4 unique exits"))
    except Exception as e:
        out.append(Check("rotation", False, repr(e)))

    # 6. latency budget
    samples = []
    for _ in range(6):
        try:
            _, ms = _get(ECHO_IP, proxy)
            samples.append(ms)
        except Exception:
            pass
    if len(samples) < 3:
        out.append(Check("latency", False, "too many failures to measure"))
    else:
        p95 = statistics.quantiles(samples, n=20)[-1] if len(samples) >= 4 else max(samples)
        out.append(Check("latency", p95 <= p95_budget_ms,
                         f"median {statistics.median(samples):.0f} ms, p95 {p95:.0f} ms"))
    return out


if __name__ == "__main__":
    PROXY = "http://USER:[e-mail]:9000"   # your config
    checks = preflight(PROXY, expect_country="US")
    width = max(len(c.name) for c in checks)
    for c in checks:
        print(f"[{'PASS' if c.ok else 'FAIL'}] {c.name:<{width}}  {c.detail}")
    raise SystemExit(0 if all(c.ok for c in checks) else 2)

Wire it into the entrypoint of any job:

def main():
    if any(not c.ok for c in preflight(PROXY)):
        log.critical("preflight failed — refusing to start crawl")
        notify("proxy preflight failed", level="page")
        sys.exit(2)
    run_crawl()

The two lines that matter most are the direct != exit_ip comparison and the rotation count. Those are the failures that otherwise masquerade as "the website is blocking us," and they are exactly the ones beginners spend whole weekends debugging.

Interpreting the results

The check output tells a story if you read the details, not just the flags:

• tunnel+auth FAIL with a 407 mention → credentials, not the site. Re-run your auth probe.
• egress FAIL with identical IPs → your library call never actually applied the proxy. Silent-direct is usually trust_env confusion or a mismatched http/https proxies dict: setting only {"http": proxy} leaves HTTPS traffic direct, which is most traffic.
• rotation FAIL with 1/4 unique exits → the gateway is serving sticky sessions and your config string is wrong. You were about to hammer one residential household's IP with your entire crawl budget.
• tier-claim FAIL → your "residential" traffic exits through a hosting ASN. Expect aggressive blocking regardless of politeness, because origin risk engines classify the ASN, not your intent.
• latency FAIL with high variance → the exit tier is contended; raise timeouts or move to a lower-latency tier for time-sensitive paths.

The preflight is also your canary

Run the same battery on a schedule — every 30 minutes during a long crawl, and once before every retry storm — and it doubles as an early-warning system. Provider-side incidents (a regional gateway degraded, a credentials pool exhausted, geo routing broken) show up in preflight telemetry before they show up as mysterious 403s in your crawler logs. That is the difference between "we noticed in 20 minutes" and "we noticed in three days."

Preflight the pool, not just the string

If you manage explicit exits rather than one rotating gateway, checking a single proxy URL tells you almost nothing — you need a population estimate. Sample K exits, run the battery on each concurrently, and gate the crawl on pool-level thresholds instead of per-exit booleans:

import asyncio
import aiohttp

async def sample_pool(exits, k=25, min_pass=0.8):
    picked = random.sample(list(exits), min(k, len(exits)))
    async with aiohttp.ClientSession() as s:
        async def one(px):
            try:
                async with s.get(ECHO_IP, proxy=px, timeout=10) as r:
                    return r.status == 200, (await r.json()).get("ip")
            except Exception:
                return False, None
        results = await asyncio.gather(*(one(px) for px in picked))
    ok = [ip for good, ip in results if good]
    rate = len(ok) / len(picked)
    print(f"pool preflight: {len(ok)}/{len(picked)} exits alive ({rate:.0%}), "
          f"{len(set(ok))} unique egress IPs")
    return rate >= min_pass and len(set(ok)) >= 2


asyncio.run(sample_pool(EXITS))

Two thresholds do the work. An alive rate catches gateway-side incidents (a region is down; expect a bimodal pass/fail per region, which is why you group results by gateway host before alerting). A unique-egress count catches the nastier silent failure: 25 exits that all resolve to the same residential IP. Your pool looks large, your rotation config looks correct, and you are about to send ten thousand requests through one household's connection. The len(set(ok)) >= 2 assertion costs nothing and has saved more than one launch.

Make the preflight a scheduled job, not a ritual

A check you run manually is a check you skip on the day it matters. Three cheap integrations turn it into real coverage:

1. Job entrypoint. sys.exit(2) on failure, before any crawl code loads. The scheduler treats it as a failed task and your alerting fires.
2. Cron. Run it every 30 minutes and emit the six pass/fail flags as metrics. When someone asks "were proxies slow Tuesday at 3?" you have a time series instead of a memory.
3. CI. Pin one proxy string per test tier and run the battery before the integration tests. A gateway outage that would otherwise surface as 40 flaky scraper tests instead surfaces as one red preflight job with a clear name.

The scheduling version also unlocks something the startup version cannot: drain-before-fail. If the periodic preflight starts failing, stop leasing new exits, let in-flight requests finish, and pause the queue. You degrade the crawl instead of corrupting the dataset with a 40-minute window of silently empty pages.

Where the preflight does not help

Be honest about the limits, because they shape how much you trust the gate. The battery cannot tell you the origin is temporarily blocking your exits (all checks pass against ipify and ipinfo, both indifferent to your traffic, while your target has blacklisted a whole ASN range). For that you need an in-crawl signal: a small canary set of known-live URLs sampled continuously, with block-rate alarms keyed on them. The preflight answers "is my plumbing up"; the canary answers "does the destination like us." You want both, and confusing them is how teams spend an afternoon debugging a site-side incident with a proxy-side tool.

Six checks, sixty seconds, and a hard gate at startup. Write the preflight once, paste it into every scraper, and the beginner debugging loop — hours of guessing whether the site or the proxy broke — mostly evaporates.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele