Not Every Block Comes From the Website: A Beginner's Taxonomy of Proxy-Layer Errors in Python

When a request fails in a proxy-based scraper, the error message rarely tells you where the failure happened. requests.exceptions.ConnectionError, a bare 502, a ReadTimeout — each of these can originate in at least three different places: your worker, the proxy gateway, the exit node, or the target website. Beginners almost universally commit the same diagnostic error: they blame the last hop. "The site blocked me" is the default explanation for everything, which leads to precisely the wrong fixes — rotating harder when the real problem is authentication, or abandoning a provider when the real problem is their own timeout budget.

This article gives you a working taxonomy: four layers, how to tell them apart from inside Python, and what each layer's failure actually obligates you to do. It's the debugging flowchart I wish I'd had in my first month of crawling through residential proxies.

The four layers

Every proxied request traverses the same path:

your worker -> proxy gateway -> exit node -> target website
   (layer 1)     (layer 2)      (layer 3)     (layer 4)

Failures at each layer have distinct signatures once you know what to look at. The two signals that separate them best are how fast the failure happens and what raised it.

Layer 1 — your worker. Local resource exhaustion, event-loop stalls, DNS resolver issues on the worker itself, TLS library misconfiguration. Signature: failures are fast, exceptions are raised before any network round trip (e.g., InvalidURL, OSError: too many open files), and they correlate with your process state, not the target.

Layer 2 — the proxy gateway. The gateway is the entry server you address directly (gateway.provider.com:port). It handles authentication, plan quota, and routing you to an exit. Failures here: 407 Proxy Authentication Required, quota-exceeded responses, instant connection resets (TCP RST within milliseconds of connect), and gateway-side 429s (the provider throttling you, not the target). Signature: fast failures — no exit was ever contacted, so nothing slow can have happened — and error text that mentions proxy/auth.

Layer 3 — the exit node and its path to the origin. The gateway picked an exit, but the exit itself is unhealthy: unreachable, bandwidth-starved, or its route to the target is broken. Failures here: ProxyError with upstream-connection wording, ConnectionResetError mid-request, ReadTimeout (the exit accepted your bytes but the answer never came back). Signature: slow failures — you paid for a round trip or most of one — and they cluster on specific exits rather than uniformly.

Layer 4 — the target website. You got a real HTTP response, generated by the origin. 403s, 429s, 200-with-CAPTCHA, 200-with-empty-shell-page. Signature: a response object exists. This is the only layer where "we got blocked" is a correct diagnosis.

The rule of thumb: no response object means layers 1–3; a response object means layer 4. Before anything else, check which side of that line you're on.

A classifier you can drop into your code

import time

import requests
from requests.exceptions import (
    ConnectionError, ProxyError, ReadTimeout, ConnectTimeout, SSLError,
)

FAST_FAIL_MS = 400  # gateway refusals are near-instant; tune to your provider


def classify(exc: Exception, elapsed_ms: float) -> str:
    """Map one failed attempt to a layer label."""
    if isinstance(exc, ProxyError):
        # ProxyError wraps the chain worker->gateway->exit. Split by speed:
        return "L2 gateway (auth/quota)" if elapsed_ms < FAST_FAIL_MS \
            else "L3 exit path"
    if isinstance(exc, ConnectTimeout):
        return "L2/L3 connect phase (gateway or exit unreachable)"
    if isinstance(exc, ReadTimeout):
        return "L3 exit path (connected, no answer)"
    if isinstance(exc, SSLError):
        return "L3 exit TLS (exit cannot complete handshake to target)"
    if isinstance(exc, ConnectionError):
        return "L1 worker" if elapsed_ms < 50 else "L3 exit path (reset)"
    return "L1 worker (local)"


def fetch_with_taxonomy(url: str, proxy: str) -> dict:
    proxies = {"http": proxy, "https": proxy}
    t0 = time.perf_counter()
    try:
        r = requests.get(url, proxies=proxies, timeout=(5, 20))
        elapsed = (time.perf_counter() - t0) * 1000
        if r.status_code in (401, 407):
            layer = "L2 gateway (auth)"
        elif r.status_code == 200:
            layer = "L4 origin OK"
        else:
            layer = "L4 origin response"
        return {"layer": layer, "status": r.status_code,
                "elapsed_ms": elapsed, "ok": r.status_code == 200}
    except requests.RequestException as e:
        elapsed = (time.perf_counter() - t0) * 1000
        return {"layer": classify(e, elapsed), "status": None,
                "elapsed_ms": elapsed, "ok": False}

Run a batch of URLs through fetch_with_taxonomy and tally the layer values. The distribution is the diagnosis:

• Predominantly L2: your credentials, plan, or gateway endpoint are the problem. No amount of exit rotation fixes a 407. Check auth string formatting (special characters in passwords need URL-encoding — a classic silent killer), quota dashboards, and whether you're hitting the right gateway hostname/port for the product you bought (rotating vs sticky gateways are often different endpoints).
• Predominantly L3 with ReadTimeout: exits are being assigned but can't reach the target. This can be provider pool quality, but before blaming them, try the same request from a direct (non-proxied) session — if direct also times out, the problem is between your region and the target, or the target is down.
• Predominantly L3 with SSLError: often a sign you're pointing an HTTPS-targeting request at a gateway/port combination that mishandles CONNECT tunneling, or the provider is MITM-ing TLS in a way your client rejects (some transparent setups do).
• L4 everything, lots of 403/429: now you have an anti-bot problem, and fingerprint/pacing work is the right investment.

The two controls that make the diagnosis honest

Two habits turn this taxonomy from a nice diagram into an actual debugging method.

Control 1: the canary. Always keep one request in your loop that goes to a target you control or a neutral echo endpoint, through the same proxy. If the canary fails at L2, your failures are not about the target at all. If the canary passes while real targets fail at L4, you've cleanly isolated origin-side blocking. The canary costs one cheap request per cycle and eliminates half of all misdiagnoses.

Control 2: the paired direct request. For any mysterious L3/L4 result, immediately retry the same URL from a direct session (trust_env=False, no proxies) on the same machine. Four outcomes: both fail (target or network), direct works + proxy fails at L2/L3 (provider path), direct works + proxy fails at L4 (the exit's IP specifically is burned — rotate and move on), both work (transient; log and continue). Every one of these four outcomes implies a different next action, and only the paired request distinguishes them.

One caution on ethics and restraint: this taxonomy exists to fix your infrastructure faster, not to bulldoze a target that's clearly refusing you. A consistent L4 pattern of 403s with a valid canary means the site has decided it doesn't want automated traffic at this shape — the right responses are lowering your rate, improving fingerprint coherence, or finding another data source, not escalating rotation. Debugging skill and scraping restraint are the same skill at different magnifications.

From taxonomy to telemetry

Once classify is in your code, don't print it — export it. A layer label per attempt, aggregated per provider and per target, turns your error logs into a monitoring dashboard: alert on L2 rate above 2% (billing/auth problems), L3 above 10% on a provider (pool degradation — time to re-benchmark or rebalance), and any sustained L4 pattern (fingerprint or pacing debt). Layer-level error budgets are to proxy crawling what SLOs are to web services: they convert a fog of "it's failing sometimes" into a small number of actionable signals.

The next time a batch dies, don't ask "which site blocked me." Ask "which layer failed me" — and you'll usually find the website was innocent.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele