Your scraper isn't blocked — your proxy is. Stop treating every failure the same way

Here's a retry loop. You've written this loop. I've written this loop.

for attempt in range(5):
    try:
        r = requests.get(url, proxies=proxies, timeout=30)
        if r.status_code == 200:
            return r.text
    except Exception:
        pass
    rotate_proxy()          # <- the bug
    time.sleep(2 ** attempt)

It looks defensive. It's actually doing something worse than nothing: it's collapsing seven genuinely different failures into one response. Rotate and retry. Rotate and retry.

Which means when your success rate drops, you have no idea whether you're being blocked, being rate-limited, being fingerprinted, or whether your proxy credentials expired forty minutes ago and every single one of those retries was doomed before it left your machine.

Let's separate them.

The seven failures, and what each one actually means

1. Can't reach the proxy at all. Connection refused, DNS failure, connect timeout to the proxy host. Your request never reached the target. Rotating your exit identity does nothing — you didn't get far enough to have one.
→ Retry the same target through a different upstream. Don't burn a rotation, don't back off aggressively.

2. 407 Proxy Authentication Required. Your credentials are wrong, expired, out of balance, or — the sneaky one — you asked for a feature on the wrong port. Requesting city-level targeting against a country-level endpoint fails here, and it looks exactly like an auth failure because it is one.
→ Never retry. This is a config bug and retrying just burns rate limit while you stay broken. Fail loud, page someone.

3. The CONNECT tunnel fails, or the proxy returns 502/503. You reached the proxy, the proxy couldn't reach the target, or the exit died mid-handshake. This is upstream, not you.
→ New identity, retry. If this class dominates your histogram, it's a provider problem — go read their status page, or change providers.

4. TLS handshake reset / EOF immediately after CONNECT. The tunnel opened and then the target dropped you at the TLS layer. This is usually fingerprinting — your client's TLS signature doesn't look like the browser it claims to be in the User-Agent.
→ Rotating IPs will not fix this. Ever. You need a different HTTP client. Every rotation you spend here is wasted money.

5. 403 / 401 from the target. You got a real HTTP response, and it says no. Identity-level block.
→ New identity + backoff. This is the one case where "rotate" is genuinely the right instinct.

6. 429 Too Many Requests. You're going too fast. Note that this is a statement about your behaviour, not your IP.
→ Back off first, rotate second — and honour Retry-After. Rotating immediately on a 429 is the single most common mistake in this list: you keep the request rate identical, just spread across more addresses, and now you're teaching the target that a whole range is misbehaving instead of one address.

7. 200 OK with an interstitial, a challenge page, or a suspiciously small body. The most expensive one, because your loop counts it as a success and writes garbage to your database.
→ Content check required. The status code lies.

Notice that the correct response differs in all seven cases, and that in two of them (2 and 4) retrying at all is actively harmful.

The classifier — Python

Zero dependencies beyond requests. Drop it in.

from enum import Enum
import requests
from requests.exceptions import (
    ProxyError, ConnectTimeout, ConnectionError as ReqConnectionError,
    ReadTimeout, SSLError,
)


class Failure(str, Enum):
    PROXY_UNREACHABLE = "proxy_unreachable"   # 1
    PROXY_AUTH        = "proxy_auth"          # 2
    UPSTREAM_DEAD     = "upstream_dead"       # 3
    TLS_REJECTED      = "tls_rejected"        # 4
    TARGET_BLOCKED    = "target_blocked"      # 5
    RATE_LIMITED      = "rate_limited"        # 6
    SOFT_BLOCK        = "soft_block"          # 7
    OK                = "ok"


# action: retry? rotate identity? back off? or stop and shout?
POLICY = {
    Failure.PROXY_UNREACHABLE: dict(retry=True,  rotate=False, backoff=1,  fatal=False),
    Failure.PROXY_AUTH:        dict(retry=False, rotate=False, backoff=0,  fatal=True),
    Failure.UPSTREAM_DEAD:     dict(retry=True,  rotate=True,  backoff=2,  fatal=False),
    Failure.TLS_REJECTED:      dict(retry=False, rotate=False, backoff=0,  fatal=True),
    Failure.TARGET_BLOCKED:    dict(retry=True,  rotate=True,  backoff=5,  fatal=False),
    Failure.RATE_LIMITED:      dict(retry=True,  rotate=False, backoff=30, fatal=False),
    Failure.SOFT_BLOCK:        dict(retry=True,  rotate=True,  backoff=10, fatal=False),
}

# tune these to your targets — they are the part you own
BLOCK_MARKERS = ("just a moment", "verify you are human", "access denied",
                 "unusual traffic", "enable javascript and cookies")
MIN_BODY_BYTES = 512


def classify_response(r: requests.Response) -> Failure:
    if r.status_code == 407:
        return Failure.PROXY_AUTH
    if r.status_code == 429:
        return Failure.RATE_LIMITED
    if r.status_code in (401, 403):
        return Failure.TARGET_BLOCKED
    if r.status_code in (502, 503, 504):
        return Failure.UPSTREAM_DEAD
    if r.status_code == 200:
        body = r.text
        low = body[:4000].lower()
        if len(body) < MIN_BODY_BYTES or any(m in low for m in BLOCK_MARKERS):
            return Failure.SOFT_BLOCK
        return Failure.OK
    return Failure.TARGET_BLOCKED


def classify_exception(e: Exception) -> Failure:
    msg = str(e).lower()

    # order matters: ProxyError subclasses ConnectionError in requests
    if isinstance(e, ProxyError):
        if "407" in msg or "authentication" in msg:
            return Failure.PROXY_AUTH
        # "tunnel connection failed" = proxy reached, target didn't answer.
        # everything else wrapped in ProxyError (incl. the generic
        # "cannot connect to proxy") means we never got there at all.
        if "tunnel connection failed" in msg:
            return Failure.UPSTREAM_DEAD
        return Failure.PROXY_UNREACHABLE
    if isinstance(e, SSLError):
        return Failure.TLS_REJECTED
    if isinstance(e, (ConnectTimeout, ReqConnectionError)):
        if "reset by peer" in msg or "eof occurred" in msg:
            return Failure.TLS_REJECTED
        return Failure.PROXY_UNREACHABLE
    if isinstance(e, ReadTimeout):
        return Failure.UPSTREAM_DEAD
    return Failure.PROXY_UNREACHABLE

Two things worth calling out, because they're where hand-rolled versions get it wrong:

• ProxyError subclasses ConnectionError in requests. Check it first or every proxy failure lands in the wrong bucket.
• ReadTimeout is not ConnectTimeout. Connect timeout means you never reached the proxy; read timeout means the tunnel opened and the far end went quiet. Different problems, different fixes.

Wiring it into a loop that respects the policy

import time
from proxy_builder_sdk import AethynClient

client = AethynClient()   # reads AETHYN_USERNAME / AETHYN_PASSWORD

def fetch(url: str, country: str = "us", max_attempts: int = 5):
    stats, run = {}, 0

    for attempt in range(max_attempts):
        # a new session id == a new exit identity; reuse it to keep one
        p = client.session(country=country, session=f"job-{run}", ttl=10)

        try:
            r = requests.get(url, proxies=p.proxies, timeout=(10, 30))
            outcome = classify_response(r)
        except Exception as e:
            outcome = classify_exception(e)

        stats[outcome] = stats.get(outcome, 0) + 1

        if outcome is Failure.OK:
            return r.text, stats

        policy = POLICY[outcome]
        if policy["fatal"]:
            # 407 or a TLS rejection: retrying cannot help. Say so, loudly.
            raise RuntimeError(f"unrecoverable: {outcome.value} via {p}")   # str(p) redacts the password
        if not policy["retry"]:
            break
        if policy["rotate"]:
            run += 1                       # next loop builds a fresh identity
        if policy["backoff"]:
            time.sleep(policy["backoff"] * (attempt + 1))

    raise RuntimeError(f"exhausted after {max_attempts}: {stats}")

str(p) is safe to log — the SDK redacts the password. Please do not be the person who pipes proxy credentials into Sentry.

The classifier — Node

Same taxonomy, undici/got error codes instead of exception classes.

import got, { HTTPError, RequestError } from "got";
import { AethynClient } from "proxy-builder-sdk";

const Failure = {
  PROXY_UNREACHABLE: "proxy_unreachable",
  PROXY_AUTH:        "proxy_auth",
  UPSTREAM_DEAD:     "upstream_dead",
  TLS_REJECTED:      "tls_rejected",
  TARGET_BLOCKED:    "target_blocked",
  RATE_LIMITED:      "rate_limited",
  SOFT_BLOCK:        "soft_block",
  OK:                "ok",
};

const BLOCK_MARKERS = ["just a moment", "verify you are human", "access denied"];
const MIN_BODY_BYTES = 512;

function classify(err, res) {
  if (res) {
    const s = res.statusCode;
    if (s === 407) return Failure.PROXY_AUTH;
    if (s === 429) return Failure.RATE_LIMITED;
    if (s === 401 || s === 403) return Failure.TARGET_BLOCKED;
    if ([502, 503, 504].includes(s)) return Failure.UPSTREAM_DEAD;
    if (s === 200) {
      const body = res.body ?? "";
      const low = body.slice(0, 4000).toLowerCase();
      return body.length < MIN_BODY_BYTES || BLOCK_MARKERS.some((m) => low.includes(m))
        ? Failure.SOFT_BLOCK
        : Failure.OK;
    }
    return Failure.TARGET_BLOCKED;
  }

  if (err instanceof HTTPError) return classify(null, err.response);

  const code = err?.code ?? "";
  const msg  = (err?.message ?? "").toLowerCase();

  if (msg.includes("407") || msg.includes("proxy authentication")) return Failure.PROXY_AUTH;
  if (code === "ECONNRESET" || code.startsWith("ERR_TLS") || code === "EPROTO") return Failure.TLS_REJECTED;
  if (code === "ECONNREFUSED" || code === "ENOTFOUND" || code === "ETIMEDOUT") return Failure.PROXY_UNREACHABLE;
  if (err instanceof RequestError && msg.includes("tunnel")) return Failure.UPSTREAM_DEAD;
  return Failure.PROXY_UNREACHABLE;
}

const client = new AethynClient();

async function fetchOnce(url, run = 0) {
  const p = client.session({ country: "us", session: `job-${run}`, ttl: 10 });
  try {
    const res = await got(url, { proxy: p.url, throwHttpErrors: false, timeout: { request: 30_000 } });
    return { outcome: classify(null, res), res };
  } catch (err) {
    return { outcome: classify(err), res: null };
  }
}

The payoff: one histogram tells you whose fault it is

This is the reason to bother. Count outcomes per run and print the distribution:

proxy_unreachable   2
upstream_dead      41
target_blocked      6
rate_limited        1
soft_block          3
ok                847

Now the diagnosis is mechanical:

Dominant class	Verdict	What to do
proxy_unreachable, upstream_dead	Provider problem. Your requests are dying before the target sees them.	Check their status. Escalate. Switch if it persists.
proxy_auth	Your config. Nothing else.	Check credentials, balance, and that your targeting features match your plan/port.
tls_rejected	Your HTTP client.	Change the client, not the proxies. No amount of rotation fixes a fingerprint.
target_blocked, soft_block	Your identity strategy.	Rotate more, slow down, use a real browser for JS-heavy targets.
rate_limited	Your request rate.	Back off. Honour Retry-After. Rotating is not a rate limit fix.

Before I had this, "the scrape is failing" was a two-hour investigation. Now it's a glance. Roughly the first three times I ran it in anger, the answer was a category I'd have bet against.

Where a proxy provider actually fits

The classifier is provider-neutral and I'd rather you kept it that way — it's the instrument, and an instrument that can't return a bad verdict about your vendor is worthless. But three properties genuinely reduce the classes it has to catch, and they're what we built Aethyn around:

• Identity is a string, not an API call. A new session id is a new exit; reuse it and you keep one for the lifetime window (1–1440 min). Rotation costs you nothing and needs no round trip. client.session({ country: "us", session: "job-7", ttl: 10 }).
• Failures are separable by design. Proxy-layer errors and target-layer responses arrive distinctly, which is what makes classes 1–3 distinguishable from 5–7 at all.
• Published per-GB pricing. A retry storm has a cost and you should be able to see it. Ours is on the pricing page — no sales call to find out.

The proxy-builder-sdk used above is MIT-licensed and on npm and PyPI. It also speaks other providers' username formats, so you can run this exact code against whatever you're using today and compare histograms. That comparison is a much better provider evaluation than anybody's marketing page, ours included.

If you want to try it against our network, the trial needs no card.

The obligatory, non-optional caveat

This is tooling for collecting public data: respect robots.txt, respect rate limits, read the terms. Class 6 exists to tell you that you're being rude, and the correct response to it is to slow down, not to find a cleverer way to keep the same pace. And there's no CAPTCHA-solving in any of the above — when a real challenge fires, the honest move is to log it and stop.

Classify your failures. Half of them aren't what you think they are.