Your Scraper's Retry Logic Is Probably Making the Block Worse

Most scrapers have one retry policy: catch the exception, sleep, try again, give up after N attempts. It's the default in every tutorial, it's what tenacity gives you in three lines, and on a healthy target it's fine.

It stops being fine the moment the failures stop being random. A rate limit, a hard block, a TLS reset and a slow origin all arrive as "the request failed," and treating them identically means you spend your entire run budget hammering something that was never going to open — while the one failure that would have succeeded on retry gets three attempts and then gets dropped.

Here's the taxonomy I actually use, and the part that took the longest to learn: the counter matters more than the backoff.

The four failures, and what each one means

429 with a Retry-After header. The friendliest failure you will ever get. The server is telling you exactly what it wants. Honour it — sleep the stated duration, then continue on the same connection and the same identity. Do not rotate anything. Rotating here is how a temporary throttle turns into a fingerprinted pattern.

429 with no header. The server wants you slower but won't say by how much. Exponential backoff with jitter is genuinely the right tool. Start higher than you think — 5 seconds, not 0.5.

403 / 401 / a challenge page. This is not a retryable condition. Something about the request was rejected, and repeating it byte-for-byte will be rejected the same way. Every retry here is pure waste, and worse, it's the pattern that gets a durable block instead of a temporary one. Short-circuit straight to whatever your fallback is — different identity, different route, or shelve the URL and move on.

Connection reset / timeout / TLS handshake failure. This one is retryable, and it's the one people wrongly lump in with 403. Transport failures are frequently just noise. Retry immediately, twice, then treat it as a hard failure.

The distinction that matters: HTTP-level rejections tell you something about your request. Transport-level failures usually tell you nothing at all.

import random, time
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}
HARD_BLOCK = {401, 403, 407, 451}

def classify(exc_or_resp):
    if isinstance(exc_or_resp, httpx.HTTPError):
        return "transport"          # retry fast, twice
    code = exc_or_resp.status_code
    if code in HARD_BLOCK:
        return "blocked"            # do not retry, change something
    if code == 429:
        return "throttled"
    if code in RETRYABLE_STATUS:
        return "server"
    return "ok" if code < 400 else "fatal"

def sleep_for(kind, attempt, resp=None):
    if kind == "throttled" and resp is not None:
        ra = resp.headers.get("retry-after")
        if ra and ra.isdigit():
            return int(ra)
        return min(60, 5 * (2 ** attempt)) + random.uniform(0, 2)
    if kind == "server":
        return min(30, 2 ** attempt) + random.uniform(0, 1)
    if kind == "transport":
        return 0.5
    return 0

The counter is the part everyone skips

Backoff handles the failure you already got. It does nothing about the one you're about to get, and on a lot of targets the limit isn't time-based at all — it's a volume budget. You get N requests, and the block arrives on N+1 regardless of how politely you spaced them.

I learned this on a regional grocery chain's stock checker. It would 429 you politely for about an hour, and then flip to a permanent 403 once you crossed some invisible total. Every backoff strategy in the world is useless against that, because the thing being counted isn't your rate — it's your volume.

The fix is a rolling per-host counter that you enforce on yourself, set well below whatever number killed the last run:

import time
from collections import defaultdict, deque

class HostBudget:
    """Rolling request counter per host. Refuses before the server does."""

    def __init__(self, max_requests=200, window_seconds=3600):
        self.max = max_requests
        self.window = window_seconds
        self.hits = defaultdict(deque)

    def allow(self, host):
        now = time.time()
        q = self.hits[host]
        while q and now - q[0] > self.window:
            q.popleft()
        if len(q) >= self.max:
            return False
        q.append(now)
        return True

    def spent(self, host):
        return len(self.hits[host])

Once that exists, a 403 stops being a signal you react to and becomes a signal you record: whatever the counter said when the block landed is your new ceiling for that host, minus a healthy margin. Write it to disk. The next run starts already knowing.

The reason the 403 is a bad signal on its own is timing. By the time you see it, you're usually already benched — the block is applied, the cooldown clock started without telling you, and nothing you do in the next hour matters. The counter is the only thing that fires before the damage.

Putting it together

def fetch(client, url, budget, max_attempts=4):
    host = httpx.URL(url).host
    if not budget.allow(host):
        raise RuntimeError(f"self-imposed budget hit for {host} ({budget.spent(host)})")

    for attempt in range(max_attempts):
        try:
            resp = client.get(url, timeout=20)
        except httpx.HTTPError as exc:
            kind = classify(exc)
            if attempt >= 2:
                raise
            time.sleep(sleep_for(kind, attempt))
            continue

        kind = classify(resp)
        if kind == "ok":
            return resp
        if kind == "blocked":
            # record the ceiling, then hand off — retrying byte-for-byte is waste
            raise BlockedError(host, budget.spent(host))
        if kind == "fatal":
            resp.raise_for_status()
        time.sleep(sleep_for(kind, attempt, resp))

    raise RuntimeError(f"exhausted attempts for {url}")

Three rules, in order of how much run time they save:

1. Never retry a hard block unchanged. It cannot succeed and it makes the block stickier.
2. Count your own requests per host and stop before the server stops you.
3. Retry transport errors fast and cheaply. They're usually nothing.

The retry loop is the least glamorous part of a scraper and the one that quietly decides whether a run finishes. Worth twenty minutes.

---

We publish code examples and testing notes for developers who scrape and automate at RoamProxy. More runnable examples: github.com/roamproxy/proxy-examples.