Put the Retry Logic in the SDK, Not in Every Script: Backoff That Understands Exit-Level State

Somewhere in every scraper codebase there is a function decorated with three layers of @retry, wrapped in a while True with an except Exception: pass, called from a supervisor that also has a retry loop. Each layer was added by a reasonable person solving a real outage. Together they form a machine that retries failed requests nine times at triple concurrency, through the same burned exit IP, at the exact moment the target site is asking it to stop.

The problem is not that retries are bad. The problem is where the retry policy lives. When retry logic is written per-script, it can only see local information — this call, this exception — while the facts that should govern the retry (which exit received the 429, how burned it is, how many workers are simultaneously retrying) live in the proxy layer. The fix is structural: retries belong in the SDK, next to the exit state, where backoff can be computed from the thing that was actually penalized.

What per-script retries get wrong

Consider the canonical hand-rolled version:

async def fetch_with_retry(url, proxy, session):
    for attempt in range(5):
        try:
            async with session.get(url, proxy=proxy) as r:
                if r.status == 200:
                    return await r.text()
        except aiohttp.ClientError:
            pass
        await asyncio.sleep(2 ** attempt)
    raise GiveUp(url)

Four assumptions hide in those ten lines, and each one is wrong often enough to matter:

1. The penalty belongs to the URL. It doesn't. A 429 is the origin telling this exit IP to slow down. Retrying the same URL through the same exit re-offends immediately, with interest.
2. Exponential backoff on wall-clock is sufficient. Backoff delays your worker, but if ten workers each back off independently and then reconverge, the origin sees a synchronized burst. The jitter usually added is uniform and small — not enough when N is large.
3. All errors are equally retriable. A proxy connection refused, a 429, a 403, a DNS failure, and a TLS handshake reset have four different remediations. Treating them uniformly means the remediation is wrong three times out of five.
4. Retries are free. Through a metered proxy pool, every retry is billed traffic — and a retry of a doomed request is the most expensive byte you'll buy all day.

The design: backoff as a function of exit state

The redesign moves retry into the SDK and makes its inputs explicit: the outcome, the exit that received it, and the recent history of that exit at that origin. Here's a compact, runnable implementation:

# sdk_retry.py
from __future__ import annotations

import asyncio
import hashlib
import random
import time
from collections import defaultdict, deque
from dataclasses import dataclass
from enum import Enum
from typing import Awaitable, Callable, Optional


class Outcome(Enum):
    OK = "ok"
    RATE_LIMITED = "429"          # origin throttles the exit
    BLOCKED = "403"               # origin refuses the exit
    PROXY_ERROR = "proxy_error"   # infra between you and exit failed
    FATAL = "fatal"               # auth, parse, logic — never retriable


@dataclass
class ExitHealth:
    proxy_url: str
    events: deque = None          # (ts, Outcome) recent history
    blocked_since: float = 0.0

    def __post_init__(self):
        if self.events is None:
            self.events = deque(maxlen=64)

    @property
    def recent_block_rate(self) -> float:
        if not self.events:
            return 0.0
        now = time.monotonic()
        recent = [o for ts, o in self.events if now - ts < 120.0]
        if not recent:
            return 0.0
        blocked = sum(1 for o in recent if o in (Outcome.BLOCKED, Outcome.RATE_LIMITED))
        return blocked / len(recent)


class ExitRetryPolicy:
    """Decides delay + whether to re-enter through the same exit."""

    def __init__(self, max_attempts: int = 4):
        self.max_attempts = max_attempts

    def verdict(self, outcome: Outcome, health: ExitHealth,
                attempt: int) -> tuple[bool, float, bool]:
        """Returns (retry?, delay_s, switch_exit?)."""
        if outcome is Outcome.OK or outcome is Outcome.FATAL:
            return False, 0.0, False

        if outcome is Outcome.RATE_LIMITED:
            # Penalty scales with how hot this exit already is.
            heat = health.recent_block_rate          # 0.0 .. 1.0
            base = 2.0 ** attempt + heat * 30.0
            jitter = random.uniform(0, base * 0.5)   # full-spread jitter
            return attempt < self.max_attempts, base + jitter, True

        if outcome is Outcome.BLOCKED:
            # Burned for a while. Long cooldown, definitely switch.
            if not health.blocked_since:
                health.blocked_since = time.monotonic()
            cooldown = 300.0 + random.uniform(0, 60.0)
            return attempt < self.max_attempts, cooldown, True

        # PROXY_ERROR: infra flake — short delay, switching is cheap insurance
        return attempt < self.max_attempts, 0.5 * (attempt + 1) + random.random(), True


class RetryingFetcher:
    def __init__(self, gateway: str, policy: Optional[ExitRetryPolicy] = None):
        self.gateway = gateway
        self.policy = policy or ExitRetryPolicy()
        self._health: dict[str, ExitHealth] = defaultdict(
            lambda: ExitHealth("", None) if False else ExitHealth(gateway)
        )
        # simpler: one health record per logical exit token
        self._exits: dict[str, ExitHealth] = {}

    def _exit(self, token: str) -> ExitHealth:
        if token not in self._exits:
            user, rest = self.gateway.split("://", 1)[1].split("@", 1)
            u, _, p = user.partition(":")
            url = f"http://{u}-sess-{token}:{p}@{rest}"
            self._exits[token] = ExitHealth(url)
        return self._exits[token]

    async def fetch(self, url: str, do: Callable[[str], Awaitable[Outcome]],
                    session_token: str = "default") -> None:
        """`do(proxy_url)` performs the request and classifies its outcome."""
        attempt = 0
        while True:
            health = self._exit(session_token)
            outcome = await do(health.proxy_url)
            health.events.append((time.monotonic(), outcome))
            retry, delay, switch = self.policy.verdict(outcome, health, attempt)
            if not retry:
                if outcome is not Outcome.OK:
                    raise RuntimeError(f"gave up on {url} ({outcome.value})")
                return
            attempt += 1
            if switch:
                session_token = self._rotate_token(session_token)
            await asyncio.sleep(delay)

    @staticmethod
    def _rotate_token(token: str) -> str:
        return hashlib.sha1(f"{token}:{time.time_ns()}".encode()).hexdigest()[:10]

The key moves, compared with the naive loop:

Delay is a function of exit heat, not just attempt count. An exit with a 60% recent block rate backs off ~18 seconds harder than a cold one, because the origin's memory of that IP is exactly what the backoff is negotiating with. A cold exit that caught a stray 429 gets a short pause and a second chance through a different address.

Blocked exits get switched, not just delayed. The switch_exit verdict means a 403 exits the doomed pairing immediately; the delay covers the cooldown bookkeeping, and the next attempt rides a new exit. Delay-without-switch is how fleets generate thousands of guaranteed-dead requests.

Jitter is proportional, not token. uniform(0, base * 0.5) spreads reconvergence across half the backoff window. With fifty workers, uniform ±100ms jitter is a thundering herd wearing a disguise.

Fatal outcomes never retry. Auth failures, parse errors, and logic bugs return immediately — retrying those is how you turn a deploy mistake into an incident.

Classifying outcomes at the call site

The fetcher needs the caller only to classify, which is a few honest lines:

import aiohttp


async def classify(proxy_url: str, url: str = "https://example.com/prices") -> Outcome:
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout, trust_env=False) as s:
        try:
            async with s.get(url, proxy=proxy_url) as r:
                if r.status == 200:
                    return Outcome.OK
                if r.status == 429:
                    return Outcome.RATE_LIMITED
                if r.status == 403:
                    return Outcome.BLOCKED
                return Outcome.FATAL
        except (aiohttp.ClientProxyConnectionError, asyncio.TimeoutError):
            return Outcome.PROXY_ERROR
        except aiohttp.ClientHttpProxyError:
            return Outcome.FATAL          # e.g. 407: credential problem


fetcher = RetryingFetcher("http://user:[e-mail]:7000")
await fetcher.fetch("https://example.com/prices",
                    lambda p: classify(p, "https://example.com/prices"),
                    session_token="prices-worker-1")

The dividend: fleet-level math becomes sane

Once retry policy lives in one place, it becomes measurable as a unit. Track attempts per successful fetch and you get a number I'd argue belongs on your dashboard next to cost-per-row: the retry multiplier. A healthy pipeline sits between 1.05 and 1.2 — five to twenty percent of requests need a second try. When it drifts past 1.5, no individual script is misbehaving; your fleet as a whole is asking too hard, and the correct response is to lower global concurrency, not to tune any single backoff.

That's the deeper point. Retries are not a private concern of each script — they are a collective negotiation between your fleet and the origins' rate limiters, conducted through your exit pool. A negotiation needs one voice. Put the retry logic in the SDK, where the exit state lives, and the voice you send to the table finally speaks for the whole fleet.

---

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele