Proxy Rotation for Scrapers: Stickiness, Rate Limits, and Backoff That Works

A scraper with a thousand exits still gets blocked in its first hundred requests if the traffic leaving those exits has the wrong shape. Blocks are rarely caused by one bad address. They come from the pattern around it: how fast you fire, how long you stay, and what you do when a target pushes back. Most teams treat rotation as one toggle; it is three, and rotating harder only touches the first.

Three independent knobs

• Granularity — how often the exit changes: every request, every session, or pinned to a target.
• Rate — requests per second, counted per exit and in aggregate.
• Backoff — what the client does when the target signals "slow down".

If the rate is wrong, a larger pool is not a fix. It spreads the same abuse across more addresses and gets them flagged as a group — worse, because replacements then inherit the reputation of the range.

Granularity: choose by task, not by preference

Granularity	Exit changes	Fits	Breaks when
Per request	Every call	Stateless pages, price feeds, parallelising one query	The target binds a session to navigation; login or CSRF flows
Sticky per session	After N requests or T minutes	Logged-in browsing, pagination, multi-step forms	TTL too long → one exit carries all volume; too short → bot-like
Pinned per target	Never, until retired	Rate-sensitive targets, account-bound tasks	The pinned exit is silently shared with another worker

If the target issues a session cookie, honour it. A real visitor does not change country between page two and page three. What matters is not your preference but which granularity the target's session semantics allow: authentication flows need a coherent exit, anonymous page fetches do not. Account-bound tasks want one fixed address per identity; bulk collection wants a wide rotating range.

Rate: the number that actually gets you blocked

Two ceilings matter, and they are not the same:

1. Per-exit ceiling — what a real line plausibly does. Keep concurrency per exit at 1–4 and pace requests to a few per second at most.
2. Aggregate ceiling — what the site tolerates from its whole visitor population. This is what triggers a range ban, and rotation does not hide it.

If the pool holds N exits at b requests per second each, your crawl rate is at most N × b. That expression answers "tonight or next week", so pool sizing is arithmetic, not a purchase decision.

Backoff: read the signal before retrying

• 429 or 503 with Retry-After — obey the header literally. Ignoring it escalates a throttle into a ban.
• 403 or 444 on one exit while others succeed — retire that exit only. Tearing down the whole run because one address is spent is how an hour becomes three days.
• Soft blocks — a 200 carrying a challenge page, an empty result set, or a truncated body. These are the expensive ones, because a naive client logs them as success and reports clean data.

Add jitter, and back off per exit rather than globally, so retries do not synchronise into a burst against a single address.

A pool that respects all three

import itertools, random, time, urllib.error, urllib.request


class Exit:
    def __init__(self, url, rps=1.0, max_fails=5):
        self.url, self.rps, self.max_fails = url, rps, max_fails
        self.next_at, self.fails = time.monotonic(), 0

    def take(self):
        """Token bucket: one slot per request, refilled at `rps`."""
        now = time.monotonic()
        if self.next_at > now:
            time.sleep(self.next_at - now)
        self.next_at = max(time.monotonic(), self.next_at) + 1.0 / self.rps

    @property
    def usable(self):
        return self.fails < self.max_fails


class SoftBlock(Exception):
    """200 OK whose body is a challenge page, not data."""


class Pool:
    def __init__(self, exits, sticky_ttl=300):
        self.exits = [Exit(u) for u in exits]
        self.ring = itertools.cycle(self.exits)
        self.sticky_ttl, self.sessions = sticky_ttl, {}

    def pick(self, key=None):
        if key and key in self.sessions:
            ex, expires = self.sessions[key]
            if expires > time.monotonic() and ex.usable:
                return ex
        for _ in range(len(self.exits) * 2):
            ex = next(self.ring)
            if ex.usable:
                if key:
                    self.sessions[key] = (ex, time.monotonic() + self.sticky_ttl)
                return ex
        raise RuntimeError("pool exhausted: every exit is retired")


def fetch(url, pool, key=None, attempts=4, min_bytes=500):
    for n in range(attempts):
        exit_ = pool.pick(key)
        exit_.take()
        opener = urllib.request.build_opener(urllib.request.ProxyHandler(
            {"http": exit_.url, "https": exit_.url}))
        try:
            with opener.open(url, timeout=20) as resp:
                body = resp.read()
            if len(body) < min_bytes:          # 200 with no payload
                raise SoftBlock("%d bytes from %s" % (len(body), exit_.url))
            exit_.fails = 0
            return body
        except (urllib.error.HTTPError, SoftBlock, OSError) as exc:
            if getattr(exc, "code", None) in (403, 429, 444) or isinstance(exc, SoftBlock):
                exit_.fails += 1               # retire the exit, keep the run
            time.sleep(min(2 ** n, 30) * random.uniform(0.5, 1.5))
    raise RuntimeError("gave up after %d attempts" % attempts)

Two details carry the value. pick() reuses a live sticky binding but still validates usability, so a retired address is never handed back. And the failure handler increments a counter instead of aborting — that is what turns a pool into a self-healing resource rather than a list you babysit.

How to tell you are being blocked

Track these per exit, not in aggregate:

• Success rate — retire anything well below the pool median, not just at zero.
• Median latency — a sharp drop usually means a cached challenge page.
• Body-size distribution — real pages cluster; block pages do not.

Common mistakes

Rotating every request against a session-bound target. Aggressive rotation is itself a signal.

Treating 403 as a network error. Retrying the same exit against the same rejection turns a soft block permanent.

FAQ

How many proxies do I need?
Set the target rate first, then divide by the per-exit budget you can defend. A pool is a rate budget, not a quantity.

Per request or per session?
Follow the target's session semantics. Cookies and coherent navigation mean sticky sessions with a TTL; stateless pages can rotate freely.

---

The fuller Chinese guide behind this piece — matching pool types to scenarios, from account warm-up to bulk collection, and pairing exits with fingerprint isolation: IP environment for automated workflows. To measure pool health before rotating blindly, see the batch proxy check.