Politeness Is Measured at the Origin: A Global Budget That Doesn't Grow With Your Proxy Pool

Somewhere in every scraping codebase is a constant like MAX_RPS = 5, and next to it, an implicit belief: if I stay under 5 requests per second, I'm being polite.

Behind a rotating proxy pool, that belief is broken in a specific and dangerous way. If each of your 20 exits draws 5 requests per second against the same origin, you're sending 100 requests per second — and every one of your per-exit limiters reports green the whole time. You didn't become polite when you bought more IPs. You just stopped being caught by the per-IP controls. The origin's actual constraint — its capacity, its caches, its on-call engineers — never changed.

The position I want to argue for: politeness is a property of the relationship between your crawl and the origin server, not between your crawl and any single address. Your politeness budget should be a global, per-origin number that does not scale with your pool size. Here's how to build that, and why it's also the self-interested move.

Why per-IP politeness is an illusion

Rate limits at the edge (CDN WAFs, per-IP token buckets) exist to stop any one client from hogging a shared resource. They are an enforcement mechanism, not a definition of fair use. Consider what 100 req/s across 20 IPs looks like from the origin:

• If the origin serves dynamic content, that's 100 req/s of compute, cache lookups, and database load. A modest origin box is now spending a double-digit percentage of its capacity on you.
• If it's behind autoscaling, you're the reason a p95 latency graph spiked at 3 a.m. and a page fired.
• If it's a small business site on a single VPS, you might be most of its traffic, spread across addresses that make it unblockable by IP.

"The site can't throttle me effectively" and "I'm being respectful" are different claims. Proxy rotation gives you the first. Only a global budget gives you the second.

And the self-interested case: origins defend themselves at layers above the IP. WAF rules escalate to ASN-level challenges, geo blocks, and fingerprint heuristics when per-IP limits stop working. A crawl that respects the origin's capacity never triggers the escalation ladder. The polite crawl and the long-lived crawl are usually the same crawl.

Architecture: a global origin budget with per-exit dispatch

The design separates two concerns that per-worker limiters mash together:

1. The origin budget — a single global rate per target origin (I set mine from the origin's own signals: response latency, 503s, Retry-After, and challenge rate — not from vibes).
2. Exit dispatch — which address the next request uses, subject to per-exit constraints (cooldowns, stickiness, fingerprint affinity).

import asyncio
import time
from collections import defaultdict
from dataclasses import dataclass, field

@dataclass
class OriginState:
    budget_per_sec: float = 2.0        # GLOBAL: does not scale with pool
    min_budget: float = 0.25
    max_budget: float = 5.0
    distress: int = 0                  # consecutive distress signals
    tokens: float = 2.0
    last_refill: float = field(default_factory=time.monotonic)

class OriginBudget:
    """One token bucket per origin hostname, shared by every worker
    and every exit. This is the politeness contract."""

    def __init__(self):
        self.origins: dict[str, OriginState] = defaultdict(OriginState)

    def state(self, origin: str) -> OriginState:
        return self.origins[origin]

    async def acquire(self, origin: str):
        s = self.state(origin)
        while True:
            now = time.monotonic()
            s.tokens = min(s.budget_per_sec,
                           s.tokens + (now - s.last_refill) * s.budget_per_sec)
            s.last_refill = now
            if s.tokens >= 1.0:
                s.tokens -= 1.0
                return
            await asyncio.sleep((1.0 - s.tokens) / s.budget_per_sec)

    def report(self, origin: str, status: int, latency: float,
               retry_after: float | None = None):
        """Feed the origin's own signals back into the budget."""
        s = self.state(origin)
        distress = False
        if status in (503, 429) or retry_after:
            distress = True
        if latency > 5.0:                       # origin struggling to keep up
            distress = True
        if status == 200 and latency < 2.0:
            s.distress = 0
            # gentle recovery toward max, never instantly
            s.budget_per_sec = min(s.max_budget, s.budget_per_sec * 1.05)
            return
        if distress:
            s.distress += 1
            if s.distress >= 3:                 # sustained, not a blip
                factor = 0.5 if retry_after is None else 0.25
                s.budget_per_sec = max(s.min_budget, s.budget_per_sec * factor)
                s.distress = 0

class ExitDispatcher:
    """Chooses WHICH exit carries the next origin-budgeted request.
    Knows nothing about politeness — only about exits."""

    def __init__(self, exits):
        self.exits = list(exits)
        self.cooldown: dict[str, float] = {}

    def pick(self) -> str:
        now = time.monotonic()
        live = [e for e in self.exits
                if self.cooldown.get(e, 0) < now]
        if not live:
            raise RuntimeError("all exits cooling down")
        # rotate round-robin among live exits for even spread
        self.exits.append(self.exits.pop(0))
        return live[0]

    def cool(self, exit_id: str, seconds: float):
        self.cooldown[exit_id] = time.monotonic() + seconds

And the crawler loop that keeps the layers honest:

import aiohttp
from urllib.parse import urlparse

async def polite_fetch(session, url, budget: OriginBudget,
                       disp: ExitDispatcher, proxy_template):
    origin = urlparse(url).netloc
    await budget.acquire(origin)                  # politeness gate (global)
    exit_id = disp.pick()                         # routing decision (local)
    proxy = proxy_template.format(session=exit_id)
    t0 = time.monotonic()
    try:
        async with session.get(url, proxy=proxy,
                               timeout=aiohttp.ClientTimeout(total=30)) as r:
            latency = time.monotonic() - t0
            ra = r.headers.get("Retry-After")
            budget.report(origin, r.status, latency,
                          float(ra) if ra and ra.isdigit() else None)
            if r.status in (403, 429):
                disp.cool(exit_id, 60)            # exit-level response
            return r.status, await r.text()
    except aiohttp.ClientError:
        disp.cool(exit_id, 300)
        return 0, ""

if __name__ == "__main__":
    async def main():
        budget = OriginBudget()
        disp = ExitDispatcher([f"s{i:03d}" for i in range(12)])
        template = "http://user-session-{session}:[e-mail]:8080"
        async with aiohttp.ClientSession() as session:
            for i in range(30):
                status, _ = await polite_fetch(
                    session, f"https://httpbin.org/delay/1?i={i}",
                    budget, disp, template)
                print(i, status,
                      f"origin_budget={budget.state('httpbin.org').budget_per_sec:.2f}/s")
    asyncio.run(main())

The two-layer split is the whole trick. Watch the output: the origin budget stays fixed at ~2/s no matter how many exits exist, while individual exits still take their own cooldowns for exit-scoped problems. Distress at the origin (sustained 503s, slowing responses) shrinks the global budget. A 429 on one exit cools that exit only — and if 429s show up across many exits at once, that's an origin-level signal, and the origin budget will read it as such because every worker reports into the same bucket.

Sizing the budget without guessing

Three inputs beat guessing:

The sitemap and crawl-delay. robots.txt Crawl-delay is a direct statement of the origin's comfort level. Even if your use case doesn't bind you to it legally, a site that says Crawl-delay: 10 is telling you its infra budget. Start there and only relax if the origin signals health.

Baseline latency. Measure p50 response time at your target rate, then again at 2x. If p50 doubles when you double your rate, you've found the origin's knee — budget below it. A healthy origin barely notices a crawl; a strained one telegraphs it in latency long before 503s appear.

Off-peak scheduling. A global budget of 3 req/s at 4 a.m. local to the origin is a different act than 3 req/s during their business peak, even though the number is identical. Put the budget in wall-clock terms when the origin has an obvious diurnal pattern.

The uncomfortable summary

Proxy pools solved the enforcement problem — per-IP limits can't stop a distributed crawl. They did not solve, and cannot solve, the fairness problem. When your pool grows tenfold and your crawl gets ten times faster against the same origin, something changed in the world, and it isn't the origin's capacity.

The engineers I respect most in this field treat the origin budget as a value they defend in code review: adding exits is a scaling decision, and scaling decisions shouldn't silently change how hard you hit someone else's server. Keep politeness at the origin, keep routing at the exit, and the two knobs will stop fighting each other.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele