Politeness at Scale: Coordinating Per-Domain Rate Limits Across Every Worker You Run

Twelve workers, each sleeping five seconds between requests. From any single machine's log, the crawl looked gentle. From the target's point of view, it was a synchronized hammer running 2.4x over budget. That postmortem is why I stopped believing in time.sleep() as a politeness strategy — and started treating politeness as a distributed-systems problem.

The incident that looked fine in every log

The setup: a price-monitoring crawl across three VMs, twelve worker processes, all configured with a 5-second delay to the target domain. One worker, one request every five seconds: 0.2 rps. Immaculate logs — request → sleep 5.0 → request → sleep 5.0, forever.

The aggregate math: 12 workers / 5 seconds = 2.4 requests per second against an origin whose Crawl-delay: 10 implied a budget of roughly 0.5–1 rps. We were 2–4x over, around the clock. Worse, because all twelve workers had been launched by the same deploy and tended to hit the same pages on the same schedule, the WAF didn't see a smooth 2.4 rps stream — it saw periodic bursts of near-simultaneous requests followed by silence. That burst pattern is exactly what bot detection flags, because humans don't arrive in phalanxes.

No single worker was misbehaving. Every individual process was polite. The fleet was not. Politeness is a property of the relationship between your entire fleet and the origin — and the origin doesn't know or care how you've sharded your work.

Why local sleeps structurally fail

You cannot fix this by tuning the sleep. The failure is structural, for three reasons:

1. No shared state. A sleep is a statement about one process's timeline, with zero visibility into eleven other timelines. The correct unit of pacing is "requests arriving at origin X from anyone," and a process-local timer cannot express that by construction.

2. Retry storms align workers. When the origin sheds load with a burst of 503s, every worker's backoff starts from the same event. If your backoff constants are similar (they usually are — everyone copies the same formula), the schedules converge and the retry wave hits as one synchronized pulse. Your politeness timer is now phase-locked to the overload you caused.

3. sleep() paces issuance, not arrival. The timer ends and you fire — but it knows nothing about the requests the other eleven workers have in flight at that instant. Six workers whose timers expire within the same 200ms window produce a 6-request burst regardless of how long each slept.

The fix has to move the pacing decision to a place every worker shares. For me that's Redis.

The design: a shared per-domain token bucket

One token bucket per domain, stored in Redis, keyed politeness:{domain}:

• Bucket state is a Redis hash: tokens (fractional) and ts (last refill timestamp, epoch seconds).
• Refill rate = 1 / max(robots.txt crawl_delay, configured_floor). For Crawl-delay: 10, that's 0.1 tokens/sec. My floor is 5 seconds — 0.2 tokens/sec — so unknown domains default to no faster than one request per 5 seconds fleet-wide.
• Capacity is 1. A bucket that holds ten tokens lets a fresh worker drain the whole budget instantly. Capacity 1 banks at most one request of credit; the fleet's steady-state rate is exactly the refill rate.
• Workers acquire a token before every request. If the bucket is empty, the script returns how long until the next token; the worker async-waits that long (plus jitter) and retries.
• Atomicity via Lua. Check-and-decrement must be one indivisible operation. GET → compute → SET from Python across twelve workers is a textbook lost-update race: all twelve read tokens = 1, all twelve decrement, all twelve fire. Redis executes a Lua script atomically — no other command interleaves — so the whole read-refill-decide-write sequence is one round trip and race-free.
• TTL of 86400 (24h), refreshed on every acquisition, so domains you stop crawling age out instead of accumulating forever.

One judgment call: the script takes "now" from the caller. Only elapsed time matters for refill, so modest clock skew is harmless; if you don't trust your fleet's clocks, use Redis's TIME command inside the script.

The build, part 1: the Lua bucket and the PolitenessGate

The Lua script — small enough to reason about line by line:

-- KEYS[1] = politeness:{domain}
-- ARGV[1] = capacity (1)
-- ARGV[2] = refill tokens/sec (e.g. 0.2 = one token per 5s)
-- ARGV[3] = now, epoch seconds (float)
-- ARGV[4] = ttl seconds
local key     = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local ttl      = tonumber(ARGV[4])

local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])
if tokens == nil then
  tokens = capacity   -- first acquisition: bucket starts full
  ts = now
end

-- refill for elapsed time, capped at capacity
local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * rate)

local wait = 0
if tokens >= 1 then
  tokens = tokens - 1
  ts = now
else
  -- time until the next full token exists
  wait = (1 - tokens) / rate
end

redis.call('HMSET', key, 'tokens', tokens, 'ts', ts)
redis.call('EXPIRE', key, ttl)
return wait

Note the fractional-credit handling: when the bucket is short of a token, we keep the fractional value and reset ts to now. Nothing is lost or double-counted — the fraction already includes all elapsed refill.

The Python wrapper, using redis.asyncio:

import asyncio
import random
import time

import redis.asyncio as redis

ACQUIRE_LUA = """<the script above>"""

class PolitenessUnavailable(RuntimeError):
    """Redis is unreachable — do NOT fall through and fetch anyway."""

class PolitenessGate:
    def __init__(self, redis_url="redis://localhost:6379/0",
                 default_interval=5.0, ttl=86400):
        self.r = redis.from_url(redis_url)
        self._acquire = self.r.register_script(ACQUIRE_LUA)
        self.default_interval = default_interval
        self.ttl = ttl
        # per-domain intervals, e.g. from robots.txt Crawl-delay
        self.intervals: dict[str, float] = {}

    def interval_for(self, domain: str) -> float:
        return self.intervals.get(domain, self.default_interval)

    async def acquire(self, domain: str) -> None:
        """Block until this worker may send ONE request to domain."""
        key = f"politeness:{canonical_domain(domain)}"
        rate = 1.0 / self.interval_for(domain)
        while True:
            try:
                wait = float(await self._acquire(
                    keys=[key],
                    args=[1, rate, time.time(), self.ttl],
                ))
            except redis.RedisError as e:
                raise PolitenessUnavailable(str(e)) from e
            if wait <= 0:
                return
            # jitter prevents waiters from re-converging into bursts
            await asyncio.sleep(wait + random.uniform(0.05, 0.25))

Usage in the crawl loop is one line: await gate.acquire(domain) before every fetch. Twelve workers, one shared bucket per domain, and the fleet's aggregate rate is capped at the refill rate no matter how many processes you add.

The build, part 2: canonicalization, the part everyone gets wrong

Two URLs can name the same rate budget while looking like different domains, and your bucket keys are only as good as your canonicalization.

The mechanical normalizations first: lowercase the host, strip a trailing dot (example.com. is legal DNS), and strip a leading www. — https://www.example.com/a and http://example.com/a share one politeness budget. Scheme is irrelevant; the origin doesn't budget by protocol.

from urllib.parse import urlsplit

def canonical_domain(host: str) -> str:
    h = host.strip().lower().rstrip(".")
    if h.startswith("www."):
        h = h[4:]
    return h

Then the trap: different hostnames, same origin. Behind a CDN, cdn.example.com, assets.example.com, and shop.example.com are frequently three DNS names for one rate limiter. Bucket them separately and you triple your effective rate against that origin even though every individual key is "within budget." The fix is explicit aliasing — a deliberate, human-reviewed map of hostnames known to share a budget:

# Hostnames that share one origin/limiter behind a CDN.
# Reviewed manually — never inferred automatically.
SHARED_BUDGET = {
    "cdn.example.com": "example.com",
    "assets.example.com": "example.com",
    "shop.example.com": "example.com",
}

def budget_domain(host: str) -> str:
    d = canonical_domain(host)
    return SHARED_BUDGET.get(d, d)

async def acquire(self, domain: str) -> None:
    key = f"politeness:{budget_domain(domain)}"
    # ... rest as above

The counter-trap matters just as much: do not auto-collapse all subdomains to the apex. foo.blogspot.com and bar.blogspot.com are entirely different sites with different owners and different budgets. My rule: apex-level shared hosting (blogspot, github.io, subdomain-per-tenant SaaS) gets no aliasing — each subdomain is its own budget — while known single-operator subdomain farms get explicit aliases. Canonicalization is a policy decision, not a string transform.

Operations: failing closed and scheduling fairly

Fail closed. When Redis is unreachable, the wrapper raises PolitenessUnavailable and the fetch does not happen. This is the right default: a crawler that silently falls back to local pacing when the coordinator dies is exactly the 12-workers-hammering fleet from the incident, except now it also lies to you in the logs. Default-deny beats default-hammer. Let the worker back off, alert, and retry the acquisition; your crawl falls behind schedule rather than falling onto the origin.

Queue depth is a scheduling signal. Track how many workers are waiting on each domain (an HINCRBY politeness:{domain} waiting on entry, decrement on exit). A domain with deep queues and an empty bucket is saturated; a domain with unused tokens is being starved. Once you have that signal, invert the dispatch loop: instead of pulling URLs from a global queue and then acquiring tokens, prefer work on domains that currently have available tokens. A fairness scheduler falls out almost for free — blocked domains accumulate backlog that drains at their refill rate the moment tokens free up.

When Redis is overkill. For a single machine with multiple processes or a single asyncio process, a per-domain fairness scheduler in one event loop does the same job with no infrastructure:

class LocalGate:
    """Single-process fallback: one shared clock, no Redis."""
    def __init__(self):
        self._next_ok: dict[str, float] = {}

    async def acquire(self, domain: str, interval: float = 5.0) -> None:
        d = budget_domain(domain)
        now = time.monotonic()
        wait = max(0.0, self._next_ok.get(d, 0.0) - now)
        if wait:
            await asyncio.sleep(wait)
        self._next_ok[d] = time.monotonic() + interval

This works because everything shares one loop and one dict — the same property Redis gives you across machines. The moment you add a second host, move to the Redis gate; the acquire() call site doesn't change.

Wrapping up

Per-process politeness is politeness for a fleet of one. The moment you run N workers, your effective rate is your sleep budget times N, and no amount of local tuning fixes it because the failure is structural: no shared clock, retry-phase-locking, invisible in-flight requests. Move the constraint to a shared per-domain token bucket — atomic via Lua, jittered on the waiting side, keyed by a deliberately canonicalized domain, fail-closed when the coordinator is down — and politeness becomes a fleet-level guarantee instead of a per-process hope. Your logs, and the WAF, will finally agree.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele