Rate Limiting Yourself: The Politeness Strategy That Keeps Scrapers Alive

Every scraping tutorial teaches you to overcome the target's rate limiting. Almost none teach you to impose your own — which is odd, because in my experience the scrapers that survive for years are the ones that throttle themselves harder than any external limiter would.

This is a post about politeness as an engineering discipline. Not ethics-performance — though being gentle with someone else's server is reasonable — but the practical observation that restraint is the cheapest block-avoidance technology that exists.

The Economics of Greed

Consider the two failure paths for a price monitor that must check 500 products hourly:

The greedy crawler fires all 500 requests as fast as concurrency allows, from rotating residential IPs. It "works" — until the target's anomaly detection notices the traffic pattern (500 synchronized fetches, hourly, forever), fingerprint-detects it, and starts serving CAPTCHAs to anything matching. Now you're paying proxy fees to collect challenge pages.

The polite crawler spreads those 500 requests across the hour with jitter: one every ~7 seconds, in randomized order, from varied IPs. Total wall time is identical (results arrive within the hour either way). Detection surface: near zero, because nothing about it is anomalous.

Same deadline. Same data. One of them looks like an attack pattern; the other looks like background noise. Politeness costs nothing and buys survival.

The Four Layers of Self-Throttling

Layer 1: Per-request jitter. The absolute baseline. Fixed intervals are a fingerprint even at low volume:

import random, time

for url in urls:
    fetch(url)
    time.sleep(random.uniform(2.0, 5.0))     # never the same interval twice

Layer 2: A global rate ceiling. Cap requests-per-minute regardless of how many workers you run. A token bucket in ten lines:

import time

class TokenBucket:
    def __init__(self, rate_per_minute):
        self.interval = 60.0 / rate_per_minute
        self.next_ok = 0.0

    def acquire(self):
        now = time.monotonic()
        wait = self.next_ok - now
        if wait > 0:
            time.sleep(wait)
        self.next_ok = max(now, self.next_ok) + self.interval

bucket = TokenBucket(rate_per_minute=12)   # deliberately modest

for url in urls:
    bucket.acquire()
    fetch(url)

Layer 3: Per-domain budgets. Your politeness obligation is per site, not global. A crawler touching 50 domains can run 50 modest budgets simultaneously; a crawler hammering one domain needs one very patient budget. Track (and cap) requests per domain per day, and let rare-to-change pages be polled rarely:

POLL_POLICY = {
    "pricing_pages":  3600 * 4,   # every 4h — prices change a few times a day
    "listing_pages":  3600 * 12,  # every 12h
    "about_pages":    3600 * 72,  # every 3 days
}

Most scrapers I audit are massively over-polling static content. Polling frequency should match the decision value of the data's freshness, not the crawler's idle capacity.

Layer 4: Circuit breakers and backoff. When a domain starts returning 429s or CAPTCHAs, the correct response is to slow down — not to rotate IPs and push harder. Back off exponentially, and when error rates exceed ~5%, pause that domain entirely for an escalating cool-down:

def on_block(domain):
    cooldowns[domain] *= 2          # 5min -> 10 -> 20 -> ...
    if error_rate_1h(domain) > 0.05:
        pause(domain, hours=1)      # let the suspicion decay

IP rotation on a flagged domain buys you requests, but it also escalates the flagging. Politeness de-escalates.

Politeness Is Also a Detection-Signal Defense

Here's the part that isn't obvious: uniform politeness patterns are themselves detectable. A scraper that polls exactly every 7.0 seconds with exactly one request per interval, forever, is still a metronome — just a slow one. The refinements that matter:

• Randomize polling order, not just intervals. Same sequence every run is a pattern.
• Randomize run start times. sleep(random.uniform(0, 900)) before each scheduled job; never start at exactly :00.
• Vary request composition. Real browsing mixes page types; a crawler that only ever hits /product/* in asin order looks like exactly what it is.
• Respect quiet hours. Traffic that runs 24/7 at constant volume has no biological signature. Humans sleep (in some timezone). A monitor that checks EU-market pages during EU daytime and pauses at 3 AM local is more plausible than one that never pauses.

What This Buys You, Quantified

Across the pipelines I run (price monitoring, SERP checks, ad verification — roughly 60–80k requests/day), the sustained success rate with the full politeness stack sits above 99% with residential proxies whose geo matches each target market (I use Thordata's rotating pool, $0.65/GB, code thor020 for 10% off). The same pipelines without self-throttling — same IPs, same headers — decayed to ~95% within two weeks as blocks accumulated. Politeness was the difference between infrastructure and incident response.

The Mindset Shift

Treat your crawler's traffic budget the way you'd want a guest to treat your bathroom: it's shared, you're unseen, and the goal is to leave no evidence you were there. Every throttling layer you add shrinks your footprint. And unlike every evasion technique — fingerprint spoofing, CAPTCHA solving, header forgery — politeness never triggers an arms race. Nobody builds defenses against a visitor who behaves.

Rate limit yourself before they rate limit you. It's the only scraping technique that gets more effective over time, not less.

---

Disclosure: my collection pipelines run on Thordata's geo-targeted residential proxies. The politeness patterns here work with any provider — they work with no provider, just slower.