Every scraper config I have inherited in the last five years has the same line in it: RATE_LIMIT = 2 requests per second. Nobody remembers where the number came from. It was copied from a repo, which copied it from a blog post, which copied it from a Stack Overflow answer written against a different site, in a different year, from a different datacenter, on a different exit pool. Two rps is not a measurement. It is a rumor.
Here is the thesis I want to argue: politeness is a measurable quantity. You can run a controlled experiment against your target, from your exits, and find the point where the site starts pushing back. That point — the knee — is the only number worth putting in your config. Operate at a fraction of it, and you get a rate you can defend when someone asks "why 2 and not 8?" The answer becomes: because we measured, and the knee was here, and we chose to sit well below it.
Rate limits are per origin-exit pair, not per crawler
The mental model most people carry is "the site allows N requests per second." That is wrong in a way that matters. Rate limits are enforced against a source, usually the client IP the site sees at its edge. What you actually have is a table keyed by (origin, exit IP) pairs. Each row gets its own allowance.
This is why the copied "2 rps" is meaningless without knowing the exit pool size. If that number was tuned by someone rotating through 200 residential exits and staying at 0.01 rps per exit, their aggregate polite throughput is 2 rps. If you run the same "2 rps" from a single datacenter IP, you are 200x over-driving your one exit, and you will meet its real ceiling almost immediately. Conversely, if you have 500 sustainable exits, your honest aggregate budget is 500x the per-exit number, and pinning yourself to "2 rps" globally leaves most of that budget unused — you are politely under-scraping your own pool.
So the correct thing to measure is per-exit. Aggregate is just arithmetic afterward: sustainable_throughput = per_exit_knee × distinct_exits_you_can_sustainably_use. The multiplication only holds if the exits are genuinely distinct and each is used within its own allowance; burning one exit hard while idling ninety-nine others does not give you the aggregate. That single sentence invalidates most rate-limit folklore, which is exactly why we go measure it ourselves.
The probe design: a staircase
The experiment is a step-up staircase. You pick one exit, one endpoint, and walk the request rate upward in stages — for example 0.5, 1, 2, 4, 8 requests per minute per exit — firing a fixed number of requests in each stage and logging what comes back. Then you watch for the first sign of pushback and stop.
Three signal families, captured per stage:
Status codes. 429s and 503s are the loud signals. But a clean 200 is not automatically "fine" — see below.
Latency percentiles. Long before a site hard-blocks you, it usually gets slow. Track p50 and p95 per stage. A knee often shows up as p95 climbing sharply while status codes are still all 200.
Soft-block markers. The most common rate-limit feedback is not a 429; it is a 200 that lies. Detect it explicitly: a sudden drop in median body length (the classic ~1 KB "sorry" page instead of your ~45 KB listing), a redirect to a login or interstitial, or a DOM sentinel missing — you check for a known element or string that must exist on a real page and is absent on a block page.
The decision rule is deliberately simple. Walk up the staircase; at the first stage that trips any signal, mark it FAILED and record the previous stage as the knee. Your production rate is then set to roughly 50–60% of the knee, not at it. The margin exists because the knee you measured is a specific moment on a specific day under a specific load from the rest of the internet; you do not want to be the operator sitting exactly on the edge where a little drift turns polite into blocked.
The code
A monotonic-clock token bucket. acquire() is awaitable and returns only when a token is available. Using loop.time() (a monotonic clock) rather than wall-clock means a clock adjustment mid-probe cannot warp your measured rate.
import asyncio
import time
class RateLimiter:
"""Token bucket: allows `rate` acquisitions per second, burst up to `burst`."""
def __init__(self, rate: float, burst: int = 1):
self.rate = float(rate)
self.burst = int(burst)
self.tokens = float(burst)
self.updated = time.monotonic()
self._lock = asyncio.Lock()
async def acquire(self) -> None:
async with self._lock:
while True:
now = time.monotonic()
self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
self.updated = now
if self.tokens >= 1.0:
self.tokens -= 1.0
return
# sleep exactly long enough for one token to accrue
await asyncio.sleep((1.0 - self.tokens) / self.rate)
The stage runner fires N requests through one proxy exit at a given per-minute rate and computes the metrics we care about.
import statistics
from dataclasses import dataclass
from html.parser import HTMLParser
import aiohttp
# DOM sentinel: a string that MUST appear on a real page of this endpoint.
# Absence is a soft-block signal even when the status code is 200.
DOM_SENTINEL = "data-listing-container"
@dataclass
class StageMetrics:
rate_rpm: float # requests per minute we asked for
n: int
success_rate: float # fraction of 200s that ALSO passed content checks
status_hist: dict
p50_ms: float
p95_ms: float
body_len_median: int
soft_block_signals: int # 200s that failed length/sentinel/redirect checks
def _looks_blocked(status: int, body: str, redirected_login: bool) -> bool:
if status in (429, 503):
return True
if status == 200 and redirected_login:
return True
if status == 200 and DOM_SENTINEL not in body:
return True
if status == 200 and len(body) < 2048:
return True # tiny "sorry" body where a real listing is ~45 KB
return False
async def run_stage(
session_factory,
url: str,
exit_proxy: str,
rate_rpm: float,
n_requests: int,
) -> StageMetrics:
limiter = RateLimiter(rate=rate_rpm / 60.0, burst=1)
statuses, latencies, body_lengths = {}, [], []
ok = soft = 0
async def one():
nonlocal ok, soft
await limiter.acquire()
t0 = time.monotonic()
async with session_factory(proxy=exit_proxy) as resp:
body = await resp.text()
redirected = any("login" in str(h) for h in resp.history) or "login" in str(resp.url)
dt_ms = (time.monotonic() - t0) * 1000.0
statuses[resp.status] = statuses.get(resp.status, 0) + 1
latencies.append(dt_ms)
body_lengths.append(len(body))
if _looks_blocked(resp.status, body, redirected):
soft += 1
else:
ok += 1
await asyncio.gather(*(one() for _ in range(n_requests)))
lat_sorted = sorted(latencies)
return StageMetrics(
rate_rpm=rate_rpm,
n=n_requests,
success_rate=ok / n_requests,
status_hist=dict(statuses),
p50_ms=lat_sorted[len(lat_sorted) // 2],
p95_ms=lat_sorted[int(len(lat_sorted) * 0.95)],
body_len_median=int(statistics.median(body_lengths)),
soft_block_signals=soft,
)
The staircase loop. A stage is considered "bad" if any 429/503 appears, or the clean-success ratio falls under 95%, or we accumulated soft-block signals. First bad stage stops the walk; the previous rate is the knee.
import math
def is_bad(m: StageMetrics) -> bool:
if m.status_hist.get(429, 0) or m.status_hist.get(503, 0):
return True
if m.success_rate < 0.95:
return True
if m.soft_block_signals:
return True
return False
async def find_knee(
session_factory,
url: str,
exit_proxy: str,
stages_rpm=(0.5, 1, 2, 4, 8, 16),
requests_per_stage=40,
margin=0.6,
) -> tuple[float, list[StageMetrics]]:
metrics: list[StageMetrics] = []
knee_rpm = stages_rpm[0]
for r in stages_rpm:
m = await run_stage(session_factory, url, exit_proxy, r, requests_per_stage)
metrics.append(m)
if is_bad(m):
break # previous stage's rate stays recorded as the knee
knee_rpm = r
recommended = knee_rpm * margin # sit at ~50-60% of the measured knee
return recommended, metrics
Example session factory wiring, so the whole thing is genuinely runnable:
import aiohttp
async def session_factory(proxy: str):
return aiohttp.ClientSession(
connector=aiohttp.TCPConnector(ssl=False),
timeout=aiohttp.ClientTimeout(total=15),
)._request.__self__ # illustrative: build a session, request through `proxy`
In production you would hold a single
ClientSessionand passproxy=exit_proxyper request rather than opening one per call; this is written for clarity.
Driving it once per exit and reading the result gives a table like this. The numbers below are illustrative — they show the shape a real staircase produces, not a claim about any specific site.
| stage (req/min/exit) | status hist | p50 ms | p95 ms | body-len median | soft-blocks | verdict |
|---|---|---|---|---|---|---|
| 0.5 | {200: 40} | 210 | 380 | 45,102 | 0 | OK |
| 1 | {200: 40} | 220 | 410 | 45,010 | 0 | OK |
| 2 | {200: 40} | 230 | 430 | 44,877 | 0 | OK |
| 4 | {200: 38, 429: 2} | 260 | 610 | 41,200 | 1 | FAIL |
| 8 | (not run) | — | — | — | — | skipped |
The knee is 2 req/min per exit. Recommended sustained rate = 2 × 0.6 = 1.2 req/min per exit. If your pool sustains 500 such exits, your honest aggregate is roughly 1.2 × 500 = 600 req/min ≈ 10 rps — a number you can actually explain, derived from measurement rather than folklore.
Two caveats I refuse to gloss over
The probe is traffic, and you spend it like any other traffic. A staircase that runs 40 requests across five stages is 200 requests per exit just to learn a number — before you fetch a single thing you actually want. Treat the probe as a cost. Run it off-peak so you are measuring your behavior, not the site's 2 a.m. maintenance-window behavior (or its busiest-hour throttling). Probe a low-value endpoint you already poll — a listing page, not the checkout — so a stage that trips a soft-block does not damage a path you care about. And cap total probe requests; do not let a runaway staircase walk to 64 rps while you are at lunch.
Knees drift, so re-measure on a schedule. The knee you found in March is not the knee in August. Targets change capacity, seasonally and deliberately. A CDN migration, a bot-detection vendor upgrade, or a new edge WAF rule can collapse a knee overnight. Two reasons to trigger a re-probe: a standing calendar (monthly is a reasonable default), and events — the day 429s start appearing at a rate you were previously comfortable at, you do not tune the old number; you re-run the staircase and let it tell you where the line moved to. The whole value of the measurement approach is that it is re-runnable. Folklore is not.
Politeness ceiling is also the cost floor
There is a quiet financial argument that lines up exactly with the ethical one. Over-driving an exit does not just risk a block; it produces 429s and soft-blocks that your retry logic then re-fetches, and those retries consume paid proxy GB — you buy the bandwidth for the request that returned a 1 KB "sorry" page, and again for the retry. The rate that keeps you politely under the knee is the same rate that stops you from paying for failed traffic. You are not choosing between being good and being cheap. Past the knee, they are the same knob. Measure once, sit at the fraction, and both budgets hold.
The honest version
The number you want in your config did not come from another project. It came from a twenty-minute staircase against your target, on your exits, on your endpoint. Copy "2 rps" and you are guessing with confidence. Measure the knee and you know where the cliff is — which means you know how far back to stand, and you can say why. That is worth more than any inherited constant, and unlike folklore, when the ground shifts you can just go measure again.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)