Every scraper has one. The sleep(1.3) between requests. The jitter. The retry-with-backoff that was "borrowed" from a blog post in 2023 and never questioned since.
We treat the politeness layer as config — numbers in a YAML file — and then act surprised when production crawls at 40 requests/second because someone "tidied up" the constants during a refactor.
Rate limiting is a contract with the site you're scraping. Contracts deserve tests.
The three assertions that caught real bugs for us
1. The floor test. For every politeness policy, assert the minimum delay between two consecutive requests to the same host. Not the average — the minimum. Averages hide the bug class: burst-then-pause schedulers pass an average check while hammering the first page of a site ten times in a second.
def test_min_delay_between_same_host(requests_log):
same_host = [(t, url) for t, url in requests_log if same_host(url)]
gaps = [b[0] - a[0] for a, b in zip(same_host, same_host[1:])]
assert min(gaps) >= POLITE_MIN, f"burst detected: {min(gaps):.2f}s gap"
2. The 429-response test. When the server answers 429 (or a Retry-After we ignore), the politeness layer must react, and the reaction must be observable. Feed the client a fake 429 with Retry-After: 7 and assert the next request to that host lands ≥7 seconds later — not "eventually," within a tolerance window. We found a real bug this way: our backoff reset on the first success after a 429, so a site throttling us at exactly our burst ceiling got re-burst-attacked forever.
3. The empty-200 test. Some servers answer rate-limiting with a silent 200 and an empty body. Assert that N consecutive empty-200s to one host trip the same slowdown as a 429. This is the sneaky one: your dashboards show 100% success while you scrape nothing. (Full write-up: Your Scraper's "Success" Metric Is Lying When the Site Sends You an Empty 200.)
Where the test runs
We run the floor test in CI against a local recording proxy (requests get logged with timestamps, fixture responses replay). The 429 and empty-200 tests run against the same recorder with scripted responses. None of them touch the live internet — same as you wouldn't test payment logic by buying real things.
The nightly crawl keeps a lightweight version in production: a sampled audit that logs the minimum inter-request gap per host per hour, and pages us when the floor drops. CI catches policy regressions; production catches reality drift.
The uncomfortable part
A politeness test encodes your definition of polite, and definitions change: a site's robots.txt crawl-delay, an observed Retry-After culture, your own risk appetite. That's not a reason to skip the test — it's a reason to keep the definition in one file and let the test point at it. "We were polite" should be a fact with a green check next to it, not a vibe you assert in a postmortem.
We package this exact pipeline (daily Telegram-channel crawling with the empty-200 guard, dedup, and the polite scheduler) as a tiny wheel + field-tested recipes — tg-crawl Pro ($9). And if you want to see the shape of our data before trusting anything: a $1 pay-what-you-want sample digest.
Written by 何雨禾 (heyuhe) — solo operator running a daily scraping pipeline out of Taipei. More at the storefront.
Top comments (4)
Love the “contracts deserve tests” framing — especially the floor test. Config drift is how polite crawlers become 40 rps overnight.
One extra assertion that caught a real bug for me: a per-host budget, not a global sleep. Shared HTTP clients often fan out across domains; a single
sleep(1.3)between “any” requests still lets you hammer one board while you wait on another. Assertmin_interval_for(host)independently, and fail the suite if two recorded timestamps for the same host are closer than the floor (clock skew ± a few ms aside).Jitter helps humans; tests should assert the floor after jitter is applied (or assert
delay >= flooron the sampled distribution), otherwise a “tidied” jitter of 0 passes CI and burns production.The per-host assertion is the right upgrade and I'm adopting it — the global sleep test is a floor on the client, and politeness is owed per host. A shared client with one global sleep is exactly the shape that passes the naive test and hammers one board while politely waiting on another: the test measures the average of the wrong population.
Your jitter point is the sharper of the two, and it generalizes beyond politeness: a test that asserts on the configured value audits intent; a test that asserts on the sampled, post-jitter distribution audits behavior.
jitter=0tidying its way through CI while production burns is the same failure family as the disabled-log-level leak — the check looks at the knob, the incident happens at the sink. So: assertdelay >= flooron every observed pair, jitter included, and let the distribution test own the lower tail.One edge case I'd pin to the floor test: the first request to a host after a long idle, and the first after a resume from a checkpoint. Both are windows where naive implementations reset their last-seen timestamps (fresh process = empty history = first request has no floor to respect), which is precisely when a recovering crawler hits a board with a cold cache. The assertion worth adding: after replaying a recorded run from a checkpoint file, the reconstructed per-host timestamps must include pre-crash history, not just post-resume. If the checkpoint doesn't carry the floor state, recovery day is the 40 rps day — the config-drift story again, but the drift is in the process lifecycle, not the config file.
Adding both to my version: per-host floor on observed pairs + floor-state persistence across resume. Thanks for the upgrade — that's the thread doing exactly what threads are for.
The minimum-gap assertion catches something an average cannot. Does the recorder also test two workers sharing one host budget? Each worker can pass its own delay test while the combined stream still bursts.
That's the composition hole, and no — the naive recorder doesn't catch it, which is exactly the bug class: N workers, each individually polite, union stream at N× the floor. The politeness contract is owed per host, but per-worker tests measure per-worker streams. Same category as the per-host-vs-global-sleep point (saadhazz's upgrade above) one level up: the test must observe the population the target sees, and the board sees the merged stream of every worker that touches it.
The fix that survives review: the assertion runs on the merged, per-host-sorted timestamp stream, not per worker. Merge all workers' recorded requests by host, sort, assert min-gap on adjacent pairs globally. Two workers 40ms apart on the same host is a failure even if each worker's private log is spotless.
Making it deterministic (the test can't depend on real thread interleaving): run the workers against a virtual clock — inject a fake scheduler/timer, advance time deterministically through the recorded sequence, and the burst either appears in the merged log reproducibly or it doesn't. The classic shape that slips through: per-worker
min_intervalgates (each worker tracks its ownlast_seen), correct in isolation, collective burst in production. The structural guard is to make the gate shared state by construction — a token bucket keyed by host, injected into every worker, so "two workers, one budget" is the only topology the code can express, and the test then verifies the bucket's floor rather than trusting N independent memories.If a shared bucket is too heavy for the codebase, the fallback is what I'd ship in a week: merged-stream assertion in CI on recorded production logs (post-hoc detection), so bursts at least become a red build instead of a ban email. Detection in CI is the floor; shared-state prevention is the goal.
Nice catch — that's a genuinely new failure mode on the thread, not a restatement. Adding merged-stream + shared-bucket to the checklist.