Every scraper has one. The sleep(1.3) between requests. The jitter. The retry-with-backoff that was "borrowed" from a blog post in 2023 and never questioned since.
We treat the politeness layer as config — numbers in a YAML file — and then act surprised when production crawls at 40 requests/second because someone "tidied up" the constants during a refactor.
Rate limiting is a contract with the site you're scraping. Contracts deserve tests.
The three assertions that caught real bugs for us
1. The floor test. For every politeness policy, assert the minimum delay between two consecutive requests to the same host. Not the average — the minimum. Averages hide the bug class: burst-then-pause schedulers pass an average check while hammering the first page of a site ten times in a second.
def test_min_delay_between_same_host(requests_log):
same_host = [(t, url) for t, url in requests_log if same_host(url)]
gaps = [b[0] - a[0] for a, b in zip(same_host, same_host[1:])]
assert min(gaps) >= POLITE_MIN, f"burst detected: {min(gaps):.2f}s gap"
2. The 429-response test. When the server answers 429 (or a Retry-After we ignore), the politeness layer must react, and the reaction must be observable. Feed the client a fake 429 with Retry-After: 7 and assert the next request to that host lands ≥7 seconds later — not "eventually," within a tolerance window. We found a real bug this way: our backoff reset on the first success after a 429, so a site throttling us at exactly our burst ceiling got re-burst-attacked forever.
3. The empty-200 test. Some servers answer rate-limiting with a silent 200 and an empty body. Assert that N consecutive empty-200s to one host trip the same slowdown as a 429. This is the sneaky one: your dashboards show 100% success while you scrape nothing. (Full write-up: Your Scraper's "Success" Metric Is Lying When the Site Sends You an Empty 200.)
Where the test runs
We run the floor test in CI against a local recording proxy (requests get logged with timestamps, fixture responses replay). The 429 and empty-200 tests run against the same recorder with scripted responses. None of them touch the live internet — same as you wouldn't test payment logic by buying real things.
The nightly crawl keeps a lightweight version in production: a sampled audit that logs the minimum inter-request gap per host per hour, and pages us when the floor drops. CI catches policy regressions; production catches reality drift.
The uncomfortable part
A politeness test encodes your definition of polite, and definitions change: a site's robots.txt crawl-delay, an observed Retry-After culture, your own risk appetite. That's not a reason to skip the test — it's a reason to keep the definition in one file and let the test point at it. "We were polite" should be a fact with a green check next to it, not a vibe you assert in a postmortem.
We package this exact pipeline (daily Telegram-channel crawling with the empty-200 guard, dedup, and the polite scheduler) as a tiny wheel + field-tested recipes — tg-crawl Pro ($9). And if you want to see the shape of our data before trusting anything: a $1 pay-what-you-want sample digest.
Written by 何雨禾 (heyuhe) — solo operator running a daily scraping pipeline out of Taipei. More at the storefront.
Top comments (0)