Retries Are Secretly Rate-Limit Violations: Auditing Your True Request Rate in Python

Here's a question that catches almost every scraper developer off guard the first time they measure it: when your polite crawler is configured to run at 2 requests per second with a retry budget of 3, what request rate does the target website actually see?

The intuitive answer is "2 per second, sometimes a bit more on retries." The measured answer, on a bad day, is closer to 5–8 per second — because retries, redirects, connection re-establishment, and library-internal re-requests all fire in addition to your scheduled pace, and none of them go through your rate limiter.

This article is about the gap between intended rate and actual rate. It's the single most common politeness bug in production scrapers, it's invisible in your code review, and it's usually the reason a "we only do 2 rps" crawler still earns itself a 429 (or worse, a silent quality-based block that no status code ever announces).

The four multipliers nobody counts

A typical Python scraper has four mechanisms that multiply outbound requests beyond what its scheduler plans:

1. Explicit retries. urllib3.util.Retry(total=3) means each logical request can become up to 4 physical requests. On flaky residential proxy paths — where a 5–10% per-request failure rate is normal, not exceptional — the multiplier is real, not theoretical.
2. Redirect following. Every 301/302 is an extra request. Targets that bounce you through a regional mirror or a bot-check interstitial routinely turn one logical fetch into two or three origin hits.
3. Partial-read re-requests. A connection reset mid-body after the origin already did the work (and logged the hit) still triggers a retry. The origin's request counter saw two requests; your success counter saw one.
4. Concurrent workers sharing a domain. Five workers each rate-limited to 1 request per 4 seconds to the same domain is 1.25 rps aggregate — unless your limiter is per-worker, in which case it's 5 rps and your politeness math was never valid in the first place.

Crucially, when you crawl through rotating proxies, the origin sees each of these physical requests arriving from a different IP address. This makes the multiplier worse, not better: a domain-level rate limit enforced per-IP won't trip, so instead of a clean 429 you accumulate reputation damage across the whole exit range until the target starts degrading your responses silently. The origin can't rate-limit you effectively per-IP — so it blocks you in bulk later. Retrying through fresh exits doesn't escape the budget; it hides the invoice until it's unpayable.

Measure first: a counting harness

You cannot audit what you don't count. The fix starts with instrumenting the actual outbound request stream — every physical request, including retries — and comparing it to what your scheduler thinks it sent.

import time
from collections import Counter

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

class AuditingAdapter(HTTPAdapter):
    """Counts every physical request the session sends, retries included."""

    def __init__(self, *args, **kwargs):
        self.events = Counter()
        self.timeline = []
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        self.events["physical_requests"] += 1
        self.events[f"physical_{request.method}"] += 1
        self.timeline.append(time.monotonic())
        return super().send(request, **kwargs)


def build_session(retries: int = 3) -> requests.Session:
    s = requests.Session()
    retry = Retry(
        total=retries,
        backoff_factor=1.0,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET", "HEAD"],
    )
    adapter = AuditingAdapter(max_retries=retry)
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    s.audit = adapter  # stash for reporting
    return s


def report(session: requests.Session, intended_rps: float, n_logical: int):
    ev = session.audit.events
    physical = ev["physical_requests"]
    wall = (session.audit.timeline[-1] - session.audit.timeline[0]
            if len(session.audit.timeline) > 1 else 1.0)
    actual_rps = physical / wall if wall > 0 else 0
    print(f"logical requests scheduled: {n_logical}")
    print(f"physical requests sent:     {physical}")
    print(f"multiplier:                 {physical / max(n_logical, 1):.2f}x")
    print(f"intended rate:              {intended_rps:.2f} rps")
    print(f"measured outbound rate:     {actual_rps:.2f} rps")

Run a real batch through build_session() and call report(). The first time I did this on a "1 rps polite" crawler with Retry(total=3) against a flaky exit path, the multiplier was 1.9x and the measured outbound rate was 2.4 rps on a bad hour. Nothing in the codebase said "2.4". The code said "1". The difference was entirely retries and redirects.

To separate retry noise from redirect noise, add a response hook:

def count_redirects(r: requests.Response, *args, **kwargs):
    if r.history:
        session.audit.events["redirect_chains"] += 1
        session.audit.events["redirect_hops"] += len(r.history)

Fixing the gap: budget retries inside the rate limiter

Once you can see the multiplier, the fix is architectural: your rate limiter must govern physical requests, not logical ones. Two changes get you there.

1. Move the budget to a global, domain-keyed limiter that the adapter itself consults. Don't sleep in your scraping loop and hope retries fit in the gaps — make the adapter acquire a token before super().send(). A minimal token-bucket version:

class RateLimitedAdapter(AuditingAdapter):
    def __init__(self, *args, rps: float = 1.0, burst: int = 2, **kwargs):
        super().__init__(*args, **kwargs)
        self.interval = 1.0 / rps
        self.burst = burst
        self.tokens = float(burst)
        self.last = time.monotonic()
        self._lock = __import__("threading").Lock()

    def _acquire(self):
        with self._lock:
            now = time.monotonic()
            self.tokens = min(self.burst,
                              self.tokens + (now - self.last) / self.interval)
            self.last = now
            if self.tokens < 1:
                time.sleep((1 - self.tokens) * self.interval)
                self.tokens = 0
            else:
                self.tokens -= 1

    def send(self, request, **kwargs):
        self._acquire()
        return super().send(request, **kwargs)

Now retries, redirects, and first attempts all draw from the same bucket. Your total outbound rate to that domain is bounded by construction — which is the only definition of politeness that survives contact with a flaky proxy path.

2. Classify failures and stop retrying the unretryable. Distinguish proxy-layer failures (the gateway refused, the exit never connected — the origin never saw anything, so retrying costs no origin budget) from origin-layer failures (you got a response: 429, 403, interstitial). Retry the first aggressively; handle the second with policy — back off, rotate identity, or park the URL for later. Blindly retrying 429s through fresh exits is the worst of both worlds: you hammer the origin and burn proxy bandwidth, and because each hit comes from a new IP, the origin's per-IP limiter never protects you from yourself.

In urllib3 v2, Retry gives you the hooks for this: catch ProxyError separately in your own retry wrapper and give it a cheap, fast retry loop (the origin wasn't touched), while reserving the slow, budgeted retry path for origin-visible failures. The distinction is one except block, and it routinely halves the effective multiplier on residential paths.

Make the multiplier a dashboard number

After you fix it, keep the instrumentation. Export two metrics from every worker: logical_requests_scheduled and physical_requests_sent, per target domain. Their ratio — call it the politeness multiplier — should sit between 1.0 and about 1.3 in healthy operation. When it starts climbing, it's telling you one of three things: your proxy path is degrading (more proxy-layer retries), the target changed its redirect behavior, or someone deployed a "small fix" that added a retry loop. All three are worth an alert, and none of them are visible in your configured rps number, which will be sitting there the whole time, perfectly innocent, saying "1.0".

The uncomfortable truth about polite scraping is that politeness isn't a configuration value — it's an outcome you have to measure at the wire. Your scheduler's intent counts for nothing; only the request stream the origin actually receives does. Count everything, budget at the adapter, and treat every retry as what it really is: another request from a machine the target is already watching.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele