Let the Server Set Your Pace: Reading Rate-Limit Headers Before You Hit 429

Every scraper eventually hits the same wall: the polite fixed delay you chose at 2 a.m. is either far too conservative (your job takes all night for no reason) or far too aggressive (you wake up to a wall of 429s and a burned IP reputation). The mistake is picking the pace yourself when the server is often telling you the pace. Modern APIs and increasingly well-configured sites publish their rate budget in response headers. Reading those headers and throttling to the server's own stated limit — before, not after, the rejection — is the single highest-leverage politeness change you can make.

This article is about the feedback loop that treats Retry-After, RateLimit-*, and X-RateLimit-* headers as a live budget you spend, rather than ignoring them and learning your limit the hard way through a 429.

The headers that actually matter

Two families of headers carry the information:

Advisory — the response you haven't sent yet. When a server does reject you, the gold mine is:

• Retry-After — seconds (or an HTTP date) to wait before the next request. Honoring this exactly is polite and efficient: it's the server computing the correct wait for you.

Budget — the standing allowance. Present on many successful responses:

• RateLimit-Limit / X-RateLimit-Limit — total requests in the window.
• RateLimit-Remaining / X-RateLimit-Remaining — requests left.
• RateLimit-Reset / X-RateLimit-Reset — when the window refills (epoch seconds or seconds-to-reset; check the doc).

The Remaining counter is the underrated one. It turns "am I about to get blocked?" from a guess into a fact.

A token-bucket that drains to the server's clock

Instead of time.sleep(1) everywhere, maintain a rate derived from the headers. If the server says 60 requests per 60 seconds, your steady-state budget is one request per second — but you learn that number from the response, not from a blog post.

import time


class RateBudget:
    """Holds the last-seen server rate-limit signal and enforces it."""

    def __init__(self):
        self.window_s = None      # seconds the limit window covers
        self.limit = None         # requests allowed per window
        self.remaining = None     # left in current window
        self.reset_at = None      # wall-clock when window refills
        self.last_wait_log = 0.0

    def observe(self, headers):
        h = {k.lower(): v for k, v in headers.items()}

        # Prefer the standard RateLimit-* trio, fall back to X- variants.
        limit = h.get("ratelimit-limit") or h.get("x-ratelimit-limit")
        remaining = h.get("ratelimit-remaining") or h.get("x-ratelimit-remaining")
        reset = h.get("ratelimit-reset") or h.get("x-ratelimit-reset")

        if limit is not None:
            self.limit = int(limit)
        if remaining is not None:
            self.remaining = int(remaining)
        if reset is not None:
            reset = int(reset)
            # Heuristic: large numbers are epoch; small ones are seconds-to-reset.
            self.reset_at = reset if reset > 1_600_000_000 else int(time.time()) + reset
            # Infer window length when limit present.
            if self.limit:
                self.window_s = max(1, self.reset_at - int(time.time()))

    def wait_seconds(self):
        """How long to sleep before the next request to stay within budget."""
        if not self.limit:
            return 0.0  # no info yet; caller uses a default floor
        if self.remaining is not None and self.remaining > 0:
            # Spread remaining budget across the time until reset.
            time_left = max(0, self.reset_at - time.time())
            return time_left / self.remaining
        if self.remaining is not None and self.remaining <= 0:
            # Out of budget: wait for the reset, plus jitter.
            return max(0, self.reset_at - time.time()) + 0.5
        # Limit known but no remaining counter: derive even spacing.
        return (self.window_s or 60) / self.limit

    def retry_after(self, headers):
        """Honor an explicit 429 Retry-After if present."""
        h = {k.lower(): v for k, v in headers.items()}
        ra = h.get("retry-after")
        if not ra:
            return None
        try:
            return max(0, int(ra))
        except ValueError:
            # HTTP-date form
            from email.utils import parsedate_to_datetime
            dt = parsedate_to_datetime(ra)
            return max(0, dt.timestamp() - time.time())


def polite_fetch(session, url, budget, default_spacing=1.0):
    import requests
    wait = budget.wait_seconds() or default_spacing
    time.sleep(wait)
    r = session.get(url, timeout=20)
    budget.observe(r.headers)
    if r.status_code == 429:
        ra = budget.retry_after(r.headers)
        if ra is not None:
            time.sleep(ra)  # server told us exactly how long
            r = session.get(url, timeout=20)
            budget.observe(r.headers)
    return r

The subtle point in wait_seconds: when the server reports remaining, divide the time left by the requests left. If it says 4 requests remain and the window resets in 12 seconds, the correct pace is 3 seconds apart — a number you never had to guess.

Why this beats a fixed delay

A fixed sleep(2) between requests assumes the server allows half a request per second forever. Reality:

• The budget refills in bursts. Header-aware pacing lets you sprint while remaining is high and crawl when it's low — finishing faster overall than a flat delay, because you're using the whole allowance instead of the average.
• You stop being surprised by 429s. You slow down as remaining → 0, so you rarely cross the line. Fewer 429s means less time in Retry-After backoff and a cleaner IP reputation.
• Multi-IP pools stay coherent. If you rotate gateways, a naive fixed delay per client process ignores that the target counts the budget across all your IPs. Read the shared header signal and you coordinate through the server's own counter.

When there are no headers: probe the limit

Many sites omit rate-limit headers. Don't guess — probe. Walk up from a conservative spacing until you get one soft signal (429, or rising 5xx), then settle at roughly half the failure spacing. Cache the discovered value per host so tomorrow's run starts from evidence, not folklore.

import json, os, time, requests

CACHE = "rate_cache.json"


def load():
    return json.load(open(CACHE)) if os.path.exists(CACHE) else {}


def discover_spacing(url, start=2.0, min_s=0.2, tries=4):
    cache, host = load(), requests.utils.urlparse(url).netloc
    spacing = cache.get(host, start)
    for _ in range(tries):
        r = requests.get(url, timeout=15)
        if r.status_code in (429, 503):
            spacing = min(8.0, spacing * 2)   # back off hard on trouble
        else:
            spacing = max(min_s, spacing * 0.9)  # creep faster while clean
        cache[host] = round(spacing, 2)
        json.dump(cache, open(CACHE, "w"))
        time.sleep(spacing)
    return cache[host]

This is a crude TCP-style ramp, and that's the point: it's self-calibrating. You run it periodically so the per-host number tracks the site's changing policy without you editing a constant.

Respect the human cost too

Header-driven pacing is about not harming the origin. Two extra courtesies cost almost nothing: identify yourself with a sane User-Agent and honor robots.txt crawl-disallow paths; and if a page serves a Retry-After longer than a few minutes, it's telling you it's busy — that's a stop sign, not a suggestion. Politeness isn't just a legal box to tick; a server that never has to throttle you is a server you can scrape for years.

The takeaway

Stop inventing delays and start listening. Every well-behaved target is already publishing its budget in RateLimit-*, X-RateLimit-*, and Retry-After. Parse them, divide remaining by time-until-reset, and let the server compute your pace. You'll finish jobs faster on good days, avoid 429 storms on tight days, and keep the relationship — and your IP reputation — intact for the long haul.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele