Build a Shared 429 Backoff Gate for Proxy Workers

#automation #devops #python #tutorial

When many workers share a proxy route or destination, handling 429 Too Many Requests inside each worker is not enough. They all discover the same limit separately, sleep separately, and often wake together.

A safer design uses a shared backoff gate. Every worker checks the gate before starting a request. The first worker that observes a limit updates the shared pause, and the rest stop creating avoidable traffic.

What the gate should store

Keep the state small and scoped:

scope_key
pause_until
probe_owner
consecutive_limits
last_status

The scope key matters. A limit may belong to an API token, account, endpoint, destination, proxy route, or a combination of those. A single global gate can unnecessarily stop healthy traffic, while a gate scoped only to an exit IP may miss an account-level limit.

Parse Retry-After carefully

Retry-After can be a number of seconds or an HTTP date. Treat a valid value as a minimum pause, not a suggestion to release the full queue at that exact moment.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(value: str | None, now: datetime) -> float | None:
    if not value:
        return None

    value = value.strip()
    if value.isdigit():
        return max(0.0, float(value))

    try:
        retry_at = parsedate_to_datetime(value)
        if retry_at.tzinfo is None:
            retry_at = retry_at.replace(tzinfo=timezone.utc)
        return max(0.0, (retry_at - now).total_seconds())
    except (TypeError, ValueError, OverflowError):
        return None

If the header is missing or invalid, use capped exponential backoff with jitter. Do not use a fixed delay across every worker.

Claim one recovery probe

After the pause expires, allowing every waiting worker to run creates another spike. Use a short lease so only one worker sends a probe.

async def before_request(scope, limiter):
    state = await limiter.get(scope)

    if state.pause_until > limiter.now():
        raise Deferred(state.pause_until)

    if state.requires_probe:
        claimed = await limiter.claim_probe(scope, ttl_seconds=15)
        if not claimed:
            raise Deferred(limiter.now_plus(seconds=5))

If the probe succeeds, reopen gradually. If it receives another 429, extend the pause and increase the backoff cap. The lease must expire automatically so a crashed probe owner does not block recovery forever.

Control rate and concurrency separately

A semaphore limits requests already in flight. A token bucket limits how quickly new requests start. You usually need both.

Long responses can exhaust concurrency even at a modest start rate. Short requests can exceed a rate limit even with low concurrency. Put both controls ahead of the shared gate and keep the queue bounded.

Do not rotate exits as the default response

A new proxy exit does not necessarily change the limiting scope. If the destination limits an account, token, session, or endpoint, aggressive rotation only adds traffic and cost.

Use rotation for authorized geographic coverage and session design. Do not use it to defeat a destination's rate limit or access controls.

Metrics that reveal whether recovery works

Track:

initial requests and retries separately;
429 responses by scope;
time spent paused;
probe success rate;
queue age and rejected work;
success per usable result;
bandwidth and compute spent on retries.

A falling 429 count is not enough if queue age and cost keep rising. The goal is stable, useful output with fewer unnecessary requests.

Production checklist

Identify the actual limiter before changing traffic.
Respect valid Retry-After values.
Share pause state across workers.
Allow one bounded recovery probe.
Reopen gradually instead of draining the backlog.
Apply both start-rate and concurrency limits.
Keep queues bounded and observable.
Never log proxy passwords, cookies, or authorization headers.
Stop or defer traffic when authorization or policy requires it.

Disclosure: I work with 98IP. This post describes reliability patterns for lawful, authorized automation. Additional proxy engineering resources: https://en.98ip.com/?k=dev

DEV Community