Python Retry with Exponential Backoff and Jitter (stdlib only)

#backend #python #softwaredevelopment

Most retry code I see in agent projects looks like this:

for i in range(3):
    try:
        return call_api()
    except Exception:
        time.sleep(2)
raise RuntimeError("gave up")

It survives a demo. In production it fails in four different ways, and each one is its own failure mode. Here's what's wrong and the stdlib-only pattern we ship instead.

Problem 1: it retries permanent errors

A 401 (bad key) and a 429 (rate-limited) are not the same failure. Retrying a 401 three times burns 6 seconds and a support ticket. Classify first:

RETRYABLE_EXCEPTIONS = (TimeoutError, ConnectionError, OSError)
RETRYABLE_STATUS = {408, 409, 425, 429, 500, 502, 503, 504}

Anything outside those lists should fail on the first attempt. In an agent loop, a permanent error you keep retrying becomes the retry spiral: the same failure five times in a row until the runtime hard-stops the turn.

Problem 2: fixed sleep creates a thundering herd

When an API wobbles, every client that went down at the same moment comes back at the same moment — fixed 2-second sleeps synchronize the retry wave and knock the service down again. The fix is exponential backoff plus jitter:

delay = min(cap, base * (2 ** attempt))   # 1.5, 3, 6, 12... capped
delay *= random.uniform(0.75, 1.25)       # jitter desynchronizes clients
time.sleep(delay)

The cap matters too. Uncapped exponential backoff means your fifth retry sleeps four minutes — through your agent's entire turn budget.

Problem 3: no total budget

This is the one that kills AI agents specifically. If each attempt gets its own timeout and each retry its own sleep, total worst-case time is timeout × (retries+1) + sleeps. A 45-second timeout with 3 retries can eat 4+ minutes — and agent platforms kill turns at 5. The retry loop needs a shared deadline across all attempts, so the last attempt gets whatever time is left, not a fresh allowance:

start = time.monotonic()
deadline = start + budget_s          # ONE budget for the whole call

for attempt in range(retries + 1):
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        break                        # budget exhausted, fail honestly
    try:
        value = run_with_timeout(fn, min(timeout, remaining))
        ...
    except RETRYABLE_EXCEPTIONS:
        if time.monotonic() < deadline - backoff_base:
            time.sleep(jittered_delay)  # only sleep if budget survives it

Problem 4: "success" that isn't

The API returned 200 with 10 of 60 results. The call "succeeded" and your agent built on garbage. A retry helper for agents should take a semantic validator, and a rejected result should count as a retryable failure:

res = resilient_call(lambda: my_llm(prompt),
                     name="llm",
                     timeout=45, retries=3,
                     validate=lambda x: len(x["results"]) == expected)

The whole thing, assembled

Put the four pieces together — classification, jittered capped backoff, shared budget, semantic validation — and you get a wrapper that returns a structured result instead of either a value or an exception:

res = resilient_call(fn, name="llm", timeout=45, retries=3,
                     budget_s=120, validate=shape_check)
if res.ok:
    use(res.value)
else:
    # res.failure = {kind, name, attempts, last_error, elapsed_s}
    journal.record(res.failure)   # "same failure 5x today" becomes a query

The structured failure record is the quiet win: your agent can reason about why a call died ("429 three times, budget exhausted") instead of hallucinating a plausible answer over stack-trace soup.

This is exactly the resilient_call module in AgentOps Autopilot (MIT, stdlib only, 24/24 tests + a chaos gauntlet that injects each failure kind). If you'd rather paste it than rebuild it, the repo is free; the paid tiers add the templates and the runbook.

Next in the series: what to do when the LLM's JSON is truncated mid-string.

DEV Community