Last month one of my automations sent the same customer email three times in four minutes. The tool call had timed out, the retry layer did exactly what it was told, and the customer got a copy for each attempt. Nothing in the logs looked like an error — every attempt was logged as an honest timeout, and every attempt had actually done its job.
That is the part of agent reliability nobody warns you about. Retries are the first thing you add and the last thing you reason about. They fix the failure you can see (a dropped request) and create the one you cannot (a side effect that happened twice, in a system that has no idea it happened once).
This post is a small resilience layer for tool calls in pure standard-library Python: exponential backoff with jitter, a circuit breaker, an idempotency key that survives retries, and a dead-letter queue that parks work instead of dropping it. Every number below comes from a measured run on a virtual clock, so you can reproduce all of it on a laptop with no API key and no network.
Two bugs that "just add retries" creates
The lost response. The request reached the service, the side effect committed, and the response was lost on the way back. Your code sees a timeout. The truth is a success plus a mystery. Retrying is now the most dangerous thing you can do, because the operation already happened — a second charge, a second email, a second PR comment.
The herd. A dependency starts failing for everyone at once, and every worker waits the same 2 seconds before trying again. They come back in lockstep, hammer the service at exactly the moment it is weakest, and turn a short wobble into a long outage. Fixed backoff is a synchronisation mechanism you did not mean to build.
Both are addressable with patterns that are older than LLMs — but only if the retry layer is designed as one unit with the side effect, not bolted on around it.
The four moving parts
-
RetryPolicy— exponential delay, capped, with jitter, so retries spread out instead of arriving together. -
CircuitBreaker— closed → open → half-open. When the dependency is down, fail fast instead of queueing. -
IdempotencyStore— one key per logical operation, reused by every attempt, so a retry can never repeat the effect. -
DeadLetterQueue— a JSONL parking lot for operations that exhausted their attempts, with areplay()to drain it once the dependency is healthy again.
A ToolExecutor wires all four together and emits one structured JSON event per attempt, so a retry storm is visible in a log rather than inferred from a billing surprise.
1. Backoff that does not synchronise
class Jitter(str, Enum):
NONE = "none"
FULL = "full" # uniform(0, cap) - best at spreading a herd
EQUAL = "equal" # cap/2 + uniform(0, cap/2)
@dataclass
class RetryPolicy:
max_attempts: int = 4
base_delay: float = 0.2
factor: float = 2.0
max_delay: float = 8.0
jitter: Jitter = Jitter.FULL
seed: Optional[int] = None
def raw_delay(self, attempt: int) -> float:
return min(self.max_delay, self.base_delay * (self.factor ** (attempt - 1)))
def delay(self, attempt: int) -> float:
cap = self.raw_delay(attempt)
if self.jitter is Jitter.NONE:
return cap
if self.jitter is Jitter.FULL:
return self._rng.uniform(0, cap)
return cap / 2 + self._rng.uniform(0, cap / 2)
The cap matters more than the curve. Without max_delay, attempt 6 at base=0.2, factor=2 waits 6.4s and attempt 8 waits 25.6s — inside a request a human is waiting on. I keep max_delay at 3–8s for interactive tools and let the dead-letter queue absorb the rest.
seed exists for the same reason the clock is injectable: a test that cannot reproduce the delay distribution is a test you will delete the second it flakes.
Here is what jitter buys, measured on 50 workers that all fail their first attempt and retry once with a 400ms base delay:
| jitter | distinct retry times | retries in the busiest 25ms | buckets used |
|---|---|---|---|
| none | 1 | 50 | 1 |
| equal | 50 | 10 | 8 |
| full | 50 | 6 | 16 |
Without jitter, all 50 arrive in the same 25ms window. With full jitter the peak is 6 and the load is spread over 16 windows. That is the difference between a dependency recovering and a dependency being retried to death.
Full jitter also wins on the tail without changing the success rate at all. Five hundred operations against a downstream failing 35% of the time per attempt, 120ms latency, up to 4 attempts:
| policy | success | failed ops | downstream calls | p50 | p95 | max |
|---|---|---|---|---|---|---|
| no retry | 63.2% | 184 | 500 | 120ms | 120ms | 120ms |
| retry, no jitter | 98.2% | 9 | 771 | 120ms | 960ms | 1,880ms |
| retry, equal jitter | 98.2% | 9 | 771 | 120ms | 937ms | 1,729ms |
| retry, full jitter | 98.2% | 9 | 771 | 120ms | 840ms | 1,577ms |
Retries take success from 63% to 98% at a cost of 54% more downstream calls. Jitter does not touch success or call volume — it only shortens the tail (p95 −12%, max −16%), which is exactly what you want from it: better worst case for the same budget.
2. A breaker, so an outage does not become a backlog
A retry policy alone is a machine for making a bad situation worse: every queued operation keeps calling a service that is already refusing connections, and each one holds a worker for the full backoff window.
def allow(self) -> bool:
if self.state is BreakerState.CLOSED:
return True
if self.state is BreakerState.OPEN:
if self.clock.monotonic() - self.opened_at >= self.cooldown:
self._set_state(BreakerState.HALF_OPEN)
self.half_open_calls = 0
else:
self.rejected += 1
return False
if self.half_open_calls >= self.half_open_max_calls:
self.rejected += 1
return False
self.half_open_calls += 1
return True
Three details that are easy to get wrong:
- Count consecutive failures, and reset on any success. A single stray 500 during a healthy hour should never contribute to opening the breaker.
-
Half-open needs a probe budget. After
cooldown, let a small fixed number of calls through (I use 2). A failed probe reopens immediately; successes close it. Without a cap, "half-open" becomes "fully open with extra steps" the moment traffic is high. -
One breaker per tool, not per process. My
charge_cardtool and mysearchtool have nothing in common, and a search outage must not stop payments.
3. One key per operation, not per attempt
This is the part that fixes the double email, and it is a design decision about identity, not about retrying.
def make_idempotency_key(tool: str, payload: Any) -> str:
blob = json.dumps({"tool": tool, "payload": payload}, sort_keys=True, default=str)
return hashlib.sha256(blob.encode()).hexdigest()[:32]
The key is derived from the tool name and the payload, so the same logical operation gets the same key on attempt 1 and attempt 4. The executor reserves it before the first call, hands the same key to every attempt, and only marks it complete on success. The downstream — or a local store, if you own both sides — records the key atomically with the effect. A retry arriving after a committed-but-lost response is then recognised as a duplicate and replays the stored answer instead of repeating the write.
Measured on 300 charge operations where 30% of responses are lost after the effect commits:
| key strategy | effects written | extra effects | duplicate attempts blocked |
|---|---|---|---|
| one key per operation | 300 | 0 | 125 |
| new key per attempt | 425 | +125 (+41.7%) | 0 |
Same code, same retry count, same 298 successes — the only difference is whether the key identifies the operation or the attempt. A uuid4() generated inside the retry loop is the single most expensive line in a retry layer, because it turns every retry into a new piece of work.
Two corollaries I learned the hard way:
- The key has to travel with the request. If the downstream does not accept an idempotency key and you do not control it, then retrying a non-idempotent verb is not safe at any backoff setting. Retry only reads, or check state afterwards.
- Never put the attempt number, timestamp, or a random nonce in the payload before hashing. It looks like harmless metadata; it silently makes the key per-attempt and reintroduces the bug.
4. Park it, do not drop it
An operation that exhausted its attempts is not garbage. It is work a person asked for, and the correct outcome is "delayed", not "lost".
def _dead_letter(self, key, payload, attempts, reason):
self.dlq.add({"tool": self.tool, "key": key, "payload": payload,
"attempts": attempts, "reason": reason})
def replay(executor: ToolExecutor, entries: list[dict]) -> int:
recovered = 0
for entry in entries:
res = executor.call(payload=entry["payload"], key=entry["key"])
if res.ok:
recovered += 1
return recovered
Replay passes the original key back in, which is what makes replay safe: if the operation actually landed before it was parked, the sink recognises the key and no second effect occurs. A dead-letter queue that replays with fresh keys is a duplicate generator with a nice name.
The executor loop
def call(self, payload=None, key=None, new_key_per_attempt=False) -> CallResult:
self.calls += 1
base_key = key or make_idempotency_key(self.tool, payload)
cached = self.store.get(base_key)
if cached is not None: # same operation, already done
self.dedup_hits += 1
return CallResult(key=base_key, ok=True, result=cached, deduplicated=True)
self.store.claim(base_key)
last_error = None
for attempt in range(1, self.policy.max_attempts + 1):
if attempt > 1:
self.clock.sleep(self.policy.delay(attempt - 1))
if not self.breaker.allow(): # fail fast, park the work
self.fast_failures += 1
self._dead_letter(base_key, payload, attempt, "circuit_open")
return CallResult(key=base_key, ok=False, fast_failed=True,
error="circuit_open", attempts=attempt)
try:
result = self.fn(key=base_key, attempt=attempt)
except TerminalError as exc: # 400s, auth, bad input: no retry
self.breaker.record_failure()
self._dead_letter(base_key, payload, attempt, f"terminal:{exc}")
return CallResult(key=base_key, ok=False, error=f"terminal:{exc}")
except RetryableError as exc: # 429, 5xx, timeouts: retry
last_error = str(exc)
self.breaker.record_failure()
continue
else:
self.breaker.record_success()
self.store.complete(base_key, result)
return CallResult(key=base_key, ok=True, result=result, attempts=attempt)
self._dead_letter(base_key, payload, self.policy.max_attempts, f"exhausted:{last_error}")
return CallResult(key=base_key, ok=False, error=f"exhausted:{last_error}")
The split between RetryableError and TerminalError is the highest-leverage line in the file. Retrying a 400 six times costs seconds and returns six identical 400s. Classify at the adapter — the one place that sees the HTTP status — not at the call site, where every tool would have to remember.
Measured end to end: a 30-second outage
Two hundred operations, one issued every 250ms, against a dependency that is dead for the first 30 seconds. Same policy in both runs, up to 4 attempts, full jitter, 100ms per call.
| configuration | succeeded live | fast-failed | parked | downstream calls | mean wait/op | max wait/op |
|---|---|---|---|---|---|---|
| retry only | 173 | 0 | 27 | 333 | 250ms | 1,620ms |
| with breaker | 72 | 127 | 128 | 229 | 60ms | 1,460ms |
The breaker run is not "better" at succeeding during the outage — it gives up locally and parks the work. Its value is the wait time: 60ms average instead of 250ms, and 31% fewer calls into a service that is refusing every one of them. Every parked operation was replayed afterwards and 200/200 were accounted for (72 live + 128 recovered), with the breaker walking open at 1.6s → half_open at 16.8s → open again at 16.9s → half_open at 32.0s → closed at 32.4s as the dependency came back at 30s.
That transition log is the second thing the breaker buys you: a timeline of when the dependency was down, produced without a monitoring stack.
Testable because time is injectable
Nothing above is testable if the tests actually sleep. Every component takes a clock, and the fake one advances virtual time:
class FakeClock(Clock):
def __init__(self):
self.now = 0.0
self.sleeps = []
def monotonic(self) -> float:
return self.now
def sleep(self, seconds: float) -> None:
if seconds > 0:
self.sleeps.append(seconds)
self.now += seconds
The suite is 23 unittest cases and runs in 4 milliseconds: backoff curve and its cap, jitter bounds under 50 draws, a breaker that opens, half-opens, rejects the third probe, and reopens on a failed probe, a key that ignores payload ordering, an expired attempt that lands exactly one effect, an open breaker that never touches the downstream, and a full outage where every parked operation comes back through replay. All four experiments in this post use the same virtual clock, so the numbers above are reproducible with python3 experiments.py.
What I would do differently
- Give the whole call a deadline, not just an attempt budget. Four attempts with growing backoff can outlive the caller. Stop retrying when the remaining budget is smaller than the next delay.
- Make the dead-letter queue replay idempotent by construction — original keys, one at a time, with a cap. A bulk replay after an outage is a herd of its own.
-
Log the outcome, not the error. One JSON line per attempt with
tool,key,attempt,latency_ms,breaker,gave_upturns "why was this slow on Tuesday" into ajqquery. - Do not retry the model. If the agent's plan was wrong, a second call with the same prompt produces the same wrong plan more expensively. Retry the tool, then re-plan.
- Treat a retried tool call as a new transaction only if you can name its identity. No key, no retry.
The pattern is not new — it is the same one payment APIs have used for a decade. What changed is that now a language model decides when to call the tool, dozens of times per task, without knowing whether the last call landed. The layer that knows is the one you wrote around it.
Run it yourself
resilience.py # policy, breaker, idempotency, DLQ, executor (~310 code lines)
experiments.py # the four measured experiments (virtual clock, no network)
tests/test_resilience.py # 23 unittest cases, ~4ms
reports/measurements.json # every number in this post
python3 -m unittest discover -s tests
python3 experiments.py
Standard library only. No key, no account, no bill.
Further Resources
The full resilience module, the experiment harness, the test suite, and the JSONL event schema I use in production are collected in my workflow pack: The Complete 2026 AI Side-Hustle Blueprint: Zero-Cost Prompts & Workflows. Everything in this post runs without it.
Note: this post contains one link to my own workflow pack. The code above is free, standalone, and does not require it.
Top comments (2)
The key section is the load-bearing part of this, and your +41.7% number is the cleanest demonstration of the point I know. One case a payload-derived key cannot express, from running this in production:
A derived key makes the conflict case invisible. If a caller retries with the same operation in mind but changed parameters (the agent re-planned between attempts, the payload mutated), the derived key changes with the payload and the retry silently becomes new work. And the mirror case: two genuinely distinct operations with the same tool and payload, the same customer charged the same amount twice on purpose, collapse into one. Mint the key before the first call instead, bind it where the effect is recorded to the digest of the parameters first seen, and the three cases separate: same key and same digest replays the saved result, same key and different digest is refused as a conflict, new key runs. The refusal matters as much as the replay: it is the only one of the three that tells the caller its retry changed the operation.
The second thing production adds is who can check. Your store knows the effect count is 1; a stranger holding only the key does not. When the recorded effect carries the digest of its canonical bytes as a receipt, a fresh client can recover the original record on the key alone, a day later in our delayed-leg test, and confirm the effect count is still 1 without trusting the store. The planted job, the recovery, and the refused changed-parameter retry are all in one public receipt: zambo.dev/run/e2568939-b651-4150-b...
"No key, no retry" is the right closing rule. I would add: no receipt, no proof it worked.
The honest-timeout log is the scariest part — every record says success while the side effect ran three times. We encode the same lesson as a rule on my side: before repeating an action that can't be undone, establish whether it already took effect. An idempotency key that survives retries is the mechanical form of that rule. Clean, stdlib-only implementation, too.