Retry context: building observability into retry decisions

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The fleet-patterns post explained the patterns that keep a fleet from drowning in its own retries. The when-to-give-up post showed when to retry at all — which layer gets to make the decision, and why most code doesn't ask the right questions. (Both of those build on the cost model and a real incident that motivated it.) This post is about what you need to know to make that decision trustworthy in the first place.

The core problem: a retry attempt looks the same whether it's recovering from a transient network hiccup or whether you're trapped in a loop burning your budget on a permanent failure. Without visibility into what's happening, your retry logic is guessing.

The foundation: idempotency keys

An idempotency key is a unique token attached to a request that tells the system "if you've seen this before, return the cached result instead of replaying the work." It's not directly a retry concern — it's a consequence of a deeper rule: retries are only safe when the operation is idempotent, and idempotency is only verifiable if the operation is labeled with a stable identity.

import uuid, time

class RetryableCall:
    def __init__(self, operation_name, user_id, resource_id):
        self.idempotency_key = f"{user_id}:{resource_id}:{operation_name}:{int(time.time() * 1000)}"
        # ^ stable within a retry window (e.g., one second), fresh across retries >1s apart
        self.attempt = 0

    def call(self, client):
        self.attempt += 1
        headers = {"X-Idempotency-Key": self.idempotency_key}
        return client.do_work(headers=headers)

The key construction matters. If your idempotency key is just a UUID per task (not per attempt), all retries of the same task share it — which is what you want. If it's per-millisecond, two retries 100ms apart get different keys, and the server will double-process. The stability window should match your retry window: if you retry up to 5 seconds, the key should be stable for 5+ seconds.

This is how you tell the server "this is attempt 3, but if you cached the result from attempt 1, use that." Without it, your retry is a replay: the server genuinely executes the operation twice.

---

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article