Retry budgets by language: Python, Go, and JavaScript

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The retry-budget argument is language-independent: your retry multiplier is set by how you recover, not by how often you fail, and a per-call cap bounds a call but never a run or a fleet of workers. That's the theory, grounded in the $200 cost of nested caps that multiplied across layers. The theory doesn't tell you where in a Python decorator, a Go for loop, or a JavaScript promise chain the budget actually lives — and each language makes a different part of it easy to get wrong.

This post is the implementation note. Same budget, three runtimes, three traps.

The one invariant, restated for code

A retry budget is a bucket shared across everything that might retry, refilled slowly, that caps retries as a fraction of throughput rather than as an absolute count per call. Two properties matter and both are easy to lose in translation:

1. The budget is shared state. If every call site owns its own counter, you don't have a budget — you have N caps that multiply.
2. A denied retry is a real outcome. When the bucket is empty the call fails now, deliberately, instead of waiting for a slot. Retrying is a privilege the system can revoke, not a right the call holds.

Here's a minimal bucket, deliberately boring, that all three languages will share the semantics of:

budget = token bucket:
  capacity C, refill R tokens/sec
  on each SUCCESS: deposit ratio_bonus (e.g. +0.1 token)
  on each RETRY:   withdraw 1 token, or DENY if empty

The refill and the success-bonus are what make it a budget: retries are funded by the work that's succeeding. When failures spike and successes stop, the bucket drains and retries stop with it. That's the whole point. Now the three ways to wire it in.

Python: the decorator hides the budget from itself

tenacity is the reflex, and it's good, but the default shape teaches you the wrong lesson:

@retry(stop=stop_after_attempt(4), wait=wait_exponential())
def call_model(payload): ...

That decorator is a per-call cap. Every function it wraps gets its own independent four attempts. Wrap five call sites and your fleet's retry ceiling is 5×4 with nothing coordinating them — exactly the "local caps compose into a global disaster" failure. The decorator is so ergonomic that it hides the fact that there's no shared state anywhere.

The fix is to make the budget an object the retry predicate consults, not a constant baked into the decorator:

class RetryBudget:
    def __init__(self, capacity, refill_per_sec, bonus=0.1):
        self.tokens, self.cap = capacity, capacity
        self.refill, self.bonus = refill_per_sec, bonus
        self.t = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.cap, self.tokens + (now - self.t) * self.refill)
        self.t = now

    def allow_retry(self):
        with self.lock:
            self._refill()
            if self.tokens >= 1:
                self.tokens -= 1
                return True
            return False

    def on_success(self):
        with self.lock:
            self._refill()
            self.tokens = min(self.cap, self.tokens + self.bonus)

BUDGET = RetryBudget(capacity=40, refill_per_sec=1.0)

def call_model(payload):
    for attempt in range(5):
        try:
            r = _do(payload); BUDGET.on_success(); return r
        except TransientError:
            if attempt == 4 or not BUDGET.allow_retry():
                raise
            time.sleep(backoff(attempt))

The trap Python makes easy: forgetting the Lock. Under threads or a thread-pool executor the read-modify-write on self.tokens races, and a "budget of 40" quietly lets through many more retries under exactly the load spike you built it for. Under asyncio the lock becomes an asyncio.Lock and the sleep an await asyncio.sleep, but the shape is identical: the budget is one object, shared, guarded.

---

Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article