Your AI retries can make an outage worse. Here is a simple fix.

When an AI starts failing, the usual fix is to just retry. Set maxRetries: 3 and just move on with your life.

But look at what that really does! Each failed call becomes up to 4 attempts: the first one, plus 3 retries. If your provider is already returning 429s or timing out, you just sent it up to four times the traffic. Thats pretty annoying, especially for your users.

Backoff with jitter spreads those retries out in time but it does not reduce how many there are.

A retry budget asks a different question

A per call limit asks: how many times can this call retry?

A retry budget asks: across all calls to this provider, how much of the recent traffic is allowed to be retries?

Once enough calls have gone through a time window, and the share of them that were retries reaches your limit, further retries stop. The call fails with an error that says so, instead of adding to the pile.

Example

This is the config in VernLLM, an open source TypeScript library I maintain:

import OpenAI from 'openai';
import { VernLLM, isLLMError } from 'vern-llm';
import { fromOpenAI } from 'vern-llm/adapters';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, maxRetries: 0 });

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-6.1-sol',
  maxRetries: 3,
  retryBudget: { windowMs: 60_000, minCalls: 10, retryRatio: 0.1 },
});

try {
  await llm.call({ userContent: 'hello' });
} catch (err) {
  if (isLLMError(err) && err.code === 'retry_budget_exhausted') {
    console.warn('retries against this target are being throttled');
  }
}

What the three numbers mean:

• windowMs: how far back it looks. Here, the last 60 seconds.
• minCalls: how many calls it needs to see before judging. A quiet start does not trip it.
• retryRatio: the share of recent traffic that may be retries. Here, 10%.

Note maxRetries: 0 on the OpenAI client. That way the SDK does not retry on its own and only one layer is in charge.

Retry budget vs circuit breaker

They sound similar but ask different things. A circuit breaker asks whether the provider is healthy. A retry budget asks whether retrying is still worth the capacity it costs.

A provider can be perfectly healthy and still need a retry on almost every call. The breaker would stay closed, but the budget would still trip. They also act at different moments: the breaker is checked once before a call starts, the budget is checked at each retry.

Things to know

• The retry budget in VernLLM is off by default. You turn it on with the retryBudget option.
• By default the numbers live in one process. If you run several instances, the vern-llm-redis package can share them.
• Retry budgets are per provider target. A fallback provider needs its own.

Links

• Docs: https://vernllm.dev
• GitHub: https://github.com/LakBud/vernLLM

If you handle retries differently, I would like to hear how!