Your LLM provider goes down. Every call in your app still runs its full retry loop: three attempts, backoff between them, then a failure. Nothing you send gets through, and every retry adds load to a provider that is trying to recover.
Retries are great for a blip. During an outage they make things worse. The fix is a circuit breaker: after enough failures in a row, stop calling the provider, fail fast, and check back later with a single trial call.
This post shows what that saves with real numbers, and how to add an LLM circuit breaker in TypeScript with VernLLM.
Retries during an outage
I sent 20 calls, one after another, to a scripted provider that returned 503 every time. Each call had maxRetries: 2, so up to 3 attempts.
| Retries only | With a circuit breaker | |
|---|---|---|
| Requests sent | 60 | 15 |
| Calls that failed fast | 0 | 15 |
| Circuit state at the end | none | open |
Without a breaker, all 20 calls spent their 3 attempts: 60 requests into a dead provider. With threshold: 5, the first 5 calls failed normally, the circuit opened, and the other 15 calls failed instantly without sending anything.
Failing fast is the point. Your users get an error, or better, a fallback answer, right away instead of after a full retry loop.
Adding a circuit breaker
npm i vern-llm openai
import OpenAI from 'openai';
import { VernLLM } from 'vern-llm';
import { fromOpenAI } from 'vern-llm/adapters';
const llm = new VernLLM({
client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
model: 'gpt-6.2-sol',
maxRetries: 2,
circuitBreaker: { threshold: 5, cooldownMs: 30_000 }, // or `true` for defaults
});
llm.getCircuitState(); // 'closed' | 'open' | 'half-open'
The breaker has three states:
- Closed. Calls go through normally. Failures are counted.
-
Open. After
thresholdfailures in a row, every call fails immediately with acircuit_openerror. Nothing is sent. -
Half open. After
cooldownMs, one call is let through as a trial. If it succeeds, the circuit closes. If it fails, the circuit opens again and the cooldown restarts.
What counts as a failure
A breaker should track provider health, not whether you liked the answer. VernLLM counts timeouts, 5xx responses, 408, 425 and 429, network errors and empty responses.
It does not count:
- JSON that failed to parse, or failed your schema. The provider answered.
- Your own invalid parameters, or a call you cancelled.
- Other 4xx responses. They describe one caller's request, and counting them would let one bad caller open the circuit for everyone.
- Local rate limit rejections that never reached the provider.
If a provider returns 200 with a useless body, you can turn that into a failure with detectSoftFailure, and it counts toward the breaker like a real outage would.
Back off between trials
A provider that is down for 20 minutes does not need a trial call every 30 seconds. cooldownBackoff grows the cooldown each time a trial fails:
circuitBreaker: {
threshold: 5,
cooldownMs: 30_000,
cooldownBackoff: { multiplier: 2, maxMs: 5 * 60_000 },
},
The growth is jittered, so many instances do not all probe at the same moment. It resets as soon as the provider recovers.
Failures that are not back to back
Some providers fail every other call instead of all at once. A rolling window catches that:
circuitBreaker: {
cooldownMs: 30_000,
tripping: { kind: 'rolling', windowMs: 60_000, minCalls: 20, failureRatio: 0.5 },
},
This opens the circuit once at least 20 calls in the last minute have a failure rate of 50% or more.
One model down, not the whole provider
If one instance serves several models, isolateByModel: true gives each model its own circuit. A broken model does not block the healthy ones on the same provider.
Get alerted
circuitBreaker: {
threshold: 5,
cooldownMs: 30_000,
onStateChange: (from, to, consecutiveFailures, model) => {
if (to === 'open') alerting.page(`circuit opened for ${model}`);
},
},
Pair it with fallback
A circuit breaker tells you a provider is down. Fallback does something about it. With a breaker on each target, an open circuit moves the call to your next model right away, without spending any attempts on the one that is down:
const llm = new VernLLM({
client: fromOpenAI(openai),
model: 'gpt-6.2-sol',
circuitBreaker: true,
fallback: {
client: fromGemini(gemini),
model: 'gemini-4-neon',
circuitBreaker: true,
},
});
Each target keeps its own breaker, so one provider going down never trips another. Running several processes? Pass a shared breaker adapter, such as the Redis backed one in vern-llm-redis, so every instance agrees on whether a provider is down.
Wrap up
Retries fix blips. A circuit breaker handles outages: stop calling, fail fast, check back with one trial, and hand the call to your fallback in the meantime.
npm i vern-llm
Docs: Circuit breaker
Top comments (0)