DEV Community

Cover image for LLM Circuit Breaker: Stop Retrying a Provider That's Down
LakBud
LakBud

Posted on AI-assisted

LLM Circuit Breaker: Stop Retrying a Provider That's Down

Your LLM provider goes down. Every call in your app still runs its full retry loop: three attempts, backoff between them, then a failure. Nothing you send gets through, and every retry adds load to a provider that is trying to recover.

Retries are great for a blip. During an outage they make things worse. The fix is a circuit breaker: after enough failures in a row, stop calling the provider, fail fast, and check back later with a single trial call.

This post shows what that saves with real numbers, and how to add an LLM circuit breaker in TypeScript with VernLLM.

Retries during an outage

I sent 20 calls, one after another, to a scripted provider that returned 503 every time. Each call had maxRetries: 2, so up to 3 attempts.

Retries only With a circuit breaker
Requests sent 60 15
Calls that failed fast 0 15
Circuit state at the end none open

Without a breaker, all 20 calls spent their 3 attempts: 60 requests into a dead provider. With threshold: 5, the first 5 calls failed normally, the circuit opened, and the other 15 calls failed instantly without sending anything.

Failing fast is the point. Your users get an error, or better, a fallback answer, right away instead of after a full retry loop.

Adding a circuit breaker

npm i vern-llm openai
Enter fullscreen mode Exit fullscreen mode
import OpenAI from 'openai';
import { VernLLM } from 'vern-llm';
import { fromOpenAI } from 'vern-llm/adapters';

const llm = new VernLLM({
  client: fromOpenAI(new OpenAI({ apiKey: process.env.OPENAI_API_KEY })),
  model: 'gpt-6.2-sol',
  maxRetries: 2,
  circuitBreaker: { threshold: 5, cooldownMs: 30_000 }, // or `true` for defaults
});

llm.getCircuitState(); // 'closed' | 'open' | 'half-open'
Enter fullscreen mode Exit fullscreen mode

The breaker has three states:

  1. Closed. Calls go through normally. Failures are counted.
  2. Open. After threshold failures in a row, every call fails immediately with a circuit_open error. Nothing is sent.
  3. Half open. After cooldownMs, one call is let through as a trial. If it succeeds, the circuit closes. If it fails, the circuit opens again and the cooldown restarts.

What counts as a failure

A breaker should track provider health, not whether you liked the answer. VernLLM counts timeouts, 5xx responses, 408, 425 and 429, network errors and empty responses.

It does not count:

  • JSON that failed to parse, or failed your schema. The provider answered.
  • Your own invalid parameters, or a call you cancelled.
  • Other 4xx responses. They describe one caller's request, and counting them would let one bad caller open the circuit for everyone.
  • Local rate limit rejections that never reached the provider.

If a provider returns 200 with a useless body, you can turn that into a failure with detectSoftFailure, and it counts toward the breaker like a real outage would.

Back off between trials

A provider that is down for 20 minutes does not need a trial call every 30 seconds. cooldownBackoff grows the cooldown each time a trial fails:

circuitBreaker: {
  threshold: 5,
  cooldownMs: 30_000,
  cooldownBackoff: { multiplier: 2, maxMs: 5 * 60_000 },
},
Enter fullscreen mode Exit fullscreen mode

The growth is jittered, so many instances do not all probe at the same moment. It resets as soon as the provider recovers.

Failures that are not back to back

Some providers fail every other call instead of all at once. A rolling window catches that:

circuitBreaker: {
  cooldownMs: 30_000,
  tripping: { kind: 'rolling', windowMs: 60_000, minCalls: 20, failureRatio: 0.5 },
},
Enter fullscreen mode Exit fullscreen mode

This opens the circuit once at least 20 calls in the last minute have a failure rate of 50% or more.

One model down, not the whole provider

If one instance serves several models, isolateByModel: true gives each model its own circuit. A broken model does not block the healthy ones on the same provider.

Get alerted

circuitBreaker: {
  threshold: 5,
  cooldownMs: 30_000,
  onStateChange: (from, to, consecutiveFailures, model) => {
    if (to === 'open') alerting.page(`circuit opened for ${model}`);
  },
},
Enter fullscreen mode Exit fullscreen mode

Pair it with fallback

A circuit breaker tells you a provider is down. Fallback does something about it. With a breaker on each target, an open circuit moves the call to your next model right away, without spending any attempts on the one that is down:

const llm = new VernLLM({
  client: fromOpenAI(openai),
  model: 'gpt-6.2-sol',
  circuitBreaker: true,
  fallback: {
    client: fromGemini(gemini),
    model: 'gemini-4-neon',
    circuitBreaker: true,
  },
});
Enter fullscreen mode Exit fullscreen mode

Each target keeps its own breaker, so one provider going down never trips another. Running several processes? Pass a shared breaker adapter, such as the Redis backed one in vern-llm-redis, so every instance agrees on whether a provider is down.

Wrap up

Retries fix blips. A circuit breaker handles outages: stop calling, fail fast, check back with one trial, and hand the call to your fallback in the meantime.

npm i vern-llm
Enter fullscreen mode Exit fullscreen mode

Docs: Circuit breaker

Top comments (0)