A model route can still return HTTP 200 while your AI product is already failing.
Latency rises.
Structured output starts failing validation.
Tool calls become unreliable.
Fallbacks get expensive.
Users wait while the system keeps sending new requests to the same degraded route.
That is not a retry problem.
It is a circuit breaker problem.
Retries are not always recovery
Retries are useful when a failure is temporary.
A network connection drops. A provider returns a short-lived 429. A request times out before it reaches the model.
But retries become harmful when the route itself is unhealthy.
Imagine a coding workflow that sends a request to its primary model route three times:
- The first request times out.
- The second request returns invalid JSON.
- The third request takes too long and misses the user-facing deadline.
Sending a fourth request to the same route is not reliability.
It is just more waiting, more cost, and less time for a fallback.
What a circuit breaker does
A circuit breaker gives a route three states:
- Closed: requests can use the route normally.
- Open: new requests avoid the route because it is currently unhealthy.
- Half-open: limited test traffic checks whether the route has recovered.
The goal is simple: stop a local route failure from becoming a product-wide failure.
When the breaker opens, the application can send eligible work to an approved fallback route, queue non-urgent work, or return a clear degraded response.
A 200 response is not enough
For AI applications, an API success response does not always mean user success.
A route may be unhealthy when it produces:
- responses that exceed the workflow time budget
- JSON that fails schema validation
- incomplete tool calls
- low-quality RAG answers
- repeated safety or content-filter failures
- expensive fallback chains
- a falling successful-task rate
A useful circuit breaker should watch the outcomes that matter to the workflow, not only provider uptime.
Scope the breaker carefully
Do not open one global circuit breaker for an entire provider.
A problem may affect only:
- one model
- one region
- one API route
- one response format
- one tool-calling workflow
- one model configuration
For example, a route that works well for support chat may be failing only for structured extraction.
Breaking too broadly removes healthy capacity. Breaking too narrowly misses the actual failure pattern.
The right scope is usually close to the real unit of risk:
provider + model + region + workflow + configuration
Define unhealthy behavior before an incident
A circuit breaker needs explicit conditions.
For example:
text
Open this route when:
- 5 requests fail within 60 seconds, or
- the error rate exceeds 25% across the last 20 requests, or
- structured-output validation fails 4 times in 10 requests, or
- median latency exceeds the remaining workflow budget
The exact thresholds will differ by product.
A real-time support chatbot needs tight latency limits. A background document-processing workflow may tolerate a slower route but care more about completion and cost.
Use retries and circuit breakers together
A practical request policy may look like this:
Send the request to the primary route.
Retry only when the error is likely transient and time remains.
Open the circuit breaker when failure thresholds are crossed.
Route new work to an approved fallback.
Send limited half-open traffic later to test recovery.
The important detail is that a fallback needs time to work.
If retries consume the entire user-facing deadline, the fallback exists only on paper.
Record why the breaker opened
When a route is disabled automatically, log the reason.
Capture the workflow, provider, model, configuration, error class, latency, validation result, retry count, fallback decision, and final task outcome.
Without this record, teams know a route was avoided but cannot tell whether the cause was provider instability, a prompt change, a schema issue, or a bad deployment.
Final thought
More model choices do not automatically make an AI product more reliable.
Reliable multi-model systems know when to retry, when to stop, and when to protect users from a route that is already failing.
Top comments (0)