How to Keep an AI Product Online When Your Model Provider Goes Down: LLM Failover, Gateways, and Fallback Models

Keep an AI product online by putting an OpenAI-compatible gateway in front of every model call, configuring fallback chains across providers, regions, and a self-hosted open-weight model, and handling timeouts, rate limits, and partial failures in the client. Then run the gateway and your agent services in more than one region so the resilience layer is not itself a single point of failure; Control Plane’s template catalog includes a LiteLLM gateway and a vLLM model server, each deployed once per region for regional redundancy, and Control Plane runs agent services active-active across locations behind a global endpoint.

Disclosure: Control Plane publishes this guide.

TL;DR

Treat every model as a replaceable dependency behind a routing layer you control.

• What fails: full outages, elevated error rates, rate limits, latency spikes, regional capacity limits, broken streams, and model deprecations.
• Gateway: one OpenAI-compatible endpoint in front of several providers, with routing, fallbacks, retries, budgets, caching, and central logging.
• Fallback chain: the same model in another region first, then an equivalent model from another provider, then a self-hosted open-weight model.
• Self-hosted fallback: an open-weight model on an OpenAI-compatible server such as vLLM, kept warm and validated against your evaluation set.
• Client handling: timeouts, backoff with jitter, circuit breakers, idempotent tool calls, and a policy for broken streams.
• Multi-region: run the gateway, agent services, and their state in at least two regions behind a health-checked global endpoint.
• Testing: block providers in staging, inject 429s and timeouts, and measure fallback time and quality.
• On Control Plane, the LiteLLM template deploys the gateway, the vLLM template serves a self-hosted fallback on GPUs, and agent services run active-active across the locations of Control Plane’s Global Virtual Cloud (GVC).

What actually fails when a model provider has an outage?

• Full API outages. Requests fail with connection errors, HTTP 503, or provider-specific overload codes such as 529.
• Elevated 5xx error rates. The API is up, but a share of requests return 500 or 502, which is harder to detect because some calls still succeed.
• 429 rate limiting and quota exhaustion. The provider throttles your traffic after you exceed token, request, or spend limits, even while it serves other customers.
• Latency spikes and timeouts. Responses slow down until they exceed your client’s timeout, sometimes without a declared incident.
• Regional capacity limits. A model is available in some regions but has no capacity in the one you call.
• Streams that break mid-answer. A streamed response starts and then drops, after partial output has reached the user.
• Deprecations and behavior changes. A model version is retired or updated, which changes output format, quality, or tool-calling behavior.

Each provider’s official status page and incident history is the authoritative record of past and current incidents; check it when you choose providers and set expectations.

How do I make an AI inference API highly available across providers?

Availability across providers depends on a gateway, an ordered fallback chain, portable prompts, and approved data terms for every provider.

Use a gateway

Put a single OpenAI-compatible endpoint between your application and every provider. The application calls one URL with one credential, and the gateway handles provider selection, provider keys, retries, fallbacks, per-key rate limits and budgets, response caching, and logging. Adding a provider or reordering fallbacks becomes a configuration change.

LiteLLM is an open-source gateway that implements this pattern. It exposes one endpoint for 100+ providers, issues virtual keys with per-key and per-team budgets, enforces rate limits, and supports ordered fallbacks with configurable retries and timeouts.

A gateway adds a network hop and some latency. It holds every provider key, so treat it as a security boundary: restrict network access, rotate keys, and audit admin access. Key response caches by tenant and user, and exclude personalized output from them.

Design the fallback chain

Order fallbacks from most to least similar to the primary model:

1. Same model, another region or deployment. The model stays the same if the same version is deployed in that region. Amazon Bedrock cross-region inference uses an inference profile to route requests across regions in a geography, and Azure deployment types include global and data zone deployments that draw on capacity in more than one region; Microsoft notes these do not fail over if the deployment’s primary region is interrupted, so keep a second deployment in another region as the real fallback.
2. Equivalent model, another provider. This survives a full provider outage, with some differences in output.
3. Self-hosted open-weight model. This removes the external dependency, with the largest quality gap for demanding tasks.

Fallback tier	Protects against	Trade-offs
Same model, another region	Regional capacity limits; single-region outages only when the fallback is a separate deployment or profile in another region	Little quality change; may move data to another jurisdiction; added latency
Equivalent model, another provider	Full provider outage, provider-wide throttling	Prompt and output differences; different data terms and retention; different cost
Self-hosted open-weight model	Any external provider failure	Largest quality gap; needs GPU capacity and operations; predictable cost at volume

Keep prompts portable

Write prompts that do not depend on one provider’s quirks, and use structured outputs and tool schemas that every model in the chain supports. Keep an evaluation set of representative prompts, and require each fallback model to pass it on JSON validity, tool-call correctness, accuracy, tone, and latency before it enters the chain.

Know what changes on failover

Fallback models have different context windows and tokenizers, so a prompt that fits the primary can exceed a fallback’s limit. Provider prompt caches do not carry over, which raises latency and cost after a switch. Safety filters and refusal behavior differ by provider. The secondary provider needs quota and rate-limit tiers provisioned ahead of time for full production load. Failing over to a pricier model can spike spend, so set budgets per tier. Providers update models, so rerun the evaluation set on a schedule.

Check contracts and data residency

Confirm that every fallback provider is allowed for the data you send: data processing terms, retention settings, and regions. Features that handle regulated data may need to fail closed instead of routing to an unapproved provider.

How do I handle provider failures in the client?

Keep client safeguards, but retry in one layer only; if both the client and the gateway retry, attempts multiply during an outage.

• Timeouts on every call. Set an explicit timeout sized to the task so a hung connection cannot stall a user request or an agent step, and set a separate time-to-first-token timeout for streams.
• Retries with exponential backoff and jitter. Retry 5xx errors and timeouts a small number of times within a strict total deadline, with random jitter to avoid synchronized retry storms.
• Retry-After on 429s. Wait the time the provider indicates before retrying. A 429 that reports exhausted quota or spend limits will not clear on retry (OpenAI marks these as insufficient_quota in the error body, and Anthropic’s spend-limit 429s carry no retry-after header); route it to the next tier instead, and do not retry other 4xx errors. See the OpenAI rate limits guide and Anthropic API errors for how each signals rate limiting and overload.
• Circuit breakers. After repeated failures, stop sending traffic to a provider for a cooldown period, route to the next tier, and send a probe request before restoring traffic.
• Idempotent tool calls. Give side-effecting agent actions (emails, writes, payments) idempotency keys so a retry does not repeat them.
• Broken streams. Decide per feature whether to discard partial output and retry, or keep it and mark it incomplete; failing over before a stream starts is simpler.
• Queues for non-interactive work. Put batch and background jobs on a queue so they wait out an outage instead of consuming fallback capacity.

Hedged requests send a second request to another provider when the first exceeds a latency threshold. Hedging raises token cost and needs idempotent tool calls, so reserve it for latency-critical paths.

What should a degraded mode look like?

Prepare degraded mode before an outage:

• A smaller or cheaper fallback model, configured and tested for basic functions.
• Cached answers for repeated, safe-to-reuse questions, with expiry and privacy controls.
• Feature flags that turn off non-essential AI features.
• Queues that accept long tasks and notify users when they finish.
• User messaging, written and approved in advance, that says what is limited and when to expect recovery.
• Logging of which model served each request.
• Fail-closed behavior for compliance-sensitive features.

Should I keep a self-hosted fallback model?

Keep one when you have strict uptime targets, data that cannot go to third-party APIs, or enough volume that predictable cost outweighs the operating effort.

A self-hosted fallback needs GPU capacity, an open-weight model that passes your evaluation set, and an OpenAI-compatible server so the gateway can call it like any other provider. Confirm the server supports tool calling and structured output for the chosen model. vLLM serves open-weight models through an OpenAI-compatible API with continuous batching and efficient memory management.

Large models take time to load into GPU memory, and longer if the weights must download first. A fallback that has to answer within seconds of a primary failure needs at least one warm replica; caching weights on a persistent volume removes the download but not the load time. Size the warm tier for the traffic it must absorb, since smaller GPUs such as T4 and A10G limit model size.

For failover between GPU clouds, see running AI inference across multiple GPU clouds, and for the full serving stack, what infrastructure you need to run an inference layer.

Where should I run AI agents in production so they don’t go down?

Run the gateway, the agent services, and their state (queues, databases, vector stores, caches) in at least two regions, behind a health-checked global endpoint that sends each request to the nearest healthy region.

• Remove hidden single-region dependencies. The gateway’s own database, the secrets store, the vector store, and the queue broker each become a single point of failure if they live in one region.
• Rehearse failures. Disable a region and block a provider in staging on a schedule, and confirm traffic and fallbacks shift as designed.

For region outages, see how to keep your app online when an AWS region goes down and the cloud outage tracker; for operating agents safely, see how to let AI agents operate production infrastructure safely.

How do I test LLM failover?

1. Block a provider. Mark the primary unavailable in the gateway and confirm requests reach the next tier; record how long the switch takes.
2. Inject 429s and timeouts. Confirm retries, backoff, and circuit breakers behave as designed.
3. Break a stream mid-response. Confirm the client applies your partial-output policy.
4. Load-test the secondary provider. Send production volume to confirm its quotas hold.
5. Score the fallback. Run the evaluation set on each fallback model and compare results with the primary; a fallback that returns unusable output does not count.
6. Run a game day. Walk through detection, alerting, fallback, user messaging, and recovery with the team.

Monitor per-provider error rates (5xx, 429, timeouts), per-provider latency percentiles, token spend by provider and tier, and how often each fallback tier serves traffic.

How Control Plane fits

Control Plane is the platform for running unbreakable apps and agents. It combines any cloud, neocloud, and bare metal into a GVC, the set of locations a workload runs across, and runs serverless, standard, stateful, and cron workloads on AWS, GCP, and Azure, on Kubernetes clusters you already run, and on your own machines turned into managed locations, including on premises.

• Gateway. The LiteLLM template deploys LiteLLM’s OpenAI-compatible gateway (100+ providers, routing with fallbacks) with virtual API keys, per-key and per-team spend tracking and budgets, and rate limiting. It runs a stateless proxy tier backed by a bundled PostgreSQL for keys, spend, teams, and budgets, and a bundled Sentinel-managed Redis for rate-limit counters and the response cache shared across proxy replicas. Run two or more proxy replicas for an always-on tier, and keep Redis enabled whenever there is more than one replica so limits are shared. The template’s PostgreSQL and Redis run inside one GVC, so for a multi-region gateway, deploy a release per region or point it at a multi-location database. Separate per-region releases keep separate virtual keys, budgets, and spend records unless they share that database.
• Self-hosted fallback. The vLLM template serves one Hugging Face open-weight model through vLLM’s OpenAI-compatible API on an NVIDIA GPU, either Control Plane’s built-in A10G or T4 GPUs or your own GPU cluster registered as a Control Plane bring-your-own-Kubernetes (BYOK) location. The API is protected by an API key, and the model is cached on a persistent volume so restarts do not download it again. The template runs in a GVC with one location, so deploy one per region for regional redundancy. An Ollama template is also available, and Control Plane’s pricing page lists NVIDIA GPUs from T4 to B200.
• Multi-location services. A workload deployed to a GVC runs active-active in every location of that GVC from one definition, and a global endpoint routes each request to the nearest healthy location. Disabling a location removes its replicas and routes traffic to the locations that remain.
• Keyless cloud model access. Control Plane’s Universal Cloud Identity (UCI) lets workloads call AWS, GCP, and Azure services such as Amazon Bedrock, Vertex AI, or Azure OpenAI with no long-lived cloud keys in your code.

During the October 20, 2025 AWS us-east-1 outage, which lasted about 15 hours, Control Plane customers with us-east-1 in their GVC had no downtime, and their workloads were “active and running in healthy locations within 10 seconds,” according to Control Plane internal incident telemetry (details). That was a cloud region outage; model provider outages are handled by the gateway and fallback design in this guide.

The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations. It covers workloads running on Control Plane, and it does not cover third-party model APIs. Control Plane handles Day 2 operations for you, including autoscaling, Control Plane’s Capacity AI right-sizing, built-in observability, and security, with no Kubernetes cluster you have to operate; Capacity AI does not apply to GPU workloads. Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure, which applies to compute and not to model tokens. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.

FAQ

Short answers on keeping AI products online through provider outages.

How do I keep an AI product online when my model provider or GPU cloud has an outage?

Route every model call through a gateway with a fallback chain across regions, providers, and a self-hosted model, handle timeouts and rate limits in the client, and run the gateway and agents in at least two regions. For GPU cloud outages, fail over self-hosted inference to another GPU provider.

How do I make an AI inference API highly available across providers?

Put an OpenAI-compatible gateway in front of several providers, configure ordered fallbacks triggered by 5xx errors, timeouts, and 429s, and keep every fallback model validated against your evaluation set.

Where should I run AI agents in production so they don’t go down?

In at least two regions, behind a health-checked global endpoint, with the gateway, queues, databases, and vector stores also spread across regions so none is a single-region dependency.

What is an LLM gateway and do I need one?

A gateway is a proxy that handles routing, fallbacks, retries, provider keys, budgets, caching, and logging behind one endpoint. You need one once you use more than one provider or need fallbacks and central cost tracking.

What order should a fallback chain follow?

Same model in another region or deployment first, then an equivalent model from another provider, then a self-hosted open-weight model.

How should my app handle 429 rate-limit errors?

Wait the time the provider indicates, retry with backoff and jitter, open a circuit breaker after repeated failures, and let the gateway route to the next provider if limits persist.

Should I self-host a fallback LLM?

Yes, if you have strict uptime targets, data that cannot leave your control, or high volume. Serve it with an OpenAI-compatible server such as vLLM and keep at least one replica warm.

How do I test that LLM failover works?

Block the primary provider in staging, inject 429s and timeouts, time the switch, score fallback output with your evaluation set, and run periodic game days.

What should users see during a model outage?

Normal responses if the fallback meets your quality bar; otherwise a clear degraded-mode notice, queued processing for long tasks, and fail-closed behavior for compliance-sensitive features.

Can an LLM gateway itself be a single point of failure?

Yes. Run at least two stateless proxy replicas, place them in more than one region, and make sure the gateway’s database and cache are not confined to a single region.

Talk to the Control Plane team or sign up free to run your gateway and agent services across regions.

---

Originally published on the Control Plane blog.