What Happens When Your LLM Provider Goes Down? Fallback Design for Indie SaaS
Your AI feature worked perfectly in the demo, then one Tuesday your provider returns 503s for forty minutes and every model call becomes a spinner. The uncomfortable truth for a one-person company is that you cannot control provider uptime — you can only control what your product does during the gap. This guide is about that gap: the four failure modes that actually hit small teams, the retry and fallback layers that fix them, and how to build all of it without doubling your bill.
Quick Answer
Assume every provider will fail intermittently, and design for it in four layers: strict timeouts, retries with jittered backoff for idempotent calls, a two-provider fallback chain behind a gateway abstraction, and a graceful degradation path that keeps the rest of your product working. Two providers with different infrastructure is usually enough for an indie SaaS. The goal is not 100% model availability — it is that a provider outage degrades one feature instead of taking down the product.
What Actually Breaks When a Provider Goes Down
The failure is rarely a clean outage. What small teams see in practice looks more like this:
- Partial degradation: one model or one region fails while the rest of the API stays healthy. Your error rate goes from 0% to 12%, which is worse than a full outage because it hides in your logs.
- Latency spikes before errors: requests hang for 60–120 seconds before timing out. Without an explicit client timeout, your web workers stay occupied and unrelated endpoints slow down too.
- Rate-limit storms: when one provider gets slow, retries pile up and you trip your own quota — sometimes on the provider that was still working. Silent quality regressions: providers route you to a different model version during capacity pressure. The call succeeds, the answer gets worse, and nothing in your monitoring notices.
Notice that only the first of those is really "the provider is down". The rest are your own architecture reacting badly. That is good news, because those you can fix.
Failure Mode to Fix, Mapped
| Failure mode | What it looks like | Cheapest effective fix |
|---|---|---|
| Hung request | Spinner never resolves; worker occupied | Hard client timeout (15–30s) plus a user-visible timeout state |
| Transient 5xx / 429 | Intermittent errors, clustered | 2–3 retries with exponential backoff and jitter, idempotency keys |
| Regional or model outage | One model fails, others fine | Fallback chain across two providers behind one interface |
| Capacity-driven rerouting | Answers get quietly worse | Log model version per response; alert on version drift |
| Cost blowup during retries | Bill spikes in the same hour as the incident | Retry budget per request; cap total attempts per user action |
Layer 1 — Timeouts and Retries That Do Not Make It Worse
Start with the boring part. Every model call needs an explicit timeout — 15 to 30 seconds for interactive features, longer only for background jobs. Most SDK defaults are generous or absent, and a missing timeout is what turns a provider hiccup into a product outage.
Then make retries disciplined. Retry only on connection errors, timeouts, 429, and 5xx. Never retry a 400 or a content-policy rejection: the second attempt will fail the same way and you pay for it twice. Use exponential backoff with jitter, cap attempts at two or three, and set a total retry budget per user action so one impatient click cannot generate a burst of ten calls.
The subtle part is idempotency. If your feature charges a credit, writes a row, or sends an email, a retry must not duplicate the side effect. Generate an idempotency key before the first attempt and pass it through every retry.
Layer 2 — The Fallback Chain
Fallback means: if the primary provider fails after its retries, send the request to a second provider. For that to work, the call has to go through one interface rather than being wired directly into your feature code. An LLM gateway — whether a self-hosted library or a hosted service — is what gives you that seam, and it is the single highest-leverage refactor for reliability. Our comparison of production LLM gateways covers the main options.
Two design decisions matter more than the tool you pick:
- Pair providers with different infrastructure. Two endpoints resold from the same upstream will fail together. Prefer providers with genuinely independent capacity — ideally one hosted API and one open-weight endpoint you could self-host in an emergency. Do not silently swap models. If your prompt is tuned for one model, falling back to a much smaller one produces confidently wrong output. Either pin a comparable model, or degrade the feature explicitly ("summary unavailable, showing raw notes") instead of pretending.
A circuit breaker is the natural third piece: after N consecutive failures, stop sending traffic to the unhealthy provider for a cooldown window and go straight to the fallback. This prevents every request from paying the primary's timeout before failing over.
Layer 3 — Degrade Gracefully Instead of Failing Loudly
The most valuable design decision is deciding in advance which features are worth waiting for and which are not. A drafting assistant can queue work and email the user when it is done. A search box cannot. Write that hierarchy down:
- Blocking features: show an honest error with a retry button. Never leave an infinite spinner.
- Enhancement features: fall back to the non-AI path. If summarization is unavailable, show the document. Users forgive a missing convenience; they do not forgive a broken page. Background features: enqueue the job, acknowledge receipt, and process when capacity returns. This is where an outage becomes almost invisible to users.
This is also where your cost discipline pays off. Queuing instead of hammering a struggling provider keeps your retry volume — and your bill — proportional to real demand. If you have already worked on keeping LLM costs predictable, an outage is the moment that discipline shows up.
Layer 4 — Make the Failure Visible
You cannot fall back from a failure you never see. At minimum, log per request: provider, model version, latency, token counts, retry count, and whether a fallback was used. Then alert on four things — error rate above your normal baseline, p95 latency, fallback rate, and model version changes. Fallback rate is the one teams forget, and it is the earliest signal that your primary is degrading.
If you already run observability for your AI calls, this is a dashboard addition, not a new system. If you do not, start with structured logs and one alert on error rate; that catches the majority of real incidents.
The Bottom Line
Provider outages are not an edge case, they are a scheduled event you have not been told about yet. For an indie SaaS, the winning posture is unglamorous: timeouts everywhere, disciplined retries with idempotency, two independent providers behind one interface, a circuit breaker, and a written degradation policy per feature. Build the first two this week — they are a few hours of work and they remove most of the blast radius. Add the fallback chain when you next touch your model-calling code, and put a gateway in front of it so the next provider you add is a config change rather than a refactor.
Related reading on PikVue:
- How to Cut Your LLM API Bill in 2026: A Guide for Indie SaaS Teams
- Best No-Code Tools for Indie SaaS Founders in 2026: When to Build vs Buy
- How to Design and Manage Webhooks for Your SaaS in 2026: A Practical Guide for I
Originally published at pikvue.com — independent 2026 reviews and comparisons of developer tools, AI products, and SaaS platforms.
Top comments (0)