Most LLM features ship with one path: prompt in, frontier model out. That path is the most expensive, slowest and least available of everything you could have built. When it degrades — and it will, because you're calling someone else's service over a network — the feature either hangs, returns something that fails a validator downstream, or throws a 500 at the user.
The fix isn't a better prompt. It's an ordered ladder of tiers, each with its own budget, plus a gate between them and one log line per request recording which tier answered. That's the part of LLMOps that survives contact with production.
Three failures, not one
Group your incidents by cause before you design anything:
- Latency. p50 is 800ms, p99 is 30s. On a voice interface users hear dead air and hang up; on a form they click again and you double your traffic. Your provider's tail is your tail.
-
Quality. HTTP 200, plausible output, wrong answer. Signalled by a downstream
KeyError, a broken citation, or nothing at all until a customer complains. - Availability. 429s, 503s, a regional outage, an expired key, or a pinned model version being deprecated out from under you.
Each needs a different response. A timeout doesn't want a retry, it wants a shorter path. A 429 wants a retry with jitter. A quality failure wants escalation, not repetition — re-running the same tier with the same prompt returns the same plausible garbage.
Retries are a load amplifier
Most SDKs default to 2–3 retries. With no jitter, at p99 every client retries in the same window and you DDoS the provider that was already struggling; you burn quota, escalate rate limits, and turn a slow minute into a 20-minute outage. I've watched this happen with a 40-line background job.
Two rules I now apply everywhere:
- Retry at most once at the same tier, only for 429/503/timeout, only with jitter, and only if the remaining deadline allows a full attempt.
- Enforce the deadline at the transport layer —
timeout=on the HTTP client — not just withasyncio.wait_for. Cancelling the coroutine doesn't reliably close the socket; the provider still finishes the request, still bills it, still counts it against your rate limit.
The ladder
Order tiers ascending by cost and latency, not by quality. For a typical text feature:
-
Exact cache, keyed on a hash of
(model_id, prompt_version, normalized_input). Free, instant. On support-style traffic this is usually the biggest single cost lever you have. - Task-specific model. A small classifier or extractor you trained or fine-tuned. For a narrow, well-defined task this isn't a compromise — a task-specific model can beat a frontier model on accuracy, latency and cost simultaneously. Don't take that on faith; measure it on 200 labelled examples from your own traffic.
- Small general model. Fast, cheap, fine for the majority of requests that are easy.
- Frontier model. Entered only when the gate rejects what came before.
-
Static fallback. A canned answer, a form, a human queue. This tier must exist and must be exercised in staging, or your graceful degradation is a
TypeError.
For voice or streaming, reorder: time-to-first-token dominates everything, so a tier that takes 3s before the first token is unusable no matter how good it is. You may jump straight to the frontier model with a hard cutoff and a shorter spoken answer instead.
The gate is a function, not a vibe
The gate decides "good enough to return." It's a schema check, an enum membership check, a length bound, or a cheap rule like "every document ID in the output exists in the retrieved set." Without one, you only escalate on exceptions — and degraded models rarely throw. They return confident nonsense that passes if text:.
Show me the ladder
python
import time, httpx
STATIC_FALLBACK = "I can't answer that
Top comments (0)