Most LLM API error handling starts out looking like normal HTTP error handling.
If it is a 429, retry later.
If it is a 500, retry with backoff.
...
For further actions, you may consider blocking this person and/or reporting abuse
The distinction between catching an SDK exception and having an actual incident policy is exactly the right line to draw. A 429 only becomes actionable once you know whether the pressure is RPM, TPM, concurrency, provider-side capacity, or your own backlog dynamics, and those lead to very different responses. I’ve found this is where teams either build a real operating model for LLM features or keep treating them like ordinary HTTP calls until the first queue pileup. This is also where trace quality matters a lot, because without a clean record of prompt size, model choice, retries, and downstream state, postmortems stay hand-wavy. Tools like agent-inspect have been useful for that local-first debugging boundary. Curious whether you’ve settled on a preferred “degrade vs queue vs fail closed” rule for user-facing flows.
That “ordinary HTTP calls until the first queue pileup” line is painfully accurate.
For user-facing flows, I usually think about it in this order:
The part I try to avoid is silent fallback that changes the product behavior without telling either the user or the operator. That tends to create the worst postmortems because, like you said, the traces are too vague to prove what actually happened.
The "signal is in the response, not the status line" point above is right, and I would push it one step further: sometimes the signal is the absence of something in a 200. A stream that delivered text but dropped its terminal frame, no finish_reason, no usage, comes back as a clean 200 with no error object, so classifyLlmFailure never runs on it at all. It is not a 429, not a 529, not a fallback returning a 200 from a different model. It is a response that looks complete and is not, and the policy only ever sees the failures that throw.
That is also where the "side effects as a first-class input" idea from this thread bites hardest, so let me connect the two. The reason an incomplete 200 is more dangerous than a clean error is that any tool call the model emitted before the stream cut already fired, with no terminal confirmation the turn finished. The classifier that would have consulted the side-effects flag never got invoked, because nothing threw. So the reconcile has to sit before the failure policy, on the happy path: check the envelope of every 200 for a terminal frame and a finish_reason before you treat the call as done, and only then hand a genuine failure to classifyLlmFailure. Otherwise side-effects-first-class is a rule you apply to the loud failures and skip on the quiet one that already did the damage.
Smaller thing on the retry table: rate_limit_tokens with "reduce token usage" needs a floor. If the failing request is a single large tool result you cannot shrink, queue-and-wait is the only safe action, and silently truncating to fit the budget is its own incident, since the model then reasons over a clipped input and looks perfectly fine doing it. Worth splitting into shrinkable versus not-shrinkable token limits, because the first is a retry knob and the second is a queue-or-fail decision.
Yeah, this is a really good catch.
The "incomplete 200" case is probably the most annoying version of this whole problem, because everything looks successful from the outside. No thrown error, no obvious fallback, no scary status code, just a response that quietly skipped the part that tells you the turn actually finished. Super fun 😅
I agree with your ordering too. The reconcile step has to happen on the happy path, before the failure policy. Basically: don't let a 200 become "success" until the envelope proves it is complete.
So something like:
finish_reason?That last bit is the part people miss, including me for a while. If the model emitted a tool call, the app may already have crossed from "LLM response" into "real-world action." At that point, retry logic is less about HTTP and more about whether the system can prove what already happened.
Also +1 on the token limit split. "Reduce token usage" is only safe if the input is actually shrinkable. Summarizing chat history or trimming optional context is one thing. Silently clipping a single large tool result is a totally different failure mode, because now the model gives you a confident answer based on damaged input lol.
I like the way you framed it: shrinkable token pressure is a retry knob; non-shrinkable token pressure is queue, wait, or fail visibly. I should probably make that distinction explicit in the table.
LLM API failure policy needs to be more explicit than normal HTTP retry logic. A 429, timeout, partial response, or safety refusal can each imply a different user experience. The fallback should be designed before the incident, not improvised during it.
Exactly. A single “retryable” flag loses too much information here. A timeout may need reconciliation, a 429 may justify waiting or falling back, a partial response may require discarding incomplete state, and a safety refusal probably shouldn’t be treated as an infrastructure failure at all. By the time an incident starts, those decisions should already be encoded in the workflow instead of left to whoever happens to be debugging it.
The signal you need is usually already in the response, just not in the status line. Providers return a retry-after plus an error type or limit header that tells you which bucket you hit (RPM vs TPM vs daily quota vs a 529/overloaded), and the correct policy branches on that field, not on the bare 429.
The other split worth hard-coding is retryable vs terminal. An overloaded_error or a 503 is worth backing off on. A 400 for context-length or an invalid request is terminal, and any loop that treats it as transient just burns quota and delays the real fix. And the one that bit me on model fallback: switching models to dodge a limit silently changes the output contract. If your caller expects a specific JSON schema or tool-call shape, a fallback model that does not honor it turns a clean outage into silent corruption downstream, which is far harder to debug. Failover has to preserve the contract, not just return a 200.
Yeah, exactly. The status line is usually the least interesting part of the failure.
The fallback point is a big one too. Returning a 200 from a different model can look like recovery while quietly breaking the contract the caller was relying on. I’ve started treating fallback as a compatibility decision, not just an availability decision: same schema expectations, same tool-call behavior, same safety assumptions, or it should probably degrade/fail instead of pretending everything is fine.
“Failover has to preserve the contract” is a really good way to put it.
Separating the decision from the label is the right cut, and the four-question shape is a clean contract. The case I would fold in explicitly is side effects, which you gesture at with stop retrying because this action has side effects but which deserves to be a first-class input to classifyLlmFailure. A 429 on a read-only summarize is trivially retryable; the same 429 on a call that already fired a tool, sent an email, or charged a card is not, and the status code can never tell them apart. In practice that means the request context needs an idempotency signal, so the policy can return retry for one and fail_closed for the other off the identical error. Retry-After parsing is the other easy win: Anthropic and OpenAI both return that header on 429, so honoring it beats any backoff curve you invent.
Totally agree. Side effects probably deserve to be a first-class input, not just a warning buried in the retry logic.
The same provider error can mean “retry safely” for a read-only summarize call and “stop, reconcile state, and fail closed” for anything that may have touched the outside world. I like the framing of passing request context into the policy instead of asking the error object to explain the whole situation.
And yes on
Retry-After. If the provider gives you a real recovery signal, ignoring it and inventing your own backoff curve is usually just making the incident noisier.The category table is the right backbone, but retry-safety needs a second axis it mostly leaves implicit: what the turn already did. The model call itself is usually cheap to retry. What isn't safe is retrying a turn where the agent already fired a tool, sent the email, wrote the row, and LLM providers won't dedupe that for you the way Stripe would. So the policy has to carry the turn's side-effect state, and the dedup key lives on your tool executions, not the model call. The other thing it underweights is cost: retry-with-jitter on token-heavy prompts during provider_overloaded multiplies your bill right when you're already degraded, so that branch wants a budget circuit-breaker next to the jitter, not just backoff.
Yeah, that second axis is important: not just why the call failed, but what already happened before it failed. Once a turn has crossed a side-effect boundary, retrying the model call can replay the plan and duplicate the real action. I’ve started treating the tool execution as the unit that needs an operation ID and idempotency protection, while the model call is just one step inside it.
Good point on the budget circuit-breaker too. Backoff limits request pressure, but it doesn’t limit how much context gets paid for again. A retry policy probably needs both an attempt limit and a token or cost budget per user-visible operation, especially when overloaded providers and long agent histories show up at the same time.
This is a great breakdown! I've been
Thanks. Looks like your comment may have gotten cut off, but I appreciate it.