The LLM API Failure Policy I Wish I Had Before My First Production Incident

Most LLM API error handling starts out looking like normal HTTP error handling.

If it is a 429, retry later.

If it is a 500, retry with backoff.

If it is a timeout, maybe retry once.

That works fine right up until your first real production incident, where the question is no longer:

"Did the API return an error?"

The question becomes:

"What should the system do now?"

That is where normal HTTP error handling starts to feel too shallow.

The problem is not the error code

A 429 from an LLM provider can mean very different things depending on the provider, account, model, and workload.

You might be hitting:

• requests per minute
• tokens per minute
• daily quota
• concurrency limits
• temporary provider-side throttling
• a model-specific capacity limit

Those are not the same operational problem.

If you hit requests per minute, slowing request frequency may help.

If you hit tokens per minute, retrying the same large prompt immediately is probably useless.

If the provider is overloaded, retrying aggressively can make the incident worse.

If your own queue is backing up, switching models may hide the symptom while the real issue keeps growing.

The HTTP status code tells you the broad class of failure. It does not tell you the correct response.

SDK error handling is not an incident policy

Most SDKs are designed to help you make a request and catch an exception.

That is useful, but it is not the same as a production policy.

An SDK can tell you:

• the request failed
• the provider returned 429
• the provider returned 529
• the request timed out
• the response body had a certain error shape

But your application still has to decide:

• should this be retried?
• should the request be queued?
• should we switch models?
• should we degrade the feature?
• should we show the user a specific message?
• should this page someone?
• should we stop retrying because this action has side effects?

That decision tree usually does not live in the SDK.

In a lot of teams, it lives in a runbook someone wrote after the first painful incident.

That is better than nothing, but it creates a gap: the code sees the error first, while the runbook explains what humans should do later.

The failure policy is the layer in between.

What I mean by a failure policy

A failure policy is a small layer near your LLM client that maps provider-specific errors into internal failure categories and actions.

Instead of spreading logic like this across your app:

if (error.status === 429) {
  retry();
}

I prefer something closer to this:

const decision = classifyLlmFailure(error, requestContext);

if (decision.action === "retry") {
  await retryWithBackoff();
}

if (decision.action === "queue") {
  await enqueueForLater();
}

if (decision.action === "fail_closed") {
  throw new UserVisibleError(decision.userMessage);
}

The important part is that the policy returns a decision, not just a label.

A useful failure policy usually answers four questions:

1. What happened?
2. How confident are we?
3. What should the system do first?
4. What should humans look at if this continues?

A small example

Here is a simplified JavaScript example.

This is not meant to cover every provider or SDK. The point is the shape of the layer.

function classifyLlmFailure(error, context = {}) {
  const status = error.status || error.response?.status;
  const headers = normalizeHeaders(error.headers || error.response?.headers || {});
  const provider = context.provider || "unknown";

  if (status === 429) {
    const resetRequests = headers["x-ratelimit-reset-requests"];
    const resetTokens = headers["x-ratelimit-reset-tokens"];
    const retryAfter = headers["retry-after"];

    if (resetTokens && !resetRequests) {
      return {
        category: "rate_limit_tokens",
        action: "queue",
        retryable: true,
        confidence: "medium",
        logLevel: "warn",
        runbook: "llm-rate-limit-tokens",
        reason: "Provider returned token reset metadata."
      };
    }

    if (resetRequests && !resetTokens) {
      return {
        category: "rate_limit_requests",
        action: "retry",
        retryable: true,
        confidence: "medium",
        retryAfterMs: parseRetryAfter(retryAfter),
        logLevel: "warn",
        runbook: "llm-rate-limit-requests",
        reason: "Provider returned request reset metadata."
      };
    }

    return {
      category: "rate_limit_unknown",
      action: "slow_down",
      retryable: true,
      confidence: "low",
      retryAfterMs: parseRetryAfter(retryAfter),
      logLevel: "warn",
      runbook: "llm-rate-limit-unknown",
      reason: "429 without enough metadata to distinguish request vs token limits."
    };
  }

  if (status === 529 || status === 503) {
    return {
      category: "provider_overloaded",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-provider-overload",
      reason: "Provider appears temporarily overloaded."
    };
  }

  if (error.name === "AbortError" || error.code === "ETIMEDOUT") {
    return {
      category: "timeout",
      action: context.hasSideEffects ? "fail_closed" : "retry",
      retryable: !context.hasSideEffects,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-timeouts",
      reason: "Request timed out before a reliable result was received."
    };
  }

  if (status >= 500) {
    return {
      category: "provider_5xx",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-provider-5xx",
      reason: "Provider returned a server-side error."
    };
  }

  if (status >= 400) {
    return {
      category: "bad_request_or_auth",
      action: "fail_fast",
      retryable: false,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-client-errors",
      reason: "Client-side request, auth, or configuration error."
    };
  }

  return {
    category: "unknown",
    action: "alert_human",
    retryable: false,
    confidence: "low",
    logLevel: "error",
    runbook: "llm-unknown-failures",
    reason: "Could not classify LLM failure."
  };
}

function normalizeHeaders(headers) {
  const result = {};

  for (const [key, value] of Object.entries(headers)) {
    result[key.toLowerCase()] = Array.isArray(value) ? value.join(",") : String(value);
  }

  return result;
}

function parseRetryAfter(value) {
  if (!value) return undefined;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(value);
  if (Number.isFinite(date)) {
    return Math.max(0, date - Date.now());
  }

  return undefined;
}

The exact categories will vary by app.

The useful part is not the specific code. It is the habit of turning provider errors into internal operational decisions.

Log the policy decision, not just the error

During an incident, a raw stack trace is rarely enough.

I want one log event that answers:

• which provider failed?
• which model was used?
• what status code came back?
• what raw response headers mattered?
• how many input and output tokens were estimated?
• what failure category did we assign?
• what action did the system take?
• which runbook should a human open?

A log line like this is much more useful than "LLM request failed":

logger.warn("llm_request_failed", {
  provider: "example-provider",
  model: "example-model",
  status: 429,
  category: decision.category,
  action: decision.action,
  confidence: decision.confidence,
  retryable: decision.retryable,
  retryAfterMs: decision.retryAfterMs,
  runbook: decision.runbook,
  estimatedInputTokens: context.estimatedInputTokens,
  estimatedOutputTokens: context.estimatedOutputTokens,
  requestId: context.requestId
});

The goal is simple: if someone is on call, they should not need 20 minutes to figure out whether this was a request-rate issue, token-rate issue, provider overload, timeout, or bad config.

If the first useful question during an incident is "what kind of failure is this?", your logs should answer that directly.

Do not retry every retryable error the same way

This is where a lot of LLM API code gets too optimistic.

"Retryable" does not mean "retry immediately."

A token-per-minute limit, request-per-minute limit, timeout, and provider overload may all be retryable, but they should not share the same retry behavior.

For example:

• rate_limit_requests: retry after the reset window, possibly with queueing
• rate_limit_tokens: reduce token usage, queue, or switch to a smaller request
• provider_overloaded: retry with jitter, but avoid synchronized retry storms
• timeout: retry only if the operation is safe to repeat
• stream_interrupted: decide whether partial output is usable or must be discarded
• tool_call_uncertain: do not blindly execute side effects twice

That last one matters a lot for agentic workflows.

If the LLM call was only generating text, a retry is usually fine.

If the LLM call was planning or triggering an external action, retrying can accidentally duplicate real work. Sending two emails, creating two tickets, charging twice, or calling the same webhook twice is a very different failure mode from "the answer took too long."

Keep the runbook close to the code

I do not think the whole incident runbook belongs inside the codebase.

Human context still matters.

But the first decision should be encoded near the client.

A practical compromise is:

• code classifies the failure
• code chooses the first action
• logs include the runbook ID
• the runbook explains investigation steps and business context

For example:

{
  category: "rate_limit_tokens",
  action: "queue",
  runbook: "llm-rate-limit-tokens"
}

Then the runbook can explain:

• where to check token usage
• which dashboards matter
• whether to reduce max output tokens
• whether fallback models are approved
• who owns quota increases
• what user-facing degradation is acceptable

That keeps institutional knowledge closer to the place where the incident starts, without turning your application code into a giant wiki page.

The checklist I use now

Before shipping a new LLM provider or model path, I try to answer these:

• Do we log raw provider error metadata?
• Can we distinguish request limits from token limits?
• Do we classify overload separately from rate limits?
• Are timeout retries safe for this workflow?
• Do streaming failures preserve partial state intentionally?
• Do tool calls have idempotency keys or operation IDs?
• Does every failure category have a first action?
• Does every common category point to a short runbook?
• Can someone on call understand the failure from one log event?

If the answer is no, the integration may still work in development, but it is probably not ready for production traffic.

Final thought

LLM APIs look like normal APIs until they fail under real workload.

Then the hard part is not catching the exception. The hard part is knowing whether to retry, slow down, queue, fallback, fail closed, or wake someone up.

That decision should not live only in a forgotten runbook.

Put a small failure policy next to your LLM client.

Your future on-call self will be a lot less annoyed.

---

I work on TokenBay, so I spend a lot of time thinking about multi-model API behavior, routing, and failure handling. This post is less about any one provider and more about the production layer I wish more LLM apps had from day one.

Comments

[Raju Dandigam]

The distinction between catching an SDK exception and having an actual incident policy is exactly the right line to draw. A 429 only becomes actionable once you know whether the pressure is RPM, TPM, concurrency, provider-side capacity, or your own backlog dynamics, and those lead to very different responses. I’ve found this is where teams either build a real operating model for LLM features or keep treating them like ordinary HTTP calls until the first queue pileup. This is also where trace quality matters a lot, because without a clean record of prompt size, model choice, retries, and downstream state, postmortems stay hand-wavy. Tools like agent-inspect have been useful for that local-first debugging boundary. Curious whether you’ve settled on a preferred “degrade vs queue vs fail closed” rule for user-facing flows.

[plasma]

That “ordinary HTTP calls until the first queue pileup” line is painfully accurate.
For user-facing flows, I usually think about it in this order:

1. If the action is read-only and the user is still waiting, queue or retry within a tight latency budget.
2. If the action can be safely degraded, return a smaller/slower/less capable path rather than pretending the original path worked.
3. If the workflow may have triggered a real side effect, fail closed and make the state explicit before retrying anything.

The part I try to avoid is silent fallback that changes the product behavior without telling either the user or the operator. That tends to create the worst postmortems because, like you said, the traces are too vague to prove what actually happened.

[Alexey Spinov]

The "signal is in the response, not the status line" point above is right, and I would push it one step further: sometimes the signal is the absence of something in a 200. A stream that delivered text but dropped its terminal frame, no finish_reason, no usage, comes back as a clean 200 with no error object, so classifyLlmFailure never runs on it at all. It is not a 429, not a 529, not a fallback returning a 200 from a different model. It is a response that looks complete and is not, and the policy only ever sees the failures that throw.

That is also where the "side effects as a first-class input" idea from this thread bites hardest, so let me connect the two. The reason an incomplete 200 is more dangerous than a clean error is that any tool call the model emitted before the stream cut already fired, with no terminal confirmation the turn finished. The classifier that would have consulted the side-effects flag never got invoked, because nothing threw. So the reconcile has to sit before the failure policy, on the happy path: check the envelope of every 200 for a terminal frame and a finish_reason before you treat the call as done, and only then hand a genuine failure to classifyLlmFailure. Otherwise side-effects-first-class is a rule you apply to the loud failures and skip on the quiet one that already did the damage.

Smaller thing on the retry table: rate_limit_tokens with "reduce token usage" needs a floor. If the failing request is a single large tool result you cannot shrink, queue-and-wait is the only safe action, and silently truncating to fit the budget is its own incident, since the model then reasons over a clipped input and looks perfectly fine doing it. Worth splitting into shrinkable versus not-shrinkable token limits, because the first is a retry knob and the second is a queue-or-fail decision.

[plasma]

Yeah, this is a really good catch.

The "incomplete 200" case is probably the most annoying version of this whole problem, because everything looks successful from the outside. No thrown error, no obvious fallback, no scary status code, just a response that quietly skipped the part that tells you the turn actually finished. Super fun 😅

I agree with your ordering too. The reconcile step has to happen on the happy path, before the failure policy. Basically: don't let a 200 become "success" until the envelope proves it is complete.

So something like:

• did the stream reach a terminal frame?
• do we have finish_reason?
• do we have usage / final metadata if the provider normally sends it?
• did any tool call start before the turn was confirmed complete?
• if yes, do we need to reconcile that tool side effect before retrying or marking the run done?

That last bit is the part people miss, including me for a while. If the model emitted a tool call, the app may already have crossed from "LLM response" into "real-world action." At that point, retry logic is less about HTTP and more about whether the system can prove what already happened.

Also +1 on the token limit split. "Reduce token usage" is only safe if the input is actually shrinkable. Summarizing chat history or trimming optional context is one thing. Silently clipping a single large tool result is a totally different failure mode, because now the model gives you a confident answer based on damaged input lol.

I like the way you framed it: shrinkable token pressure is a retry knob; non-shrinkable token pressure is queue, wait, or fail visibly. I should probably make that distinction explicit in the table.

[Alex Shev]

LLM API failure policy needs to be more explicit than normal HTTP retry logic. A 429, timeout, partial response, or safety refusal can each imply a different user experience. The fallback should be designed before the incident, not improvised during it.

[plasma]

Exactly. A single “retryable” flag loses too much information here. A timeout may need reconciliation, a 429 may justify waiting or falling back, a partial response may require discarding incomplete state, and a safety refusal probably shouldn’t be treated as an infrastructure failure at all. By the time an incident starts, those decisions should already be encoded in the workflow instead of left to whoever happens to be debugging it.

[Dipankar Sarkar]

The signal you need is usually already in the response, just not in the status line. Providers return a retry-after plus an error type or limit header that tells you which bucket you hit (RPM vs TPM vs daily quota vs a 529/overloaded), and the correct policy branches on that field, not on the bare 429.

The other split worth hard-coding is retryable vs terminal. An overloaded_error or a 503 is worth backing off on. A 400 for context-length or an invalid request is terminal, and any loop that treats it as transient just burns quota and delays the real fix. And the one that bit me on model fallback: switching models to dodge a limit silently changes the output contract. If your caller expects a specific JSON schema or tool-call shape, a fallback model that does not honor it turns a clean outage into silent corruption downstream, which is far harder to debug. Failover has to preserve the contract, not just return a 200.

[plasma]

Yeah, exactly. The status line is usually the least interesting part of the failure.

The fallback point is a big one too. Returning a 200 from a different model can look like recovery while quietly breaking the contract the caller was relying on. I’ve started treating fallback as a compatibility decision, not just an availability decision: same schema expectations, same tool-call behavior, same safety assumptions, or it should probably degrade/fail instead of pretending everything is fine.

“Failover has to preserve the contract” is a really good way to put it.

[Skillselion]

Separating the decision from the label is the right cut, and the four-question shape is a clean contract. The case I would fold in explicitly is side effects, which you gesture at with stop retrying because this action has side effects but which deserves to be a first-class input to classifyLlmFailure. A 429 on a read-only summarize is trivially retryable; the same 429 on a call that already fired a tool, sent an email, or charged a card is not, and the status code can never tell them apart. In practice that means the request context needs an idempotency signal, so the policy can return retry for one and fail_closed for the other off the identical error. Retry-After parsing is the other easy win: Anthropic and OpenAI both return that header on 429, so honoring it beats any backoff curve you invent.

[plasma]

Totally agree. Side effects probably deserve to be a first-class input, not just a warning buried in the retry logic.

The same provider error can mean “retry safely” for a read-only summarize call and “stop, reconcile state, and fail closed” for anything that may have touched the outside world. I like the framing of passing request context into the policy instead of asking the error object to explain the whole situation.

And yes on Retry-After. If the provider gives you a real recovery signal, ignoring it and inventing your own backoff curve is usually just making the incident noisier.

[Valentin Monteiro]

The category table is the right backbone, but retry-safety needs a second axis it mostly leaves implicit: what the turn already did. The model call itself is usually cheap to retry. What isn't safe is retrying a turn where the agent already fired a tool, sent the email, wrote the row, and LLM providers won't dedupe that for you the way Stripe would. So the policy has to carry the turn's side-effect state, and the dedup key lives on your tool executions, not the model call. The other thing it underweights is cost: retry-with-jitter on token-heavy prompts during provider_overloaded multiplies your bill right when you're already degraded, so that branch wants a budget circuit-breaker next to the jitter, not just backoff.

[plasma]

Yeah, that second axis is important: not just why the call failed, but what already happened before it failed. Once a turn has crossed a side-effect boundary, retrying the model call can replay the plan and duplicate the real action. I’ve started treating the tool execution as the unit that needs an operation ID and idempotency protection, while the model call is just one step inside it.
Good point on the budget circuit-breaker too. Backoff limits request pressure, but it doesn’t limit how much context gets paid for again. A retry policy probably needs both an attempt limit and a token or cost budget per user-visible operation, especially when overloaded providers and long agent histories show up at the same time.

[Frank]

This is a great breakdown! I've been

[plasma]

Thanks. Looks like your comment may have gotten cut off, but I appreciate it.