DEV Community

plasma
plasma

Posted on

The LLM API Failure Policy I Wish I Had Before My First Production Incident

#ai #llm #javascript #webdev

Most LLM API error handling starts out looking like normal HTTP error handling.

If it is a 429, retry later.

If it is a 500, retry with backoff.

If it is a timeout, maybe retry once.

That works fine right up until your first real production incident, where the question is no longer:

"Did the API return an error?"

The question becomes:

"What should the system do now?"

That is where normal HTTP error handling starts to feel too shallow.

The problem is not the error code

A 429 from an LLM provider can mean very different things depending on the provider, account, model, and workload.

You might be hitting:

  • requests per minute
  • tokens per minute
  • daily quota
  • concurrency limits
  • temporary provider-side throttling
  • a model-specific capacity limit

Those are not the same operational problem.

If you hit requests per minute, slowing request frequency may help.

If you hit tokens per minute, retrying the same large prompt immediately is probably useless.

If the provider is overloaded, retrying aggressively can make the incident worse.

If your own queue is backing up, switching models may hide the symptom while the real issue keeps growing.

The HTTP status code tells you the broad class of failure. It does not tell you the correct response.

SDK error handling is not an incident policy

Most SDKs are designed to help you make a request and catch an exception.

That is useful, but it is not the same as a production policy.

An SDK can tell you:

  • the request failed
  • the provider returned 429
  • the provider returned 529
  • the request timed out
  • the response body had a certain error shape

But your application still has to decide:

  • should this be retried?
  • should the request be queued?
  • should we switch models?
  • should we degrade the feature?
  • should we show the user a specific message?
  • should this page someone?
  • should we stop retrying because this action has side effects?

That decision tree usually does not live in the SDK.

In a lot of teams, it lives in a runbook someone wrote after the first painful incident.

That is better than nothing, but it creates a gap: the code sees the error first, while the runbook explains what humans should do later.

The failure policy is the layer in between.

What I mean by a failure policy

A failure policy is a small layer near your LLM client that maps provider-specific errors into internal failure categories and actions.

Instead of spreading logic like this across your app:

if (error.status === 429) {
  retry();
}
Enter fullscreen mode Exit fullscreen mode

I prefer something closer to this:

const decision = classifyLlmFailure(error, requestContext);

if (decision.action === "retry") {
  await retryWithBackoff();
}

if (decision.action === "queue") {
  await enqueueForLater();
}

if (decision.action === "fail_closed") {
  throw new UserVisibleError(decision.userMessage);
}
Enter fullscreen mode Exit fullscreen mode

The important part is that the policy returns a decision, not just a label.

A useful failure policy usually answers four questions:

  1. What happened?
  2. How confident are we?
  3. What should the system do first?
  4. What should humans look at if this continues?

A small example

Here is a simplified JavaScript example.

This is not meant to cover every provider or SDK. The point is the shape of the layer.

function classifyLlmFailure(error, context = {}) {
  const status = error.status || error.response?.status;
  const headers = normalizeHeaders(error.headers || error.response?.headers || {});
  const provider = context.provider || "unknown";

  if (status === 429) {
    const resetRequests = headers["x-ratelimit-reset-requests"];
    const resetTokens = headers["x-ratelimit-reset-tokens"];
    const retryAfter = headers["retry-after"];

    if (resetTokens && !resetRequests) {
      return {
        category: "rate_limit_tokens",
        action: "queue",
        retryable: true,
        confidence: "medium",
        logLevel: "warn",
        runbook: "llm-rate-limit-tokens",
        reason: "Provider returned token reset metadata."
      };
    }

    if (resetRequests && !resetTokens) {
      return {
        category: "rate_limit_requests",
        action: "retry",
        retryable: true,
        confidence: "medium",
        retryAfterMs: parseRetryAfter(retryAfter),
        logLevel: "warn",
        runbook: "llm-rate-limit-requests",
        reason: "Provider returned request reset metadata."
      };
    }

    return {
      category: "rate_limit_unknown",
      action: "slow_down",
      retryable: true,
      confidence: "low",
      retryAfterMs: parseRetryAfter(retryAfter),
      logLevel: "warn",
      runbook: "llm-rate-limit-unknown",
      reason: "429 without enough metadata to distinguish request vs token limits."
    };
  }

  if (status === 529 || status === 503) {
    return {
      category: "provider_overloaded",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-provider-overload",
      reason: "Provider appears temporarily overloaded."
    };
  }

  if (error.name === "AbortError" || error.code === "ETIMEDOUT") {
    return {
      category: "timeout",
      action: context.hasSideEffects ? "fail_closed" : "retry",
      retryable: !context.hasSideEffects,
      confidence: "medium",
      logLevel: "warn",
      runbook: "llm-timeouts",
      reason: "Request timed out before a reliable result was received."
    };
  }

  if (status >= 500) {
    return {
      category: "provider_5xx",
      action: "retry_with_jitter",
      retryable: true,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-provider-5xx",
      reason: "Provider returned a server-side error."
    };
  }

  if (status >= 400) {
    return {
      category: "bad_request_or_auth",
      action: "fail_fast",
      retryable: false,
      confidence: "medium",
      logLevel: "error",
      runbook: "llm-client-errors",
      reason: "Client-side request, auth, or configuration error."
    };
  }

  return {
    category: "unknown",
    action: "alert_human",
    retryable: false,
    confidence: "low",
    logLevel: "error",
    runbook: "llm-unknown-failures",
    reason: "Could not classify LLM failure."
  };
}

function normalizeHeaders(headers) {
  const result = {};

  for (const [key, value] of Object.entries(headers)) {
    result[key.toLowerCase()] = Array.isArray(value) ? value.join(",") : String(value);
  }

  return result;
}

function parseRetryAfter(value) {
  if (!value) return undefined;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return seconds * 1000;
  }

  const date = Date.parse(value);
  if (Number.isFinite(date)) {
    return Math.max(0, date - Date.now());
  }

  return undefined;
}
Enter fullscreen mode Exit fullscreen mode

The exact categories will vary by app.

The useful part is not the specific code. It is the habit of turning provider errors into internal operational decisions.

Log the policy decision, not just the error

During an incident, a raw stack trace is rarely enough.

I want one log event that answers:

  • which provider failed?
  • which model was used?
  • what status code came back?
  • what raw response headers mattered?
  • how many input and output tokens were estimated?
  • what failure category did we assign?
  • what action did the system take?
  • which runbook should a human open?

A log line like this is much more useful than "LLM request failed":

logger.warn("llm_request_failed", {
  provider: "example-provider",
  model: "example-model",
  status: 429,
  category: decision.category,
  action: decision.action,
  confidence: decision.confidence,
  retryable: decision.retryable,
  retryAfterMs: decision.retryAfterMs,
  runbook: decision.runbook,
  estimatedInputTokens: context.estimatedInputTokens,
  estimatedOutputTokens: context.estimatedOutputTokens,
  requestId: context.requestId
});
Enter fullscreen mode Exit fullscreen mode

The goal is simple: if someone is on call, they should not need 20 minutes to figure out whether this was a request-rate issue, token-rate issue, provider overload, timeout, or bad config.

If the first useful question during an incident is "what kind of failure is this?", your logs should answer that directly.

Do not retry every retryable error the same way

This is where a lot of LLM API code gets too optimistic.

"Retryable" does not mean "retry immediately."

A token-per-minute limit, request-per-minute limit, timeout, and provider overload may all be retryable, but they should not share the same retry behavior.

For example:

  • rate_limit_requests: retry after the reset window, possibly with queueing
  • rate_limit_tokens: reduce token usage, queue, or switch to a smaller request
  • provider_overloaded: retry with jitter, but avoid synchronized retry storms
  • timeout: retry only if the operation is safe to repeat
  • stream_interrupted: decide whether partial output is usable or must be discarded
  • tool_call_uncertain: do not blindly execute side effects twice

That last one matters a lot for agentic workflows.

If the LLM call was only generating text, a retry is usually fine.

If the LLM call was planning or triggering an external action, retrying can accidentally duplicate real work. Sending two emails, creating two tickets, charging twice, or calling the same webhook twice is a very different failure mode from "the answer took too long."

Keep the runbook close to the code

I do not think the whole incident runbook belongs inside the codebase.

Human context still matters.

But the first decision should be encoded near the client.

A practical compromise is:

  • code classifies the failure
  • code chooses the first action
  • logs include the runbook ID
  • the runbook explains investigation steps and business context

For example:

{
  category: "rate_limit_tokens",
  action: "queue",
  runbook: "llm-rate-limit-tokens"
}
Enter fullscreen mode Exit fullscreen mode

Then the runbook can explain:

  • where to check token usage
  • which dashboards matter
  • whether to reduce max output tokens
  • whether fallback models are approved
  • who owns quota increases
  • what user-facing degradation is acceptable

That keeps institutional knowledge closer to the place where the incident starts, without turning your application code into a giant wiki page.

The checklist I use now

Before shipping a new LLM provider or model path, I try to answer these:

  • Do we log raw provider error metadata?
  • Can we distinguish request limits from token limits?
  • Do we classify overload separately from rate limits?
  • Are timeout retries safe for this workflow?
  • Do streaming failures preserve partial state intentionally?
  • Do tool calls have idempotency keys or operation IDs?
  • Does every failure category have a first action?
  • Does every common category point to a short runbook?
  • Can someone on call understand the failure from one log event?

If the answer is no, the integration may still work in development, but it is probably not ready for production traffic.

Final thought

LLM APIs look like normal APIs until they fail under real workload.

Then the hard part is not catching the exception. The hard part is knowing whether to retry, slow down, queue, fallback, fail closed, or wake someone up.

That decision should not live only in a forgotten runbook.

Put a small failure policy next to your LLM client.

Your future on-call self will be a lot less annoyed.

I work on TokenBay, so I spend a lot of time thinking about multi-model API behavior, routing, and failure handling. This post is less about any one provider and more about the production layer I wish more LLM apps had from day one.

Top comments (2)

Collapse
 
dipankar_sarkar profile image
Dipankar Sarkar •

The signal you need is usually already in the response, just not in the status line. Providers return a retry-after plus an error type or limit header that tells you which bucket you hit (RPM vs TPM vs daily quota vs a 529/overloaded), and the correct policy branches on that field, not on the bare 429.

The other split worth hard-coding is retryable vs terminal. An overloaded_error or a 503 is worth backing off on. A 400 for context-length or an invalid request is terminal, and any loop that treats it as transient just burns quota and delays the real fix. And the one that bit me on model fallback: switching models to dodge a limit silently changes the output contract. If your caller expects a specific JSON schema or tool-call shape, a fallback model that does not honor it turns a clean outage into silent corruption downstream, which is far harder to debug. Failover has to preserve the contract, not just return a 200.

Collapse
 
skillselion profile image
Skillselion •

Separating the decision from the label is the right cut, and the four-question shape is a clean contract. The case I would fold in explicitly is side effects, which you gesture at with stop retrying because this action has side effects but which deserves to be a first-class input to classifyLlmFailure. A 429 on a read-only summarize is trivially retryable; the same 429 on a call that already fired a tool, sent an email, or charged a card is not, and the status code can never tell them apart. In practice that means the request context needs an idempotency signal, so the policy can return retry for one and fail_closed for the other off the identical error. Retry-After parsing is the other easy win: Anthropic and OpenAI both return that header on 429, so honoring it beats any backoff curve you invent.