Safe Retries for OpenAI-Compatible APIs: HTTP Errors, Retry-After, and Partial SSE Output

Retrying an API request sounds simple until the request generates text, invokes a tool, or consumes billable model capacity. A disconnected POST /v1/chat/completions is different from a failed GET /v1/models: the server may have accepted and executed the generation before the client saw an error. A stream that has already emitted output is different again, because replaying the whole request can duplicate visible text, tool calls, side effects, or cost.

This guide builds a conservative error and retry boundary for OpenAI-compatible APIs in Node.js and TypeScript. It covers four questions:

1. What kind of failure occurred?
2. What diagnostic data can be logged without retaining request or response bodies?
3. Did a streaming response already produce meaningful output?
4. Does the caller know that replay is safe within its attempt and time budgets?

The goal is not to build another API client. It is to make the decision inputs explicit before any retry loop runs.

The examples use the published @ai-router/openai-compatible-errors@0.1.0. Its source, tests, and release workflow are public on GitHub.

Why retryable: true is incomplete

Most retry logic starts with status codes:

• Retry 429 because it indicates rate limiting.
• Retry 502, 503, and 504 because an upstream service may recover.
• Retry a transport timeout because the network may recover.
• Do not retry 400 or 401 because the request or credential needs correction.

Those are useful error classifications, but they do not establish replay safety.

Consider a generation POST that times out while awaiting response headers. Three different events can look identical to the caller:

1. The request never left the process.
2. The request reached a proxy, but the proxy never sent it upstream.
3. The model completed the request, while the response was lost on the return path.

Only the first case is obviously safe to replay. A generic network error often cannot distinguish the three. Returning retryable: true collapses “the failure may be transient” and “repeating the operation is safe” into one dangerous bit.

A safer model separates them:

type Transience = "transient" | "permanent" | "unknown";
type ReplaySafety = "safe" | "unsafe" | "unknown";
type RetryAction = "retry" | "do_not_retry" | "manual_decision";

An automated retry should require both a transient error and explicit replay safety. Unknown evidence should stay unknown.

Classify the error before planning a retry

OpenAI-compatible services can expose failure evidence in several places:

• HTTP status: 401, 403, 408, 413, 429, 5xx.
• JSON body: error.message, error.type, error.code.
• Headers: Retry-After, x-request-id, request-id, correlation IDs.
• SDK error: status, statusCode, request_id, responseHeaders.
• Transport exception: AbortError, timeout, DNS, connection reset.
• SSE event: { "error": ... } or a Responses-style response.failed event inside an HTTP 200 stream.

Status alone is not always enough. A 429 can mean a temporary request-rate limit or an exhausted quota that will not recover after a two-second delay. Check structured error codes and types before assigning the category.

A useful taxonomy can remain small:

Category	Examples	Transience
Authentication	401, invalid API key	Permanent
Permission	403, access denied	Permanent
Rate limit	429 + rate-limit signal	Transient
Quota	insufficient quota, billing/credit signal	Permanent
Validation	400/409/422	Permanent
Timeout	408, timeout exception	Transient
Network	DNS, connection reset, Fetch failure	Transient but replay may be unknown
Upstream	502/503/504	Transient but replay may be unknown
Stream	premature EOF	Transient only before output and when replay is safe
Unknown	insufficient evidence	Unknown

The category should be a stable application contract. Keep raw vendor strings out of that contract.

Reproduce the decision without an API key

The following fixture runs entirely on loopback. It returns a synthetic 429 with Retry-After: 2, then evaluates the same error twice: once when replay is explicitly safe, and once when replay safety is unknown.

npm install @ai-router/openai-compatible-errors@0.1.0

import { once } from "node:events";
import { createServer } from "node:http";
import {
  decideOpenAICompatibleRetry,
  normalizeOpenAICompatibleResponse,
} from "@ai-router/openai-compatible-errors";

const server = createServer((_request, response) => {
  response.writeHead(429, {
    "content-type": "application/json",
    "retry-after": "2",
    "x-request-id": "req_loopback_demo",
  });
  response.end(JSON.stringify({
    error: {
      type: "rate_limit_error",
      code: "rate_limit_exceeded",
      message: "Synthetic fixture; excluded from the result by default.",
    },
  }));
});

server.listen(0, "127.0.0.1");
await once(server, "listening");
const { port } = server.address();

try {
  const startedAt = Date.now();
  const response = await fetch(`http://127.0.0.1:${port}/v1/demo`, {
    method: "POST",
  });
  const error = await normalizeOpenAICompatibleResponse(response);
  if (!error) throw new Error("Expected a synthetic API error");
  const context = {
    method: "POST",
    phase: "http_error",
    attempt: 1,
    elapsedMs: Date.now() - startedAt,
  };

  console.log({
    error,
    safePlan: decideOpenAICompatibleRetry(error, {
      ...context,
      replaySafety: "safe", // Safe only for this side-effect-free fixture.
    }),
    unknownPlan: decideOpenAICompatibleRetry(error, {
      ...context,
      replaySafety: "unknown",
    }),
  });
} finally {
  await new Promise((resolve) => server.close(resolve));
}

The safe result is retry with a 2,000 ms server-directed delay. The unknown result is manual_decision, even though both plans are based on the same transient 429. A complete, defensively checked copy of this fixture is available in the repository examples.

Build diagnostics that are safe by construction

Logging the original SDK error is convenient, but it can retain more than a status and message. Depending on the SDK and wrapper, the object may contain full headers, request body values, response text, nested causes, or a stack whose message includes provider text.

Redaction after the fact is fragile. A stronger default is to construct a new error object that never stores those fields:

interface SafeApiError {
  name: "OpenAICompatibleError";
  message: string; // A fixed message owned by the normalizer.
  category: string;
  source: "http" | "fetch" | "openai_sdk" | "ai_sdk" | "sse" | "unknown";
  status?: number;
  code?: string;
  type?: string;
  requestId?: string;
  retryAfterMs?: number;
}

The default message should be something predictable such as Rate limit exceeded, not a copy of error.message. Provider messages can echo an API key, prompt fragment, file name, customer identifier, or upstream HTML. If a provider message is genuinely needed, make it an explicit opt-in, redact common credential forms, truncate it, and document that secret redaction is not PII anonymization.

For arbitrary diagnostic objects, use a bounded sanitizer:

• Never invoke user-defined getters or toJSON hooks while traversing unknown objects.
• Detect circular references.
• Limit depth, node count, array items, object keys, and string length.
• Replace oversized strings rather than preserving a prefix that might contain a partial secret.
• Redact authorization, API-key, cookie, password, token, prompt, message-list, and request/response body fields.
• Redact Bearer/Basic values, known key shapes, JWTs, URL userinfo, and sensitive query parameters in free text.

Even a strong sanitizer is defense in depth. The best way to protect a prompt is not to send it to the logger.

Honor Retry-After without shortening it

HTTP defines Retry-After as either a delay in seconds or an HTTP date. Some SDKs and gateways also expose a millisecond hint such as retry-after-ms.

function parseRetryAfter(value: string, nowMs = Date.now()): number | undefined {
  const trimmed = value.trim();
  if (/^\d+$/.test(trimmed)) {
    const seconds = Number(trimmed);
    return Math.round(seconds * 1_000);
  }

  const targetMs = Date.parse(trimmed);
  if (Number.isNaN(targetMs)) return undefined;
  return Math.max(0, targetMs - nowMs);
}

Do not clamp a one-hour server delay down to a local ten-second maximum and retry early. If the server hint exceeds the caller's single-delay or total-time budget, the safe result is to stop retrying. A local maximum is a budget, not permission to ignore the server's lower-bound advice.

When there is no valid server hint, capped exponential backoff with jitter is reasonable:

const cap = Math.min(baseDelayMs * 2 ** (attempt - 1), maxDelayMs);
const delayMs = Math.round(Math.random() * cap); // Full jitter.

Check the attempt budget and elapsed-time budget before returning a retry plan. Count the initial request as attempt one so the policy is unambiguous.

Treat SSE output as a replay boundary

Server-sent events are framed as lines. Multiple data: lines form one event, events end with a blank line, and a UTF-8 character or JSON object can be split across arbitrary network chunks. Searching each chunk for "error" is therefore not a parser.

A stream inspector needs incremental state:

interface StreamState {
  eventsSeen: number;
  malformedEvents: number;
  hasOutput: boolean;
  done: boolean;
  unexpectedEof: boolean;
  error?: SafeApiError;
}

For Chat Completions, meaningful output includes non-empty choices[].delta.content, tool-call deltas, or function-call arguments. For Responses-style streams, output text or function-call argument delta events cross the same boundary.

Once hasOutput is true, a whole-request retry should default to do_not_retry. This remains true even if the terminal error is a normally transient 429 or 503. Transience describes the failure; it does not erase partial output.

An early SSE error can arrive inside an HTTP 200 response:

event: error
data: {"error":{"code":"rate_limit_exceeded","message":"..."}}

The error contract should normalize this event the same way as a non-2xx JSON body while recording source: "sse". It should not retain the raw event.

Premature EOF also needs an explicit state. If the stream emitted data but never sent [DONE] or a completion event, mark unexpectedEof. Whether that is retryable still depends on hasOutput and replay safety.

A fail-closed retry decision

The decision order matters. Evaluate the strongest reasons to stop first:

1. Caller abort: do not retry.
2. Partial SSE output: do not retry the whole request.
3. Completed request: do not retry.
4. Permanent error: do not retry.
5. Replay explicitly unsafe: do not retry.
6. Unknown phase or replay safety: require a manual decision.
7. Unknown error category: require a manual decision.
8. Attempt or time budget exhausted: do not retry.
9. Server delay exceeds the budget: do not retry; never shorten it.
10. Only now may a transient, replay-safe failure return retry.

In code with @ai-router/openai-compatible-errors:

import {
  decideOpenAICompatibleRetry,
  normalizeOpenAICompatibleResponse,
} from "@ai-router/openai-compatible-errors";

const error = await normalizeOpenAICompatibleResponse(response);

if (error) {
  const plan = decideOpenAICompatibleRetry(error, {
    method: "POST",
    phase: "http_error",
    replaySafety: "unknown",
    attempt: 1,
    elapsedMs: Date.now() - startedAt,
  });

  if (plan.action === "retry" && plan.delayMs !== undefined) {
    console.info("Caller may schedule a retry", { delayMs: plan.delayMs });
  } else {
    console.warn("Do not automatically replay this request", {
      category: error.category,
      action: plan.action,
      reason: plan.reason,
    });
  }
}

The library never schedules the retry. Keeping execution outside the package makes rate budgets, cancellation, observability, and endpoint-specific semantics visible to the application.

What should set replaySafety?

The method is evidence, not the complete answer. GET and HEAD are commonly designed to be replay-safe, while a generation POST is often ambiguous. An idempotency key helps only when the endpoint contract says how the key is scoped, retained, and applied to the operation. The mere presence of a header called Idempotency-Key is not proof.

Set replaySafety: "safe" only when the application owns enough evidence, such as:

• The operation is read-only and the endpoint contract supports replay.
• The service documents idempotent behavior for this exact operation and key.
• The failure occurred before the request could be sent, and the request phase is trustworthy.

Use unknown when a proxy, timeout, or connection reset makes acceptance ambiguous. Use unsafe after observable side effects or when replay can duplicate user-visible work.

Validation checklist

Before trusting an error boundary, test behavior rather than descriptions:

• 429 transient rate limit versus 429 exhausted quota.
• Retry-After seconds, HTTP date, invalid value, past date, and value beyond budget.
• 401, 403, 413, 422, 502, 503, and 504.
• Loopback raw Fetch and pinned OpenAI Node SDK client catch paths, plus OpenAI/AI SDK-style structural errors.
• A raw Fetch Response whose original body remains unread.
• A body that is text or HTML rather than JSON.
• Error messages containing synthetic API keys, Bearer tokens, JWTs, URL credentials, prompts, and response fragments.
• Circular objects, throwing getters, revoked proxies, deep arrays, and oversized strings.
• SSE frames split across bytes inside a UTF-8 character.
• An error before output, an error after output, malformed JSON, [DONE], and premature EOF.
• Attempt, elapsed-time, and per-delay budget boundaries.
• ESM, CommonJS, and TypeScript declaration imports from the packed tarball.

Fixtures should use synthetic canaries, never copied production credentials or customer prompts.

When a new dependency is not justified

Use the error types already provided by the official OpenAI SDK when one SDK and endpoint cover your application. Use a multi-provider normalizer when your primary problem is one taxonomy across OpenAI, Anthropic, Gemini, and other native APIs. Use provider utilities when you are implementing an AI SDK provider and already depend on that stack.

Add a separate boundary only when you need its narrower guarantees: default body-free diagnostics, multiple OpenAI-compatible error shapes, SSE output tracking, and an explicit replay-safety decision. Otherwise, another package increases maintenance without reducing risk.

References

• HTTP Semantics: Retry-After
• HTML Living Standard: Server-sent events
• OpenAI Node SDK error handling
• @ai-sdk/openai-compatible
• llm-errors

Install the package from npm, inspect the public source and test matrix, or report a suspected security issue through the repository's private advisory form.

AI ROUTER maintains @ai-router/openai-compatible-errors as an independent, provider-neutral developer utility. It is not affiliated with or endorsed by OpenAI.