DEV Community

Cover image for Make your LLM calls survive 429s and overloads, with any client
Himanshu Sharma
Himanshu Sharma

Posted on

Make your LLM calls survive 429s and overloads, with any client

#ai #typescript #opensource #webdev

You ship an LLM feature. It works. Then, in production, you start seeing this:

429 Too Many Requests
529 overloaded_error
503 Service Unavailable
Enter fullscreen mode Exit fullscreen mode

So you reach for a retry library — p-retry, async-retry, a hand-rolled
backoff loop — and wrap the call. And it mostly works, until you notice it's
retrying a 400 Bad Request (pointless), ignoring the Retry-After: 2 the
server politely sent (so you wait the wrong amount), and hammering a 429 that
you could have paced under in the first place.

The problem is that an LLM endpoint doesn't fail like a generic HTTP service, and
generic retry libraries don't know the difference.

What's actually different about LLM failures

  • A 429 might be a quota limit or a momentary burst. Either way the server usually tells you how long to wait, in a Retry-After (seconds or an HTTP date) or a retry-after-ms header. A blind exponential backoff ignores it and guesses.
  • Anthropic adds a 529 overloaded_error — "the service is saturated, try again." That's retryable, and it's a different thing from a quota 429. Most retry predicates don't even have 529 in their list.
  • Providers report your remaining budget in x-ratelimit-remaining-requests / -tokens (OpenAI) and anthropic-ratelimit-* headers. You can pace against that instead of waiting to be told "no."

The official SDKs know all of this. The openai and @anthropic-ai/sdk clients
retry 429/5xx and honor retry-after out of the box. The catch: that logic
lives inside the client. The moment you call through a gateway, a proxy, your
own fetch, a second provider, or a streaming client, you leave it behind and
you're back to a generic loop.

retry-wire

retry-wire is that provider-aware
retry logic, unbundled — a zero-dependency policy you wrap around any fetch
or any async call. Two functions, same knowledge inside both.

A drop-in fetch wrapper:

import { retryFetch } from "retry-wire";

const rfetch = retryFetch(fetch, { provider: "openai", maxRetries: 5 });

const res = await rfetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  body,
  signal,
});
Enter fullscreen mode Exit fullscreen mode

rfetch has the exact fetch signature, so it drops in anywhere a fetch is
accepted. It retries 429 / 529 / 5xx / 408 / 409 and network errors, honors
Retry-After, drains the response body before retrying so nothing leaks, and
returns non-retryable responses (200, 400, …) untouched — just like fetch.

Or wrap an SDK method — anything that returns a promise:

import { withResilience } from "retry-wire";

const out = await withResilience(
  (signal) => client.messages.create(params, { signal }),
  {
    provider: "anthropic", // treats 529 overloaded as retryable, distinct from 429
    maxRetries: 5,
    backoff: { initial: 500, max: 60_000, jitter: "full" },
    onRetry: ({ attempt, delayMs, reason }) =>
      console.warn(`retry ${attempt} in ${delayMs}ms: ${reason}`),
  },
);
Enter fullscreen mode Exit fullscreen mode

It reads the thrown error's status and headers (matching how the SDK error
objects are shaped), forwards an AbortSignal to your call, and never retries an
abort or a non-retryable error.

Rule packs: the provider knowledge, as data

What makes it provider-aware is a rule pack — plain data, not hardcoded
branching:

interface RulePack {
  name?: string;
  isRetryable(ctx): boolean;                 // is this outcome worth retrying?
  retryAfterMs(headers): number | undefined; // the server's advised wait
  readLimits?(headers): RateLimitInfo;        // remaining request/token budget
}
Enter fullscreen mode Exit fullscreen mode

genericPack, openaiPack, and anthropicPack ship built in. anthropicPack
is the one that knows 529; the OpenAI and Anthropic packs also parse the
provider's rate-limit headers. For a gateway, Bedrock, Vertex, or anything with
its own rules, defineRulePack gives you a typed custom pack:

import { defineRulePack, parseRetryAfter } from "retry-wire";

const bedrockPack = defineRulePack({
  name: "bedrock",
  isRetryable: (ctx) => ctx.status === 429 || ctx.status === 503,
  retryAfterMs: (headers) => parseRetryAfter(headers),
});
Enter fullscreen mode Exit fullscreen mode

Don't just react to 429 — avoid it

Retrying reacts to a rate limit. Throttling avoids it: pace your own calls
under the tier before the provider has to push back. It's opt-in, built on token
buckets over requests- and tokens-per-minute:

const rfetch = retryFetch(fetch, {
  provider: "openai",
  throttle: {
    rpm: 50,
    tpm: 40_000,
    estimateTokens: (input, init) => countTokens(init?.body),
  },
});
Enter fullscreen mode Exit fullscreen mode

A call that would outrun the budget waits on an abortable timer until a bucket
token is available — so you glide under the limit instead of bouncing off it.

It composes with sse-wire

If you stream completions, you've probably met
sse-wire — a fetch-based SSE client
that reconnects a stream only when the connection actually drops. The two solve
different halves of the same problem: sse-wire reconnects the stream;
retry-wire resilience-wraps the request that opens the stream, so a 429 or
529 on the way in is retried before a byte streams.

import { retryFetch } from "retry-wire";
import { sse } from "sse-wire";

const rfetch = retryFetch(fetch, { provider: "openai", maxRetries: 5 });

for await (const event of sse("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
  body: JSON.stringify({ model, stream: true, messages }),
  fetch: rfetch, // open the stream through the resilient fetch
})) {
  if (event.data === "[DONE]") break;
  render(JSON.parse(event.data));
}
Enter fullscreen mode Exit fullscreen mode

retry-wire is part of a small line of zero-dependency LLM dev tools, each
useful on its own:

  • retry-wire — provider-aware retry + throttle for the request. (this one)
  • sse-wire — fetch-based SSE client: POST, headers, abort, opt-in reconnection.
  • trickle-json — assemble streamed token deltas into the best valid partial value on every chunk.
  • coerce-json — repair and coerce that value to your Zod / JSON Schema, logging every fix.
  • trickle-react — React bindings for the streaming parse.
  • expect-llm — assertions for LLM output in tests.

Why not just use the SDK's retries?

Because they only cover calls made through that SDK. One app that talks to
OpenAI and Anthropic has two retry implementations with different knobs; add a
gateway or a raw fetch and you have a third, generic one. retry-wire is one
policy — openai / anthropic / generic packs plus custom — across every
transport: SDKs, gateways, raw fetch, and streaming.

Try it 

npm install retry-wire
Enter fullscreen mode Exit fullscreen mode

If it retries something it shouldn't — or doesn't retry something it should, or
misreads a Retry-After — open an issue with the status/headers you saw. The
provider-aware part is the whole point, so I want to know. A star helps if it
saves you a hand-rolled backoff loop.

Top comments (4)

Collapse
 
baumgaerben profile image
baumgaerben •

Bài viết rất thực tế — 429 là "nỗi đau" hiển nhiên khi scale LLM feature nhưng thường bị bỏ qua ở giai đoạn dev.

Một điều ít người nhắc: retry strategy phải phân biệt rõ giữa rate limit (429) và overload (503/504). Với 429, server thường trả Retry-After header — respect nó thay vì hardcode backoff. Với 5xx, exponential backoff + jitter mới hợp lý. Gộp chung hai case dễ dẫn đến thundering herd khi service vừa recover.

Còn một điểm hay gặp: client-side queue với priority. Không phải request nào cũng quan trọng bằng nhau — user-facing chat cần ưu tiên hơn background summarization. Implement priority queue + token bucket per priority tier giúp tránh starvation cho traffic quan trọng khi quota bị chặt.

Ai đã thử pattern "hedged requests" cho LLM call chưa? Gửi song song 2 request (primary + backup provider/model), lấy kết quả nào về trước rồi cancel cái kia. Tốn token gấp đôi nhưng giảm tail latency đáng kể cho critical path. Trade-off thế nào ở chỗ các bạn? (site: labagent .tech)

Collapse
 
ai_adam profile image
ai_adam •

Xử lý 429 và overload thực sự là bài toán "phải có" chứ không phải "nên có" khi đưa LLM vào production. Vài điều mình rút ra sau vài lần đau khổ:

Exponential backoff + jitter là baseline, nhưng quan trọng hơn là đọc header Retry-After (nếu provider trả về) thay vì tự đoán. Nhiều team bỏ qua điều này và bị ban tiếp vì retry quá dồn dập.

Circuit breaker giúp tránh "thundering herd" khi service đang down — ngừng gọi hẳn sau N lỗi liên tiếp, chuyển sang fallback (model nhỏ hơn, cached response, hoặc graceful degradation) thay vì tiếp tục bắn request vào hố đen.

Client-side rate limiting (token bucket/leaky bucket) trước khi request ra network tiết kiệm chi phí và giảm áp lực lên provider. Đặc biệt quan trọng khi có nhiều worker/instance cùng chia sẻ quota.

Observability: log đầy đủ request_id, model, latency, retry_count, error_code. Không có dữ liệu này thì debug production là mò kim đáy bể.

Test chaos: dùng tool như toxiproxy hoặc mock server trả 429 ngẫu nhiên trong CI/CD để verify retry logic trước khi deploy.

Mình thấy nhiều team chỉ handle try/catch đơn giản rồi hy vọng may mắn — production sẽ d — found it via LabAgent, site: labagent .tech

Some comments may only be visible to logged-in visitors. Sign in to view all comments.