Your First AI Feature Will Break on Retries — the TypeScript Fix

#typescript #node #ai #reliability

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Your codebase already has a retry helper. It wraps flaky HTTP calls, it backs
off, it has served you well for years. So when you add an LLM call, you wrap
that too.

const res = await withRetry(() =>
  client.messages.create({ model, max_tokens: 4096, messages }),
);

That looks like diligence. Applied to a model call it is closer to a footgun,
because three assumptions your retry helper was built on are all false here.

Assumption one: a retry is cheap

For a REST call, a retry costs a round trip. For a model call it costs the
whole generation again, and on a long response, the second attempt may be the
most expensive request your service makes that hour.

The case that hurts most is a read timeout. Your client gives up at 30
seconds; the provider is still generating and will finish. You are billed for
that generation whether or not you read it. Then you retry and are billed
again.

// this is two full generations, billed, one of them thrown away
const res = await withRetry(() => client.messages.create(params), {
  timeout: 30_000,
});

Streaming makes the timeout question mostly go away, because you get tokens
continuously rather than waiting for a complete response. If you are timing
out non-streaming calls on long outputs, the fix is usually to stream rather
than to tune the timeout.

Assumption two: the same request gives the same answer

An idempotent GET returns the same body twice. A model call does not.

That matters when the retry is triggered by something downstream of a
successful generation — a parse failure, a validation error, your own
network blip while reading the response. The retry produces a different
answer, and now your logs contain one request id with two different outputs,
only one of which is the one you acted on.

// which of these did the user see?
const first = await client.messages.create(params);   // succeeded, unread
const second = await client.messages.create(params);  // different answer

The fix is to be precise about what you are retrying. Separate the call
from the parse:

async function generate(params: MessageCreateParams) {
  return withRetry(() => client.messages.create(params), {
    retryOn: isTransportError,      // 429, 5xx, socket reset — nothing else
  });
}

async function extract(doc: string): Promise<Invoice> {
  const res = await generate(buildParams(doc));
  const parsed = Invoice.safeParse(JSON.parse(textOf(res.content)));
  if (parsed.success) return parsed.data;
  return repairWithModel(res, parsed.error);   // a NEW turn, not a retry
}

Transport failures get a retry. Content failures get a follow-up turn that
includes the previous answer and the validation error, which is both cheaper
and far more likely to succeed, because the model can see what was wrong.

Assumption three: retrying is safe

If the call is a plain completion, a duplicate is waste. If the call is part
of an agent turn that already executed a tool, a duplicate is a second side
effect.

// turn N: model asks to send an email; you send it
// response read fails; retry re-sends the same messages
// model asks to send the email again; you send it again

The retry helper has no idea that the array it is re-sending describes work
already done. Anything with side effects needs an idempotency key derived from
the operation, not the attempt, that is a whole topic, but the minimum is
knowing your retry wrapper cannot make it safe on its own.

What a retry policy for models looks like

type Attempt = { n: number; err: unknown };

export async function callModel(
  params: MessageCreateParams,
  opts: { maxAttempts?: number; budget?: Budget } = {},
) {
  const max = opts.maxAttempts ?? 3;
  let last: unknown;

  for (let n = 1; n <= max; n++) {
    try {
      opts.budget?.assertCanSpend(estimateCost(params));
      const res = await client.messages.create(params);
      opts.budget?.record(params.model, res.usage);
      return res;
    } catch (err) {
      last = err;
      if (!isRetryable(err)) throw err;
      if (n === max) break;
      await sleep(delayFor(err, n));
    }
  }
  throw new ModelUnavailable(max, last);
}

Four things it does that a generic helper does not.

isRetryable is narrow. 429 and 5xx and socket errors. A 400 means your
request is malformed and will be malformed again. A content-policy refusal is
not a transient fault. Retrying either burns your attempts and delays the real
error.

const isRetryable = (e: unknown) =>
  e instanceof APIError &&
  (e.status === 429 || e.status === 408 || (e.status ?? 0) >= 500);

It honours Retry-After. The server knows when it will accept you; your
backoff curve is guessing.

function delayFor(err: unknown, n: number) {
  const hinted = retryAfterMs(err);
  if (hinted) return hinted;
  const base = Math.min(1000 * 2 ** (n - 1), 20_000);
  return base + Math.random() * base * 0.3;   // jitter
}

Jitter matters more than usual here. Rate limits are account-wide, so every
concurrent request in your service gets 429'd at the same instant. Without
jitter they all retry together and trigger the limit again — a thundering herd
you built yourself.

It checks the budget before each attempt. Three attempts at a large
request is three times the cost, and a retry loop is exactly where an
unbounded bill comes from.

It throws a typed error. ModelUnavailable carrying the attempt count and
the last cause lets the caller decide between a cached answer, a smaller
model, or a plain error page.

The one nobody expects: overloaded is not rare

Providers return an overloaded/capacity error under load. It is transient and
it is not rare at peak, and because it looks like a 5xx, a generic retry
handles it by hammering a service that is already saturated.

Treat it as its own case with a longer floor:

if (isOverloaded(err)) {
  await sleep(5_000 + Math.random() * 5_000);
  continue;
}

If you have a fallback model, this is where it earns its place — degrading to
a smaller model beats failing, for most features.

What to log

logger.warn("model retry", {
  attempt: n,
  status: (err as APIError)?.status,
  errorType: err?.constructor?.name,
  delayMs: delay,
  promptVersion: PROMPT.version,
  costSoFarUsd: budget?.spent,
});

Retry counts are a leading indicator. A slow climb in attempt-2 rate means you
are approaching your rate limit before you hit it, and that is the window in
which raising concurrency limits is a planned change rather than an incident.

The short version

Retry transport failures only, with jitter and Retry-After. Send content
failures back as a correction turn, not a retry. Check a budget before every
attempt. Never let a generic helper wrap a call that has already caused a side
effect.

The helper you already have does none of that, and it will look like it is
working right up until the invoice.

If this was useful

AI That Answers covers the failure
modes of a first LLM app in TypeScript — retries, timeouts, streaming, cost
accounting, and the parse boundary that decides which failures are even
retryable.