Retry logic that survives 429 and 503, tested against a live API

#api #javascript #testingapis #node

Almost every retry helper I've reviewed had at least one of these bugs:

It retries a 400, so a bad request gets sent four times instead of once.
It ignores Retry-After and hammers a server that already told it when to come back.
It has no timeout, so one hung connection blocks the whole job.
Every client waits exactly the same time, so they all come back at the same moment and trigger the next 429.

The reason these survive is that retries are hard to test. Your real API rarely fails on demand. So I added failure endpoints to testingapis.com (a free fake API I build) and used them to check my own helper. Here's the helper and what happened.

What to retry

Status	Retry?	Why
408, 429	Yes	The server asked you to slow down or try again
500, 502, 503, 504	Yes	Usually temporary
Network error, timeout	Yes	The request may never have arrived
400, 401, 403, 404, 422	No	Sending it again gives the same answer

One warning: retrying a POST that timed out can create the order twice. Only retry non-idempotent requests if the API supports an idempotency key.

The helper

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

export async function fetchWithRetry(url, opts = {}, { tries = 4, baseMs = 500, maxMs = 10_000 } = {}) {
  for (let attempt = 1; ; attempt++) {
    let res;
    try {
      res = await fetch(url, { ...opts, signal: AbortSignal.timeout(5_000) });
      if (!RETRYABLE.has(res.status) || attempt === tries) return res;
    } catch (err) {
      if (attempt === tries) throw err;          // network error or timeout
    }
    const retryAfter = Number(res?.headers.get("retry-after"));
    const backoff = Math.min(maxMs, baseMs * 2 ** (attempt - 1));
    const wait = retryAfter > 0 ? retryAfter * 1000 : Math.random() * backoff; // full jitter
    console.log(`attempt ${attempt}: ${res?.status ?? "network error"}, waiting ${Math.round(wait)}ms`);
    await new Promise((r) => setTimeout(r, wait));
  }
}

Three decisions worth explaining:

Retry-After wins. If the server says "wait 1 second", that's better information than any formula. (It can also be an HTTP date. If you call APIs that send dates, parse those too.)
Full jitter. Waiting a random time between 0 and the backoff spreads clients out. AWS wrote the classic post on why.
A per-attempt timeout. AbortSignal.timeout is built into Node 18+ and browsers. Without it, fetch can wait for minutes.

Testing it against real failures

Three cases, one each for "retry and succeed", "don't retry" and "time out":

import { fetchWithRetry } from "./retry.mjs";
const base = "https://api.testingapis.com/api";
const key = `devto-${Date.now()}`;   // a fresh counter for this run

// Fails twice with 429, then succeeds
let res = await fetchWithRetry(`${base}/chaos/retry/${key}?succeedAfter=2&failStatus=429`);
console.log("final:", res.status, "x-attempt", res.headers.get("x-attempt"));

// A 400 must not be retried
res = await fetchWithRetry(`${base}/status/400`);
console.log("400 final:", res.status);

// Never answers within our 5s timeout
const t = Date.now();
res = await fetchWithRetry(`${base}/chaos/timeout?seconds=30`, {}, { tries: 2 }).catch((e) => e);
console.log("timeout:", res.name, `after ${Math.round((Date.now() - t) / 1000)}s`);

Output from my run (Node 24):

attempt 1: 429, waiting 1000ms
attempt 2: 429, waiting 1000ms
final: 200 x-attempt 3
400 final: 400
attempt 1: network error, waiting 466ms
timeout: TimeoutError after 10s

What this shows:

The 429s came with Retry-After: 1, and the helper waited exactly that, not its own backoff. The third attempt succeeded (x-attempt: 3).
The 400 came back on the first try, with no "attempt" lines. Good.
The timeout case gave up after two 5-second attempts plus a random jitter wait (yours will differ), and threw a TimeoutError instead of hanging for 30 seconds.

Turning it into a test

Once it works, pin it down so nobody "simplifies" it later:

import { test } from "node:test";
import assert from "node:assert/strict";
import { fetchWithRetry } from "./retry.mjs";

const base = "https://api.testingapis.com/api";

test("retries 429 until it succeeds", async () => {
  const res = await fetchWithRetry(`${base}/chaos/retry/ci-${Date.now()}?succeedAfter=2&failStatus=429`);
  assert.equal(res.status, 200);
  assert.equal(res.headers.get("x-attempt"), "3");
});

test("does not retry a 400", async () => {
  const res = await fetchWithRetry(`${base}/status/400`);
  assert.equal(res.status, 400);
});

Run it with node --test.

Other failures worth throwing at it

/api/chaos/flaky?failRate=0.3: a 503 about 30% of the time, like a real flaky dependency.
/api/Products/1?delay=3: slow but successful. Check your timeout doesn't fire too early.
/api/chaos/partial: the headers promise 10,000 bytes and the connection closes after about 64. Most clients throw while reading the body, after fetch resolved, so the retry loop above doesn't catch it. Worth knowing.

The full list of failure endpoints is in the mock error responses guide. It's free and needs no sign-up.

How does your team handle retries on POST requests? Idempotency keys, or just no retries at all?

DEV Community