DEV Community

oleksoleks07
oleksoleks07

Posted on Edited on AI-assisted

Your Retry Logic Can Turn One Timeout Into a Traffic Storm

A request to a payment or inventory API times out after eight seconds. Your client never receives a response.

Did the server fail before doing anything? Did it commit the change and lose the reply? Is it still processing?

You cannot infer the answer from the timeout.

Retrying might restore availability. It might also charge a customer twice or add more traffic to a service that is already struggling.

A retry is not just “try again after a delay.” It is a decision about semantics, load, and time.

Before adding retries, answer three questions:

  1. Is this failure likely to be temporary?
  2. Is this operation safe to repeat?
  3. Does another attempt fit inside the caller's deadline?

1. Retry only failures that might recover

For an HTTP client, separate transport failures from HTTP responses.

fetch() rejects for network-level failures, but it does not reject just because the server returned 429 or 503. Those are normal Response objects. You must inspect response.status yourself.

A reasonable starting policy for a read request might look like this:

Failure Retry by default? Why
Network error Usually, with a limit The connection may have failed temporarily
408 Request Timeout Often The server did not complete the request in time
429 Too Many Requests Yes, but respect Retry-After The server is explicitly applying a rate limit
502, 503, 504 Often A gateway or service may be temporarily unavailable
500 Internal Server Error Depends on the API Some servers use it for transient faults; others do not
400, 401, 403, 404, 422 Usually not Repeating the same request rarely fixes invalid input or permissions

This is a policy, not a universal list. Follow the API's documented behavior. A retry loop should not turn a permanent error into three identical requests.

2. A timeout does not make a write safe to repeat

This is the most important distinction.

Imagine a client sends:

POST /payments
Enter fullscreen mode Exit fullscreen mode

The server creates the payment, but the response is lost. The client times out and sends the same request again.

From the client's point of view, both attempts failed. From the server's point of view, the first one may have succeeded.

For writes, retry only when the API provides a documented idempotency mechanism or another way to make duplicate submissions safe. If the API supports an Idempotency-Key header:

  • Generate one key for the logical operation.
  • Reuse that exact key for every retry of that operation.
  • Do not generate a new key for each attempt.
  • Check the provider's retention window and what it guarantees for repeated keys.

A client-side timeout or abort does not roll back a side effect that the server has already committed.

The helper below deliberately retries safe HTTP methods only. That is a useful default for shared code. Add write retries only after verifying the server-side contract.

3. Respect Retry-After

When a server sends Retry-After, treat it as a minimum wait rather than ignoring it and using your own tiny delay.

The value can be either:

  • A number of seconds, such as Retry-After: 30
  • An HTTP date, such as Retry-After: Fri, 20 Nov 2026 15:00:00 GMT

The behavior is described in RFC 9110. The 429 status code may also include this header; see RFC 6585, section 4.

If a service asks you to wait 30 seconds, retrying after 200 milliseconds is not resilience. It is ignoring the service's feedback.

4. Add backoff, jitter, and one overall deadline

If every client retries after exactly one second, they can all hit the recovering service at the same moment.

Use exponential backoff with jitter:

attempt 1: random delay between 0 and 200 ms
attempt 2: random delay between 0 and 400 ms
attempt 3: random delay between 0 and 800 ms
Enter fullscreen mode Exit fullscreen mode

The random part spreads retries across time instead of creating another synchronized burst. AWS explains the failure-amplification problem and the value of jitter in its timeouts, retries, and backoff guidance.

Also set a total deadline for the logical operation. Do not give every retry a fresh eight-second timeout if the caller only has eight seconds in total.

5. A small Node.js implementation

This example uses Node.js with built-in fetch, AbortSignal.timeout(), and AbortSignal.any() (Node 20.3+ for any()). It:

  • Allows only safe methods.
  • Retries a small set of potentially transient failures.
  • Understands both forms of Retry-After.
  • Uses full jitter when the server does not specify a delay.
  • Shares one deadline across all attempts.
  • Stops if the caller cancels the operation.
const RETRYABLE_STATUS = new Set([408, 429, 500, 502, 503, 504]);

function parseRetryAfter(value) {
  if (!value) return null;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) {
    return Math.max(0, seconds * 1000);
  }

  const date = Date.parse(value);
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

function sleep(ms, signal) {
  return new Promise((resolve, reject) => {
    const onAbort = () => {
      clearTimeout(timer);
      reject(signal.reason ?? new Error("Aborted"));
    };

    const timer = setTimeout(() => {
      signal.removeEventListener("abort", onAbort);
      resolve();
    }, ms);

    signal.addEventListener("abort", onAbort, { once: true });
    if (signal.aborted) onAbort();
  });
}

async function fetchWithRetry(url, {
  maxAttempts = 3,
  timeoutMs = 8_000,
  baseDelayMs = 200,
  maxDelayMs = 2_000,
  signal: callerSignal,
  ...init
} = {}) {
  if (!Number.isInteger(maxAttempts) || maxAttempts < 1) {
    throw new RangeError("maxAttempts must be at least 1");
  }

  const method = (init.method ?? "GET").toUpperCase();

  if (!["GET", "HEAD", "OPTIONS"].includes(method)) {
    throw new Error(
      "This helper retries safe methods only. Add write retries only when the API documents idempotency support."
    );
  }

  const timeoutSignal = AbortSignal.timeout(timeoutMs);
  const signal = callerSignal
    ? AbortSignal.any([callerSignal, timeoutSignal])
    : timeoutSignal;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    signal.throwIfAborted();

    let response;

    try {
      response = await fetch(url, { ...init, signal });
    } catch (error) {
      if (
        signal.aborted ||
        !(error instanceof TypeError) ||
        attempt === maxAttempts
      ) {
        throw error;
      }

      const ceiling = Math.min(
        maxDelayMs,
        baseDelayMs * 2 ** (attempt - 1)
      );

      await sleep(Math.random() * ceiling, signal);
      continue;
    }

    if (
      response.ok ||
      !RETRYABLE_STATUS.has(response.status) ||
      attempt === maxAttempts
    ) {
      return response;
    }

    const retryAfterMs = parseRetryAfter(
      response.headers.get("retry-after")
    );

    const ceiling = Math.min(
      maxDelayMs,
      baseDelayMs * 2 ** (attempt - 1)
    );

    const jitterMs = Math.random() * ceiling;

    // Retry-After is a minimum; add a little jitter to spread clients out.
    const delayMs = retryAfterMs === null
      ? jitterMs
      : retryAfterMs + Math.random() * Math.min(250, ceiling);

    await response.body?.cancel().catch(() => {});
    await sleep(delayMs, signal);
  }

  throw new Error("Retry loop exhausted unexpectedly");
}
Enter fullscreen mode Exit fullscreen mode

Usage:

const response = await fetchWithRetry(
  "https://api.example.com/catalog",
  {
    headers: { authorization: "Bearer " + token },
    timeoutMs: 5_000,
    maxAttempts: 3
  }
);

if (!response.ok) {
  throw new Error("Catalog request failed: " + response.status);
}

const catalog = await response.json();
Enter fullscreen mode Exit fullscreen mode

A few details are intentional:

  • The timeout covers the whole retry sequence, not each attempt independently.
  • The last HTTP response is returned so the caller can inspect and handle it.
  • A network failure is retried only when it surfaces as a TypeError; aborts and other errors are not retried.
  • 500 is included as a candidate status, but remove it if your provider documents it as non-transient.

For a production client, also validate the URL and configuration at the boundary, and test the helper against the actual HTTP client and runtime versions you deploy.

6. Avoid retry multiplication

Retries become dangerous when several layers each implement their own policy.

If an API client makes three attempts, a service layer makes three attempts, and a job runner makes three attempts, one logical operation can create up to:

3 × 3 × 3 = 27 requests
Enter fullscreen mode Exit fullscreen mode

Choose one layer to own retries whenever possible. Other layers should pass deadlines and cancellation signals down rather than silently starting their own retry loops.

Measure retries as well as failures. At minimum, record:

  • Logical operation name and final outcome
  • Attempt count and total elapsed time
  • HTTP status or classified network error
  • Delay chosen and whether Retry-After was present
  • Whether the caller deadline or cancellation stopped the operation

Do not log authorization headers, raw idempotency keys, or sensitive request bodies.

A practical review checklist

Before merging retry logic, ask:

  • [ ] Are only transient failures retried?
  • [ ] Are writes protected by a server-supported idempotency contract?
  • [ ] Is Retry-After respected?
  • [ ] Is there jitter and a maximum attempt count?
  • [ ] Is there one deadline for the entire operation?
  • [ ] Can the caller cancel queued work and in-flight requests?
  • [ ] Is retry behavior owned by one layer and visible in telemetry?

Retries should make a system more tolerant of brief failures, not hide a broken dependency or amplify an outage.

What has caused more trouble in your systems: missing retries, retries that happen too aggressively, or duplicate side effects after a timeout?

Top comments (0)