Stop Retry Storms: Backoff Is Not Enough Without a Budget

A backend can safely process 10,000 requests per second. Traffic rises to 10,100. The extra 100 requests fail, so the clients immediately retry them.

One second later, the backend receives 10,200 requests. Then 10,300. Soon it spends more time rejecting retries than completing original work.

The failure was small. The recovery code turned it into an outage.

A retry is load, not a second chance

Retries are usually introduced like local error handling: catch an exception, wait, try again. But the retry executes on the system that just told you it could not complete the first attempt.

Google's SRE book walks through the 10,000 QPS scenario above. A backend overloaded by only 100 QPS begins receiving an additional 100 retry QPS every round. The feedback loop can consume CPU, memory, file descriptors, and eventually crash replicas. Google says this pattern has contributed to several real cascading failures across RPC, browser, and offline-sync clients.

10,100 original QPS
   100 fail
   100 retry  ──┐
   200 fail     │ positive feedback
   200 retry  ──┘

Dropping original traffic back to 9,000 QPS may not recover the service. If only a fraction of replicas remains healthy, or rejecting a request is expensive, retries can keep the backend above its new reduced capacity.

The first design question is not “how long should I wait?” It is “may this system afford another attempt?”

Layers turn three retries into 64 attempts

Now put the request behind a browser, frontend, API, and database adapter. Each layer sees an error and performs three retries: four total attempts per layer.

The database does not receive four attempts. It can receive:

browser × frontend × backend
   4    ×    4     ×    4    = 64 attempts

That 64-attempt example comes from Google's guidance on cascading failures. The fix is architectural: one layer owns the retry for a failing dependency. The layer immediately above that dependency usually has the best error detail and the smallest amount of duplicated work.

A browser may retry a whole checkout while the API client inside the server also retries the payment call. Even perfect exponential backoff cannot repair that ownership bug; it only spreads 64 attempts over time.

Carry attempt metadata across boundaries, such as attempt: 2 or a request ID. Then a downstream “do not retry” decision can survive the trip back up instead of being translated into a generic 500 that every layer interprets as permission.

Retry only work that is safe to repeat

Transport failure creates an uncomfortable ambiguity: the server may have committed the operation and lost the response. Retrying a read is usually harmless. Retrying POST /charges can charge twice.

RFC 9110 defines PUT, DELETE, and safe methods as idempotent. It also says clients should not automatically retry a non-idempotent request unless they know its semantics are idempotent or know the original request was never applied.

HTTP method alone is not enough. A broken DELETE handler might send a second email, while a POST protected by an idempotency key can be safe. The server needs to make the business operation repeatable:

async function createOrder(request) {
  const key = request.headers.get('Idempotency-Key');
  if (!key) return new Response('key required', { status: 400 });

  return db.transaction(async (tx) => {
    const previous = await tx.idempotency.find(key);
    if (previous) return Response.json(previous.body, { status: previous.status });

    const order = await tx.orders.insert(await request.json());
    await tx.idempotency.insert({ key, status: 201, body: order });
    return Response.json(order, { status: 201 });
  });
}

The idempotency record and side effect must commit atomically. A “check, then insert” outside one transaction has a race: two copies can both observe no record and both create an order. Give the key a unique constraint and define its scope, retention period, and response-replay behavior.

Do not retry authentication failures, validation errors, or malformed requests. No amount of waiting turns a missing field into a valid payload.

Backoff without jitter schedules the next spike

Exponential backoff increases the maximum delay after each failure. If 1,000 clients fail together and all wait exactly 100 ms, then 200 ms, then 400 ms, they remain synchronized. You have replaced one spike with a series of scheduled spikes.

AWS simulated 100 clients contending to update one object. With plain exponential backoff, calls still arrived in clusters. Adding full jitter—choosing a random delay from zero to the exponential cap—cut the call count by more than half compared with no jitter and also improved completion time.

Here is a compact policy for JavaScript:

const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);

async function fetchWithRetry(url, options = {}) {
  const maxAttempts = 3;
  const baseMs = 100;
  const capMs = 2_000;
  const deadlineMs = performance.now() + 5_000;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const remainingMs = deadlineMs - performance.now();
    if (remainingMs <= 0) throw new DOMException('retry deadline exceeded', 'TimeoutError');

    try {
      const response = await fetch(url, {
        ...options,
        signal: AbortSignal.timeout(Math.ceil(Math.min(1_500, remainingMs))),
      });

      if (!RETRYABLE.has(response.status) || attempt === maxAttempts - 1) {
        return response;
      }

      const retryAfter = parseRetryAfter(response.headers.get('retry-after'));
      const windowMs = Math.min(capMs, baseMs * 2 ** attempt);
      const delayMs = retryAfter ?? Math.random() * windowMs;
      if (delayMs >= deadlineMs - performance.now()) return response;
      await sleep(delayMs);
    } catch (error) {
      if (attempt === maxAttempts - 1) throw error;
      const remainingAfterFailure = deadlineMs - performance.now();
      if (remainingAfterFailure <= 0) throw error;
      const windowMs = Math.min(capMs, baseMs * 2 ** attempt);
      await sleep(Math.min(Math.random() * windowMs, remainingAfterFailure));
    }
  }
}

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

function parseRetryAfter(value) {
  if (!value) return null;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
  const dateMs = Date.parse(value);
  return Number.isFinite(dateMs) ? Math.max(0, dateMs - Date.now()) : null;
}

The example is intentionally conservative, not universal. Your API must classify which failures are transient. A 500 from a deterministic bug will fail again. A connection error after sending a non-idempotent body is ambiguous. A response body or domain error can override the status-code default.

Respect server direction. RFC 9110 allows Retry-After on a 503 as either a delay in seconds or an HTTP date. Treat it as a lower bound, validate absurd values, and still enforce the caller's overall deadline. Waiting 120 seconds is not recovery when only 800 ms remains.

A retry budget breaks the feedback loop

Attempts cap the cost of one logical request. A retry budget caps the cost of all requests during a failure.

Google describes two controls working together: up to three attempts per request and a per-client retry ratio below 10%. The per-request limit alone can push worst-case traffic close to 3×. Adding the 10% ratio holds general growth to about 1.1×.

That is the difference between “every failure gets two retries” and “retries may consume at most 10% of current traffic.” When the budget is empty, fail fast. The user sees an error, but the dependency receives the quiet period it needs to recover.

AWS SDKs implement the same principle with a token bucket. Their standard mode uses full-jitter backoff and a retry quota; widespread failures drain tokens, after which calls return errors without retrying. Successful first attempts replenish the budget.

Set budgets per dependency and, where noisy neighbors matter, per tenant or operation class. A healthy inventory service should not donate its retry capacity to a failing recommendation service. Interactive reads and asynchronous jobs should not compete under one policy either.

Observe attempts, not just errors

A dashboard showing a rising 503 rate cannot tell whether retries are helping or feeding the incident. Record these dimensions together:

• original request rate and retry request rate;
• attempts per logical operation;
• success-after-retry versus final failure;
• delay chosen, including whether Retry-After won;
• budget exhaustion and “do not retry” decisions;
• dependency saturation: queue depth, latency percentiles, rejected work.

Alert on retry amplification: (originals + retries) / originals. A ratio of 1.02 during isolated network noise may be healthy. A ratio climbing toward 1.5 while dependency throughput falls is a feedback loop, not resilience.

During an incident, the fastest mitigation may be to disable retries, shed low-priority work, or return a clear overload response. Adding replicas can fail when cold caches and synchronized retries overwhelm each new process as soon as it starts.

The policy in one review checklist

Before approving a retry loop, demand concrete answers:

1. Which exact failures are transient?
2. Is the operation idempotent, or protected by an atomic idempotency key?
3. Which single layer owns the retry?
4. What are the attempt cap and overall deadline?
5. Is delay randomized with full jitter, and is Retry-After honored?
6. What global budget stops retries during widespread failure?
7. Which metric exposes amplification?

Retries spend capacity to buy another probability of success. Spend that capacity only when the failure is transient, the operation is safe, and a budget proves the system can afford it.

Which retry in your stack currently has no owner, no budget, or no idempotency guarantee?