Why "just retry it" can be dangerous

Your Node.js service calls an external API. The request times out, returns 503 Service Unavailable, or gets a 429 Too Many Requests response. A retry sounds reasonable: wait briefly, send the same request again, and continue.

But not every failed-looking request actually failed on the server.

A timeout means the client did not receive the expected result in time. The server might have completed the operation and lost the response on the way back. If the request created a payout, order, or exchange operation, a blind retry could create it twice.

The practical goal isn't to retry everything. It's to retry only when the operation is safe, with a clear limit and a delay that respects the server.

We'll build a small JSON HTTP client using built-in Node.js APIs, then test transient errors, backoff, Retry-After, timeouts, and duplicate-request safety.

Scope: This is a standalone educational example, not the internal infrastructure of any cryptocurrency platform. It uses Node.js 22-compatible APIs, supports GET and explicitly opted-in POST, and expects a JSON response body on success. It is not a production-ready transaction client.

First, classify the failure

These cases require different decisions:

What happened	Suggested behavior in this example
Network failure or request timeout	Retry an eligible operation, within a bounded attempt limit
429 Too Many Requests	Respect Retry-After when provided; otherwise use backoff
503 Service Unavailable, 502, 504	Consider a bounded retry for eligible operations
408 Request Timeout or selected 500 errors	Retry only according to a deliberate policy
400, 401, 403, 404	Do not automatically retry without a change or an explicit contract
POST with an unknown result	Do not retry blindly; confirm idempotency support or reconcile the operation

fetch() doesn't reject just because a server responds with HTTP 503. You still need to inspect response.status. Network errors and aborts, on the other hand, reject the promise.

1. Put a timeout on each attempt

Modern Node.js has fetch() and AbortSignal.timeout() built in:

const response = await fetch('https://api.example.test/data', {
  signal: AbortSignal.timeout(3000),
});

The URL above is deliberately a placeholder. Replace it with a real endpoint that you control or are authorized to query.

With a timeout signal, a request is aborted after the configured interval. Crucially, aborting the client-side request does not undo a remote operation that the server may already have started or completed.

Also distinguish two kinds of cancellation:

• Per-attempt timeout: One attempt took too long. An eligible read can be retried.
• Caller cancellation: The user, job, or parent request no longer needs the result. Stop the entire retry loop.

AbortSignal.any() combines the two signals. Our client will also read the JSON response inside the attempt's timeout window, instead of returning a Response and parsing its body later.

2. Add exponential backoff with jitter

Retrying immediately after a failure can make an overloaded service even slower. A simple exponential backoff increases the potential delay between attempts:

Attempt 1 fails → max backoff 300 ms
Attempt 2 fails → max backoff 600 ms
Attempt 3 fails → max backoff 1,200 ms
Attempt 4       → no retry after final failure

These are delay caps, not the exact sleeps. We'll use full jitter: choose a random delay between zero and the current capped value. That spreads clients out, helping avoid a synchronized retry spike.

The policy is bounded: four total attempts means one initial request plus at most three retries—not four retries.

3. Respect Retry-After without retrying too early

For 429 and 503, servers may include Retry-After:

HTTP/1.1 429 Too Many Requests
Retry-After: 5

That means wait five seconds before another attempt. The header can also be an HTTP date.

Our implementation has a maximum amount of time it is willing to wait. If the server asks for an hour but the operation has only a ten-second wait budget, the client stops. It must not quietly truncate the server's required delay and retry too early. A higher-level job scheduler may choose to resume later.

4. Build a reusable JSON retry client

Create retry-json.mjs:

import { setTimeout as delay } from 'node:timers/promises';

const RETRYABLE_STATUSES = new Set([408, 429, 500, 502, 503, 504]);
const defaultWait = (ms, signal) => delay(ms, undefined, { signal });

export function parseRetryAfter(value, now = Date.now()) {
  if (value == null) return null;
  const text = value.trim();
  if (/^\d+$/.test(text)) {
    const seconds = Number(text);
    return Number.isSafeInteger(seconds) && seconds <= Number.MAX_SAFE_INTEGER / 1000
      ? seconds * 1000 : Infinity;
  }
  // The commonly used IMF-fixdate HTTP-date format, e.g. Fri, 23 Oct 2026 12:00:03 GMT.
  if (!/^[A-Za-z]{3}, \d{2} [A-Za-z]{3} \d{4} \d{2}:\d{2}:\d{2} GMT$/.test(text)) {
    return null;
  }
  const timestamp = Date.parse(text);
  return Number.isNaN(timestamp) ? null : Math.max(0, timestamp - now);
}

export async function fetchJsonWithRetry(url, {
  method = 'GET',
  headers = {},
  body,
  maxAttempts = 4,
  requestTimeoutMs = 3000,
  baseDelayMs = 300,
  maxBackoffMs = 3000,
  maxRetryAfterMs = 10_000,
  allowPostRetry = false,
  idempotencyKey,
  signal,
  fetchImpl = fetch,
  wait = defaultWait,
  random = Math.random,
  now = Date.now,
} = {}) {
  const httpMethod = method.toUpperCase();
  if (!['GET', 'POST'].includes(httpMethod)) {
    throw new TypeError('This JSON example supports GET and POST only');
  }
  if (!Number.isInteger(maxAttempts) || maxAttempts < 1 || maxAttempts > 8 ||
      !Number.isSafeInteger(requestTimeoutMs) || requestTimeoutMs < 1 ||
      !Number.isSafeInteger(baseDelayMs) || baseDelayMs < 1 ||
      !Number.isSafeInteger(maxBackoffMs) || maxBackoffMs < baseDelayMs ||
      !Number.isSafeInteger(maxRetryAfterMs) || maxRetryAfterMs < 0) {
    throw new RangeError('Invalid retry configuration');
  }
  if (httpMethod === 'POST' && allowPostRetry &&
      (typeof idempotencyKey !== 'string' || !idempotencyKey.trim() || typeof body !== 'string')) {
    throw new TypeError('Retrying POST requires a stable key and repeatable string body');
  }

  // POST retries are an explicit opt-in; the remote API must guarantee deduplication.
  const canRetry = httpMethod === 'GET' || (httpMethod === 'POST' && allowPostRetry);
  const requestHeaders = new Headers(headers);
  if (idempotencyKey != null) requestHeaders.set('Idempotency-Key', idempotencyKey);

  const jitter = (attempt) => {
    const cap = Math.min(maxBackoffMs, baseDelayMs * 2 ** (attempt - 1));
    return Math.floor(random() * (cap + 1)); // Full jitter in [0, cap].
  };

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    signal?.throwIfAborted();
    const timeoutSignal = AbortSignal.timeout(requestTimeoutMs);
    const attemptSignal = signal
      ? AbortSignal.any([signal, timeoutSignal])
      : timeoutSignal;
    let response;

    try {
      response = await fetchImpl(url, {
        method: httpMethod,
        headers: requestHeaders,
        body,
        signal: attemptSignal,
      });
    } catch (error) {
      if (signal?.aborted) throw signal.reason;
      const transient = error instanceof TypeError || timeoutSignal.aborted;
      if (!canRetry || attempt === maxAttempts || !transient) throw error;
      await wait(jitter(attempt), signal);
      continue;
    }

    if (response.ok) {
      try {
        // Read within the timeout window, not after returning the Response.
        return await response.json();
      } catch (error) {
        if (signal?.aborted) throw signal.reason;
        if (!canRetry || attempt === maxAttempts || !timeoutSignal.aborted) throw error;
        await wait(jitter(attempt), signal);
        continue;
      }
    }

    const status = response.status;
    const header = response.headers.get('Retry-After');
    // Release the body before retrying to avoid holding connections unnecessarily.
    if (response.body) {
      try { await response.body.cancel(); } catch { /* already closed */ }
    }
    if (!canRetry || !RETRYABLE_STATUSES.has(status) || attempt === maxAttempts) {
      throw new Error(`HTTP ${status}`);
    }
    const serverWaitMs = parseRetryAfter(header, now());
    if (serverWaitMs !== null && serverWaitMs > maxRetryAfterMs) {
      // Do not silently cap Retry-After and hit the server too early.
      throw new Error(`Retry-After exceeds local wait budget (HTTP ${status})`);
    }
    await wait(serverWaitMs ?? jitter(attempt), signal);
  }
}

A few choices are intentional:

• The helper only retries GET by default. It supports POST, but does not retry it unless the caller explicitly opts in and supplies a stable idempotency key and reusable string body.
• The server—not the header alone—must actually implement idempotency semantics for that endpoint. Sending an arbitrary Idempotency-Key to an API that ignores it provides no protection.
• The sample treats 408, 429, 500, 502, 503, and 504 as retryable HTTP statuses only for requests eligible for retry. This is an illustrative policy, not a universal rule for every API.
• Invalid JSON is not treated as a transient network problem. The request succeeded at the HTTP level but its response is not valid for this JSON client.
• The example accepts standard Retry-After seconds and the common IMF-fixdate HTTP-date format. Other historical date formats are outside its scope.

5. Use it for a read-only API request

import { fetchJsonWithRetry } from './retry-json.mjs';

// Replace this placeholder with your own JSON GET endpoint.
const data = await fetchJsonWithRetry('https://api.example.test/rates', {
  maxAttempts: 4,
  requestTimeoutMs: 3000,
  baseDelayMs: 300,
  maxBackoffMs: 3000,
});

console.log(data);

This is a usage illustration. The .test file below uses mock responses instead of making real HTTP requests, so you can reproduce the results without an API key or an internet connection.

6. What about POST and duplicate operations?

HTTP defines GET as safe and idempotent, assuming the server implements the method correctly. A POST may create a new resource or trigger a financial side effect; repeating it can repeat the side effect.

Imagine creating a payout operation:

1. Your client sends POST /payouts.
2. The server creates the payout.
3. The response gets lost, and your client times out.
4. A naive retry sends another POST /payouts.

There could now be two payouts. A client-side timeout is not proof that the first payout failed.

Some APIs explicitly support a stable idempotency key for operations like this. If—and only if—the provider documents and guarantees that contract for the endpoint, you may opt in:

import { fetchJsonWithRetry } from './retry-json.mjs';

// ILLUSTRATIVE ONLY: the target endpoint must document idempotency support.
const payload = JSON.stringify({
  amount: '100.00',
  currency: 'USD',
});

const operation = await fetchJsonWithRetry('https://api.example.test/payouts', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: payload,
  allowPostRetry: true,
  idempotencyKey: 'payout-workflow-123',
});

The same key and the exact same payload are sent on every attempt. In real systems, persist the key before calling the provider, bind it to the logical business operation, and keep it stable across process restarts. The provider must reject conflicting payloads under the same key and implement its own deduplication; key retention periods and response-replay behavior depend on that provider's contract.

If you cannot verify the remote operation's outcome or the provider's idempotency semantics, reconcile before retrying. That may mean querying the operation by your stable reference or escalating the uncertain result to a reconciliation worker.

7. Test the retry policy without external requests

The helper allows fetch, waiting, and randomness to be injected. This lets us simulate different failures quickly and deterministically.

Create retry-json.test.mjs. Here is a focused set of runnable tests:

import test from 'node:test';
import assert from 'node:assert/strict';
import { fetchJsonWithRetry } from './retry-json.mjs';

const url = 'https://api.example.test/data';
const reply = (status, data = {}, headers = {}) =>
  new Response(JSON.stringify(data), { status, headers });

test('GET retries 503 twice, then succeeds', async () => {
  const steps = [reply(503), reply(503), reply(200, { ok: true })];
  const waits = [];
  const result = await fetchJsonWithRetry(url, {
    fetchImpl: async () => steps.shift(),
    baseDelayMs: 100,
    random: () => 0.5,
    wait: async (ms) => waits.push(ms),
  });
  assert.deepEqual(result, { ok: true });
  assert.deepEqual(waits, [50, 100]);
});

test('429 honors Retry-After', async () => {
  const steps = [reply(429, {}, { 'Retry-After': '2' }), reply(200)];
  const waits = [];
  await fetchJsonWithRetry(url, {
    fetchImpl: async () => steps.shift(),
    wait: async (ms) => waits.push(ms),
  });
  assert.deepEqual(waits, [2000]);
});

test('does not retry a POST by default', async () => {
  let calls = 0;
  await assert.rejects(fetchJsonWithRetry(url, {
    method: 'POST',
    body: '{"amount":"100"}',
    fetchImpl: async () => { calls++; return reply(503); },
  }), /HTTP 503/);
  assert.equal(calls, 1);
});

test('opted-in POST reuses its idempotency key', async () => {
  const headersSeen = [];
  const result = await fetchJsonWithRetry(url, {
    method: 'POST',
    body: '{"amount":"100"}',
    allowPostRetry: true,
    idempotencyKey: 'operation-123',
    fetchImpl: async (_, options) => {
      headersSeen.push(options.headers.get('Idempotency-Key'));
      return headersSeen.length === 1 ? reply(503) : reply(200, { ok: true });
    },
    wait: async () => {},
  });
  assert.deepEqual(result, { ok: true });
  assert.deepEqual(headersSeen, ['operation-123', 'operation-123']);
});

Run them with Node.js 22 or another compatible current version:

node --test retry-json.test.mjs

An extended local test suite of 16 cases also checks timeouts, caller cancellation, invalid JSON, excessive Retry-After values, HTTP dates, validation, and retry exhaustion. All 16 passed on Node.js 22.16.0. These are mock-based tests, not performance benchmarks or tests against a real payment provider.

8. Production concerns beyond this demo

A short retry helper is only one layer of resilience. Before adopting a policy in a production backend, consider:

• End-to-end deadline: The demo limits each attempt and each Retry-After wait, but does not impose one explicit absolute deadline for the whole operation. A total deadline, queue deadline, or request budget is often needed.
• Retry amplification: If a frontend, API gateway, job worker, and provider SDK all retry independently, one failure can multiply into many requests. Choose where retries belong.
• Idempotency and persistence: A stable key should be persisted and associated with the intended operation. Your own database may also need unique constraints, event deduplication, and transactional outbox processing.
• Rate limiting: Retry-After helps, but real clients may also need shared rate limits, concurrency limits, and circuit breakers.
• Observability: Record the operation ID, attempt number, HTTP status, elapsed time, and final outcome. Avoid logging access tokens, request secrets, or sensitive payloads.
• Ambiguous outcomes: A timed-out write should enter a reconciliation path when the remote result is unknown. Do not turn uncertainty into an automatic second payment.
• Replayable bodies: The example permits retries only for POST requests with a reusable string body. Streams and other one-shot bodies need different handling.

Key takeaways

Reliable retries require more than a for loop:

1. Timeout each attempt, and recognize that a timeout says nothing definitive about remote side effects.
2. Retry only eligible operations and carefully selected transient failures.
3. Back off with jitter, and honor the server's Retry-After instructions.
4. Bound the total number of attempts and stop when the server's requested delay exceeds your wait budget.
5. Never assume an idempotency key works unless the provider explicitly implements it for that endpoint.
6. Reconcile uncertain writes rather than blindly repeating them.

These practices won't eliminate network failures. They make your Node.js service less likely to turn a temporary problem into extra load, duplicate operations, or incorrect financial state.

References

• Node.js: Global fetch, AbortSignal.timeout, and AbortSignal.any
• Node.js: Timers Promises API
• RFC 9110: HTTP idempotent methods
• RFC 9110: Retry-After
• MDN: 429 Too Many Requests
• AWS: Timeouts, retries, and backoff with jitter

AI disclosure: This initial draft and code examples were generated with AI assistance. The runnable example was tested with Node.js 22.16.0 using mock responses. Human technical review is required before publication.