DEV Community

ThalynRift3485
ThalynRift3485

Posted on

Node.js Fintech Cleanup: How to Retry Failed Webhook Callbacks with Delayed Jobs in 2026

Short answer: put each cleanup command in a durable queue, give it a stable idempotency key, retry only transient callback failures with delayed jobs, and move exhausted work to a dead-letter queue for deliberate replay.

Pattern Survives a process restart? Retry control Best fit
In-process timer No Hand-built Disposable local work
Cron-triggered scan Depends on the job store Per scan Coarse reconciliation
Durable delayed queue Yes Per job Fintech cleanup with callbacks

Use the durable delayed queue for the primary path, then run a small scheduled reconciliation scan as a backstop. This choice is about ownership of state, not brand preference: a web request shouldn't remain open while a reservation hold expires, and a timer in one Node.js process can't be the record of work after that process exits.

The catch is operational weight. A queue adds persistence, workers, metrics, and a replay procedure. For a nightly, fully idempotent batch with no per-item deadline, stick with a cron-triggered database scan. It has fewer moving parts and makes the database the obvious source of truth.

How should a Node.js queue retry failed webhook callbacks?

Start by separating business completion from delivery completion. In this example, the business event is hold.expired; the callback is only a notification about it. The cleanup worker should commit the hold's final state before it attempts the webhook. Otherwise a slow receiver can extend a financial reservation merely by withholding an acknowledgment.

Treat callback outcomes by class. A 2xx response acknowledges delivery. A 408, 429, or a network timeout can enter a delayed retry because another attempt may succeed. Most other 4xx responses need review rather than blind repetition: retrying malformed authentication or an invalid URL ten times creates noise, not resilience. HTTP defines 429 as Too Many Requests, and a server may include Retry-After; honor that signal when it is present.

Keep the policy bounded. For example, five attempts with exponential delay and jitter produce a clear terminal state without synchronizing every failed callback on the same second. Those numbers are an example policy, not a universal optimum. I'm not sure any fixed schedule is right for every receiver; the evidence that resolves it is your own distribution of recovery time, rate-limit responses, and callback latency.

No magic here.

The queue record should carry an immutable event ID, account ID, hold ID, target URL identifier, attempt count, and notBefore timestamp. Don't put mutable credentials or an entire customer object in the job. Resolve secrets at execution time, and fetch the current business record before applying a transition. That keeps queued data small and prevents a delayed job from reviving stale state.

Idempotency is the actual boundary

Delivery is normally at least once at the application boundary: a worker can send a callback successfully and lose its acknowledgment before it records success. The next worker cannot know whether the receiver acted. Pretending the ambiguity doesn't exist is how duplicate ledger side effects appear.

Use one stable key for one logical event, such as hold-expired:<holdId>:<version>. Store it with a uniqueness constraint beside the state transition. The producer may enqueue again after a timeout, but only one transaction creates the outbox event. On the receiving side, send that same key in an idempotency header and require consumers to record processed event IDs before performing non-repeatable work.

This is the important split: retry count belongs to a delivery attempt, while the idempotency key belongs to the logical event. Generating a fresh key on every retry defeats deduplication. Reusing one key for several distinct hold versions suppresses valid work. Both mistakes can look fine in a happy-path demo.

The database transaction should also guard the cleanup itself. An update shaped like WHERE status = 'active' AND expires_at <= now() makes a second execution a no-op after the first succeeds. The callback payload can then report the committed version. Exactly-once delivery isn't the useful promise; one observable business effect is.

Implement the worker contract in TypeScript

The useful abstraction is tiny. It doesn't need an SDK-sized configuration surface. A producer schedules a typed command, a worker leases due work, and each outcome becomes ack, retry, or deadLetter. The adapter behind this interface can use a hosted queue or a self-managed broker without leaking vendor-specific fields into business code.

type CleanupJob = {
  eventId: string;
  holdId: string;
  accountId: string;
  callbackRef: string;
  notBefore: string;
  attempt: number;
};

type Result =
  | { kind: "ack" }
  | { kind: "retry"; runAt: string; reason: string }
  | { kind: "deadLetter"; reason: string };

interface JobStore {
  enqueue(job: CleanupJob): Promise<void>;
  ack(eventId: string): Promise<void>;
  retry(job: CleanupJob, runAt: string, reason: string): Promise<void>;
  deadLetter(job: CleanupJob, reason: string): Promise<void>;
}
Enter fullscreen mode Exit fullscreen mode

The handler below keeps policy explicit. expireHold must be a transactional, idempotent database operation. resolveCallback looks up the current endpoint and secret by reference. The callback gets a stable event ID; retries change the attempt number, not that ID.

const MAX_ATTEMPTS = 5;
const BASE_DELAY_MS = 30_000;

function retryDelayMs(attempt: number, random = Math.random): number {
  const exponential = BASE_DELAY_MS * 2 ** Math.max(0, attempt - 1);
  const jitter = Math.floor(exponential * 0.2 * random());
  return exponential + jitter;
}

async function deliver(job: CleanupJob): Promise<Result> {
  const committed = await expireHold(job.holdId, job.accountId);
  const callback = await resolveCallback(job.callbackRef);

  const response = await fetch(callback.url, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": job.eventId,
      authorization: `Bearer ${callback.token}`,
    },
    body: JSON.stringify({
      id: job.eventId,
      type: "hold.expired",
      holdId: job.holdId,
      version: committed.version,
    }),
    signal: AbortSignal.timeout(10_000),
  });

  if (response.ok) return { kind: "ack" };

  const retryable = response.status === 408 || response.status === 429;
  if (!retryable || job.attempt >= MAX_ATTEMPTS) {
    return { kind: "deadLetter", reason: `callback status ${response.status}` };
  }

  const delayMs = retryDelayMs(job.attempt);
  return {
    kind: "retry",
    runAt: new Date(Date.now() + delayMs).toISOString(),
    reason: `callback status ${response.status}`,
  };
}

async function run(job: CleanupJob, store: JobStore): Promise<void> {
  try {
    const result = await deliver(job);
    if (result.kind === "ack") await store.ack(job.eventId);
    if (result.kind === "retry") {
      await store.retry({ ...job, attempt: job.attempt + 1 }, result.runAt, result.reason);
    }
    if (result.kind === "deadLetter") await store.deadLetter(job, result.reason);
  } catch (error) {
    if (job.attempt >= MAX_ATTEMPTS) {
      await store.deadLetter(job, error instanceof Error ? error.name : "unknown error");
      return;
    }
    const runAt = new Date(Date.now() + retryDelayMs(job.attempt)).toISOString();
    await store.retry({ ...job, attempt: job.attempt + 1 }, runAt, "transport failure");
  }
}
Enter fullscreen mode Exit fullscreen mode

Production code should parse Retry-After, sign webhook bodies, cap response reads, and distinguish cancellation from a transport failure. Those are deliberate omissions from the compact example. They aren't excuses to bury behavior in configuration: keep the retry classifier and delay function reviewable in source control.

For technology selection, compare the behavior you can prove. BullMQ provides delayed jobs on Redis, RabbitMQ offers acknowledgments and dead-letter exchanges, and Amazon SQS provides visibility timeouts plus dead-letter queues. Their persistence, delay semantics, ordering, and operating model differ, so test the exact adapter rather than assuming the generic contract erases those differences. A Redis-backed queue can suit a team already operating Redis; a broker is attractive when routing is central; a managed queue trades local control for less infrastructure ownership. None of those facts picks a winner without workload constraints.

Test recovery, not just submission

The fastest queue API is irrelevant if replay is guesswork. Benchmark time-to-first-call during evaluation, but make the acceptance test about state transitions. Start with an active hold named hold_1042, an expiry in the past, and event ID hold-expired:hold_1042:3. Enqueue it twice to prove that producer duplication does not create two outbox records. Let the first worker commit the conditional expiry and send the callback, then terminate that worker after the receiver stores the event ID but before the worker acknowledges its queue lease. On restart, the queue should deliver the command again. The cleanup update now affects zero rows because the hold is already expired; the receiver also recognizes the same event ID, so the test finishes with one hold transition and one downstream effect even though it observed two deliveries. Next, configure the test receiver to return 429 with a valid Retry-After, verify that no callback arrives before the indicated time, and then allow 2xx. Run a separate case that exhausts the five-attempt budget, inspect the dead-letter record for its event ID and attempt history, repair the receiver configuration, and replay without minting a new identity. Finally, run the reconciliation scan over the same row and prove the uniqueness constraint prevents another logical event. This sequence is slower than asserting that enqueue() returned. It also tests the ambiguity that matters.

Break it on purpose.

Track queue age, attempts by outcome, callback latency, dead-letter growth, and the age of the oldest dead letter. A queue depth of ten may be harmless at midnight and alarming five minutes before settlement; age carries the context depth lacks. Logs need eventId, holdId, attempt, scheduled time, start time, HTTP outcome, and final disposition. Avoid logging tokens or full financial payloads.

Deploy producers before consumers only when the old consumer can safely ignore or reject the new schema. A simpler rule is versioning the job body and supporting the previous version during a rolling deployment. Keep a reconciliation query too: select expired active holds with no outbox event, enqueue their stable keys, and let the uniqueness constraint collapse overlap. Schedule that scan with cron or a workflow scheduler, but don't make its clock the only record of each deadline.

This is also where cost belongs: measure retained job volume, callback egress, broker operations, and engineer time spent on upgrades. A cheap request price can be erased by a week of bespoke replay tooling. Your mileage may vary — especially when compliance rules determine retention and access controls — so put those requirements in the evaluation sheet before comparing throughput charts.

When should cron or a database scan win?

Use cron for coarse, repeatable maintenance where a missed invocation can be recovered by the next scan. Cron is a time-based job scheduler, and hosted workflow systems can also trigger work on a schedule, but scheduled execution does not itself define per-item leases, retry ownership, or idempotency. Your application still owns those decisions.

A database scan is the runner-up for fintech cleanup when the table is already authoritative, deadlines tolerate the scan interval, and the operation is naturally expressed as a conditional update. It is not suitable when millions of items share one deadline, each receiver needs independent backoff, or operators must pause and replay one tenant without rescanning everything. Pick the queue there.

Keep the scan when simplicity wins. Pick the durable queue when each delayed job needs its own retry clock, audit trail, and dead-letter decision. In both cases, the invariant remains the same: stable event identity plus a conditional business transition. The scheduler is replaceable. That invariant isn't.

References

Top comments (0)