DEV Community

leiferiksson8493
leiferiksson8493

Posted on

Practical Handling of Webhook Task Retries After Failure in Delayed Queue Node.js Systems

Short answer: handle failed webhook task retries with a durable event record, delayed exponential backoff, a leased worker, and receiver-side idempotency. Treat delivery as at-least-once, then make duplicate attempts harmless.

A failed webhook task needs two separate decisions: when to try again, and how to make a repeated attempt harmless. The queue can provide at-least-once work delivery. It cannot make the receiver idempotent for you.

My default design is a durable delivery record, exponential backoff with a cap, a lease around each attempt, and a dead-letter boundary after a small fixed number of failures. The sender records the event ID before it sends. The receiver treats that ID as a deduplication key. This keeps the delivery guarantee explicit: no silent loss, and duplicates are possible but controlled.

The revenue-per-hour test matters here. I want the smallest mechanism that can explain every retry in a log and recover without a manual database edit. I ship weekly, so I outsource undifferentiated queue plumbing to a standard queue or a small internal worker. I keep the delivery state and idempotency rules in my application.

Keep the first version boring.

That is enough.

What guarantee should a delayed webhook retry provide?

Start with the guarantee, not the timer.

For an outbound webhook, the practical contract is usually at-least-once delivery: a task remains eligible until the receiver returns an accepted HTTP response or the task reaches an explicit terminal state. At-most-once delivery avoids duplicates only by accepting a greater chance of loss. Exactly-once delivery across two independent systems is not a property a retry loop can honestly promise.

That leaves one useful rule: retry for eventual delivery, and design the operation to tolerate repetition. A timeout is especially important. The sender may not know whether the receiver committed the request before the connection failed. Retrying can produce two deliveries even when the first attempt succeeded.

A delivery record makes this visible. It should contain an event ID, endpoint, payload version, attempt count, next eligible time, last error class, and terminal status. The event ID stays constant across attempts. The attempt ID changes. Those are different identifiers, and merging them is a common source of bad incident reports.

The smallest worker that preserves the boundary

The worker below assumes a queue has already made a task visible after its delay. It does not delete a task until the delivery state has been updated. The lease prevents two workers from actively processing the same record, but it does not pretend a crash can never create overlap.

The failure sequence is easy to miss. Worker A acquires event evt_42, sends the POST, and waits for a response. The receiver commits the side effect. The connection drops before A receives the response, so A records a network failure. Worker B later sends evt_42 again. Without a unique event key at the receiver, the business action runs twice even though the queue did exactly what it was asked to do. With the key, B gets the stored result and the delivery record can move to delivered. That is the boundary I care about more than a perfect retry count.

type Delivery = {
  eventId: string;
  endpoint: string;
  payload: unknown;
  attempt: number;
  nextAttemptAt: number;
  status: "pending" | "delivered" | "dead";
};

const MAX_ATTEMPTS = 6;
const BASE_DELAY_MS = 1_000;
const MAX_DELAY_MS = 15 * 60_000;

function retryDelay(attempt: number): number {
  const exponential = Math.min(
    MAX_DELAY_MS,
    BASE_DELAY_MS * 2 ** Math.max(0, attempt - 1),
  );
  const jitter = Math.floor(exponential * (0.8 + Math.random() * 0.4));
  return jitter;
}

async function deliver(delivery: Delivery): Promise<void> {
  const lease = await store.acquireLease(delivery.eventId, 30_000);
  if (!lease) return;

  try {
    const response = await fetch(delivery.endpoint, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "idempotency-key": delivery.eventId,
      },
      body: JSON.stringify(delivery.payload),
    });

    if (response.ok) {
      await store.markDelivered(delivery.eventId);
      return;
    }

    const attempt = delivery.attempt + 1;
    if (attempt >= MAX_ATTEMPTS || response.status >= 400 && response.status < 500) {
      await store.markDead(delivery.eventId, `http-${response.status}`);
      return;
    }

    await store.scheduleRetry(
      delivery.eventId,
      attempt,
      Date.now() + retryDelay(attempt),
      `http-${response.status}`,
    );
  } catch (error) {
    const attempt = delivery.attempt + 1;
    if (attempt >= MAX_ATTEMPTS) {
      await store.markDead(delivery.eventId, "network-error");
      return;
    }

    await store.scheduleRetry(
      delivery.eventId,
      attempt,
      Date.now() + retryDelay(attempt),
      error instanceof Error ? error.name : "unknown-error",
    );
  } finally {
    await lease.release();
  }
}
Enter fullscreen mode Exit fullscreen mode

The status test intentionally separates likely permanent client errors from transient failures. That is a policy choice, not a universal HTTP law. Some APIs use a 4xx response for a temporary quota condition; their contract should override this default. A queue retry policy that cannot classify failures will either hammer a broken endpoint or discard work too early.

The markDelivered operation must be idempotent. So must scheduleRetry and markDead. A worker can crash after the remote server accepts the request and before the local transaction commits. The next worker must be allowed to repeat the request and safely record the same final state.

Why the receiver needs a durable idempotency check

A header is a useful transport for an idempotency key. It is not idempotency by itself. The receiver needs a unique constraint on the event ID, or an equivalent atomic insert, before applying a side effect. A read followed by a write leaves a race between two overlapping attempts.

The receiver can claim the key, apply the business change in the same transaction, and return the previous result for a later attempt. If the business operation calls another service, that boundary needs its own idempotency rule or an outbox. Otherwise the first database transaction may be safe while the downstream side effect is duplicated.

This is also why I do not use a payload hash as the primary key. Two legitimate events can have identical JSON, and one event can have a changed representation after serialization. The producer-owned event ID is the stable identity.

Backoff, leases, and the dead-letter boundary

A fixed one-second retry looks friendly in a demo and becomes a synchronized thundering herd in production. Exponential delay with jitter spreads attempts. The cap keeps recovery bounded enough to reason about. Six attempts is an example policy, not a magic number; choose it from the endpoint's recovery window and the business value of the event.

The lease duration must exceed the normal request timeout, with room for the state write. For the 30,000 ms lease in the example, a request that can run longer needs either a longer lease or renewal. If it expires while the request is still running, another worker may start the same attempt. That is acceptable only because the receiver is idempotent. A longer lease reduces overlap but delays recovery after a crashed worker.

After the retry budget is exhausted, move the task to a dead-letter state with the event ID, endpoint, attempt history, and last response class. A dead-letter queue is a recovery boundary, not a trash can. It needs an operator path: inspect, fix the cause, replay with the same event ID when the receiver supports deduplication, and preserve the original attempt history. Queue documentation commonly describes this separation as a way to isolate messages that cannot be processed successfully; the application still owns replay safety.

Measure the states that affect delivery guarantees: pending age, attempts by endpoint, time to first success, dead-letter count, response classes, and duplicate suppression count. A single success-rate graph hides a queue that is quietly aging.

What I would change at scale

At larger volume, I would partition work by endpoint or tenant so one unhealthy destination cannot consume every worker. I would add per-endpoint concurrency limits and a circuit breaker that pauses new attempts without erasing pending records. I would also separate payload storage from the queue message, because a queue should carry a durable reference rather than become the only copy of a large body.

Those changes cost operational complexity. They pay back when one receiver has a long outage or a strict rate limit. Before that point, a delivery table, a leased worker, jittered backoff, and a dead-letter workflow are easier to operate than a many-layer scheduler. The trade-off is deliberate: this small design does not fit workflows that require atomic commits across the sender, receiver, and a third service; use an explicit transaction or saga boundary there.

The decision rule is short: choose at-least-once delivery when loss is unacceptable, make the receiver idempotent with a stable event ID, classify permanent failures, and make every terminal state observable. Retries then become a controlled recovery process instead of an optimistic loop around fetch.

Further reading

Top comments (0)