DEV Community

EngelbertPierce7942
EngelbertPierce7942

Posted on

Node.js Marketplace Recovery — Queue and Cron for 5 Failed Webhook Jobs

Short answer: put each failed webhook job in a delayed queue, keep reservation expiry in the database, and reserve cron for a small recovery audit. That is the simplest architecture I would ship for fixed-window marketplace holds because every interrupted delivery remains a named piece of work.

Choice What survives a restart? Recovery action Use it when
Cron over reservation rows Reservation state Recompute due work on the next scan Late notification is acceptable and there is no per-attempt history
Delayed queue Job state and retry timing Reclaim or retry the named job Delivery attempts need individual ownership and inspection
Queue with a cron audit Job state plus an independent completeness check Retry known jobs; reconcile missing ones A stale hold must be recoverable without reading application logs

Recommendation: choose the third row for reservation-expiry webhooks. The queue runs ordinary delivery. The cron audit asks whether every expired reservation has a corresponding terminal or active delivery record. It does not become a second dispatcher.

The extra component has to earn its keep. For a one-person SaaS, it does so only if the recovery question becomes boring: given a reservation ID, can I tell what should happen next in one query? Revenue per engineering hour favors that answer over a clever scheduler. Ship weekly. Outsource the undifferentiated mechanics, but keep the state model yours.

How do queue and cron compare for failed webhook jobs?

Cron should own a recurring question. The queue should own a particular delivery. Those roles sound similar until a worker stops between sending an HTTP request and recording its result.

A reservation and its notification have separate lifecycles. The database can mark a hold expired at its configured deadline even while the notification remains scheduled, leased, retryable, delivered, or dead. The webhook must never extend the hold. Otherwise, a downstream delay leaks into marketplace inventory rules.

Use five delivery states: scheduled, leased, retryable, delivered, and dead. A stable event ID ties them to one reservation expiry. leased needs a lease deadline so work doesn't belong to a vanished process forever. retryable needs a next-attempt time. dead means the automatic policy ended and an operator must decide whether replay is safe.

Five is a design choice, not a standard. Some queue systems combine scheduled and retryable, while others model waiting and delayed jobs separately. I prefer explicit names at the application boundary because they make the recovery runbook portable. The exact storage representation can change later.

This separation gives the cron audit one narrow invariant to test: every expired reservation that requires a webhook must map to a delivery record in an active or terminal state. It should report violations. It may recreate a missing job using the same stable event ID, but it should not reschedule all failed deliveries on every tick. That would create two retry clocks, and nobody needs that at 2 a.m.

No second clock.

Rollout starts with a recovery ledger

Operational recovery is the deciding criterion. A timer tells me when code might run. A ledger tells me which business event exists, who owns it, and what evidence is left after an interruption.

For each expiry delivery, retain the event ID, reservation ID, current state, attempt count, next eligible time, lease deadline, and last recorded outcome. Keep payload handling appropriate to the data involved; the operational record should contain enough context to identify the work without turning logs into a second database. The receiver should treat the stable event ID as its idempotency key so replaying an uncertain attempt does not repeat the inventory effect.

Sign the exact serialized request body. RFC 2104 defines HMAC as keyed hashing for message authentication, which is useful for authenticating a webhook body. It doesn't schedule retries, provide transport security, or make a repeated business action idempotent. Those remain separate responsibilities.

The long part of this design is the interruption map. Suppose illustrative reservation R-6041 expires and produces event reservation-expired:R-6041:v1. If the process stops before the delivery record is created, the cron audit finds an expired reservation with no matching event. If it stops after the job is leased but before the request, the lease deadline makes the work eligible again. If it stops after sending but before recording success, the outcome is uncertain, so the same event ID is delivered again and the receiver's idempotency check decides whether the effect was already applied. If the retry policy ends, the job stays visible as dead. Each stop maps to a query and an action; none requires reconstructing intent from a timestamp buried in a log line.

That's the bar.

The HTTP worker integration boundary

The worker shouldn't know how cron scans reservations. It receives one delivery, makes one signed HTTP request, and returns an outcome that the queue adapter maps to a state transition. A small boundary is easier to test during the normal weekly release cycle.

import { createHmac } from "node:crypto";

type ExpiryDelivery = {
  eventId: string;
  reservationId: string;
  expiredAt: string;
};

type AttemptOutcome =
  | { kind: "delivered"; status: number }
  | { kind: "retryable"; status?: number }
  | { kind: "dead"; status?: number };

type RetryPolicy = {
  delayAfter(attempt: number): number | undefined;
  canRetry(status?: number): boolean;
};

function signatureFor(body: string, secret: string): string {
  return createHmac("sha256", secret).update(body).digest("hex");
}

async function attemptDelivery(
  delivery: ExpiryDelivery,
  attempt: number,
  endpoint: URL,
  secret: string,
  policy: RetryPolicy,
): Promise<AttemptOutcome> {
  const body = JSON.stringify(delivery);

  try {
    const response = await fetch(endpoint, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "x-event-id": delivery.eventId,
        "x-webhook-signature": signatureFor(body, secret),
      },
      body,
    });

    if (response.ok) {
      return { kind: "delivered", status: response.status };
    }

    const delay = policy.delayAfter(attempt);
    return delay !== undefined && policy.canRetry(response.status)
      ? { kind: "retryable", status: response.status }
      : { kind: "dead", status: response.status };
  } catch {
    return policy.delayAfter(attempt) !== undefined && policy.canRetry()
      ? { kind: "retryable" }
      : { kind: "dead" };
  }
}
Enter fullscreen mode Exit fullscreen mode

The policy stays outside the worker because retry delays depend on the marketplace's notification promise and the receiver's behavior. I'm not sure what delay sequence is right without those inputs, and a universal set of numbers would be fake precision. Define the maximum acceptable notification lag, the automatic-attempt ceiling, and the manual-review expectation before filling in RetryPolicy.

Test the transition surface, not a happy-path fetch alone. Cover a successful response, a response the configured policy marks retryable, a response it marks terminal, a network exception, and the final allowed attempt. Then test the adapter separately: retryable becomes a delayed job, dead becomes reviewable work, and delivered cannot be claimed again. Terminate a test worker after each durable write and verify that the audit can still classify the event.

BullMQ is one documented Node.js implementation of queues, workers, retries, and related job states. Its documentation is useful when mapping this generic boundary onto that library. Keep application terms such as eventId and dead independent of a library's internal names; that keeps recovery logic readable if the backing component changes.

Govern the interruption audit

Run the audit often enough to meet the notification-lag promise, and bound every scan. Its useful outputs are exceptions: an expired reservation without a delivery record, a lease past its deadline, a retryable job past its next eligible time, or a dead job awaiting a decision. Alerting on the oldest item in each category is usually more actionable than counting every historical success.

Recovery needs a written action for each output. Recreate missing work with the original event ID. Release an expired lease through the queue's ownership mechanism. Let the queue make an overdue retry eligible. Inspect a dead job before replay, and reuse its event ID if replay is approved. Don't silently delete the evidence after an operator acts; record the decision and resulting state.

A deployment should exercise the same machinery. Stop claiming new jobs, allow active HTTP calls a bounded drain period, then leave unfinished work reclaimable through lease expiry. The release check is concrete: can an operator locate every non-delivered expiry and say whether it will retry automatically or needs a decision? A green process-health check cannot answer that.

This is also where scope stays under control. Start with the five states, one worker boundary, one audit query, and one replay path. Avoid building a visual control plane until the query and runbook become painful. The first useful interface may be an authenticated internal command that accepts an event ID and records a replay decision. Fancy dashboards can wait.

Keep it small.

Test cron-only and queue-only limits

Stick with cron-only when the database mutation is the whole job, one scan interval of delay is acceptable, and individual HTTP attempts do not need their own history. Expiring abandoned drafts is a better fit than notifying an external marketplace participant. One database lock strategy and one deployment may be the right revenue-per-hour choice at low complexity.

Queue-only can also be enough when job creation is already atomic with the authoritative state change, the queue exposes all recovery evidence you need, and an independent completeness audit would repeat an existing guarantee. The catch is that this assumption must survive a restart test. If nobody can prove where missing work would appear, queue-only is optimism rather than architecture.

The recommended queue-plus-audit design is not suitable when nobody will review dead jobs or maintain the reconciliation rule. It adds a queue, leases, delayed retries, a cron process, and an operational promise. If a workflow spans several independently owned systems or needs coordinated compensation, choose a workflow design matched to that consistency problem instead of stretching this small state machine.

The decision is deliberately unglamorous: choose the least complex design that leaves a durable answer after every interruption. For fixed-window reservation expiry with failed webhooks, that usually means a delayed queue for attempts and cron for proof that no business event disappeared.

References

Top comments (0)