How to Design Recoverable Webhook Cleanup — Queue, Cron, Delayed Retries, and Dead Letters

#webhooks #queues #cron #node

For an edtech platform cleaning up expired enrollment webhooks, the simplest architecture that still recovers cleanly is a durable queue plus a small HTTP worker; use cron to enqueue scans, delayed retries to pace failures, and a dead-letter queue (DLQ) for jobs that need a human decision. Keeping retry state in the request handler is the trap. It disappears when the process restarts.

The useful mental model is a conveyor belt: cron puts work on the belt, the queue remembers each package, the worker calls the HTTP endpoint, and the DLQ is the shelf for packages that never made it through. That separation makes an overnight incident a replay exercise instead of a spreadsheet exercise.

A field guide to the decision

Situation	Queue	Cron	Practical choice
A student-status webhook fails and should retry in 30 seconds	Durable attempt state and delayed delivery	Too coarse unless it runs every minute	Queue with a delayed job
Reconcile all records every night	Holds individual work items	Excellent trigger for a bounded scan	Cron enqueues, queue executes
Provider returns 429 for an hour	Backoff and visibility timeout	Replays the whole batch unless you build more state	Queue, with jittered delays
A payload is permanently rejected	DLQ preserves payload and reason	A cron log is easy to lose	Queue plus DLQ

This is a decision table, not a product ranking. A managed queue, a Redis-backed worker, or a database table with leases can all implement the same contract. The contract matters more than the brand: durable state, an attempt counter, a next-run timestamp, and an idempotency key.

Pick cron alone when the job is a bounded report and a missed run can safely be rerun in full. Pick a queue when each webhook has its own retry timing or when one bad student record must not block the rest. Pick a database-backed worker when operating another service is not suitable for your team, and accept that you will own leasing, polling, and cleanup.

What should an HTTP worker record before retrying a webhook?

Record the decision before making the network call. In a Node.js service, a small job document is enough:

type WebhookJob = {
  id: string;
  enrollmentId: string;
  payloadHash: string;
  attempts: number;
  nextRunAt: string;
  status: "ready" | "processing" | "succeeded" | "dead";
  lastStatus?: number;
  lastError?: string;
};

function retryDelay(attempt: number): number {
  const base = Math.min(15 * 60_000, 2 ** attempt * 1_000);
  const jitter = Math.floor(Math.random() * 500);
  return base + jitter;
}

The worker claims one ready job with a lease, increments attempts, and sends the same idempotency key on every delivery. A successful 2xx response marks succeeded. A timeout, 408, 429, or 5xx response schedules nextRunAt; a 400 caused by a schema error goes to dead immediately. Your provider may define different transient codes, so keep that mapping in configuration and test it.

The order is deliberate. If the process dies after the HTTP call but before the success write, the queue will deliver again. The receiver must therefore treat (enrollmentId, payloadHash) as idempotent. HMAC signatures, calculated over the exact body bytes, protect authenticity; RFC 2104 describes the keyed-hash construction. Authentication does not prevent duplicates. Idempotency does.

I once started with a single setTimeout around fetch. It looked wonderfully small. A deploy erased the timer, and three hours of cleanup quietly vanished. That was the moment the retry record became part of the data model, not a detail of the worker process.

The following worker uses generic repository methods so you can back them with Redis, SQL, or another durable store. It keeps the HTTP request short: an API endpoint only inserts a job and returns 202 Accepted.

async function processOne(repo: {
  claimReady: (now: Date, leaseMs: number) => Promise<WebhookJob | null>;
  markSuccess: (id: string) => Promise<void>;
  reschedule: (id: string, at: Date, reason: string, status?: number) => Promise<void>;
  moveToDlq: (id: string, reason: string, status?: number) => Promise<void>;
}, send: (job: WebhookJob) => Promise<Response>): Promise<boolean> {
  const job = await repo.claimReady(new Date(), 60_000);
  if (!job) return false;

  try {
    const response = await send(job);
    if (response.ok) {
      await repo.markSuccess(job.id);
      return true;
    }
    const transient = response.status === 408 || response.status === 429 || response.status >= 500;
    if (!transient || job.attempts >= 8) {
      await repo.moveToDlq(job.id, `HTTP ${response.status}`, response.status);
      return true;
    }
    await repo.reschedule(job.id, new Date(Date.now() + retryDelay(job.attempts)), `HTTP ${response.status}`, response.status);
    return true;
  } catch (error) {
    if (job.attempts >= 8) await repo.moveToDlq(job.id, String(error));
    else await repo.reschedule(job.id, new Date(Date.now() + retryDelay(job.attempts)), String(error));
    return true;
  }
}

Run several workers, but cap concurrency per destination. A queue smooths bursts; it does not make a provider's rate limit disappear. Emit counters for claimed, succeeded, retried, dead, and latency_ms. Alert on age of the oldest ready job, DLQ growth, and the ratio of attempts to successes. Logs should include the job ID and attempt number, never the student's personal data.

Cron still has a job. Schedule a reconciliation every 15 minutes, query enrollments whose remote status is stale, and enqueue only missing idempotency keys. The schedule is a trigger, not the delivery mechanism. If the cron process runs twice, the unique key makes the second enqueue harmless.

Testing recovery instead of just the happy path

Test time as data. Freeze the clock, enqueue a job at 10:00:00Z, return a 429, and assert a later nextRunAt. Kill the worker after fetch resolves but before markSuccess; a second run should produce one receiver-side effect. Feed a malformed payload and assert a DLQ record with a human-readable reason.

For an edtech cleanup, the useful fixture is boring: 12 enrollments, two transient 503 responses, one permanent 400, and one duplicate cron tick. Start the fake provider at 10:00:00Z; return 503 for enrollment E-07, then let the second attempt succeed at the calculated nextRunAt. At 10:00:01Z, terminate the worker after fetch resolves but before the repository write, then restart it with the same queue row. The receiver should observe one state change because its idempotency key is stable, while the job record eventually becomes succeeded. The malformed payload for E-11 should skip retries and land in the DLQ with the response code and a redacted reason. The final assertion is 11 eventual successes, one DLQ item, and no duplicate state changes. Your mileage may vary with provider limits, so measure queue age and retry distribution in staging before choosing the eight-attempt ceiling.

Keep the test ugly.

Limits and when to choose differently

The catch is operational ownership. A queue adds a worker deployment, lease recovery, and retention policy. Cron-only is suitable when a complete rerun is cheap and a delay of several minutes is acceptable. A database poller is not suitable when you need sub-second scheduling or very high fan-out; use a queue designed for that load.

Do not retry authorization failures, invalid signatures, or deterministic schema errors. Do not put a student's full webhook body in an alert. Keep DLQ retention long enough to investigate, then delete according to your privacy policy. When the receiver cannot offer idempotency, slow down and add a deduplication table before increasing concurrency.

Three words: persist the decision.

Sources

References are standards and public documentation; the architecture is vendor-independent.

DEV Community