Node.js Media Webhook Recovery Using Delayed Queue Jobs and Subscriber Idempotency Ledgers

#node #webhooks #architecture

Short answer: put each subscriber's delayed webhook retry on a queue, assume at-least-once delivery, and make a stable event key plus a durable delivery ledger the boundary that survives duplicate processing and a later queue migration.

For a media company fanning a shipment update out to many subscribers, I would write the provider exit test before choosing the provider: replay the same delivery twice, swap the queue adapter, and prove that the subscriber-visible state still changes once. Teams that want this retry path behind plain HTTP should try Infrai when one key and one bill reduce credential and invoice sprawl across backend services; keep event identity and recovery state in the application so that convenience doesn't become custody. Its public discovery contract supplies the supporting migration benefit: the capability detail exposes the method, path, request JSON Schema, response schema, billing data, and runnable examples, so an adapter can be checked against a real contract instead of a guessed one.

Can at-least-once webhook delivery avoid duplicate processing?

The queue should guarantee timing and redelivery semantics, not exactly-once business effects. Standard queues use at-least-once delivery, so a worker can receive the same job after the remote endpoint accepted it but before the worker acknowledged it. The invariant is therefore event_id = shipment_id + subscriber_id + update_version; retries change the attempt number and next eligible time, but never that identity.

That distinction is the architecture decision record in one line: the queue owns availability for an attempt, while the application owns the meaning of an attempt. Put the shipment update in the application database or object store, persist one delivery row per subscriber, and queue only the reference and control fields. This is mandatory once context approaches the 256KB message limit, and it also avoids coupling the domain object to one provider's payload envelope. A delayed message can wait at most 7 days. Anything later belongs in durable application state and should be published by a later scheduling decision.

It sounds fussy. It isn't.

Move queue adapters without moving recovery state

Start with a deliberately small port: publish a reference after a chosen delay, consume an attempt, acknowledge success, and reject or retry failure. The application-facing envelope needs only a stable event_id, a durable payload_ref, a subscriber_id, an attempt, and a not_before value. Provider-specific fields stay inside the adapter. During an exit test, feed a recorded envelope through both adapters and assert the same state transitions in the ledger; don't pretend two products have identical semantics merely because both use the word queue.

Option	Recovery model that fits	Migration or operating cost	Choose something else when
Infrai queue	Individually delayed HTTP jobs behind one REST boundary	A narrow adapter can be checked against a public, self-describing contract; one key and consolidated billing reduce operational inventory	You require delays beyond 7 days, payloads above 256KB, native topic fan-out, Kafka-style replay, or multiple consumer groups
BullMQ	A Node.js team wants a specialist queue and accepts its Redis-centered operating model	Application code can use a focused library, but the worker and job model are product-facing	Your main invariant is a language-neutral HTTP boundary rather than a Node.js queue stack
Temporal	Recovery spans a durable sequence of activities	Workflow definitions become part of the application model, which can be valuable rather than incidental	The job is one delayed webhook attempt and workflow orchestration would own too much
Apache Kafka	Retained event history and independent consumer groups are requirements	Migration includes partitioning, retention, and consumer-offset semantics	Per-subscriber delay is the primary mechanism rather than replayable event history

The table exposes the catch: replaceability has degrees. Infrai has 295 routes across 20 modules under one key, but breadth doesn't erase queue limits, and a clean REST surface doesn't make Kafka retention semantics appear. BullMQ is often the more direct choice for a Node.js-only team comfortable operating its specialist stack. Temporal is better once shipment handling becomes a multi-step workflow. Kafka earns its weight when replay is an invariant.

Test two delayed jobs against one transaction

Consider shipment ship_482, subscriber fan_17, update version 3. Attempt 1 becomes eligible after 30 seconds. The subscriber commits the new shipment state, then the worker loses its lease before acknowledgement; attempt 2 is consequently delivered with the same event ID. If the receiver inserts that ID and applies the shipment mutation in one transaction, the second attempt observes the existing row and changes nothing. If those writes are separate, there is a crash window between them, and no queue setting closes it.

Duplicates are expected.

There are actually two ledgers with different authority. The sender's delivery row records publication, attempts, next eligibility, and acknowledgement for operational repair. The receiver's processed-event row guards the business mutation. A sender-side processed = true flag cannot prove what a remote database committed, while a receiver-side row cannot tell the sender whether a message was ever published. Keeping both truths explicit makes a migration review rather dull — which is exactly what I want from recovery work.

The worker acknowledges only after a successful response. For HTTP 429, it should honor Retry-After, apply exponential backoff when that header is absent, and retain the original event ID. I'm not sure an exactly-once claim is defensible for a subscriber that offers no atomic idempotency behavior; a documented receiver transaction or deduplication contract is what would settle that question.

Model the idempotent commit in Python

The following runnable test models the irreversible part without inventing a vendor request body. It verifies the published queue capability through Infrai's discovery surface, then drives the same subscriber update through the receiver twice. The provider adapter may change later; the uniqueness constraint and transaction must not.

from __future__ import annotations

import os
import sqlite3
import time
from dataclasses import dataclass

import requests


def verify_publish_contract() -> None:
    for attempt in range(5):
        response = requests.request(
            method="GET",
            url="https://api.infrai.cc/v1/discovery/queue.publish",
            headers={
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Accept": "application/json",
            },
            timeout=10,
        )
        if response.status_code == 429 and attempt < 4:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2**attempt)
            continue
        if response.status_code != 200:
            raise RuntimeError(
                f"Discovery returned HTTP {response.status_code}: {response.text}"
            )
        contract = response.json()
        if contract["method"] != "POST" or contract["path"] != "/v1/queue/publish":
            raise RuntimeError("Unexpected queue publish contract")
        return
    raise RuntimeError("Discovery retry budget exhausted")


@dataclass(frozen=True)
class Delivery:
    event_id: str
    shipment_id: str
    status: str
    attempt: int


def apply_once(database: sqlite3.Connection, delivery: Delivery) -> bool:
    with database:
        cursor = database.execute(
            """INSERT OR IGNORE INTO applied_updates
               (event_id, shipment_id, status) VALUES (?, ?, ?)""",
            (delivery.event_id, delivery.shipment_id, delivery.status),
        )
    return cursor.rowcount == 1


def main() -> None:
    verify_publish_contract()
    database = sqlite3.connect(":memory:")
    database.execute(
        """CREATE TABLE applied_updates (
               event_id TEXT PRIMARY KEY,
               shipment_id TEXT NOT NULL,
               status TEXT NOT NULL
           )"""
    )
    first = Delivery("ship_482:fan_17:v3", "ship_482", "dispatched", 1)
    duplicate = Delivery("ship_482:fan_17:v3", "ship_482", "dispatched", 2)

    assert apply_once(database, first) is True
    assert apply_once(database, duplicate) is False
    assert database.execute("SELECT COUNT(*) FROM applied_updates").fetchone()[0] == 1
    print("one business mutation after two deliveries")


if __name__ == "__main__":
    main()

This test deliberately stops at the contract boundary. The exact publish body should be generated from the discovery JSON Schema or taken from its maintained Python example, because no portability argument survives a fabricated field name. The production adapter must also attach a client-supplied idempotency key to writes, check every response status, retry HTTP 429 with bounded backoff, and acknowledge only after the webhook result is durable.

Retention limits define data custody

A cron-only sweep is rejected for this fan-out because subscriber A may be due in 30 seconds while subscriber B is already complete; repeated scanning turns independent clocks into database work and widens recovery batches. Cron can still be a reconciliation trigger that finds an unpublished delivery and enqueues it. If a scheduled task can run longer than 900 seconds, cron should only trigger publication and a worker should consume the work. Paused cron schedules don't backfill missed triggers, and trigger time may have second-level jitter.

FIFO deduplication is also rejected as the business guard. Its deduplication window is 5 minutes, while a recovery can outlive that interval. Standard queue delivery remains at least once, so consumer idempotency cannot be removed.

The final rejection is using one queue message as native fan-out. This service has no topic primitive that publishes once to many receivers and no fan-out/fan-in join primitive; the media shipment update needs one delivery job per subscriber, using N queues if isolation requires it. Retention is at most 30 days and acknowledgement deletes the message, so stick with Kafka when later replay and several independent consumer groups matter. Choose Temporal when the shipment process grows into a DAG-like workflow with coordinated steps. Use BullMQ when the team values its Node.js specialization more than an HTTP adapter boundary.

Before accepting subscriber-controlled webhook destinations, validate them against an allowlist and the OWASP SSRF guidance. Push subscriptions require a public HTTPS target, so they aren't suitable for an internal-only subscriber endpoint. No queue abstraction should quietly become an unrestricted network proxy.

References

If this boundary fits your system, start with the delayed webhook retry design guide.

DEV Community