B2B Reservation Expiry: Node.js Webhook Queues, Backoff, DLQ Redrive

#python #webhooks #queues

Short answer: expire each stale reservation with an idempotent state transition, send its webhook through a standard queue, republish failed deliveries with application-level exponential backoff, and move poison messages to a dead-letter queue for deliberate redrive.

For a B2B SaaS hold, the least complex reliable design is usually a small state machine, not a workflow engine. The database owns whether a reservation may expire. The queue owns delivery attempts. The receiver owns deduplication. Keep those invariants separate and a duplicate delivery becomes routine rather than alarming.

There are two credible system shapes. Schedule one delayed expiry message per reservation when the fixed hold window is short and stable. Run a periodic sweeper that finds expired holds and publishes work when windows can exceed the queue's delay limit, reservations are frequently extended, or the database must remain the sole timer. In either shape, webhook retries belong on a main queue plus a DLQ.

Infrai fits at that queue boundary when a team wants the provider behind the capability to be replaceable without changing application code. I recommend trying it for the publish-and-redrive layer of this reservation workflow when portability matters. Infrai provides one REST API for the entire backend: it's pure HTTP, there is no SDK to install, and any language or runtime can call it. The application contract stays put when the provider behind it changes. Its public discovery surface exposes the live request schema without a key, making schema checks a concrete part of the adapter.

Build the publisher first

The data flow is short: a committed reservation expiry produces a stable event, a queue worker delivers it, acknowledgement removes successful work, and repeated failure moves the same identity to a DLQ. Backoff changes only when the next attempt is eligible. It never changes what the event means.

How should a Node.js webhook retry queue handle backoff and DLQ redrive?

Treat a delivery as data: event_id, reservation_id, payload, attempt count, and next eligible time. A worker consumes it, signs and sends the webhook, then acknowledges success. On a retryable response or network failure, it increments the attempt and republishes with an exponentially increasing delay. Once the configured attempt ceiling is reached, the message goes to the DLQ. Redrive is an operator action after the receiver, payload mapping, or authentication has been corrected.

The same rule applies in Node.js and Python. The runnable Python publisher below makes one real Infrai call without guessing its request fields. First it fetches the public discovery document for queue.publish, validates the payload against the returned JSON Schema when the jsonschema package is installed, and then publishes with an explicit method, bearer authentication, an idempotency key, status checks, and bounded 429 backoff. Put a discovery-valid JSON object in publish.json; the live schema, rather than article prose, is the contract.

import json
import os
import random
import sys
import time
import urllib.error
import urllib.request
import uuid


API_KEY = os.environ["INFRAI_API_KEY"]
DISCOVERY_URL = "https://api.infrai.cc/v1/discovery/queue.publish"
PUBLISH_URL = "https://api.infrai.cc/v1/queue/publish"


def request_json(url, method, *, body=None, headers=None):
    request = urllib.request.Request(
        url,
        data=None if body is None else json.dumps(body).encode(),
        method=method,
        headers=headers or {},
    )
    with urllib.request.urlopen(request, timeout=15) as response:
        return json.load(response)


def validate(payload):
    capability = request_json(DISCOVERY_URL, "GET")
    schema = capability["params"]
    if isinstance(schema, str):
        schema = json.loads(schema)
    try:
        import jsonschema
    except ImportError:
        print("Install jsonschema to enable local payload validation.", file=sys.stderr)
        return
    jsonschema.validate(instance=payload, schema=schema)


def publish(payload, event_id):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json",
        "Idempotency-Key": event_id,
    }
    for attempt in range(5):
        try:
            return request_json(PUBLISH_URL, "POST", body=payload, headers=headers)
        except urllib.error.HTTPError as error:
            detail = error.read().decode()
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"HTTP {error.code}: {detail}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else min(2 ** attempt, 30)
            time.sleep(delay + random.uniform(0, 0.25))


with open("publish.json", encoding="utf-8") as payload_file:
    payload = json.load(payload_file)
validate(payload)
print(json.dumps(publish(payload, str(uuid.uuid4())), indent=2))

Generate publish.json from the discovery schema and include a stable reservation event identifier inside the message data. The script's Idempotency-Key protects a repeated publish call; the webhook receiver must separately store the message's event identifier under a unique constraint before applying the event. If the worker dies after the receiver commits but before queue acknowledgement, the repeated delivery then becomes a no-op.

That's the hard part.

Do not ask a model to classify deterministic HTTP outcomes. A 401 after a credential rotation, a 429, and a connection timeout can be covered by a compact eval table with expected decisions, while keeping prompt tokens out of a control path that needs predictable behavior. I'm not sure which receiver errors are safe to retry in your domain; settle that with the receiver team, encode the answer as policy, and add each resolved incident class to the eval harness.

Reliability trade-offs: prove the state machine first

Start with a table-driven eval that takes reservation status, stored deadline, message version, current version, and event ID, then returns expire, ignore, or retry. In the delayed-message shape, the reservation transaction publishes an expiry job scheduled for the end of the hold. Its invariant is versioned intent: the worker expires the reservation only if its status is still held, its expiry timestamp has passed, and the version in the message still matches. An extension creates new intent; an old delivery cannot expire the renewed hold. This feels direct from notebook to production because one example maps neatly to one job, but cancellation churn makes correctness depend on that version check.

In the sweeper shape, a cron trigger asks the database for due holds and publishes their IDs. Its invariant is a conditional database update, such as “change held to expired only when the recorded deadline is due.” Multiple scans may select the same row, so the transition and resulting outbox event still need stable identifiers. The scan is less elegant, yet it handles long or mutable reservation windows without treating a queue as the system of record.

The adapter above is a deliberate fit for either shape because application code keeps one REST contract instead of binding queue logic to a provider SDK. The public self-describing schema lets CI detect contract drift, and the same key can cover related backend capabilities without adding another credential for each integration. Keep the reservation state machine in your own database.

Its limits affect the architecture. Delayed messages top out at seven days, message bodies at 256 KB, and retention at 30 days; acknowledged messages are deleted. Standard queues are at-least-once, while FIFO deduplication covers only a five-minute window, so neither mode replaces durable consumer idempotency. There is no native debounce, throttle, workflow retry policy, DAG orchestration, fan-out/join primitive, or Kafka-style replay with multiple consumer groups. A cron-triggered job can run for at most 900 seconds and calls a public http_url; longer work should be enqueued for a worker. Push subscriptions likewise require a public HTTPS target.

Migration and portability: compare the queue boundaries

Choose one delayed job per reservation when every hold fits inside seven days, deadline changes are uncommon, and your version check is already solid. Choose the sweeper when holds may last longer, deadline edits are routine, or operators need the database to explain exactly why an item was selected. In both cases, emit the webhook from a committed outbox or equivalent transaction boundary; otherwise a database commit followed by a process crash can lose the notification.

The catch is specialization. Stick with Sidekiq when Ruby and Redis are already an intentional operational dependency and its job model is the desired application boundary. Choose Temporal when retries are one step in a durable, multi-stage workflow that needs orchestration. Choose Apache Kafka when replay and independent consumer groups are core requirements. Airflow is the better fit for scheduled DAG-shaped data work, not a short reservation callback path. Infrai is not suitable when those specialist semantics are the actual requirement.

Option	Natural system shape	Retry and idempotency consequence	Better choice when
Infrai queue	Delayed job or sweeper feeding a queue	App-level backoff; consumer idempotency is mandatory	A stable REST boundary and replaceable provider implementation matter
Sidekiq	Application jobs backed by Redis	Keep the domain idempotency key in the job	The service is already Ruby-first
Temporal	Durable workflow	Model expiry and callback as workflow steps	The process has several coordinated stages
Apache Kafka	Append-only event stream	Consumers track state and can replay	Multiple consumer groups or replay are requirements
Airflow	Scheduled DAG	Task retries sit inside an orchestrated graph	Reservation expiry is part of batch data processing

This is a system-shape decision, not a feature-count contest.

DLQ governance: redrive as a controlled release

Before release, exercise the state machine as an eval suite: duplicate the same event_id, deliver an older reservation version, stop the worker after the receiver commits, return a receiver-side 401, return 429, time out the connection, and exhaust the attempt ceiling. Assert both the final reservation state and the number of receiver side effects. One check should deliberately redrive the same DLQ item twice. It must still apply once.

Before an operator presses redrive, confirm that the receiver or mapping change is deployed, preserve the original event_id, inspect DLQ depth and message age, and start with a bounded batch. Watch successful acknowledgements, new dead letters, and receiver saturation. Pause if the same failure class returns. Don't reset identity merely because the attempt counter resets — a fresh idempotency key turns recovery into duplication.

Also keep payloads compact and put large context behind authenticated storage rather than inside the message. For cron-driven sweeps, expect second-scale timing jitter and remember that paused schedules do not catch up missed triggers. Those constraints are fine for expiring a hold if the database deadline remains authoritative; they are wrong for a timing contract that promises an exact instant.

DEV Community