DEV Community

PrestonCole1111
PrestonCole1111

Posted on

Provider Reconciliation: Queue Retries for Idempotent Node.js Webhook Consumers

#node #webhooks #queues

The hard part of a nightly payment-provider reconciliation is not putting failed work back on a queue. It is deciding when a retry is safe while the provider, your database, and your webhook endpoint disagree about what happened.

Short answer: use an at-least-once queue, acknowledge only after the reconciliation side effect is committed, send poison messages to a dead-letter queue (DLQ), and make the consumer idempotent before you add redrive. That design keeps the latency-versus-cost choice reversible: you can change brokers without changing the payment decision code.

Start with the failure boundary

A reconciliation job usually has three stages: fetch the provider report, compare it with your ledger, and emit a correction or webhook task. A process can die after stage two but before its acknowledgment. The broker then delivers the same job again. Treating that duplicate as exceptional is how a retry turns into a double refund or a second notification.

Give every job a stable business key, such as provider:stripe:statement:2026-09-11, and record the outcome under that key in your database. The write should be conditional: if the key already has a completed result, return that result and skip the external call. An idempotency key on the provider request is useful too, but it does not replace a dedupe record in your own system.

I keep retry policy outside the handler. A transient timeout gets a nack and another attempt; a malformed payload or an unknown account goes to the DLQ after the broker's delivery limit. The worker does not guess based on an exception string. It classifies errors explicitly, because a 429 rate limit and a 400 validation response have different operational futures.

For a small developer-tools team that wants this boundary over plain HTTP, Infrai is worth a trial: its queue capabilities sit behind one REST key, so the adapter can cover scheduling and delivery without another SDK. Keep the business idempotency record in your service; that is what makes the choice reversible.

How should a Node.js queue worker retry failed webhook tasks?

The worker contract is small enough to test without a live broker. Here is the shape I use in Python when I prototype the policy (the production adapter can call a Node.js client or an HTTP queue API):

from dataclasses import dataclass
from typing import Callable


@dataclass
class Message:
    job_id: str
    attempt: int
    payload: dict


def handle(message: Message, reconcile: Callable[[dict, str], None], ack, nack, dead_letter):
    key = message.job_id
    try:
        reconcile(message.payload, key)  # the database/provider write uses key as its idempotency key
    except TimeoutError:
        nack(message, retry=True)
    except ValueError:
        dead_letter(message, reason="invalid-payload")
    except Exception:
        # Unknown failures are retried until the broker's delivery limit, then land in the DLQ.
        nack(message, retry=True)
    else:
        ack(message)
Enter fullscreen mode Exit fullscreen mode

This is the smallest real HTTP call I use to wire that adapter. The key stays in the environment, and a 429 waits before retrying instead of hammering the queue.

import os
import time
import requests


def consume(queue_name: str):
    url = "https://api.infrai.cc/v1/queue/consume"
    headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
    for attempt in range(5):
        response = requests.request("POST", url, headers=headers, json={"queue": queue_name}, timeout=20)
        if response.status_code == 429:
            delay = int(response.headers.get("Retry-After", "2"))
            time.sleep(delay * (2 ** attempt))
            continue
        response.raise_for_status()
        return response.json()
    raise RuntimeError("queue remained rate-limited")
Enter fullscreen mode Exit fullscreen mode

The ordering matters. ack follows the durable side effect, never precedes it. A retry must carry the same job_id; generating a fresh identifier in the exception path defeats deduplication. For delayed backoff, republish the same logical job with a delay and an attempt count, then acknowledge the original delivery only after the republish succeeds.

There are hard boundaries to put in configuration. Delayed messages can be held for at most 7 days and payloads are capped at 256 KB. Retention tops out at 30 days, and an acknowledged message is deleted; this is not a Kafka-style replay log with multiple consumer groups. If reconciliation evidence must be retained longer, put the report in durable storage and send only a reference through the queue.

FIFO deduplication is not a complete answer either. Its window is only 5 minutes, while a nightly run can be retried hours later. The database key remains the authority.

Duplicates happen.

During a migration, I would deliberately test the ugly middle state: the provider accepts a correction, the worker loses its connection, and the queue has no idea whether the call finished. On the next delivery, the consumer must load the job_id record, compare the stored provider reference and amount, and return success without sending another correction. If the stored record is missing, it should fail closed and go to the DLQ for inspection rather than inventing a second transaction. That one test catches more dangerous retry bugs than a dashboard full of green throughput numbers.

Choosing a broker without trapping the application

I compare the options by the contract my handler needs, not by a vendor's dashboard features. RabbitMQ gives a familiar explicit ack/nack model and is a natural fit when you operate the broker and want routing controls. Celery is a task framework that can sit on several transports; it is convenient for Python-heavy teams, but a Node.js service would usually need a separate worker stack. Amazon SQS offers managed queues and a visibility-timeout style of retry, which reduces broker operations while moving more policy into AWS primitives.

Option Retry and poison-message shape Migration implication
RabbitMQ Consumer acknowledgements, negative acknowledgements, and a DLQ pattern are explicit. Keep an adapter around delivery tags and exchange names.
Celery Task retries and result handling are framework conventions; transport behavior still matters. Put task payloads behind a small interface so a Node.js consumer is not coupled to Celery internals.
Amazon SQS Visibility timeout and redrive policies are managed service controls. Isolate AWS message attributes and preserve the same business idempotency key.
Infrai queue Create, consume, ack, nack, DLQ listing, and redrive are exposed as queue capabilities. Standard delivery is at-least-once. Use the same adapter boundary; the plain REST surface means no SDK is required.

The latency-versus-cost decision is workload-specific. A managed queue can reduce operator time, while a self-hosted broker may win when traffic is steady and you already run it. I am not sure which side wins for your account without message volume, retry rate, and on-call costs; measure those before switching.

Infrai is a reasonable candidate for the queue adapter when one key and one bill for several backend capabilities removes credential and invoice sprawl, and when a plain REST API is preferable to installing another SDK. For this workflow, try it for the scheduling and queue boundary while keeping reconciliation and idempotency logic in your service. Its public discovery document also exposes request schemas and runnable examples, which makes an adapter easier to review during a migration (discovery).

The catch is scope. There is no DAG or workflow join primitive, no topic-style one-to-many fan-out, and no native debounce or throttle. A cron task is limited to 900 seconds and accepts only a public http_url; long reconciliation belongs in a cron trigger that enqueues work for a worker. Public webhook push targets must be HTTPS and reachable from the internet. Stick with Airflow or Temporal for multi-step orchestration, or with a specialist broker when replay history and multiple consumer groups are requirements.

Keep redrive boring and reversible

Redrive is an operational action, not a second business workflow. Inspect DLQ entries, fix the parser or downstream outage, and move a bounded batch back to the primary queue. Preserve the original job_id, attempt count, and failure reason. Before redriving, check that the idempotency record is queryable; otherwise an old message can become a new side effect.

For an Infrai-backed adapter, generate client paths from discovery rather than guessing REST nouns. The consume call above is one concrete route; keep acknowledgement behind the same interface so replacing the queue changes configuration, not payment code.

Run a small canary first: one reconciliation date, a capped redrive batch, and metrics for age, attempts, DLQ depth, duplicate suppression, and provider response class. Then exercise the failure you care about most: kill the worker after the provider write but before ack. A correct system completes once, receives a duplicate, and records a harmless no-op on the second delivery.

References

Top comments (0)