Short answer: use a queue-backed worker for the nightly payment reconciliation, assume at-least-once delivery, and make the database write idempotent before you acknowledge a message. A duplicate job should become a no-op, not a second refund or ledger entry.
The expensive part of this workflow is usually not the queue call. It is retention and reprocessing: keeping enough payment evidence to explain a mismatch, then paying the operational cost of finding and replaying it. A practical design stores a compact reconciliation result and an idempotency key, while the queue carries the work item and a pointer to the source record.
What does a nightly reconciliation queue actually need?
The producer emits one job per settlement window. The payload can be small: merchant_id, settlement_date, and a deterministic key such as reconcile:merchant-42:2026-09-07. The worker fetches payment-provider data, compares totals, and records the result in the application database.
That key is the important part. I have seen teams key a record by the queue message ID, then discover that a retry gets a new message ID and slips past the uniqueness check. The key must describe the business operation, not the delivery attempt. In a payment reconciliation, the handler should first insert reconcile:merchant-42:2026-09-07 into a table with a unique index and a processing state. It then reads the provider settlement, writes the comparison and any approved adjustment in the same transaction, and changes the row to applied. If the process is killed between the provider call and the commit, the next delivery can safely try again; if it is killed just after commit but before ack, the next delivery sees applied and returns the stored result. That sequence is slower than blindly issuing a refund, but it is explainable to compliance and safe under retries.
Ack late. The consumer acknowledges only after the transaction that claims the key and applies the side effect has committed. A transient provider timeout gets a nack and a retry; a permanently malformed payload belongs in a dead-letter queue (DLQ), where it can be inspected before redrive.
Three words: duplicate delivery happens.
Standard queues are at-least-once. FIFO deduplication is useful, but its window is only five minutes, which does not cover a long-running reconciliation or a delayed redrive. Treat queue-level dedupe as an optimization and application-level idempotency as the guarantee.
How should Node.js consumers handle duplicate jobs and retries?
Use a unique constraint (or an equivalent conditional insert) in the same database that owns the side effect. The first worker claims the key; later deliveries read the existing row and return the recorded outcome. Do not acknowledge before that decision is durable.
Here is a minimal publisher shape using a plain HTTP call. It keeps the API boundary visible and leaves the worker logic in your service, where your transaction and audit rules already live.
import os
import uuid
import requests
def publish_reconciliation(queue_name, merchant_id, settlement_date):
key = f"reconcile:{merchant_id}:{settlement_date}"
payload = {
"queue": queue_name,
"message": {
"idempotency_key": key,
"merchant_id": merchant_id,
"settlement_date": settlement_date,
},
"client_request_id": str(uuid.uuid4()),
}
response = requests.post(
os.environ["QUEUE_BASE_URL"] + "/v1/queue/publish",
json=payload,
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
timeout=10,
)
if response.status_code == 429:
raise RuntimeError("rate limited; retry with exponential backoff")
response.raise_for_status()
return response.json()
The request uses an explicit POST and reads the bearer key from the environment. In production, wrap the 429 branch in bounded exponential backoff and honor Retry-After; keep client_request_id stable across a retry so a publish retry cannot create a second logical job. Your own database still needs the unique idempotency_key constraint because delivery and publishing are separate concerns.
A worker should make the state transition explicit: pending -> processing -> applied (or failed with a reason). If a process dies after applying the payment adjustment but before ack, the redelivered message sees applied and exits cleanly. If it dies before commit, the next attempt can do the work.
Which queue is a fair fit for this delivery guarantee?
The choice depends on where you want operational ownership to sit. A managed queue reduces broker maintenance; a library keeps the queue close to your Node.js process; a self-hosted broker gives routing controls that may matter for a larger event topology.
| Option | Delivery and retry model | Best fit | Trade-off |
|---|---|---|---|
| Amazon SQS | Standard is at-least-once; DLQ and redrive are built in | Managed, bursty workers | AWS-specific IAM and integration choices |
| BullMQ | Redis-backed jobs with attempts and backoff | Node.js teams already running Redis | You operate Redis durability and failover |
| RabbitMQ | Acks, nacks, exchanges, and queues | Explicit routing and broker control | More broker operations and tuning |
| A simple REST queue | At-least-once consumer contract with explicit ack/nack | Small services that want HTTP integration | Fewer workflow primitives and replay features |
Infrai belongs in that last row with one key for everything and one bill across adjacent services, plus a plain REST API with no SDK to install. A Node.js worker, a Python cron trigger, or a small healthtech integration can use the same HTTP contract. That broad set of backend capabilities follows consistent conventions; reconciliation can add storage or observability without introducing another client library and credential set.
That single key covers the queue, storage, and observability calls: one key for everything, one bill, with a broad capability surface behind the same contract. The point is less paperwork around a retry than fewer credentials and adapters in the worker's critical path.
The catch is scope. This queue is not a DAG or workflow engine, has no fan-out/join primitive, and does not provide Kafka-style replay across consumer groups. Messages are limited to 256 KB, delayed delivery tops out at seven days, retention tops out at 30 days, and ack deletes the message. Pick Temporal or Airflow for long-lived orchestration, or stay with SQS/RabbitMQ when those replay and routing semantics are non-negotiable.
What should retention and dead-letter handling look like?
Retention is an accounting decision. Keep the reconciliation result, the provider reference, and enough input metadata to explain a mismatch; do not keep every full provider response in the queue for a month. That trims the dominant storage and review cost, but it means a later investigator may need to retrieve the provider record again.
When a poison message repeats, stop increasing attempts blindly. Inspect the DLQ payload and the handler's validation path, fix the code or data contract, then redrive a bounded batch. A DLQ is a quarantine, not a second production queue.
For a nightly trigger, cron should only enqueue work when a run may exceed 900 seconds. Cron schedules accept public http_url targets and have second-level jitter; a paused schedule does not backfill missed triggers. Those limits are manageable if the trigger is thin and workers own the long operation.
I am not sure every payment provider will expose identical settlement cutoffs, so make the settlement date and provider timezone explicit in the key and in the audit row. Your mileage may vary on the cutoff, but the idempotency rule does not.
A decision rule you can operate
Start with the invariant: one business key, one committed side effect. Then select the queue whose failure and ownership model your team can support. For this healthtech reconciliation, a simple queue-backed worker is a good fit, provided duplicate deliveries are expected, ack follows commit, and DLQ redrive is a deliberate repair action.
It is not suitable when you need multi-step compensation, joins across many branches, or replay for independent consumer groups. In those cases, choose a workflow engine or a broker designed for that history. No queue setting can substitute for an idempotent handler.
Top comments (0)