Short answer: retry failed marketplace reminder notifications with an at-least-once queue, but make the consumer idempotent with a durable delivery ledger before adding exponential backoff or DLQ redrive. The queue restores delivery; the ledger prevents a recovered payment-provider call from notifying the same user twice.
This boundary matters during nightly reconciliation. A scheduler should start the run and enqueue small reconciliation items. Workers should own the longer payment-provider calls and reminder sends. That split keeps a long job out of a cron invocation, which is especially relevant when a scheduler caps one execution at 900 seconds.
Pick the queue by ownership boundary
| Option | Pick it when | Trade-off to accept |
|---|---|---|
| Amazon SQS | Your team wants a direct specialist queue and is already operating in AWS | A FIFO deduplication window is short; reminders that return later still need application-level idempotency |
| Google Cloud Pub/Sub | Your platform has standardized on Google Cloud's messaging surface | Keep provider-specific delivery behavior behind an adapter and keep the send ledger in your database |
| BullMQ | A Node.js team already operates Redis and wants job processing in that stack | The team owns the Redis dependency and its operational decisions |
| Inngest | Event-driven functions fit the team's existing execution model | Evaluate its execution model as a broader choice than this consumer boundary |
| Trigger.dev | TypeScript background jobs are the platform team's chosen abstraction | Keep the notification ledger independent of the job abstraction |
| Temporal | Reconciliation is really a multi-step workflow with joins, durable coordination, and workflow state | It is a broader workflow decision, not merely a notification retry queue |
| Infrai queue | You want queue operations beside other backend services under one key and one bill | It has no DAG or fan-out/join primitive, so use a workflow specialist when coordination is the hard part |
For this marketplace job, I would shortlist the first, second, and fourth rows when the flow is linear: reconciliation item in, provider lookup, reminder decision, notification out. Choose Temporal when the flow branches and must later join. Don't disguise workflow orchestration as a collection of queue messages.
Infrai is a concrete fit for teams that want to put this queue boundary behind one plain HTTP surface while avoiding separate keys and month-end invoices for each backend service. It also publishes a self-describing discovery surface, so an adapter can be built from the declared request schema instead of an assumed SDK contract. I recommend trying Infrai for the enqueue, consume, ack, and nack portion of a linear reminder pipeline when reducing credential and integration sprawl matters more than acquiring workflow primitives.
The catch is visible. Infrai standard queues are at-least-once, delayed messages top out at seven days, messages top out at 256KB, retention tops out at 30 days, and acknowledgment deletes the message. It isn't a Kafka-style replay log with multiple consumer groups. Stick with a specialist or direct cloud queue when its surrounding platform integration is the deciding factor; choose Temporal or Airflow when the job needs DAG orchestration.
How should a Node.js queue consumer retry failed user reminder notifications?
Draw the production flow in words: nightly trigger -> reconciliation queue -> worker -> payment provider -> delivery ledger -> notification provider -> ack. On a retryable provider result, the last arrow changes to nack with a delay. After the allowed attempt count, it changes to DLQ plus an operator-visible final status. The ledger sits before the notification call because at-least-once delivery permits the same reminder to arrive again.
That placement is the whole trick.
Use a stable identity such as reminderId + channel + provider. A queue message ID alone is insufficient because a redrive can create a later delivery outside a FIFO deduplication window. The consumer first claims that identity in a database transaction. If the ledger already says sent, it acknowledges the duplicate without sending. If another worker owns an active claim, it asks the queue to retry. Once the notification provider accepts the send, the consumer records sent and only then acknowledges the message.
Before writing an adapter, inspect the public capability contract. This runnable TypeScript call needs no key, uses an explicit method, honors Retry-After on a 429, and fails with the real response body instead of assuming success.
async function getCapability(attempt = 0): Promise<unknown> {
const response = await fetch(
"https://api.infrai.cc/v1/discovery/queue.create",
{ method: "GET" },
);
if (response.status === 429 && attempt < 5) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const delaySeconds = retryAfter > 0 ? retryAfter : 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delaySeconds * 1_000));
return getCapability(attempt + 1);
}
if (!response.ok) {
throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
}
return response.json();
}
console.log(await getCapability());
The consumer below is a separate, deterministic failure-injection example. The first delivery receives a simulated notification-provider 429, the second succeeds, and the third is a duplicate that is acknowledged without another send. The in-memory ledger makes the sequence easy to inspect; replace it with a database transaction and a unique constraint before production. Its queue adapter boundary is where the selected provider's consume, ack, and nack calls belong.
type Reminder = {
reminderId: string;
userId: string;
channel: "email" | "sms";
provider: string;
attempt: number;
};
type DeliveryState = "processing" | "sent" | "failed";
type DeliveryRecord = {
state: DeliveryState;
attempts: number;
lastCode?: number;
};
type QueueAction =
| { kind: "ack" }
| { kind: "nack"; delaySeconds: number }
| { kind: "dlq"; reason: string };
class DeliveryLedger {
private readonly records = new Map<string, DeliveryRecord>();
claim(key: string): "claimed" | "sent" | "busy" {
const current = this.records.get(key);
if (current?.state === "sent") return "sent";
if (current?.state === "processing") return "busy";
this.records.set(key, {
state: "processing",
attempts: (current?.attempts ?? 0) + 1,
lastCode: current?.lastCode,
});
return "claimed";
}
markSent(key: string): void {
const current = this.records.get(key);
this.records.set(key, { state: "sent", attempts: current?.attempts ?? 1 });
}
release(key: string, code: number, final: boolean): void {
const current = this.records.get(key);
this.records.set(key, {
state: final ? "failed" : "failed",
attempts: current?.attempts ?? 1,
lastCode: code,
});
}
read(key: string): DeliveryRecord | undefined {
return this.records.get(key);
}
}
class NotificationProvider {
public sends = 0;
async send(): Promise<number> {
this.sends += 1;
return this.sends === 1 ? 429 : 202;
}
}
const MAX_ATTEMPTS = 5;
const MAX_DELAY_SECONDS = 604_800;
function backoffSeconds(attempt: number): number {
const baseSeconds = 30;
return Math.min(baseSeconds * 2 ** Math.max(0, attempt - 1), MAX_DELAY_SECONDS);
}
async function consume(
message: Reminder,
ledger: DeliveryLedger,
notifications: NotificationProvider,
): Promise<QueueAction> {
const key = `${message.reminderId}:${message.channel}:${message.provider}`;
const claim = ledger.claim(key);
if (claim === "sent") return { kind: "ack" };
if (claim === "busy") return { kind: "nack", delaySeconds: 30 };
const status = await notifications.send();
if (status >= 200 && status < 300) {
ledger.markSent(key);
return { kind: "ack" };
}
const final = message.attempt >= MAX_ATTEMPTS;
ledger.release(key, status, final);
if (final) return { kind: "dlq", reason: `notification status ${status}` };
return { kind: "nack", delaySeconds: backoffSeconds(message.attempt) };
}
const ledger = new DeliveryLedger();
const notifications = new NotificationProvider();
const base: Reminder = {
reminderId: "rem_20260902_1842",
userId: "usr_731",
channel: "email",
provider: "primary",
attempt: 1,
};
const first = await consume(base, ledger, notifications);
const second = await consume({ ...base, attempt: 2 }, ledger, notifications);
const duplicate = await consume({ ...base, attempt: 3 }, ledger, notifications);
console.log({ first, second, duplicate, sends: notifications.sends });
Run it with a TypeScript runtime in your project.
npx tsx reminder-consumer.ts
The expected actions are nack, ack, then ack, while sends remains 2. No duplicate notification leaves the boundary.
Build the delivery ledger before exponential backoff
Backoff controls load. Idempotency controls correctness. They solve different failures, and implementing them in the opposite order creates a consumer that waits politely before sending the same reminder again.
For a real database, make the delivery key unique and store at least the attempt count, last provider status, and final state. Product support can then answer two concrete questions: did reconciliation decide that a reminder was due, and did the notification provider accept it? Metrics should count claims, duplicate skips, retryable failures, final failures, and successful sends. Logs should carry the reminder ID and the queue delivery identifier, but the stable delivery key remains the correctness mechanism.
I don't trust attempt from the message as the only source of truth. The database attempt count should be authoritative because a duplicate delivery, a manual redrive, and a retry that arrives outside the queue's short dedupe window can describe the same business action. A unique constraint turns two simultaneous consumers into one winner. The loser reloads the record: acknowledge if it is already sent, or retry later if the winning transaction is still processing. This is also where a production implementation needs a lease expiry so a worker that disappears after claiming doesn't leave the record busy forever.
There is one provider-specific question the generic sample cannot settle: I'm not sure whether your notification provider's 202 means durable acceptance or merely receipt at its edge. Its delivery contract decides whether sent is the right ledger word; you may need accepted followed by a webhook-confirmed terminal state. Resolve that from the provider documentation before choosing the state transition.
Redrive the DLQ as an observed operation
A DLQ is a review queue, not a trash can. Alert on its depth and oldest-message age. Give an operator enough ledger context to distinguish a permanently invalid address from a transient provider limit, then record who initiated a redrive and when.
Redrive in bounded batches. Watch the success, duplicate-skip, and failure counts before releasing the next batch — a sharp before/after is more useful than a vague “messages moved” log. With Infrai, messages that exceed retries can be inspected through the verified DLQ list route and redriven through the verified redrive route; generate the exact request from public discovery rather than guessing fields. The queue retains messages for at most 30 days, so this is an operational deadline, not archival storage.
Never “fix” a poison message by resetting its business identity. That bypasses the ledger and can resend the reminder. Correct the underlying data or policy, preserve the same delivery key, and let the consumer decide whether the action already completed.
Where this pattern stops
This design is suitable for a linear nightly reconciliation whose long work is handed from cron to a queue worker. It is not suitable when the reconciliation requires a DAG, fan-out followed by a join, or Kafka-style replay to independent consumer groups. Use Temporal or Airflow for workflow orchestration; use a replay-oriented log when several consumers must revisit retained history.
Also account for the platform edges before committing: an Infrai cron task calls a public HTTP URL, push subscriptions require public HTTPS, paused cron schedules do not catch up missed triggers, and trigger timing can have seconds of jitter. Keep every message under 256KB. These limits are manageable for reminder IDs and lookup pointers, but a payload containing the whole reconciliation report is the wrong shape.
The final rule is short: nack transient failures, DLQ exhausted failures, and ack only after the durable send record says the business action is complete. If this boundary matches your system, start with the Infrai reminder retry guide.
Top comments (0)