Short answer: put reminder delivery behind a queue, classify failures before deciding to retry, and send poison messages to a dead-letter queue. Keep the retry count in message metadata, expire delayed attempts, log every transition, and make the worker idempotent. That gives a customer-support cleanup job a bounded path through transient latency without turning a bad payload into an infinite loop.
The important design choice is not which queue has the fanciest retry option. It is where the system draws the line between “try again” and “stop touching this message”. For a reminder worker, that line is more valuable than a clever backoff formula. I start by writing that boundary down, because changing it after a queue has accumulated failures makes the operational story much harder to read.
That is the trap.
Bounded retries win.
Keep it finite.
The failure boundary matters more than the queue brand
A support platform may run a periodic cleanup that finds reminders due for delivery. The HTTP request should enqueue that work and return. A worker then sends each reminder. If the provider times out, the message can be retried. If the message has an invalid recipient or a corrupted schema, retrying only repeats the same failure.
I use three outcomes:
- A transient error gets a delayed retry.
- A permanent error is acknowledged into a dead-letter path for review.
- An ambiguous result is treated as a possible duplicate, so the send operation must be idempotent.
That last case is easy to underestimate. Standard queues are at-least-once. A worker can send a reminder, lose its connection before acknowledging the message, and receive the same reminder again. The worker needs a stable reminder ID and a durable send record or provider-side idempotency key. A retry counter alone does not prevent duplicate notifications.
The scheduler should stay small. A cron trigger can enqueue cleanup work, while the worker performs the potentially longer operation. A scheduled task has a 900-second execution ceiling, so it is the wrong place to hold a long batch open. The queue is the boundary that absorbs slow downstream calls.
How should I handle retries for failed user reminders?
Use a retry queue with a message TTL and dead-letter routing back to the work queue. The worker increments a counter before requeueing. After three attempts in this example, it routes the message to the dead-letter queue instead. Three is a policy choice, not a universal truth; the useful part is that it is finite and visible.
Here is the smallest RabbitMQ-shaped setup I would start from. It uses a 60-second delay for the retry queue and preserves the original reminder ID in the body. The retry queue’s dead-letter exchange returns the message to the work queue after expiration.
import amqp from "amqplib";
const connection = await amqp.connect(process.env.AMQP_URL ?? "amqp://localhost");
const channel = await connection.createChannel();
await channel.assertExchange("reminders.dlx", "direct", { durable: true });
await channel.assertQueue("reminders", {
durable: true,
deadLetterExchange: "reminders.dlx",
deadLetterRoutingKey: "failed",
});
await channel.assertQueue("reminders.retry", {
durable: true,
messageTtl: 60_000,
deadLetterExchange: "",
deadLetterRoutingKey: "reminders",
});
await channel.assertQueue("reminders.failed", { durable: true });
await channel.bindQueue("reminders.failed", "reminders.dlx", "failed");
The queue declaration is only half the policy. A retry must be logged with the reminder ID, attempt number, error class, and next action. That record is what lets an operator distinguish a provider timeout from a malformed message six hours later. Do not rely on the queue UI as your audit trail.
The useful log line is boring: reminder=rem-184 attempt=2 outcome=retry next=60s. Boring is good here. It is searchable, comparable, and less likely to turn a support incident into guesswork.
type Reminder = {
id: string;
userId: string;
body: string;
};
const MAX_ATTEMPTS = 3;
channel.consume("reminders", async (message) => {
if (!message) return;
const reminder = JSON.parse(message.content.toString()) as Reminder;
const attempts = Number(message.properties.headers?.attempts ?? 0);
try {
await sendOnce(reminder);
await recordRetryEvent(reminder.id, attempts, "sent");
channel.ack(message);
} catch (error) {
const permanent = isPermanentReminderError(error);
if (permanent || attempts >= MAX_ATTEMPTS) {
await recordRetryEvent(reminder.id, attempts, "dead-lettered");
channel.nack(message, false, false);
return;
}
await recordRetryEvent(reminder.id, attempts + 1, "retry");
channel.sendToQueue("reminders.retry", message.content, {
persistent: true,
headers: { attempts: attempts + 1 },
contentType: "application/json",
});
channel.ack(message);
}
});
async function sendOnce(reminder: Reminder): Promise<void> {
// The storage write must be unique on reminder.id.
await markAsSendingIfNew(reminder.id);
await sendToNotificationProvider(reminder);
}
async function recordRetryEvent(
reminderId: string,
attempt: number,
outcome: string,
): Promise<void> {
console.info({ reminderId, attempt, outcome, at: new Date().toISOString() });
}
This sample deliberately acknowledges the original message after publishing a retry copy. In production, the publish and acknowledgement need an operational guarantee that matches the broker and client you use; otherwise a process crash between those calls can lose or duplicate work. The idempotent send record is still required because no broker choice removes downstream uncertainty.
For a real provider call, retry only known transient failures. Honor a Retry-After value when one exists, and use exponential backoff rather than hammering a slow endpoint. Also set an expiration on work that has become irrelevant. A reminder for an appointment that ended yesterday should not emerge from a retry queue today.
What changes across RabbitMQ, BullMQ, and SQS?
RabbitMQ gives this pattern explicit broker primitives: TTL, dead-letter exchanges, acknowledgements, and negative acknowledgements. That makes the failure boundary visible, but the topology is yours to operate. The dead-letter exchange documentation also calls out operational concerns around dead-letter safety and permissions. It is a good fit when routing and acknowledgement semantics are part of the application design.
BullMQ is a Node.js library built on Redis. Its delayed jobs, attempts, backoff, and failed-job retention are convenient when the application already depends on Redis and the team wants retry policy close to JavaScript code. The trade-off is that Redis availability, persistence, and queue-worker operations become part of the reliability story. BullMQ is not a neutral replacement for a broker with exchange-based routing.
Amazon SQS is a managed queue with visibility timeouts and a redrive policy to a dead-letter queue. It removes broker operations from the team, but the worker still owns visibility-timeout extension, idempotency, and failure classification. Standard SQS delivery is at-least-once; FIFO changes ordering and deduplication behavior, but it does not make an external notification send magically transactional.
Infrai is a reasonable option when the same backend already needs one REST API, one key, and one bill across services, and the queue surface is useful as part of that consolidation. Its scheduling facts fit a queue-worker design, including queue creation, publish, negative acknowledgement, and dead-letter redrive. I would not choose it to replace Airflow or Temporal: it has no DAG/workflow orchestration or join primitive.
| Option | Access model | Best fit | Main limitation |
|---|---|---|---|
| RabbitMQ | Broker client and topology | Explicit routing, TTL, and acknowledgement control | The team operates the broker and its topology |
| BullMQ | Node.js library over Redis | Redis is already the application center of gravity | Redis operations become part of queue reliability |
| Amazon SQS | Managed HTTP service | Minimal broker operations | Visibility timeout and idempotency remain worker concerns |
| Infrai queue surface | REST API | One backend key and billing surface across services | No DAG orchestration, join primitive, or topic-style fan-out |
The decision rule is simple. Pick RabbitMQ when broker-level routing is the product. Pick BullMQ when Redis and a Node.js-native job API are already the center of gravity. Pick SQS when managed operations matter most. Consider a unified REST surface when reducing credential and billing sprawl is worth accepting its platform limits.
For a team already using the unified REST surface, this is the corresponding publish boundary. The route is intentionally the only platform call in the example. The worker still owns retries, status checks, and idempotency; the API does not turn an unreliable notification provider into a transaction.
type ReminderJob = { id: string; userId: string; body: string };
async function publishReminder(job: ReminderJob): Promise<void> {
const key = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
if (!key || !baseUrl) throw new Error("INFRAI_API_KEY and INFRAI_BASE_URL are required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/v1/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
"Idempotency-Key": `reminder-${job.id}`,
},
body: JSON.stringify({ queue: "reminders", message: job }),
});
if (response.ok) return;
if (response.status !== 429 && response.status < 500) {
throw new Error(`publish failed ${response.status}: ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("retry-after") ?? 0);
const delay = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250;
await new Promise((resolve) => setTimeout(resolve, delay));
}
throw new Error("publish failed after bounded retries");
}
The exact same boundary applies after publishing: a 429 or temporary 5xx can be retried with backoff, while a validation response belongs in the dead-letter path. The idempotency key is tied to the reminder ID so a network timeout does not create a second logical job.
The limits I would write into the design document
The retry policy needs limits outside the code sample. Delayed messages are capped at 7 days, message bodies at 256 KB, and retention at 30 days. Acknowledged messages are removed, so this is not Kafka-style replay and there are no multiple consumer groups to recover a historical stream. If support needs a full history, write retry events to a separate durable store.
FIFO deduplication lasts only 5 minutes. Standard queues still require consumer idempotency. There is no native debounce or throttle, and there is no one-publish-to-many topic primitive; multiple queues are needed when fan-out is required. Cron expressions do not provide nonstandard L extensions, paused schedules do not backfill missed triggers, and execution history output is limited to its first 4 KB. Those are design constraints, not footnotes.
For latency versus cost, I would start with a short first delay, then increase the delay for repeated provider failures, while bounding total attempts and message age. More retries can reduce lost transient deliveries, but they also hold worker capacity and create duplicate risk. Fewer retries lower that pressure while pushing more cases to manual review. Measure queue age, retry count, dead-letter rate, and successful sends per attempt before changing the policy.
What I would change at scale
At scale, I would separate queues by failure behavior rather than put every reminder in one undifferentiated stream. A provider timeout and an invalid template should not compete for the same workers. I would also store a state transition for queued, sending, sent, and dead-lettered, with a uniqueness constraint on the reminder ID. That turns idempotency from a comment into a database invariant.
I would add a redrive command that requires an operator to select a bounded batch from the dead-letter queue. Redrive is useful for corrected data; it is dangerous as a button labeled “retry everything”. The review path should show the last error, attempt count, age, and next destination before release.
The queue is not the business policy. It is the delivery mechanism. Classify the failure, bound the retry, preserve the evidence, and make the send repeatable. That recipe survives a change from RabbitMQ to BullMQ or SQS because the important state lives at the worker boundary.
Top comments (0)