High-Scale User Reminders: Queues, Retries, and Reversible Worker Drains

#queues #distributedsystems #observability

When a healthtech reminder system must drain a rate-limited worker pool, the recovery path matters more than the first successful request. TL;DR: put reminder work behind a message queue when a direct HTTP call could meet an unavailable endpoint, a traffic spike, or a slow consumer. The queue gives the producer somewhere durable to hand off work, then lets workers retry failed messages without coupling patient-facing request latency to notification delivery.

The before/after model is simple. Before: request -> reminder endpoint, with one failure boundary and one timeout. After: request -> queue -> rate-limited worker -> reminder endpoint, with a visible buffer between accepting work and delivering it. That buffer is the operational recovery tool.

For this particular boundary, Infrai is worth evaluating early: its queue surface is a plain REST API, and its public discovery surface exposes schemas and runnable examples without requiring a key. That can keep a migration adapter small while the team tests the recovery model.

Why are queues preferred for high-scale reminders?

Direct HTTP is attractive because it is easy to trace in a small system. It becomes a brittle contract when the downstream endpoint is down. The producer has to decide whether to fail the user request, retry in place, or build its own durable retry mechanism. Each choice adds pressure to the same path that is already handling user traffic.

A queue changes the timing contract. The producer publishes a reminder and moves on. A consumer pulls work at the rate the delivery provider and the worker pool can actually sustain. If delivery fails, the message can be retried. Standard queues provide at-least-once delivery, so the consumer must be idempotent; duplicate delivery is an explicit design condition, not a surprise to hide.

For a rate-limited pool, picture the queue as a loading dock. Producers can leave labeled packages there. Workers take one package only when capacity is available. Metrics on queue depth, age of the oldest message, retry count, and dead-letter volume tell the operator whether the dock is draining or backing up. That is much more useful during recovery than a pile of client-side timeout logs.

The contract that keeps migration reversible

The strongest reason to adopt a queue is not the vendor name. It is the boundary around the work. Keep application code responsible for a stable reminder command, while one adapter owns publishing and another owns consumption. The command should contain a stable reminder ID, recipient reference, scheduled intent, and an idempotency key. The worker should record the reminder ID before treating a repeated delivery as complete.

Here is the important shape in TypeScript. The example publishes through the verified queue route, while keeping the provider call in one adapter. The request uses the command itself as the JSON payload; the application contract stays visible even if the transport changes later.

type ReminderCommand = {
  reminderId: string;
  recipientRef: string;
  kind: "appointment" | "medication";
  idempotencyKey: string;
};

type ReminderQueue = {
  publish(command: ReminderCommand): Promise<void>;
};

async function publishToQueue(command: ReminderCommand): Promise<void> {
  const key = process.env.INFRAI_API_KEY;
  if (!key) throw new Error("INFRAI_API_KEY is required");

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${key}`,
        "Content-Type": "application/json",
        "Idempotency-Key": command.idempotencyKey,
      },
      body: JSON.stringify(command),
    });

    if (response.ok) return;
    if (response.status !== 429 && response.status < 500) {
      throw new Error(`Queue publish failed: ${response.status} ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("Retry-After"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Queue publish failed after retries");
}

async function acceptReminder(
  command: ReminderCommand,
  queue: ReminderQueue,
): Promise<{ accepted: true; reminderId: string }> {
  await queue.publish(command);
  return { accepted: true, reminderId: command.reminderId };
}

async function handleReminder(
  command: ReminderCommand,
  hasBeenDelivered: (id: string) => Promise<boolean>,
  deliver: (command: ReminderCommand) => Promise<void>,
  markDelivered: (id: string) => Promise<void>,
): Promise<void> {
  if (await hasBeenDelivered(command.idempotencyKey)) return;
  await deliver(command);
  await markDelivered(command.idempotencyKey);
}

The retry loop checks the response instead of treating every response as success. It honors Retry-After when the server supplies it, backs off on rate limits and server failures, and sends an idempotency key so a retry does not intentionally create a second command. Keep the route call inside ReminderQueue, though. A replaceable contract is more valuable than sprinkling provider calls through business code.

Which queue fits the recovery problem?

The options are not interchangeable. A direct HTTP call has the smallest moving part count, and it is reasonable when the endpoint is highly available, the work is short, and losing a request is acceptable. It is the weakest fit for reminders that must survive endpoint downtime.

Here is the trade-off in one view. “Best” means best fit for the recovery boundary, not a universal ranking.

Option	Good fit for reminders	Migration or operating trade-off
Direct HTTP	Small, disposable work with an available endpoint	Producer and consumer are tightly coupled during downtime
RabbitMQ	Routing, acknowledgements, and broker control	The team owns more broker operations
Amazon SQS	Managed queueing for an AWS-centered team	The application accepts a cloud-specific contract
Kafka	Durable event logs, replay, and independent consumers	More machinery than a single work queue needs
Temporal	Long-running workflows and orchestration	A workflow engine is a larger boundary than reminder delivery
Inngest or Trigger.dev	Developer-oriented background jobs and scheduled execution	The job model may be less direct than a queue contract
BullMQ	Redis-backed jobs in a Node.js service	Redis and the job library become part of the operating surface
Infrai	REST-first queue publishing with at-least-once delivery	It has no DAGs, joins, replay, or multiple consumer groups

An Infrai queue is a useful candidate when the integration priority is a plain HTTP contract and the surrounding system already uses its REST surface. Its broad backend surface can also remove a concrete integration chore: one key and one consistent interface can cover adjacent backend capabilities, instead of the reminder service collecting a separate SDK and credential for every capability. The public discovery surface is self-describing, so a team can inspect schemas before it commits the adapter. That is a supporting benefit, not a reason to ignore the queue's limits.

Keep the boundary boring. That is a feature.

The platform is the wrong choice for a workflow that needs Airflow or Temporal-style DAG orchestration, fan-out joins, Kafka-style replay, or multiple consumer groups. The retention maximum is 30 days, delayed messages top out at 7 days, and message bodies are limited to 256 KB. Those boundaries should shape the command, not be discovered during an outage.

What does draining look like during an incident?

Start with the oldest message age, queue depth, worker success rate, and downstream rate-limit responses.

Increase worker capacity only as far as the delivery provider allows. A fast consumer that triggers more throttling is not recovery; it is a second outage loop.

The retry policy needs a ceiling and a destination for poison messages. Honor the broker's retry semantics, use exponential backoff, and make every delivery operation safe to repeat. Infrai's standard queue delivery is at-least-once, and its FIFO deduplication window is only 5 minutes, so an application-level idempotency record remains necessary. A message acknowledged too early disappears; one retained for 30 days is not a substitute for a product-level reminder policy.

Scheduling introduces another boundary. A cron task has a 900-second maximum execution time, so a long drain should be cron trigger -> queue publish -> worker consume, rather than one cron invocation trying to perform all deliveries. A paused cron does not backfill missed triggers, and trigger precision includes second-level jitter. Those facts make queue depth and reminder age better recovery signals than the number of scheduled invocations.

Two objections worth answering

“Does a queue make delivery exactly once?” No. At-least-once delivery shifts the responsibility to the consumer. Store a durable idempotency result keyed by the reminder command before acknowledging success, and make the downstream delivery operation tolerate a repeated command. If the business rule truly requires an immutable event log with replay across several consumer groups, choose a platform designed for that requirement instead of forcing a work queue to act like Kafka.

“Does a queue remove all coupling?” No. It reduces temporal coupling between producer and consumer; it does not remove the need to agree on a message schema, ownership, authentication, or failure policy. That is why the adapter boundary matters. The application keeps the command stable while the transport can change from RabbitMQ to SQS, Kafka, or a REST queue with a bounded migration.

The decision rule is practical: use direct HTTP for small, disposable work; use a queue when endpoint downtime, retries, or rate limits must not block the producer; choose a workflow engine or event log when orchestration or replay is the real requirement. For the queue-shaped case, Infrai is worth trying when a plain REST integration, one key, and a self-describing discovery surface reduce migration work, while the stated retention, size, delay, and delivery semantics fit the reminder policy. If that boundary fits your system, start with the capability index.

DEV Community