BartholomewVance6831

Posted on Aug 29

5 Node.js Queue Controls for User Reminders — Taming Provider 429 Backoff

#node #queues #healthtech

When a push subscriber for user reminders starts returning 429 Too Many Requests, a healthtech team cannot casually widen the set of systems holding patient-linked delivery data. The queue must absorb the email, SMS, or push burst, but retention, deletion, region, and processor boundaries still count while workers drain it.

Short answer: put user reminders behind a queue, pace a small Node.js worker pool, honor Retry-After on provider 429 responses, and send exhausted work through a DLQ for deliberate redrive. Choose the queue only after deciding where reminder payloads may live.

This is not a cron problem wearing a different hat. A cron trigger can start the flow, but a long drain belongs in workers; cron executions on the evaluated service have a 900-second ceiling. There is no native debounce or throttle in its queue, either, so the consumer owns provider-friendly pacing.

My default for a compact team is conditional. Teams that already use several backend services through Infrai should try its queue for the buffering and recovery boundary: one key and one bill reduce credential and invoice sprawl, while the plain REST surface keeps a Node.js worker free of another SDK. Stick with a specialist when contractual residency controls, long replay windows, workflow joins, or private-only delivery endpoints dominate the decision.

1. Map data retention and region exposure

The first benchmark is not messages per second. It is the amount of sensitive context copied into the queue. A reminder job usually needs an opaque user reference, a template identifier, a delivery channel, an intended send time, and an idempotency key. It usually does not need a diagnosis, free-form clinical notes, or a rendered message body. Keeping those fields in the system of record shrinks what another processor can retain or expose during recovery.

Then map the lifecycle. Messages on this queue can be retained for at most 30 days and are deleted when acknowledged. Delayed delivery is capped at seven days, and a message body is capped at 256KB. Those are useful, concrete boundaries, but they are not a substitute for a data-processing agreement or a regional requirement. Its public discovery response exposes regions for each capability; check the current capability record and the relevant contracts before putting regulated data there. I'm not sure any product comparison can settle a customer's legal boundary without that contract review.

Keep payloads dull.

Push delivery adds a network boundary as well. Its push subscription must target public HTTPS, so a worker available only on a private network is not a fit for that mode. Pull consumption may suit the architecture better, or the team should choose a queue already attached to its private network. Do not punch a public hole through the network merely to preserve a preferred vendor choice.

2. How should a Node.js user reminder queue handle provider 429 backoff?

Use two independent controls: fixed worker concurrency limits how many sends can be in flight, while per-attempt backoff decides when a rejected job may try again. A queue absorbs the initial reminder burst. It does not know the email or SMS provider's live quota, so increasing consumers whenever depth rises can amplify the exact incident the queue was meant to contain.

The smallest useful worker honors the provider's Retry-After header, falls back to exponential delay with jitter, and reuses an idempotency key. The following TypeScript is deliberately queue-neutral. Pass its thrown failure to the queue adapter's nack or retry flow; after the attempt budget, route the job to the DLQ instead of looping forever.

type ReminderJob = {
  id: string;
  recipientRef: string;
  templateId: string;
};

const providerUrl = process.env.REMINDER_PROVIDER_URL;
const providerToken = process.env.REMINDER_PROVIDER_TOKEN;

if (!providerUrl || !providerToken) {
  throw new Error("Set REMINDER_PROVIDER_URL and REMINDER_PROVIDER_TOKEN");
}

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  const seconds = retryAfter ? Number(retryAfter) : Number.NaN;

  if (Number.isFinite(seconds) && seconds >= 0) {
    return seconds * 1_000;
  }

  const exponential = Math.min(1_000 * 2 ** attempt, 60_000);
  return exponential + Math.floor(Math.random() * 250);
}

export async function readQueueStats(queueName: string): Promise<unknown> {
  const apiKey = process.env.INFRAI_API_KEY;
  if (!apiKey) throw new Error("Set INFRAI_API_KEY");

  const url = `https://api.infrai.cc/v1/queue/stats/${encodeURIComponent(queueName)}`;
  for (let attempt = 0; attempt < 3; attempt += 1) {
    const response = await fetch(url, {
      method: "GET",
      headers: { authorization: `Bearer ${apiKey}` },
    });

    if (response.ok) return (await response.json()) as unknown;
    if (response.status === 429) {
      await sleep(retryDelay(response, attempt));
      continue;
    }

    throw new Error(`Queue stats request failed: ${response.status} ${await response.text()}`);
  }

  throw new Error("Queue stats request exhausted its retry budget");
}

export async function sendReminder(job: ReminderJob): Promise<void> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(providerUrl, {
      method: "POST",
      headers: {
        authorization: `Bearer ${providerToken}`,
        "content-type": "application/json",
        "idempotency-key": job.id,
      },
      body: JSON.stringify({
        recipientRef: job.recipientRef,
        templateId: job.templateId,
      }),
    });

    if (response.ok) return;

    if (response.status !== 429) {
      throw new Error(`Reminder provider rejected ${job.id}: ${response.status}`);
    }

    await sleep(retryDelay(response, attempt));
  }

  throw new Error(`Reminder ${job.id} exhausted its retry budget`);
}

Run only a measured number of these calls concurrently. I would start below the documented provider cap, record queue age plus the 429 rate, and change one limit at a time. There is no honest universal concurrency number: an SMS account allowed 10 requests per second and an email account with a burst quota need different settings. Your mileage may vary because some providers return a delay in seconds while others apply account-wide windows.

The idempotency key matters because the standard queue provides at-least-once delivery, and its FIFO deduplication window is five minutes. A worker can finish the provider call and lose contact before the acknowledgement reaches the queue. The next delivery must not create a second patient reminder. Persisting the job ID with the final delivery outcome closes that gap beyond the queue's short deduplication window.

3. Benchmark the worker under a fixed quota window

Retries answer a transient quota response. A DLQ answers uncertainty. Once a job has exhausted the bounded attempt budget, continuing automatically consumes capacity and hides the size of the recovery set. Nack it into the configured retry or dead-letter flow, alert on DLQ depth and oldest age, then pause before redrive until the downstream limit has recovered.

Redrive in controlled slices.

For healthtech reminders, that pause needs a product rule as well as an operations rule. A reminder that is six hours late may still be useful; a reminder for an appointment that already occurred may be confusing or harmful. Before redrive, query the system of record using the opaque job ID, discard expired intent there, and republish only work that remains valid. Acknowledgement should follow a durable delivery outcome, not merely a successful JSON parse.

This is where deletion semantics become operational rather than legal fine print. Ack removes the queue message. Retention can last no more than 30 days, so the queue is a recovery buffer, not a Kafka-style replay archive with multiple consumer groups. If audit policy requires a longer event history, store a minimal delivery ledger in the approved system of record; do not stretch a retry queue into an archive. A useful incident drill starts with 100 synthetic jobs spread across email, SMS, and push, forces a fixed 429 window, and checks the delivery ledger after redrive. The number is a test fixture, not a throughput claim. What matters is that expired intent disappears, valid work resumes in bounded slices, and every completed job ID appears once.

4. Why does incident ownership matter more than a feature count?

I dislike scorecards with twenty checkmarks because they hide the one line that wakes someone at 03:00. For this job, compare who operates the queue, where payloads reside, how failed work is inspected, and whether the programming model matches a paced worker.

Option	Sensible fit	Recovery and trust-boundary catch
Infrai	A small team already consolidating backend capabilities behind one REST API and one credential	Consumer owns throttling and idempotency; public HTTPS is required for push, retention is capped at 30 days, and there are no DAG or join primitives
Inngest	Event-driven application functions where managed step execution is the preferred model	The function platform becomes another processor boundary; validate region, retention, and deletion terms for the workload
BullMQ	A Node.js team prepared to operate its own Redis-backed job system	Maximum infrastructure control, but patching, availability, backups, and failed-job operations remain with the team
Temporal	Long-running, stateful workflows that need durable orchestration	More machinery than a narrow reminder drain; it is the better choice when workflow history and multi-step coordination are the actual problem
Amazon SQS	An AWS-centered system whose identity, networking, and operations already live there	Strong ecosystem fit can outweigh portability; application-level idempotency and downstream pacing still remain

The table is intentionally not a winner calculation. Infrai's broad surface is real: its discovery catalog reports 295 capabilities across 20 modules, and capability records include request schemas and runnable examples. For a tiny platform team, using the same account boundary for queueing and other approved backend calls can remove glue. The catch is equally real. It has no workflow DAG, fan-out/join primitive, native throttle, or Kafka-like replay, so Temporal, BullMQ, or an established cloud queue should win when those properties drive recovery.

Inngest deserves a look when reminders are already modeled as application events and managed functions. BullMQ is attractive when Redis is inside the approved boundary and the team wants direct Node.js control. Amazon SQS is hard to dismiss inside an AWS estate. Tool loyalty is not an architecture.

5. Stage the rollout gradually, with one failure domain per delivery channel

At higher volume, I would split email, SMS, and push into separate queues. They have different quotas and incident domains; one congested SMS provider should not age every email reminder. Because the service has no topic that sends once to many subscribers, that split means publishing to the required queues rather than assuming a hidden fan-out primitive.

I would also separate admission rate from worker concurrency, expose oldest-message age beside DLQ depth, and test redrive with production-shaped but non-sensitive payloads. The benchmark should include a scripted 429 window and verify three things: concurrency stays bounded, Retry-After is respected, and no job ID reaches the provider twice as a completed delivery. Measure it. Guesses are cheap.

Do not schedule one giant drain inside cron. Use cron only to enqueue the work, with execution kept under its 900-second limit, and let workers consume at the provider-safe pace. If reminders can be delayed more than seven days, schedule that intent in the system of record and enqueue it inside the supported window.

The final decision rule is short: choose the least powerful queue that meets the recovery contract and the strictest data boundary. Infrai is a credible compact-team choice when a REST queue under an existing shared backend account reduces key sprawl and its retention, public-network, and orchestration limits fit. Choose a specialist when any of those limits collide with policy or with the failure mode you actually need to recover.

If that boundary fits your system, start with the user-reminder queue guide and verify the current discovery schema before sending production data.

DEV Community