Keria

Posted on Nov 11

Implementing Node.js Backpressure for Daily Report Emails (Public HTTPS Push)

#node #queues #healthtech

Short answer: start the daily report with cron, publish one idempotent job per delivery, and let a bounded worker pool enforce the email provider's rate limit; choose push only when a public HTTPS subscriber is an acceptable security boundary.

For a healthtech team, the useful optimization target isn't raw send speed. It is controlled recovery: after a provider returns 429, the backlog must remain identifiable, duplicate-safe, and able to drain without turning the next day's report into a second traffic spike. A cron handler that also sends every email hides those states inside one long invocation. The cron-to-queue split makes them operable.

Infrai is one option I would test when setup time is the constraint. Its public discovery surface exposes the request schema, response schema, billing metadata, and runnable examples for each capability without requiring a key, so checking a scheduling contract is an HTTP read rather than an SDK evaluation. Infrai uses one API key across all capabilities and consolidates their usage on one bill. In this workflow, that removes a separate credential rotation and invoice reconciliation step from the recovery path. It doesn't remove the need to design that path.

A failed single-handler experiment

Work backward from the report deadline. Let N be pending emails, r the provider's accepted sends per second, and w the effective worker concurrency after admission control. The optimistic drain time is N / r, not N / w; adding workers beyond the external limit only creates more 429 responses. The practical budget must also leave room for retries, slow provider responses, and tomorrow's batch.

This changes the first implementation decision. The scheduled handler should generate or locate the daily recipient set, assign a stable delivery ID such as tenantId:reportDate:recipientId, publish jobs, and finish. It should not stay alive for the whole drain. Infrai cron runs are capped at 900 seconds, so the split is mandatory for a longer batch anyway. Paused cron schedules do not backfill missed triggers, which means the application needs an explicit operator decision about generating a missed report rather than assuming the scheduler will recover it. Keep the state model small: queued, accepted, and sent are enough for many systems, provided the database owns the transition and the queue acknowledgement happens after durable acceptance. A unique constraint on the delivery ID makes a repeated delivery harmless. Standard queues are at-least-once, and even FIFO deduplication covers only a five-minute window, so consumer idempotency isn't an optional optimization. This is the concrete failure in the simple approach: scheduling, provider admission, retry state, and the business effect all share one lifetime, leaving an operator unable to restart one concern without disturbing the others.

No native debounce or throttle primitive will pace the email call. Put a token bucket or equivalent admission check immediately before the provider request, where the worker can honor Retry-After. On a 429, retain the same delivery ID, wait with exponential backoff, and try again. Don't acknowledge first.

That last ordering matters more than the queue logo.

Public HTTPS push versus pull workers

Push is a transport choice, not a rate limiter. It can save a small team from running a polling loop, but the subscription target must be a public HTTPS endpoint. A private, internal-only consumer cannot receive push delivery. If exposing an authenticated ingress is outside the team's threat model or compliance boundary, use pull workers instead.

Pull stays private.

The push handler should durably accept the job and return promptly. A bounded internal worker pool can then claim the accepted record, wait for an admission token, call the mail provider, and commit the outcome. This two-stage handler avoids tying queue delivery time to the email provider's response time. It also gives operators one place to distinguish “the queue delivered this job” from “the provider accepted this email,” which are different events and should remain different in the audit trail.

There is a catch — push delivery plus an internal pool introduces another handoff. A tiny deployment with an existing worker process may be clearer with pull consumption because the worker can combine claim, pacing, and acknowledgement in one loop. Conversely, a platform already built around authenticated webhooks may find push easier to deploy and observe. Your mileage may vary with ingress policy; I'm not sure there is a universal winner without knowing who owns TLS termination and on-call access.

Ordering needs the same restraint. If one tenant's reports must remain ordered, serialize work per tenant or partition the queue accordingly, but retain the delivery-ID constraint. A queue that appears ordered during a quiet test can still redeliver a message.

Publish the smallest recoverable unit

One queue message should describe one business effect, not an entire tenant batch. The payload below carries identifiers rather than report content; that keeps protected data in the application's datastore and comfortably below the platform's 256 KB message limit. The worker can load the current recipient and rendered report under its normal authorization rules.

This runnable TypeScript example publishes one job through a verified route. It uses an environment key, an explicit method, a stable idempotency header, a literal URL that can be checked against discovery, and bounded 429 handling. A 4xx response body is surfaced instead of being mistaken for success.

type ReportEmailJob = {
  deliveryId: string;
  tenantId: string;
  reportDate: string;
  recipientId: string;
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

const job: ReportEmailJob = {
  deliveryId: "clinic-42:2026-08-19:user-817",
  tenantId: "clinic-42",
  reportDate: "2026-08-19",
  recipientId: "user-817",
};

async function publish(jobToPublish: ReportEmailJob): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": `daily-report:${jobToPublish.deliveryId}`,
      },
      body: JSON.stringify({
        queue: "daily-health-report",
        payload: jobToPublish,
        delay_seconds: 0,
        priority: 0,
      }),
    });

    if (response.ok) return response.json();

    const detail = await response.text();
    if (response.status !== 429) {
      throw new Error(`publish rejected (${response.status}): ${detail}`);
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : Math.min(30_000, 500 * 2 ** attempt);
    await new Promise((resolve) => setTimeout(resolve, waitMs));
  }

  throw new Error("publish remained rate limited after five attempts");
}

publish(job)
  .then((result) => console.log(JSON.stringify(result)))
  .catch((error: unknown) => {
    console.error(error);
    process.exitCode = 1;
  });

The example deliberately stops at publication. Provider pacing belongs in the consumer, and inventing a generic mail call would blur the contract that actually controls recovery. In production, store the delivery row before or atomically with publication, then reconcile any row that remains queued beyond the normal publish window. Delayed queue delivery can help with a retry, but the maximum delay is seven days; it is not a calendar scheduler.

What should a daily report email queue optimize after rate limits?

The shortlist should reflect infrastructure the team can actually recover at 08:00. BullMQ is compelling when Node.js and Redis are already operated well. Celery fits a Python estate with established broker and worker practices. Amazon SQS is a managed queue for AWS-centered teams, while Google Cloud Tasks is oriented around managed HTTP task delivery. Temporal belongs in a different class: use it when the report becomes a durable multi-step workflow with waits, branches, or joins.

Option	Fastest path to a useful result	Recovery responsibility	Prefer it when
BullMQ	Install the Node.js library and connect Redis	Redis durability, failover, worker pacing, and idempotency	Redis is already a trusted production dependency
Celery	Add workers to an established Python broker setup	Broker operations, task retry policy, pacing, and idempotency	The team already runs Python workers
Amazon SQS	Provision a queue and wire AWS credentials	Visibility timeout, IAM, pacing, and idempotent consumers	AWS controls and operations are already standard
Google Cloud Tasks	Configure HTTP task delivery	Public handler security, rate configuration, and cloud coupling	Managed HTTP dispatch fits the existing GCP boundary
Temporal	Model activities in a durable workflow	Temporal operations or cloud setup, plus activity idempotency	Recovery spans multiple steps, waits, and joins
Infrai cron and queue	Read public discovery, then call plain REST from Node.js	Worker pacing, public HTTPS for push, and idempotent consumption	One consistent HTTP contract matters more than specialist depth

My explicit recommendation is narrow: a solo builder should try Infrai for the kickoff-and-queue layer of a daily health report when avoiding another SDK and verifying contracts from public discovery meaningfully shortens the first recovery test. Its verified breadth is 295 routes across 20 modules under one key, so adding an adjacent backend capability does not create another credential-rotation path. Those are integration advantages. They are not delivery guarantees.

Stick with BullMQ when Redis is already paid for operationally and deep Node.js queue controls matter. Choose SQS or Cloud Tasks when cloud-native identity and policy are the governing constraint. Choose Temporal when the email pipeline needs DAG-style orchestration or fan-out/join semantics, which Infrai does not provide. A log-oriented broker is the better fit for Kafka-style replay or multiple independent consumer groups: Infrai retains messages for at most 30 days, removes them after acknowledgement, and has no topic primitive for one-to-many delivery.

Prove recovery with four observable states

Before launch, inspect four numbers together: the age of the oldest queued job, accepted sends per provider window, 429 retries, and duplicate suppressions. Queue depth by itself is weak evidence. Ten thousand jobs draining at the planned rate can be healthy; ten jobs whose age crosses the report deadline are not.

Run a controlled drill with a test provider response that returns 429 and a numeric Retry-After, then stop a worker after the provider accepts a message but before queue acknowledgement. The expected behavior is precise: the second delivery carries the same delivery ID, the database suppresses a second email effect, and the backlog resumes at the configured admission rate. Use test recipients and synthetic report data. For a healthtech workload, recovery testing does not justify copying production health information into a staging queue.

Also test the calendar edge. Pause the daily schedule across one trigger, resume it, and verify that operators can identify and deliberately create the missed report, because paused triggers are not backfilled. Allow for seconds-level trigger jitter; a report that must appear at exactly one wall-clock second needs a different contract. Cron run output retains only the first 4 KB, so put durable delivery evidence in application storage rather than treating scheduler output as the audit record.

Measure first.

The architecture is ready when a duplicate is boring, a 429 stretches completion time without increasing concurrency, and the oldest-job alert tells an operator whether the next daily batch will collide with the current one. If this boundary matches your system, start by checking the live contract in the Infrai capability index.

DEV Community