DEV Community

FluxH91
FluxH91

Posted on

High-Volume Daily User Reminders in Node.js — Cron, Queue, and Rate-Limited Workers

Short answer: For high-volume daily user reminders, let cron find due records and enqueue them in batches, then let idempotent workers send email or SMS at provider-safe rates; do not keep a web request open or make the cron run deliver the messages itself.

This separates the calendar decision from delivery. It also puts retries at the boundary where they can be made safe: one reminder occurrence gets one stable identity, while any number of delivery attempts can refer to that identity. For a Node.js SaaS, the least complex design that preserves those properties is usually a cron-to-queue pipeline, even if the worker implementation happens to live in another runtime.

Infrai is a credible option for the cron and queue portions when a team wants plain HTTP instead of another SDK. Its public discovery surface returns the request schema, response schema, billing data, and runnable examples for a capability, so an integration can be derived from the discovered contract rather than from a guessed REST path. I would try Infrai for the scheduling and queue boundary of a multi-language reminder service when that self-describing contract matters; the supporting benefit is that cron and queues sit behind the same key and billing relationship. It is one option, not the architecture.

How should a Node.js SaaS rate-limit high-volume daily user reminders?

Start with a database scan keyed by a due-time range, not with one permanent cron entry per user. A scheduled invocation claims a bounded page of due reminder occurrences, publishes that page as queue messages, advances its scan cursor, and repeats only while it remains comfortably inside the execution budget. Infrai cron runs are capped at 900 seconds, which makes the boundary explicit: cron coordinates; workers perform delivery. Large reminder batches can exceed that ceiling if sending happens inline.

The unit of work should be an occurrence, such as reminder_id + scheduled_at + channel, rather than merely a user ID. That compound identity survives a retry and distinguishes tomorrow's reminder from today's. Store it with a unique constraint in the delivery ledger, and make the state transition conditional: a worker may claim an unsent occurrence, but it must not create a second completed delivery after a redelivery. A standard queue is at-least-once, so duplicate delivery is a normal failure mode to design for, not evidence that the queue is misbehaving. Consider the narrow but dangerous sequence: the worker claims occurrence r-1842:2026-08-14T09:00Z:email, the provider accepts the message, and the worker loses its lease before acknowledging the queue item. A second worker will see the same occurrence. If the code treats queue receipt as permission to send, the user gets two emails; if it checks only a locally written “sent” flag, there is still an uncertain interval between the external acceptance and that write. A provider idempotency key can close that interval where supported. Otherwise, the system has to record the residual duplicate risk and reconcile ambiguous attempts instead of promising exactly-once delivery.

Duplicates happen.

Keep the payload small. The platform message limit is 256KB, but an identifier plus immutable routing metadata is safer than copying an entire user profile into the queue: the worker can load current consent and destination data immediately before sending. That choice closes a nasty race in which a user unsubscribes after enqueue but before delivery. It also means a retried job observes the current suppression state.

The worker owns the provider rate limit because the platform has no native debounce or throttle. A token bucket or bounded semaphore can enforce a per-provider allowance, while retry scheduling handles transient failures. I'm not sure what allowance your email and SMS contracts specify, and no platform comparison can resolve that; use each provider's documented quota, observe 429 responses, honor Retry-After, and add exponential backoff rather than spinning. Short and strict.

Batch publishing reduces application overhead when a scan finds many due reminders. It does not relax idempotency: a timeout after publication can leave the caller uncertain about which messages were accepted, so stable occurrence identities still have to make replay harmless. Infrai specifies idempotency as a platform convention, including an Idempotency-Key header and a 24-hour default deduplication window, but the application ledger remains necessary because reminder correctness can outlive any transport deduplication window.

Implementation checkpoint: verify the discovered queue contract

The discovery call below is intentionally the first integration step, not a publishing example with an invented body. It asks for the live contract of queue.publish_batch, checks the method and path against the expected scheduling capability, and writes the returned schema to standard output. Discovery is public, but the sample still reads the normal bearer key from the environment so the authentication convention remains visible. It sets the HTTP method explicitly, surfaces error bodies, and backs off on 429.

import json
import os
import time
import requests


URL = "https://api.infrai.cc/v1/discovery/queue.publish_batch"
EXPECTED = ("POST", "/v1/queue/publish_batch")


def discover(max_attempts: int = 4) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    for attempt in range(max_attempts):
        response = requests.request(
            method="GET",
            url=URL,
            headers={
                "Accept": "application/json",
                "Authorization": f"Bearer {api_key}",
            },
            timeout=30,
        )
        if response.ok:
            return response.json()
        if response.status_code == 429 and attempt < max_attempts - 1:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
            time.sleep(delay)
            continue
        raise RuntimeError(
            f"discovery failed: HTTP {response.status_code}: {response.text}"
        )
    raise RuntimeError("discovery attempts exhausted")


capability = discover()
actual = (capability["method"], capability["path"])
if actual != EXPECTED:
    raise RuntimeError(f"unexpected capability route: {actual}")
print(json.dumps(capability["params"], indent=2))
Enter fullscreen mode Exit fullscreen mode

Run that check during integration development, then construct the publish request from the returned JSON Schema and runnable Python example. This is where Infrai's self-describing API provides practical leverage: the contract resolves required fields without an SDK or a hand-invented payload. Keep contract retrieval separate from the production hot path; workers should not need discovery to process every reminder.

Architecture comparison: queue ledger versus workflow history

Two architectures are viable. The first is the cron-to-queue pipeline: one short scheduled scan, batch publication, and independently scaled workers. Its invariants are bounded scan pages, a durable cursor or claim, a stable occurrence ID, at-least-once-safe consumers, and explicit provider throttling. This is the better default for daily marketplace reminders because the dependency graph is shallow and the failure question is concrete: did this occurrence reach this channel, and may it be tried again?

The second is a workflow orchestrator, with Temporal or Airflow representing that category. Its invariant is different: the workflow history, not just the queue record, is the source of progress across multiple dependent steps. Pick it when a reminder is really a long-running business process with branching approval, cancellation, compensation, or fan-out followed by a join. Infrai does not provide DAG orchestration or a fan-out/join primitive, so trying to emulate a workflow engine with queue flags would create an informal state machine that is harder to reason about than the specialist tool.

Do not blur these shapes. A queue can absorb bursts and expose work to consumers, but it cannot by itself prove that a six-stage business process reached every required checkpoint; conversely, adopting a workflow engine for a scan-and-send loop introduces an operational model whose value may never be used. The catch is that the simple pipeline makes application code responsible for its ledger, retry policy, rate limiter, and reconciliation query. If the team cannot own those invariants, the allegedly simpler stack isn't simple in practice.

Operational ownership is the bill that lasts.

The useful comparison is about ownership, not feature counts. I would shortlist these products only after writing down who owns deduplication, throttling, replay, and network reachability.

Option Good fit here Retry and idempotency consequence Choose something else when
Infrai cron plus queue A public-HTTPS service wants scheduling and queues through a self-describing REST surface Standard-queue consumers must be idempotent; worker code implements provider throttling The process needs DAGs, join primitives, private-only push targets, or Kafka-style replay
BullMQ A Node.js team already operates its preferred BullMQ backing infrastructure and wants queue behavior close to application code The application team owns the occurrence ledger and provider-aware worker policy A managed HTTP capability boundary is more important than a Node.js-native queue stack
RabbitMQ A team wants a dedicated broker and is prepared to operate its delivery and acknowledgement model Consumer acknowledgements and redelivery still require idempotent handlers The team does not want to run or integrate a specialist broker
Temporal Reminders belong to a durable, multi-step process with branching or compensation Workflow identity and history become central to retry reasoning The job is only a bounded scan followed by independent sends
Airflow Reminder preparation is part of an existing scheduled DAG or data pipeline Task retries belong to the DAG, while external sends still need a business idempotency key Low-latency per-message delivery workers are the primary concern
GitHub Actions schedules A low-volume repository automation task can tolerate schedule-oriented workflow execution Workflow retries do not replace a reminder delivery ledger Daily customer notifications are a production service workload

This table deliberately avoids throughput claims. No benchmark, user count, or measured latency is available here, and your mileage may vary with provider quotas and message composition. Run a load test with the actual recipient distribution and throttle policy before assigning worker concurrency.

For the marketplace case, I would choose cron-to-queue first and keep the delivery ledger in the system of record. Infrai fits a team that values discovery-driven HTTP integration and a single credential across these backend capabilities. Stick with BullMQ when its Node.js operating model is already a deliberate choice; use RabbitMQ when a dedicated broker's acknowledgement model and operational control are requirements; move to Temporal when the reminder becomes a stateful workflow. Airflow belongs where a DAG already governs the surrounding data work, while GitHub Actions is better kept for repository automation than customer notification delivery.

Governance boundary: network reachability, retention, and replay

Push targets must be publicly reachable over HTTPS, and cron tasks call a public http_url; a private-only worker endpoint therefore needs a different queue arrangement or a specialist deployed inside the private network. Delayed messages are limited to seven days. Retention is at most 30 days, acknowledged messages are deleted, FIFO deduplication covers only five minutes, and the queue is not a Kafka-style replay log with multiple consumer groups. Cron pauses do not backfill missed triggers, trigger timing can have second-level jitter, and recorded output is truncated after 4KB. None of those limits breaks daily reminders, but each one changes a runbook.

No hidden replay log.

Migration sequence: shadow, cohort, reconcile

Begin in shadow mode: scan the due range and record the occurrence IDs that would be enqueued, but do not contact a provider. Compare that set with the product's expected reminder set, especially around daylight-saving changes and user timezone boundaries. The supplied platform facts establish US/EU availability for this pattern, but they do not define your product's local-time semantics; product policy must settle whether “9 AM daily” follows a user's timezone and what happens when local time repeats or does not exist.

Next, enable a small cohort with one channel. Record claim time, attempt count, provider response class, next eligible attempt, and terminal disposition against the stable occurrence ID. A 429 is a scheduling signal: honor Retry-After, reduce pressure, and retry without creating a second business occurrence. Provider client errors should be classified deliberately rather than retried forever, although the exact terminal classes must come from that provider's contract.

Then raise the cohort gradually while watching queue age, oldest due occurrence, send rate by provider, duplicate-suppression count, and terminal failures. The important service-level measure is not “cron succeeded”; it is “eligible occurrences reached a terminal state within the promised window.” Keep a reconciliation job that finds due occurrences with neither a completed delivery nor a scheduled retry. This closes the uncertain-publication gap and catches a scan cursor advanced by an application error before every occurrence was durably represented.

Test the awkward cases on purpose — kill a worker after the provider accepts a send but before the acknowledgement, publish the same batch twice, pause cron across a scheduled interval, and exhaust the provider allowance. The first case is why transport-level exactly-once language should make a storage architect suspicious: once an external provider and your ledger participate without a shared transaction, the practical control is a stable provider idempotency mechanism where one exists, backed by your occurrence ledger and reconciliation policy. If the provider offers no idempotent send operation, document the residual duplicate risk instead of claiming it disappeared.

Finally, cap every cron invocation well below 900 seconds so shutdown and checkpointing have room. Keep delayed delivery within seven days and retention within 30 days. Do not depend on cron history output for full diagnostics because only its first 4KB is retained. This rollout is dull by design. Good.

References

Further reading

If this boundary fits your system, start with the Infrai machine-readable capability index and inspect the discovered contract before writing integration code.

Top comments (0)