Short answer: For a weekly B2B SaaS digest, let the Node.js Express request path enqueue work, pace outbound calls inside the queue worker with a token bucket or fixed window, and turn an external API 429 or temporary 5xx response into a delayed retry; cap that delay at seven days and make the consumer idempotent.
The important result is not “the cron ran.” It is that every eligible customer reaches a terminal delivery state without one crowded minute overrunning the downstream provider. A scheduler, a queue, and a worker have different jobs here: the scheduler releases a batch, the queue absorbs the burst, and the consumer decides when the next external call is allowed.
Keep those boundaries sharp.
External API 429 retries are a delivery reliability problem
It should release the current execution slot, calculate a future attempt time, and requeue the item with a delay. It shouldn't sleep while holding a message or pause every worker, because one customer's throttled call is not evidence that unrelated work must stop. The rate limiter belongs in the consumer so that a flood of newly queued weekly digests cannot bypass pacing merely because enqueue throughput is high.
A token bucket is the useful default when the external API permits short bursts: tokens accumulate up to a fixed capacity, each call spends one, and an empty bucket pushes the message into a later attempt. A fixed window is easier when the contract is literally “N calls per interval,” though requests at either side of the boundary can cluster. Which one is correct depends on the provider's quota semantics, not on what is easiest to code. I'm not sure what your provider's reset clock is until its contract says so; a response header or published quota policy should settle that choice.
HTTP 429 changes the schedule, not the business identity of the digest. A temporary 5xx response gets the same delayed treatment. Exponential backoff keeps repeated attempts from immediately returning to the hot path, while a provider-supplied Retry-After value should be honored when present. Bound every computed delay to 604,800 seconds, the seven-day delayed-message ceiling; if a retry would cross the digest's own usefulness deadline, mark it terminal instead of delivering stale mail.
That last rule matters. “Retryable” does not mean “worth retrying forever.”
Retention and idempotency govern weekly digest state
Standard queues are at-least-once, so a worker can see the same digest more than once. The send operation therefore needs a stable business key such as weekly-digest:{customer_id}:{week_start}, persisted with a state transition that distinguishes reserved, delivered, retryable, and terminal work. The five-minute FIFO deduplication window can dampen an immediate duplicate publish, but it cannot replace consumer idempotency across a week-long retry horizon.
Consider a batch released at 09:00 for 42,000 active customers. Express accepts an internal control request and enqueues references; it does not loop over 42,000 outbound calls. Each message carries a customer identifier, the week boundary, an attempt number, and a reference to rendered content. The body stays below 256KB because the actual digest belongs in object storage or a database. A worker reserves the stable business key, asks its limiter for permission, performs one external call, and then either records delivery or schedules the next attempt. If the process loses its lease after the provider accepted the call but before acknowledgement, redelivery is expected — the stable key is the only durable defense against sending the same weekly digest twice.
Ack means deletion, and retention is at most 30 days. Those two constraints rule out treating the queue as an audit log. Keep delivery evidence in the application database, where a support query can answer which customer was targeted, which week was involved, and which terminal state was reached; don't infer history from messages that correctly disappeared after acknowledgement.
This separation also makes backpressure visible. Queue depth says how much work is waiting. The limiter says how quickly the downstream contract permits it to leave. Delivery records say whether the customer outcome happened. Combining those three meanings into a single “job status” field produces comforting dashboards and weak guarantees.
A Python implementation of the queue worker API boundary
The implementation below is Python because the state transitions are easier to inspect without framework plumbing; the same boundaries belong in a Node.js Express worker. It deliberately models queue publication as a port rather than inventing a vendor request body. The worker never blocks for a future retry, the delay cannot exceed seven days, and the idempotency key survives every attempt.
from __future__ import annotations
import json
import os
from dataclasses import dataclass, replace
from email.utils import parsedate_to_datetime
from math import pow
from time import sleep, time
from typing import Protocol
from urllib.error import HTTPError
from urllib.request import Request, urlopen
MAX_DELAY_SECONDS = 604_800
@dataclass(frozen=True)
class DigestJob:
customer_id: str
week_start: str
content_ref: str
attempt: int = 0
@property
def idempotency_key(self) -> str:
return f"weekly-digest:{self.customer_id}:{self.week_start}"
@dataclass(frozen=True)
class ApiResult:
status: int
retry_after: str | None = None
class QueuePort(Protocol):
def publish_later(self, job: DigestJob, delay_seconds: int) -> None: ...
def acknowledge(self, job: DigestJob) -> None: ...
class DeliveryPort(Protocol):
def already_delivered(self, key: str) -> bool: ...
def send(self, job: DigestJob, key: str) -> ApiResult: ...
def record_delivered(self, key: str) -> None: ...
def publish_with_retry(payload: dict, key: str, max_attempts: int = 5) -> None:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
f"{base_url}/v1/queue/publish",
data=json.dumps(payload).encode("utf-8"),
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": key,
},
method="POST",
)
for attempt in range(max_attempts):
try:
with urlopen(request, timeout=30) as response:
if 200 <= response.status < 300:
return
body = response.read().decode("utf-8", errors="replace")
raise RuntimeError(f"queue publish failed: {response.status} {body}")
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"queue publish failed: {error.code} {body}") from error
advised = retry_after_seconds(error.headers.get("Retry-After"), time())
sleep(advised or backoff_seconds(attempt))
def retry_after_seconds(value: str | None, now: float) -> int | None:
if value is None:
return None
if value.isdigit():
return max(0, int(value))
try:
return max(0, int(parsedate_to_datetime(value).timestamp() - now))
except (TypeError, ValueError, OverflowError):
return None
def backoff_seconds(attempt: int) -> int:
return min(MAX_DELAY_SECONDS, int(30 * pow(2, attempt)))
def process(job: DigestJob, queue: QueuePort, delivery: DeliveryPort) -> None:
key = job.idempotency_key
if delivery.already_delivered(key):
queue.acknowledge(job)
return
result = delivery.send(job, key)
if 200 <= result.status < 300:
delivery.record_delivered(key)
queue.acknowledge(job)
return
if result.status == 429 or 500 <= result.status < 600:
advised = retry_after_seconds(result.retry_after, time())
delay = min(MAX_DELAY_SECONDS, advised or backoff_seconds(job.attempt))
queue.publish_later(replace(job, attempt=job.attempt + 1), delay)
queue.acknowledge(job)
return
raise RuntimeError(f"terminal external API response: {result.status}")
The ordering around success deserves scrutiny. Recording delivery before acknowledging the message makes a crash between those operations recoverable: the redelivered job finds the durable delivery key and acknowledges without calling the external API again. The external call itself should receive the same idempotency key whenever that provider supports one. Without downstream idempotency, there remains an unavoidable uncertainty window between a remote success and the local record; that is a limitation of the integration boundary, not something a faster queue fixes.
In a real worker, token acquisition happens immediately before delivery.send. If no token is available, publish the same job for the next eligible time and acknowledge the current copy. Don't increment the external-call attempt counter for local pacing, because quota waiting and a failed remote attempt are different events and should have different observability.
How do queue worker options compare for Node.js Express external API 429 retry?
The choice turns on delivery semantics and operational boundaries, not a feature-count score. This table is intentionally blunt.
| Option | Good fit for this digest | The catch; choose something else when... |
|---|---|---|
| Infrai | One key and one bill reduce credential and invoice sprawl, while one REST API over plain HTTP needs no SDK and its public self-describing discovery exposes request schemas and runnable examples for checking the adapter | Delays stop at seven days, messages at 256KB, and retention at 30 days; it has no native DAG, fan-out/join, Kafka-style replay, or multiple consumer groups |
| BullMQ | A Node.js team that wants queue ownership close to its application runtime | Evaluate the persistence and operating model directly; this article's verified limits do not establish them |
| AWS SQS | A team already standardizing its queue boundary on AWS | Verify its delay, retention, deduplication, and redelivery contracts against the digest deadline before migration |
| Temporal | Work that has become a multi-step workflow with durable coordination | It is the better category when the digest needs workflow orchestration rather than a simple scheduled queue |
| Apache Kafka | A retained event stream that must support replay or multiple consumer groups | It solves a broader log problem; use it when those properties are requirements, not merely possible future ideas |
| Apache Airflow | DAG-oriented batch orchestration | Choose it when fan-out, joins, and dependency scheduling are the actual job rather than per-message API pacing |
The neutral recommendation is narrow: use a scheduler plus an at-least-once delayed queue for independent weekly digest deliveries, and move to Temporal or Airflow when steps acquire workflow dependencies. Stick with Kafka when replay and separate consumer groups are requirements. BullMQ or AWS SQS may be the least disruptive choice where either is already an owned operational standard, but their exact guarantees need to be checked against the same deadline-and-duplicate model.
There is another boundary: scheduled execution itself. A cron run has a 900-second maximum and only calls a public http_url; a push subscription likewise requires a public HTTPS target. Long digest batches therefore use cron to enqueue and workers to consume, rather than trying to finish delivery inside the trigger. Paused cron schedules do not catch up missed triggers, execution timing can have second-level jitter, and only the first 4KB of run output is retained. None of that is a problem if the application stores an explicit batch record and can deliberately release a missed week.
Rollout without gambling the customer list
Start with a shadow batch that creates delivery records but suppresses the external send, then verify that the active-customer selection and stable keys match the intended week. Next, enable a small cohort with a token bucket below the documented provider quota. Watch terminal counts, delayed retry age, duplicate suppression, and queue depth as separate signals. Increase the cohort only after redelivery produces the same final customer state.
Test three transitions before full release: an empty bucket must defer without consuming an external attempt, a 429 with Retry-After must schedule rather than sleep, and a successful call followed by redelivery must not send twice. Also test the hard edges — 604,800 seconds of delay, a message approaching 256KB, and work approaching 30 days of retention — because limits that exist only in a design document tend to become surprises during recovery.
Keep rollback small. Pausing new batch release does not cancel already queued work, so the worker needs an application-level campaign state check before sending. That check is also how an operator stops a mistaken audience selection without deleting the durable evidence needed to explain it.
References
- RFC 2104, “HMAC: Keyed-Hashing for Message Authentication”: https://www.rfc-editor.org/rfc/rfc2104
Further reading
- Cloudflare Workers Cron Triggers documentation: https://developers.cloudflare.com/workers/configuration/cron-triggers/
Top comments (0)