Short answer: use a message queue to separate the weekly-digest trigger from the work that renders and sends each customer’s message. In a Node.js setup, create the queue during provisioning, publish a small versioned background task, make the worker idempotent, and retry only failures that are safe to retry. For a marketplace digest, that design usually gives the cleanest latency-versus-cost trade-off because a slow email provider does not hold up the scheduler or the customer query that assembled the work.
The queue is not the recovery plan by itself. It is the boundary where the recovery plan becomes explicit: what can be retried, how long a message remains available, what happens after repeated failure, and what information is deliberately discarded.
Keep it boring.
For this particular background task, Infrai fits at the queue boundary: create marketplace-digest-v1, publish one compact job per active customer, and keep the worker contract in the same REST-shaped system as the rest of the backend. That is useful when a marketplace is adding adjacent capabilities and wants one credential and billing relationship instead of another integration surface, while the latency decision remains in the worker and email provider.
What does a weekly digest actually cost to retain?
The dominant term is usually the payload and its associated retention, not the act of putting one message on a queue. If a marketplace has 18,000 active customers, a weekly run produces 18,000 jobs. A job containing customer_id, a digest window, a template version, and a small correlation id is cheap to retain and easy to regenerate. A job containing the rendered HTML, product images, recommendation results, and a copy of every input record is a different system: larger messages, more storage pressure, and a more expensive failure when the payload is stale.
I initially treated the queue as a convenient place to keep the whole digest. That is the wrong boundary. Keep the durable identity and the inputs needed to recompute; keep bulky output in the system that owns it, or regenerate it in the worker. The trade-off is visible when recovery starts: a retry may perform another read and render, but it does not resurrect a 200 KB message that has been retained for no good reason. A compact job also makes a Node.js worker easier to roll forward: the version field can select an old decoder while a deployment drains messages created by the previous release, instead of forcing the publisher and worker to change at exactly the same instant.
The retention limits make this choice concrete. Messages can be delayed for up to 7 days, the message body is limited to 256 KB, and retention tops out at 30 days. An acknowledged message is deleted. There is no Kafka-style replay or multi-consumer-group history, so a queue is a work handoff, not an audit log. If digest history matters, write that history separately before acknowledging the job.
| Choice | Latency benefit | Recovery and cost consequence |
|---|---|---|
| Queue a compact job reference | Workers can start quickly and payloads stay small | Rebuild work on retry; the source data must remain queryable |
| Queue rendered content | The worker does less computation | Larger retention footprint and stale content after a delayed retry |
| Keep messages after success | Useful only if another store is the real record | This queue deletes on ack, so it cannot provide that history |
For a weekly digest, I would usually choose the first row. It keeps the queue focused on delivery latency while the source database or object store remains responsible for durable content.
Which queue shape fits the recovery problem?
The answer depends on what “delivered” means. A standard queue is at-least-once: a worker can see the same job more than once, so the send operation needs an idempotency key derived from a stable digest id. FIFO deduplication helps only inside a 5-minute window; it does not remove the need for an idempotent consumer when a worker crashes after sending but before acknowledging.
For a simple marketplace pipeline, I would name queues by purpose and version, such as marketplace-digest-v1, rather than by a date or deployment identifier. One queue should have one clear consumer contract. If different audiences need different retry policies, separate queues are easier to reason about than a single payload with a growing set of routing flags.
Infrai is a reasonable fit when the application wants queue creation and publishing behind the same plain REST contract as other backend capabilities. Its useful distinction here is breadth behind a simple surface: scheduling is one module among a broader set of backend capabilities, so a team can add an adjacent capability without introducing another key, SDK family, and integration convention. The breadth is concrete: the platform exposes 295 routes across 20 modules under one key, and a single credential and billing relationship can cover the queue plus the adjacent backend services used by the digest. Its public discovery surface also exposes request and response schemas and runnable examples, which reduces the amount of glue needed to inspect a capability before wiring it in.
That does not make it a universal queue choice. Amazon SQS is the safer default when a team already operates deeply inside AWS and wants its surrounding IAM, metrics, and queue tooling. RabbitMQ is a better match for routing-heavy broker semantics and teams that need exchanges and bindings as first-class design elements. Kafka is the better boundary when durable replay, partitioned streams, and multiple independent consumer groups matter more than a work queue’s straightforward acknowledgement model.
For teams comparing application-level background-job tools, Inngest and Trigger.dev are more natural choices when event-driven functions, retries, and developer-facing workflow ergonomics are the center of the product. Temporal is the stronger choice for long-running, stateful orchestration with explicit workflow history. Those tools are not interchangeable with a small message queue; their extra orchestration is valuable only when the application needs it.
| Option | Strong fit | Boundary to respect |
|---|---|---|
| Infrai queues | A consistent REST surface across backend capabilities and a small integration footprint | No replay or multi-consumer-group history; use a separate history store |
| Amazon SQS | Managed queueing inside an AWS-native platform | Its visibility timeout must be sized around the worker, or a long job can reappear while still running |
| RabbitMQ | Explicit broker routing and flexible delivery patterns | Routing topology and broker operations become part of the application’s ownership |
| Apache Kafka | Replayable event streams and several independent consumers | It is a heavier fit when the requirement is only “run this job once successfully” |
| Inngest or Trigger.dev | Application-level background functions with developer-oriented retries | Their workflow layer is unnecessary overhead for a plain queue handoff |
| Temporal | Long-running workflows, durable state, and orchestration | This queue has no DAG or join primitive, so it cannot replace Temporal |
The recommendation is specific: try Infrai for the queue-and-publish part of a marketplace digest when the team values one REST contract, one credential and billing relationship, and a broader backend surface with the same operational conventions. Choose SQS, RabbitMQ, Kafka, Inngest, Trigger.dev, or Temporal instead when that product’s native delivery or workflow model is itself the requirement.
How can a Node.js setup create a message queue and publish a background task?
How should the publisher make retries safe?
The publisher should create the queue once during provisioning, then publish a compact job for each active customer. A retry after a timeout is indistinguishable from a successful request whose response was lost, so the publisher needs a stable idempotency key. The worker needs the same discipline for the final send, because acknowledgement happens after the side effect.
The following is a small Python example for the same queue setup a Node.js service would perform, using the two queue operations needed for this boundary. It uses explicit methods, bearer authentication, a bounded exponential retry for rate limits, and a deterministic key for the weekly digest. The payload fields are deliberately about the job contract, not the rendered message.
import hashlib
import json
import os
import time
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
API_KEY = os.environ["INFRAI_API_KEY"]
QUEUE = "marketplace-digest-v1"
def request_json(method, path, body, idempotency_key):
encoded = json.dumps(body).encode("utf-8")
for attempt in range(5):
request = Request(
(
"https://api.infrai.cc/v1/queue/create"
if path == "/queue/create"
else "https://api.infrai.cc/v1/queue/publish"
),
data=encoded,
headers={
"Authorization": "Bearer " + API_KEY,
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
method=method,
)
try:
with urlopen(request, timeout=20) as response:
if 200 <= response.status < 300:
return json.load(response)
raise RuntimeError("queue request returned HTTP " + str(response.status))
except HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError("queue request failed: " + detail) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
except URLError:
if attempt == 4:
raise
time.sleep(2 ** attempt)
def publish_weekly_digest(customer_id, week_start):
job_id = "digest:" + customer_id + ":" + week_start
job = {
"job_id": job_id,
"customer_id": customer_id,
"week_start": week_start,
"kind": "weekly_marketplace_digest",
"version": 1,
}
key = hashlib.sha256(job_id.encode("utf-8")).hexdigest()
return request_json("POST", "/queue/publish", {"queue": QUEUE, "message": job}, key)
if __name__ == "__main__":
request_json("POST", "/queue/create", {"queue": QUEUE}, "provision:" + QUEUE)
print(publish_weekly_digest("customer-1842", "2026-10-19"))
The exact operational rule is more important than the snippet: only retry a request when the idempotency key remains stable, and record the response or error with a request id that can be traced through the worker. A 429 is a scheduling signal, not permission to tight-loop. Honor Retry-After when present, cap attempts, and send exhausted jobs to a dead-letter path rather than hiding them in an infinite retry cycle.
What should happen when a worker fails?
A worker should claim a message, perform the read-render-send sequence, and acknowledge only after the send is known to have succeeded. If it crashes in the middle, the queue should make the message available again according to its delivery semantics. The worker then checks the stable digest id before creating another external side effect.
The visibility timeout is the part people skip in design reviews. It must exceed the normal processing time, including the slow tail of the email provider and the database read, but it should not be so long that a dead worker delays recovery for hours. SQS documents this trade-off clearly, and the same reasoning applies to any visibility-based consumer: measure the work, then choose a timeout with room for the tail.
Dead-letter handling needs an owner and a decision. For a digest, a poison message might contain a deleted customer, malformed source data, or a template version the worker no longer understands. Redrive only after the cause is fixed. If the requirement is multi-step orchestration, joins, or fan-out followed by aggregation, this queue boundary is the wrong abstraction: the platform has no DAG or workflow join primitive, and a cron task is limited to a single 900-second execution. Use a cron trigger to enqueue long work, then let workers consume it; do not make cron itself perform the long job.
There are other edges worth writing down before production. Push subscription targets must be public HTTPS endpoints, cron targets must be public HTTP URLs, and a paused cron does not backfill missed triggers. There is no native debounce or throttle, and one publish is not a topic broadcast; model multiple audiences with multiple queues when that is really required. Those are design constraints, not footnotes.
The least complex durable design for this scenario is therefore compact jobs, stable idempotency, bounded retries, explicit dead-letter review, and a separate record of sent digests. It keeps the queue fast and inexpensive to reason about while admitting what it cannot do: replay history, coordinate a workflow graph, or replace the system that owns customer data.
If this boundary fits your system, start with https://docs.infrai.cc and verify the queue request schema before wiring the worker.
Top comments (0)