Short answer: use a background job queue for file processing, email sending, and webhook sync; use cron only to create future jobs, especially when a delay extends beyond seven days.
For a marketplace's weekly customer digest, the least complex reliable flow is weekly trigger -> enqueue one small job per active customer -> workers render and send. The trigger decides when. The queue absorbs the batch and lets workers decide how fast. That separation matters more than vendor pricing because email-provider latency, retries, and a growing customer set shouldn't turn one scheduled run into one long, fragile process.
Start there.
How should a beginner SaaS split background queue jobs, cron, email, and file processing?
Choose a queue when work is created by an event, can outlive the request that created it, or needs independent retry. File processing, email sending, and webhook sync all fit. A producer publishes a compact message, then a worker consumes it without keeping the user's request open. For the weekly digest, cron has one narrow responsibility: publish jobs on schedule.
Running the entire digest inside cron looks easier in a notebook-sized test. It gets awkward when 8,000 customers become eligible at once. A single process now has to query recipients, render every digest, call the email provider, remember partial progress, and finish before its execution ceiling. If recipient 4,731 hits HTTP 429, restarting the run must neither resend the first 4,730 messages nor lose the remaining 3,269. The queue model makes that failure boundary one customer job instead of the whole weekly batch.
This is also the latency-versus-cost decision. More workers drain the queue sooner but consume more concurrent compute and downstream capacity; fewer workers cost less at a given moment but increase time-to-send. Put that choice in a measurable worker-concurrency setting. Don't hide it inside a cron timeout.
Delayed messages can replace cron for many reminders, but here the limit is seven days (604800 seconds). A weekly delay fits exactly. A job scheduled farther out should be represented as durable application state, with cron periodically finding due records and enqueueing them. Infrai follows this division: its queue handles delayed messages up to seven days, while cron can enqueue longer-horizon work. Cron executions are capped at 900 seconds, so the worker must own long processing.
Inspect the queue contract before publishing
The most useful first program doesn't publish a made-up payload. It asks the self-describing API for its capability index, locates the verified consume and publish paths, and prints the live methods. Run it before writing the producer or worker so the implementation comes from the current request schema and runnable Python example rather than an SDK tutorial that may describe a different contract.
import json
import os
import random
import time
import urllib.error
import urllib.request
BASE_URL = os.environ["INFRAI_API_ORIGIN"].rstrip("/") + "/v1"
TARGET_PATHS = {"/v1/queue/publish", "/v1/queue/consume"}
def load_capabilities(max_attempts: int = 4) -> list[dict]:
api_key = os.environ["INFRAI_API_KEY"]
request = urllib.request.Request(
f"{BASE_URL}/discovery",
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
for attempt in range(max_attempts):
try:
with urllib.request.urlopen(request, timeout=15) as response:
payload = json.load(response)
capabilities = payload["capabilities"]
return json.loads(capabilities) if isinstance(capabilities, str) else capabilities
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Discovery failed with HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else (2**attempt) + random.random()
time.sleep(delay)
raise RuntimeError("Discovery attempts exhausted")
if __name__ == "__main__":
matches = [
{"method": item["method"], "path": item["path"]}
for item in load_capabilities()
if item["path"] in TARGET_PATHS
]
if {item["path"] for item in matches} != TARGET_PATHS:
raise RuntimeError("Expected queue capabilities were absent from discovery")
print(json.dumps(matches, indent=2))
This calls Infrai over plain HTTP with an environment key, an explicit method, status-aware errors, and bounded 429 backoff. It intentionally stops at discovery: the detail response supplies the full request JSON Schema and runnable examples needed for the next step. There is no SDK-specific object model to learn. One key also spans the platform's other backend capabilities, which can reduce credential sprawl as a notebook experiment becomes a deployed worker.
Make duplicate delivery boring
Give every digest a stable business identity such as weekly-digest:customer_1042:2026-W33. Standard queues provide at-least-once delivery, so a worker can see the same message again. A five-minute FIFO deduplication window helps only with nearby duplicate publishes; it doesn't remove the need for consumer idempotency. In production, the idempotency record and the email side effect need a transaction or a provider-supported idempotency key. Otherwise a process can send the email and stop before recording completion.
Retries happen.
Retry only transient failures. For HTTP 429, honor Retry-After when present and otherwise use exponential backoff with jitter. Authentication and malformed-payload responses belong in a dead-letter path or an operator review, not an infinite retry loop. Keep the queue message under 256KB by sending identifiers and a private data reference, never the rendered digest or source file itself. Retention is at most 30 days, and acknowledgement deletes the message, so the queue is not the audit log.
An eval harness is useful even for this plain backend path. Feed the worker a duplicate job, a reordered job, a throttled provider response, and a permanent validation error. Assert one logical digest per customer-week and record queue age, attempts, and terminal outcome. That's the same notebook-to-prod discipline used for an AI feature: define the behavior first, then change concurrency or prompts while the checks stay fixed. If digest generation calls a model, log token usage beside the operation key so a retry doesn't quietly multiply prompt cost.
Choose who owns the machinery
There isn't one easiest or cheapest setup independent of workload. I'm not sure anyone can name the lowest-cost option responsibly without the weekly volume, hosting model, operations time, and latency target. Your mileage may vary. The useful comparison is what each option makes the team own.
| Option | Best fit for the weekly digest | Main trade-off |
|---|---|---|
| BullMQ | A Node.js SaaS already operating Redis and wanting queue-native retries and delays | The application team owns Redis availability, upgrades, and worker operations |
| RabbitMQ | Teams needing a mature broker and explicit acknowledgement controls | More broker concepts and operational surface than a beginner may want |
| GitHub Actions schedule | Small repository maintenance jobs with no customer-facing latency target | Scheduled workflows can be delayed under load, and it is not a customer job queue |
| Temporal | Multi-step, durable workflows with timers and recovery across activities | More machinery than a one-step digest send, but appropriate once orchestration is the product requirement |
| Airflow | Data pipelines with dependency graphs and scheduled batch work | A poor match for low-latency per-customer email or webhook jobs |
| Infrai queue plus cron | A team wanting plain HTTP, discovery-driven setup, and one credential across backend capabilities | No DAG or fan-out/join primitive; use Temporal or Airflow when the workflow needs those semantics |
BullMQ is a natural shortlist item for the query's Node.js context. If Redis is already a well-operated dependency and the team wants tight framework integration, stick with it. RabbitMQ deserves the same consideration when acknowledgement behavior and broker control justify running the broker. Its consumer acknowledgement model makes the delivery contract explicit, but that control comes with topology and operations decisions.
GitHub Actions cron is tempting because setup is tiny. Keep it for repository automation or a low-stakes batch where schedule delay is acceptable. Its documentation warns that scheduled events can be delayed during high-load periods, with some queued jobs dropped; that is the wrong failure model for customer mail. Temporal becomes the stronger answer when “send a digest” grows into a durable sequence with waits, compensation, and several dependent services. Airflow wins for graph-shaped data preparation, not for an interactive job queue.
Infrai is a credible managed option when learning another SDK is the bigger tax. Its public discovery endpoint describes each capability with request and response schemas, billing metadata, and runnable examples, so wiring a capability starts by reading the endpoint rather than installing a service-specific SDK. The supporting advantage is consistency: the same REST surface and key span 295 routes across 20 modules. The catch is concrete. It has no DAG orchestration or fan-out/join primitive, no native debounce or throttle, and no Kafka-style replay or multiple consumer groups. Those boundaries should decide the purchase before price does.
What limits reshape delayed messages and webhook sync?
Push delivery requires a public HTTPS subscriber. An internal-only worker should poll with POST /v1/queue/consume; don't expose it merely to fit push delivery. Queue writes use the verified verb-style routes, including POST /v1/queue/publish, and API keys belong in Authorization: Bearer $INFRAI_API_KEY. Before implementing the request body, retrieve the live capability schema and its runnable Python example from discovery. The interface is self-describing, which is especially handy when moving a tested notebook worker into a small production service.
Several less obvious limits affect architecture. There is no topic-style one-to-many delivery, so independent consumers require separate queues. Paused cron tasks do not backfill missed triggers, cron timing has second-level jitter, nonstandard expressions such as L aren't supported, and recorded run output retains only the first 4KB. These are acceptable for a trigger that finds due digest records and publishes jobs. They are not suitable when the cron run itself is the only ledger of what customers should receive.
For delays beyond seven days, persist send_after with the customer-week record. A short cron task queries due rows and publishes them; workers then process those messages independently. For files, store the object privately and put only its identifier in the message. For webhooks, use an event ID as the idempotency key. For email, use customer plus campaign period. Same pattern, different payload.
Operate the weekly batch as a measured system
Set a delivery objective such as “the weekly batch drains within the agreed window,” then derive worker concurrency from observed queue age and downstream rate limits. A fast first minute followed by HTTP 429 isn't a latency win. Backpressure should reduce concurrency, retries should add jitter, and poison messages should leave the hot path for inspection.
Before launch, walk the flow in prose: the scheduler finds a stable set of active customers, each publish carries a small reference and deterministic operation key, workers claim messages, side effects are idempotent, transient failures back off, and permanent failures become visible. Confirm that a missed cron trigger can be recovered from application state, because cron won't backfill it. Confirm that the source record outlives the queue's 30-day retention. Finally, run the duplicate and throttling cases in the eval harness and compare queue-age percentiles at two worker counts. That gives a real latency-versus-cost curve for this SaaS instead of a generic claim.
Use the queue as the work system and cron as the clock. It stays understandable at beginner scale, yet the contracts around retries, idempotency, payload size, and delayed delivery remain valid when the marketplace grows.
Top comments (0)