A cron process should create one durable scheduling record and exit. A worker queue should own report generation, subscriber fan-out, retries, and delivery state. That boundary keeps a slow daily shipment report from turning a timer into an accidental job runner, while still leaving room to choose latency or cost deliberately.
Short answer: run the scheduler at a fixed time, claim the day with an idempotency key, enqueue one report job, and let workers split the resulting shipment update into subscriber-sized deliveries. The scheduler needs a small timeout. The report worker needs a lease, bounded concurrency, durable progress, and a clear rule for permanent failure.
The concrete case here is a fintech operation that sends a daily shipment-status report by email to many subscribers. Report generation may scan a large event range; the final update may fan out to thousands of recipients. The report is one logical job, but delivery is many independently retryable jobs. Treating those as one unit is the first design mistake.
How should I combine a cron job trigger with a worker-based queue?
The clock is not a guarantee that work happened. It is only a request to start work. This combining of a cron trigger with worker-based execution is useful because the two components have different failure boundaries. I define these invariants before choosing a queue library or storage engine:
Keep it boring.
The queue is the boundary.
- At most one report run is active for a tenant and report date, unless an operator explicitly starts a replacement.
- A worker may execute a job more than once, but a subscriber must not receive an unbounded number of copies because of a retry.
- A generated report is retained long enough for delivery retries, and its status is observable without reading worker logs.
- One slow subscriber cannot hold the report-generation lease forever.
- A failure has a category: retryable, permanent, or awaiting operator action.
The identifier should be derived from stable business data, such as tenant_id + report_date + report_type, rather than from the scheduler attempt. Otherwise two timer invocations create two valid-looking reports. A uniqueness constraint at the persistence boundary is more convincing than a promise that the cron process will never overlap.
Here is the decision record I use for the critical path:
| Boundary | Option | Latency benefit | Cost or failure trade-off |
|---|---|---|---|
| Timer to queue | enqueue one run key | Low scheduler overhead | Requires durable deduplication |
| Report to delivery | fan out child jobs | Slow recipients are isolated | More queue records and state transitions |
| Retry policy | bounded exponential delay | Absorbs short outages | Delays visibility of a bad dependency |
| Result storage | immutable artifact plus status row | Delivery can resume | Retention and access controls become explicit work |
This is a latency-versus-cost decision, not a contest between cron and queues. More parallel delivery reduces wall-clock time but increases connection pressure, provider calls, and duplicate-delivery exposure.
Why should cron stop before the report is finished?
A timer process has a narrow responsibility: determine the intended run, record it, and publish a message. It should not wait for a database scan, render a large attachment, or send email. A long-lived cron invocation makes timeout semantics ambiguous: did the process die before enqueueing, after enqueueing, or after the work completed but before acknowledging it? Those are long-running tasks, so the trigger should return before the expensive work begins.
A generic Node.js scheduler can hand the durable parts to a queue client like this. The interface is intentionally abstract; the important detail is the stable key and the separation between schedule time and execution time.
from datetime import date
def schedule_daily_report(store, queue, tenant_id: str, run_date: date) -> None:
run_key = f"shipment-report:{tenant_id}:{run_date.isoformat()}"
created = store.insert_run_if_absent(
key=run_key,
tenant_id=tenant_id,
report_date=run_date,
status="queued",
)
if created:
queue.publish(
topic="report.generate",
key=run_key,
payload={
"run_key": run_key,
"tenant_id": tenant_id,
"report_date": run_date.isoformat(),
},
)
The insert and publish operations need a defined relationship. If they cannot share a transaction, use an outbox record in the same transaction as the run row, then have a small publisher relay unsent outbox records. That adds storage and a relay, but it closes the gap where the run exists and the message does not. A queue's acknowledgement alone cannot repair that gap.
How do workers survive retries and partial fan-out?
The report worker claims a run with a lease, reads the required event range, writes an immutable artifact, and records the artifact checksum or version. It then creates delivery jobs keyed by (run_key, subscriber_id). The fan-out step is a state transition, not a loop whose progress lives only in memory.
I use a lease such as 15 minutes only when the report can make progress or renew it; a fixed lease that outlives the worker's real liveness delays recovery. A reaper can return expired work to the queue, but it must not assume that an expired lease means the old worker stopped. The delivery write needs its own idempotency key, and the email boundary needs an explicit duplicate policy because a process can lose its acknowledgement after sending.
def deliver_one(store, mailer, delivery_id: str) -> None:
delivery = store.claim_delivery(delivery_id, lease_seconds=900)
if delivery is None:
return
if delivery.status == "sent":
return
try:
mailer.send(
recipient=delivery.recipient,
artifact_ref=delivery.artifact_ref,
idempotency_key=delivery.idempotency_key,
)
store.mark_sent(delivery_id)
except TemporaryDependencyError as error:
store.release_for_retry(delivery_id, reason=str(error))
except PermanentDeliveryError as error:
store.mark_failed(delivery_id, reason=str(error))
The code cannot make an external email provider transactional with the database. That is a real failure boundary. If the send succeeds and the process crashes before mark_sent, a retry may send again. The practical answer is to use provider-supported idempotency where available, or make the message tolerate duplicates and expose the duplicate risk to operations. Claiming exactly-once delivery without controlling both sides is a misleading label.
Which failure modes deserve a separate alarm?
A queue depth alarm is not enough. A growing queue can mean a report scan is slow, delivery is throttled, or workers are repeatedly receiving poison messages. Those require different actions.
Track schedule lag, time from run creation to artifact completion, fan-out count, retry age, lease expirations, permanent failures, and delivery-provider responses. Keep the run key in every log and metric. The useful question at 09:30 UTC is not “are workers up?” but “which tenant and report date cannot reach a terminal state, and at which boundary?”
Security belongs in the same record of decisions. A report worker that fetches a callback URL, attachment, or remote logo must validate the destination and restrict network access; an attacker should not be able to turn report generation into a server-side request forgery primitive. The OWASP guidance distinguishes allowlisting approaches for trusted destinations from broader controls, and the right choice depends on whether destinations are fixed or user-supplied.
Do not let retries amplify that risk. A permanent validation error should move to a review state, not be retried every minute. Conversely, a transient database connection failure should not be marked permanent merely because the first attempt exceeded a short worker timeout.
When is a single scheduled worker enough?
A queue with separate generation and delivery jobs has a price: more rows, more instrumentation, more deployment coordination, and more operational states. For one tenant, a small report, and a recipient list that fits comfortably inside the execution budget, one scheduled worker can be the better design. It should still persist a run key and an outcome; simplicity does not remove the need for recovery.
Choose the split when report duration is variable, subscribers have uneven delivery behavior, or a missed retry has business consequences. Choose the simpler worker when the work is short, the recipient set is small, and the team can inspect and replay one run without reconstructing hidden progress. My decision rule is to buy parallelism only where it removes a measured bottleneck or isolates a failure boundary. Otherwise it is operational interest on a system that has not earned the complexity.
The durable architecture is therefore modest: a timer creates intent, an outbox protects publication, a worker materializes one report, and child jobs make fan-out independently recoverable. Latency improves through controlled concurrency; cost stays visible through queue volume, storage retention, and provider calls. The design is credible because it names what it cannot guarantee.
Top comments (0)