TL;DR: In Node.js, make the nightly cron trigger enqueue a small, immutable reconciliation manifest, then let a scheduled queue worker claim bounded pages. Do not make one process run all long-running background jobs. The recovery unit should be one provider-account page, not the whole night, and completion should be derived from durable page states rather than from a process surviving longer than the 15-minute limit.
That decision separates three clocks that are often confused: when work becomes eligible, how long one worker may run, and how long operators retain evidence. It also keeps the observability bill tied to useful recovery facts. A queue can absorb a long run; a single invocation cannot.
How should cron trigger a scheduled Node.js queue worker?
Start with the business boundary. For an edtech platform, a nightly reconciliation compares internal payment records with a payment provider's records for a closed interval. The interval must be explicit, such as [window_start, window_end), because “last night” changes meaning across time zones, retries, and daylight-saving transitions. The manifest stores that interval, the provider account, a schema version, and a stable run key.
Four invariants carry most of the design:
- A schedule tick may create or discover one manifest for a run key; repeating the tick must not create a second logical run.
- Every page has a stable identity derived from the run key and its cursor or partition. A retry may repeat a read, but it cannot silently create a new page identity.
- Writing reconciliation results and advancing page state must have an explicit consistency boundary. If they cannot share one transaction, the result write needs an idempotency key and the page remains retryable until both facts are durable.
- A run is complete only when no page is pending or leased and every discovered continuation has been recorded. “The queue looked empty” is not a completion condition.
The failure boundary follows from those invariants. The scheduler owns manifest creation and enqueue intent. A worker owns one page lease. The finalizer owns the transition from fully settled pages to a closed run. None of them needs to remain alive for the duration of the full reconciliation.
Use a lease with an expiry rather than a permanent processing flag. If a worker disappears, another worker can claim the page after the lease expires. The lease duration should exceed the ordinary page-processing time but remain short enough to meet the recovery objective. Extend it only while useful work is progressing.
Keep that boundary small.
Architecture decision and alternatives
The primary axis is operational recovery, so the comparison is about what an operator can safely replay at 03:00, not which option has the shortest setup guide.
| Option | Failure boundary | Replay unit | Long-run behavior | Evidence cost | Decision |
|---|---|---|---|---|---|
| Scheduler performs the full scan | Entire invocation | Whole interval | Bound to one runtime window | Low event count, poor diagnosis | Reject for this job |
| Scheduler starts one resumable process | Process checkpoint | Cursor checkpoint | Can cross invocations if checkpoints are correct | Moderate | Valid for strictly serial sources |
| Scheduler creates a manifest; workers process pages | One leased page | Stable page identity | Total run may outlive any worker | More state, controlled labels | Adopt |
The adopted shape adds bookkeeping. That cost buys a narrower retry boundary and a legible recovery procedure: find expired leases, requeue those page identities, and finalize only after all pages settle. It also supports deliberate concurrency limits when the provider enforces request quotas.
The limitation is real: a manifest and queue introduce more durable states, leases, and recovery tooling than a single cron callback. This alternative is unsuitable for a tiny cleanup job or report with a proven, tightly bounded runtime; in that case, one scheduled invocation is easier to operate. It is also a poor fit when the upstream source permits only one strictly serial cursor, unless the queue enforces a single active page for that account. Choose the extra state only when partial replay and long-running recovery justify it.
Ordering deserves care. A reconciliation usually needs deterministic coverage, not global execution order. If pages are independent, forcing one total order reduces recovery throughput without improving correctness. If a provider cursor is strictly serial, store the continuation before acknowledging the current page and keep that account on one logical chain. FIFO queue semantics can help preserve order and suppress some duplicate sends, but application-level idempotency is still the safer business boundary because a queue cannot decide whether a payment result was already committed.
The critical path as an HTTP contract
The following calls illustrate a generic control plane. They are deliberately small: the schedule creates a manifest, workers claim bounded work, and results are committed with stable keys. Authentication and transport retry policy belong outside the example.
curl --fail-with-body --retry 3 \
--request PUT \
--header 'Content-Type: application/json' \
--header 'Idempotency-Key: reconcile-acct_42-2026-10-11' \
--data '{"account_id":"acct_42","window_start":"2026-10-11T00:00:00Z","window_end":"2026-10-12T00:00:00Z","schema_version":3}' \
https://jobs.example.net/reconciliation-runs/reconcile-acct_42-2026-10-11
curl --fail-with-body --retry 3 \
--request POST \
--header 'Content-Type: application/json' \
--data '{"worker_id":"worker-07","lease_seconds":120}' \
https://jobs.example.net/reconciliation-pages/claim
curl --fail-with-body --retry 3 \
--request PUT \
--header 'Content-Type: application/json' \
--header 'Idempotency-Key: reconcile-acct_42-2026-10-11-page-000184' \
--data '{"run_key":"reconcile-acct_42-2026-10-11","page_id":"page-000184","lease_token":"opaque-token","status":"succeeded","matched":247,"missing_internal":2,"missing_provider":1}' \
https://jobs.example.net/reconciliation-pages/page-000184/result
These are interface examples, not prescribed public routes. PUT makes the stable identities visible, while the idempotency keys express the replay contract. A real implementation must reject a stale lease token, validate that the interval cannot mutate after creation, and atomically record any continuation needed to discover the next page.
Keep the worker deadline smaller than the hosting limit. For a 15-minute ceiling, a worker that begins new provider I/O at minute 14 has created an ambiguous handoff. A 10-minute work budget, followed by checkpoint and release time, is an illustrative policy rather than a universal constant. Measure page latency, then choose the budget from its upper tail and the time needed for a clean lease handoff.
One trap is subtle: acknowledging the queue message before the result commit turns a worker crash into missing work. Committing a result and then failing before acknowledgment produces a duplicate delivery instead. Prefer the second failure mode, provided the commit is idempotent. Duplicate computation is visible and recoverable; an unrecorded gap is harder to prove absent.
Make duplicates boring.
Count telemetry as retained bytes, not reassuring events
Recovery needs enough evidence to answer three questions: which run owns this page, who currently holds its lease, and what durable outcome was recorded? Everything else must justify its storage. Logging every matched payment identifier makes the common path expensive and increases exposure. Store compact counts for successful pages, and place discrepancy details in a restricted result record with a retention policy matched to the audit requirement.
Cardinality grows multiplicatively. Suppose an illustrative deployment has 600 provider accounts, 200 pages per nightly run, three attempts per page in a bad night, and four per-page state events. That is 600 × 200 × 3 × 4 = 1,440,000 events before request traces. If each retained event averages 900 bytes after indexing and metadata, one troubled run is about 1.30 GB in decimal units. Thirty days of similarly verbose retention would be about 38.9 GB. These are planning inputs, not benchmark results; measure the encoded event size in the actual telemetry pipeline.
Labels are often worse than volume. status, operation, and a coarse provider class have bounded value. Payment ID, page cursor, lease token, and raw error text are high-cardinality fields and should not become metric labels. Keep identifiers in logs or traces only when they support a concrete lookup, and never log secrets or payment data merely to make a dashboard convenient.
Sampling must respect failure asymmetry. Successful page spans can be sampled after aggregate counters prove coverage. Failed commits, expired leases, manifest conflicts, and finalization refusals should be retained because they define the recovery queue. Even there, repeated identical errors can be aggregated by a stable, low-cardinality error class. The question is not “Can this event be collected?” It is “Which recovery decision becomes impossible if it expires?”
A practical retention model has three tiers: short-lived attempt detail for debugging, longer-lived run and page outcomes for operational replay, and the minimum business reconciliation evidence required by policy. Assign byte budgets to each tier. Review actual ingestion and indexed size after a representative run, because compression and indexing make source-line length a poor proxy for stored cost.
Keep less, on purpose.
Why reject one large scheduled invocation?
A monolithic invocation has a valid use case: small, bounded work whose worst-case duration remains comfortably below the execution limit and whose safe replay unit is the entire operation. A nightly cache refresh over a fixed list may fit. The approach is attractive because it has fewer moving parts and fewer state transitions to observe.
Payment reconciliation does not have those properties. Provider latency, pagination depth, throttling, and discrepancy handling can expand the run. Retrying the whole interval repeats already completed reads, while checkpointing inside the monolith gradually recreates a queue without explicit leases or page states. Once operational recovery requires answering “what remains?”, represent that answer directly.
The resumable single-process option remains reasonable when the upstream API exposes a strictly serial cursor and parallel work would violate its contract. Persist the cursor with the processed result, enforce one active lease per account, and relaunch from the last committed boundary. It trades throughput for a smaller state model. This is a constraint-driven choice, not a default.
The final decision rule is concise: schedule eligibility, persist coverage, lease bounded pages, and make result commits replayable. Then retain only the telemetry that proves those transitions. A runtime ceiling stops being the architecture's timer; it becomes one worker's local deadline.
Top comments (0)