For a Node.js cron health check that must catch a missed pricing rollout job, record every expected tenant-scoped run and alert when its deadline passes without matching success. The deciding constraint is cost attribution: a generic heartbeat can say that a worker is alive, but it cannot prove that the new customer-support pricing rule reached each intended tenant or explain which retries consumed work.
Short answer: use an external deadline monitor backed by an append-only run ledger. Emit started, succeeded, and failed events with a stable run ID, tenant ID, rule version, attempt number, and cost units. Keep retries idempotent. This catches a job that never starts, a job that stalls, and a job that finishes only part of its audience without treating “the process is up” as evidence of completion.
How should a Node.js cron health check detect a missed job?
A scheduler, a worker, and a pricing-rule write are three separate failure boundaries. The scheduler can miss its trigger. The worker can accept work and then exceed its timeout. The rollout can update 799 of 800 tenants and still return control normally if completion is inferred from process exit rather than checked against the intended audience. That last case matters in customer support: a new rule may change how message, agent-assist, or automation usage is charged. If the rollout has no tenant-level evidence, aggregate infrastructure metrics cannot tell finance which account received which rule, and a retry can quietly double-count work even when the final configuration is correct. The invariant is precise: for every expected (schedule, tenant, rule_version) tuple, there must be exactly one terminal business outcome before its deadline. Attempts may be many; the attributed billable outcome must remain one. No pulse can replace that invariant. Use two clocks. The scheduler creates the expectation before dispatch, using a UTC deadline derived from the schedule. The worker reports progress with wall-clock timestamps, while the monitor decides lateness from its own clock. This avoids allowing a stalled or misconfigured worker to extend its own deadline. It also distinguishes missed-job detection from ordinary uptime monitoring: one asks whether a promised business action occurred, while the other asks whether a process or endpoint responds.
That distinction matters.
Decision record and failure boundaries
The chosen design is an external deadline monitor plus an append-only event ledger. “External” means outside the scheduled worker's process and failure domain; it can still be a free, self-hosted service. A Healthchecks-style dead-man switch is one alternative for simple schedules, but the monitor described here also reconciles tenant outcomes and attempt costs. It periodically finds expectations whose deadlines have passed and that lack a success event, then opens one alert per stable run ID.
| Option | Detects never-started work | Proves per-tenant completion | Supports retry cost attribution | Main boundary |
|---|---|---|---|---|
| In-process timer | No | Only while the process survives | Weak | Dies with the worker |
| Host or container liveness probe | No | No | No | Measures runtime availability |
| External heartbeat with a single success ping | Yes | Only if each tenant has a distinct expectation | Limited | Payload and retention policy |
| External deadline monitor with an event ledger | Yes | Yes | Yes | Ledger availability and cardinality |
This is an accounting decision as much as an alerting decision. Store the expected audience at rollout creation time, not after execution, because the source tenant set may change during a retry. Attribute operational cost units to attempts, but attach the resulting pricing-rule state to the stable run ID. That separation answers two different questions: “What did execution consume?” and “What business change became effective?”
The trade-off is storage and cardinality. This design is not suitable for a low-impact cron job where a single late ping is enough and nobody needs per-tenant reconciliation; a basic dead-man switch is easier to operate there. It is also a poor fit if the team cannot retain stable tenant identifiers under its data policy. In that case, aggregate expectations by an approved non-identifying scope and accept that account-level attribution will be unavailable.
Keep the event vocabulary small. expected establishes obligation, started proves pickup, succeeded proves the postcondition, and failed records a terminal attempt. A retry creates another attempt under the same run ID. It must not create a second obligation.
The critical path in code
The monitoring endpoint does not need to know how Node.js schedules work. It needs a durable contract. The following Python sketch shows the receiver-side rules because all code here is deliberately implementation-neutral; a Node.js worker can send the same JSON fields with its normal HTTP client.
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Literal
EventState = Literal["expected", "started", "succeeded", "failed"]
@dataclass(frozen=True)
class RunEvent:
run_id: str
tenant_id: str
rule_version: str
attempt: int
state: EventState
occurred_at: datetime
deadline_at: datetime
cost_units: int
def accept_event(event: RunEvent, ledger) -> bool:
if event.occurred_at.tzinfo is None or event.deadline_at.tzinfo is None:
raise ValueError("timestamps must include a UTC offset")
if event.attempt < 0 or event.cost_units < 0:
raise ValueError("attempt and cost_units cannot be negative")
event_key = (
event.run_id,
event.tenant_id,
event.attempt,
event.state,
)
return ledger.append_once(event_key, event)
def overdue_expectations(ledger, now: datetime):
assert now.tzinfo is not None
for expected in ledger.expected_before(now):
scope = (
expected.run_id,
expected.tenant_id,
expected.rule_version,
)
if not ledger.has_success(scope):
yield scope
append_once is the idempotency boundary. Put a unique constraint behind that key, and make duplicate delivery return the original acceptance result rather than adding cost again. The success check includes rule_version; otherwise a late success from the previous rollout could close the current alert.
Retries need a budget, not optimism. Retry transient dispatch or transport failure with bounded attempts and jitter, but do not move the original business deadline. Once that deadline passes, page or ticket the responsible team and let remediation use a new attempt under the same run. Permanent validation errors should fail immediately. An invalid rule will not become valid because it slept for 30 seconds.
Postcondition checks should read the effective rule version for the tenant after the write. An HTTP 200 or a resolved promise proves only that one operation returned successfully. It does not, by itself, prove that the intended tenant population converged.
Operating the monitor without noisy alerts
Choose the deadline from the business tolerance, then include the scheduler's maximum expected delay and the worker's execution budget. Avoid learning that threshold from the same job being monitored; a long-running failure can otherwise normalize itself. For a rollout expected at 02:00 UTC with a 20-minute completion objective, the expectation should carry the explicit timestamp used by the monitor. Concrete timestamps also remove daylight-saving ambiguity from the evaluation path.
Alert identity should be (run_id, tenant_id, rule_version), not the attempt number. The first overdue transition opens the alert; later scans update it. A verified success resolves it. This keeps three retry attempts from becoming three pages while retaining all three attempt costs in the ledger.
Watch the watcher. Its own health is measured by the age of the latest completed scan, the number of expectations evaluated, ledger write failures, and alert-delivery failures. A dashboard query over an analytical store can help investigate large histories, but alert correctness should not depend on a slow exploratory query. ClickHouse, for example, documents an analytical database rather than a scheduler contract; storage choice does not define the missed-run semantics.
There is a compliance edge too. Tenant IDs and rule versions are enough for correlation. Do not place message bodies, phone numbers, email addresses, OTPs, or support transcripts in heartbeat payloads. Logs used for operational accounting tend to live longer and reach more people than application request data, so data minimization belongs in the event schema rather than in a cleanup promise.
Why reject a single process heartbeat?
A repeating “worker alive” pulse was rejected because it conflates capacity with completion. It can be green while one tenant's rollout is absent, and it cannot distinguish a cheap first attempt from repeated expensive retries. The signal is attractive because it has low cardinality. That is also why it loses the evidence needed here.
The option still has a valid use case: use a process heartbeat to detect that a worker pool has stopped polling entirely. Pair it with per-run expectations when the business obligation matters. These signals answer different questions and should generate different alerts.
The rollout is complete only when the expected tenant set reconciles to terminal outcomes, the new rule version is verified, and attempt costs can be summed without duplicating the business result. That gives operations a missed-job detector, finance an attribution trail, and support a clear answer when one account sees a different price rule.
Top comments (0)