Short answer: don't treat a process heartbeat as proof that a Node.js cron background job ran; record each scheduled run's start and completion, evaluate that record against a deadline, and page only when the notification outcome requires action.
The page says checkout-notifications missed its 02:10 run. It also says what the on-call needs next: the expected schedule, the last successful completion, the deployment revision, how many delivery attempts were accepted or rejected, and the trace or log correlation key. A green host heartbeat is absent because it wouldn't answer the incident question. The process can be healthy while its scheduler never invokes the callback, and the callback can return while downstream delivery work is still incomplete.
That distinction is the whole design.
Minimize the incident record before storing it
Process liveness and scheduled-work completion answer different questions. The first says a runtime can respond; the second says a specific notification obligation reached a known outcome. An incident responder needs the second answer on the page, with enough evidence to recover the sequence of events, but the record should stop there rather than becoming another store of customer data.
Emit the terminal fact before evaluating silence
An alert should prove either that a known run failed or that an expected run cannot be accounted for. Those are separate states. An explicit failure can be reported as soon as the job reaches a terminal error; a missed run has no error path, so another component must notice that the completion deadline passed without evidence.
How can a Node.js cron background job detect a missed run without heartbeat polling?
For an e-commerce notification service, the evidence should follow the business unit of work. A useful run record contains a stable job name, scheduled time, actual start time, completion time, terminal outcome, deployment revision, and aggregate delivery counts. It should not contain customer email addresses, message bodies, or raw order details. Article 5 of the GDPR establishes data minimization: personal data must be adequate, relevant, and limited to what is necessary. Article 17 also creates erasure obligations. Keeping notification telemetry operational and aggregate reduces the amount of personal data that observability storage has to govern, search, and erase.
The word completion matters. A start-only heartbeat proves that a callback began. It doesn't prove that a queue batch drained, that retry decisions were persisted, or that the job reached its declared terminal state. Conversely, a completion-only event can detect silence but leaves a thin reconstruction trail. Recording both edges lets the responder distinguish "not started" from "started but not finished" without opening five dashboards and guessing from CPU graphs.
Use a state machine small enough to explain during an incident:
| Observed state at deadline | Interpretation | Action |
|---|---|---|
| No start, no completion | Scheduler or invocation gap | Page the job owner |
| Start, no completion | Stalled or over-deadline execution | Page with the run ID and age |
| Completion with failure | Known terminal failure | Page immediately if delivery is affected |
| Completion with success | Run accounted for | No page |
This isn't a generic liveness check. It is an assertion about one scheduled obligation.
Reconstruct the incident from the page backward
Start at 02:19, with the on-call looking at the notification failure page. Suppose the job is scheduled for 02:10, its declared maximum useful runtime is six minutes, and the team allows three minutes for scheduler jitter and event ingestion. Those values are an example policy, not universal constants; your mileage may vary, and queue depth plus the delivery provider's contract should determine them. At 02:19, the evaluator asks for evidence attached to the 02:10 schedule slot.
If the ledger shows a 02:10 start and no terminal event, the page reports an overdue execution. If it shows neither event, the page reports a missing invocation. If it shows a terminal failure at 02:12, there is no reason to wait for the deadline: the explicit failure path should have paged already. This gives the responder a timeline rather than a symptom. The most useful payload is terse — job, slot, observed state, last good slot, revision, and correlation key — because the alert is an index into evidence, not a compressed dashboard.
Now work backward. The signal that should have fired earlier is the terminal failure emitted by the job wrapper after the notification batch outcome became durable. The fallback signal is the independent deadline evaluator. They cover different holes: execution errors and silence. A host probe, scheduler-process heartbeat, or log search alone covers neither contract cleanly; each observes activity, while the page needs to observe fulfillment of a scheduled obligation.
The detector should run outside the failure domain of the job scheduler. If both share the same event loop, process, or deployment, one frozen runtime can suppress the job and the code intended to report its absence. Independence doesn't require a commercial service. A separate worker can read a durable run ledger and evaluate due slots, provided its own inability to evaluate is visible to the on-call through a distinct mechanism.
Don't start with the dashboard.
Dashboards are useful after the page, particularly for examining delivery rejection trends and queue age, but a chart that is only inspected during an incident is not missed-job detection. The page must be generated from a machine-evaluable invariant: every required schedule slot reaches exactly one recognized terminal outcome before its deadline.
Test the detector against missing evidence
In the Node.js service, wrap the cron callback so it creates a run identity from the job name and scheduled slot, writes a start event, performs the notification work, and writes one terminal result only after the result is durable. The transport can be an internal HTTPS endpoint, a queue, or a database table. The contract matters more than the transport: writes should be idempotent by run identity, timestamps should use one time basis, and retries must not create a second logical run.
The following Go example is the independent evaluator side, honoring the article's Go-only code convention. It deliberately accepts a generic ledger interface, so the detection rule can be tested without binding the incident logic to a monitoring product.
package missedrun
import (
"context"
"fmt"
"time"
)
type Run struct {
Job string
ScheduledAt time.Time
StartedAt *time.Time
CompletedAt *time.Time
Outcome string
Revision string
Correlation string
}
type Ledger interface {
Find(ctx context.Context, job string, scheduledAt time.Time) (Run, bool, error)
}
type Alert struct {
Summary string
Run Run
}
func Evaluate(
ctx context.Context,
ledger Ledger,
job string,
scheduledAt time.Time,
deadline time.Time,
now time.Time,
) (*Alert, error) {
run, found, err := ledger.Find(ctx, job, scheduledAt)
if err != nil {
return nil, fmt.Errorf("read run ledger: %w", err)
}
if found && run.CompletedAt != nil && run.Outcome == "failure" {
return &Alert{Summary: "notification run failed", Run: run}, nil
}
if now.Before(deadline) || (found && run.CompletedAt != nil) {
return nil, nil
}
if found && run.StartedAt != nil {
return &Alert{Summary: "notification run exceeded its completion deadline", Run: run}, nil
}
return &Alert{
Summary: "notification run did not start before its completion deadline",
Run: Run{Job: job, ScheduledAt: scheduledAt},
}, nil
}
There is an important policy decision hidden in Outcome. A notification batch may partially fail. The wrapper needs an explicit rule for whether that is terminal success, terminal failure, or a completed run with degraded delivery that routes to a ticket instead of a page. Don't let the detector infer severity from log wording. Define it alongside the service-level objective and make the run producer send the resulting state.
Test the missing evidence, not merely the happy path. A focused suite should cover no record after deadline, start without completion, explicit failure before deadline, late success, duplicate terminal writes, evaluator clock skew, daylight-saving changes in business schedules, and a deployment during the schedule boundary. For calendar schedules, persist the intended slot and its time zone rather than recomputing history from the evaluator's local clock. I'm not sure any single grace period remains correct through a major campaign; queue-runtime percentiles and incident review are what should resolve that uncertainty.
Deployment deserves its own check. During a rolling release, two replicas may both believe they own the same slot unless leader election or an external scheduler provides exclusivity. The run identity must collapse duplicate observations, but idempotent telemetry does not make duplicate customer notifications harmless. Execution ownership and observation idempotency are two controls, and the postmortem should examine both.
False positives consume the response budget
The catch is that a deadline tight enough to catch every small delay can train the on-call to distrust the page. Set it too loose and the notification window is already gone when the alert fires. This design is not suitable when jobs have no meaningful completion deadline or when nobody can act on a miss; in that case, record the outcome for review and use a ticket or report instead of waking someone. Stick with a simple process-health check when the actual obligation is only "keep this worker available" and no discrete scheduled result exists.
Thresholds should come from the schedule contract, expected runtime distribution, ingestion delay, and the time left for a useful response. The earlier 02:19 threshold was deliberately easy to inspect, not a recommended default. Re-evaluate it after campaigns, provider changes, and material batch-size shifts. Also define how many pages one outage may create: a five-minute job should not produce twelve independent pages during an hour-long incident. Group consecutive missing slots into one incident while preserving each slot in the reconstruction record.
There is no clean threshold without a cost.
A conservative starting policy is to page immediately on an actionable terminal failure, page once when a required slot passes its deadline without completion, and suppress later alerts into the open incident until a successful run or explicit operator resolution closes it. Measure false positives as pages where the job completed within the accepted service window and no action was possible or necessary. Measure false negatives through schedule audits that compare expected slots with terminal records. Those two reviews turn alert tuning into evidence rather than preference.
Finally, rehearse the page. Give an engineer only the alert payload and the run ledger, then ask for the sequence of scheduler invocation, batch execution, persistence, and notification delivery. If the timeline cannot be reconstructed, adding another dashboard panel won't fix the missing evidence. Change the events.
Top comments (0)