Short answer: detect a missed Node.js cron job from an expectation created outside the job, then use heartbeats, retries, and result records to explain the miss; a heartbeat generated by the job cannot detect that the job never started.
For a scheduled customer-support import, the useful page is not “cron is down.” It is “run support-import-0200 passed its 02:20 UTC result deadline with no result; the last evidence was retry 2 at 02:11.” That sentence gives the responder an affected operation, a deadline, and a place to begin reconstruction. A green dashboard does not.
The implementation below is a free, self-hosted monitoring alternative in the literal sense that it has no required hosted service: a small detector reads an expected run plus append-only evidence and makes one deadline decision. The real cost is operational ownership. Someone must preserve the evidence, test the page, and answer for the detector at 3 a.m.
Write the failing replay before building the monitor
Begin with a deterministic failing case: an expected support import at 02:00 UTC, no worker evidence, and an evaluation clock fixed at 02:03. The assertion is PAGE never_started. Then add a second case in which the worker starts at 02:01, retries at 02:11, and still has no result at 02:21; that assertion is PAGE result_missing_after_retry attempt=2. Writing these outcomes first forces every later field to earn its place in incident reconstruction, and it prevents a polished dashboard from becoming the unstated specification.
The post-incident question is “Why did the 02:00 support import produce no usable result by 02:20?” Process liveness, a scheduler log, and a heartbeat are supporting evidence, while the expected result window is the primary signal.
An expectation must be created independently of the code being watched. A control-plane timer, a separate scheduler, or a precomputed schedule can materialize run_id, scheduled_at, start_deadline, and result_deadline. If the Node.js process is responsible for both doing the work and declaring that the work ought to exist, a failed process leaves no absence to query. Silence looks normal.
Keep the states small. They are evidence labels, not a workflow engine.
| Evidence at the result deadline | What can be asserted | Page reason |
|---|---|---|
| No event for the expected run | Dispatch or worker start was missed | never_started |
started, but no result |
Work began and did not produce a terminal record | result_missing |
One or more retry events, but no result |
Recovery ran but did not meet the run deadline | result_missing_after_retry |
result with zero records |
Transport completed; business usefulness is undecided | no transport page |
result with records |
The scheduled result exists | no page |
Zero needs care. A zero-record import can be a valid quiet period or a broken upstream query, and heartbeat monitoring cannot settle that business question. Treat it as a separate rule based on expected support volume; don't silently translate zero into infrastructure failure.
This is the first correction many implementations need: retries must not move the deadline. If each attempt receives a fresh full timeout, an import can remain “healthy” indefinitely while producing nothing. Preserve one logical run ID and one result deadline across all attempts. Attempts add evidence; they do not rewrite the promise.
How can a self-hosted Node.js cron health check detect a missed job?
Run the detector outside the Node.js cron process and give it an expected run even when the evidence file is empty. The example uses Go because one copyable implementation is easier to audit than fragments in several languages; a Node.js worker only has to append equivalent JSON records to durable storage. No SDK or vendor-specific route is involved.
Save this as main.go.
package main
import (
"bufio"
"encoding/json"
"flag"
"fmt"
"os"
"sort"
"time"
)
type Event struct {
RunID string `json:"run_id"`
Kind string `json:"kind"`
ObservedAt time.Time `json:"observed_at"`
Attempt int `json:"attempt"`
Records int `json:"records"`
}
func readEvents(path, runID string) ([]Event, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
events := make([]Event, 0)
scanner := bufio.NewScanner(file)
for scanner.Scan() {
var event Event
if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
return nil, fmt.Errorf("decode evidence: %w", err)
}
if event.RunID != runID {
continue
}
if event.Kind != "started" && event.Kind != "retry" && event.Kind != "result" {
return nil, fmt.Errorf("unknown evidence kind %q", event.Kind)
}
if event.ObservedAt.IsZero() || event.Attempt < 1 || event.Records < 0 {
return nil, fmt.Errorf("invalid evidence for run %q", runID)
}
events = append(events, event)
}
if err := scanner.Err(); err != nil {
return nil, err
}
sort.Slice(events, func(i, j int) bool {
return events[i].ObservedAt.Before(events[j].ObservedAt)
})
return events, nil
}
func decision(events []Event, now, startDeadline, resultDeadline time.Time) string {
for _, event := range events {
if event.Kind == "result" && !event.ObservedAt.After(resultDeadline) {
return fmt.Sprintf("OK result_records=%d attempt=%d", event.Records, event.Attempt)
}
}
if now.Before(startDeadline) {
return "WAIT start_window_open"
}
if len(events) == 0 {
return "PAGE never_started"
}
if now.Before(resultDeadline) {
return "WAIT result_window_open"
}
last := events[len(events)-1]
if last.Kind == "retry" {
return fmt.Sprintf("PAGE result_missing_after_retry attempt=%d", last.Attempt)
}
return fmt.Sprintf("PAGE result_missing last_kind=%s attempt=%d", last.Kind, last.Attempt)
}
func mustTime(value string) time.Time {
parsed, err := time.Parse(time.RFC3339, value)
if err != nil {
fmt.Fprintln(os.Stderr, "invalid RFC3339 time:", err)
os.Exit(2)
}
return parsed
}
func main() {
runID := flag.String("run", "", "expected logical run ID")
evidence := flag.String("evidence", "events.jsonl", "JSON Lines evidence file")
scheduled := flag.String("scheduled", "", "scheduled time in RFC3339")
nowValue := flag.String("now", "", "evaluation time in RFC3339")
startGrace := flag.Duration("start-grace", 2*time.Minute, "allowed start delay")
resultTimeout := flag.Duration("result-timeout", 20*time.Minute, "whole-run result deadline")
flag.Parse()
if *runID == "" || *scheduled == "" || *nowValue == "" {
fmt.Fprintln(os.Stderr, "run, scheduled, and now are required")
os.Exit(2)
}
scheduledAt := mustTime(*scheduled)
events, err := readEvents(*evidence, *runID)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
fmt.Println(decision(
events,
mustTime(*nowValue),
scheduledAt.Add(*startGrace),
scheduledAt.Add(*resultTimeout),
))
}
The detector deliberately fails closed on malformed evidence with exit code 2 rather than pretending the monitored job is healthy. In production, the evidence writer needs idempotency on a key such as (run_id, kind, attempt), and the store needs an ordering rule that rejects a late event from an older attempt if it would obscure newer evidence. Those are storage invariants. A chart cannot repair them later.
There is also a boundary here. This example evaluates one supplied expectation; it does not calculate cron calendars. Keep calendar interpretation in one scheduler and feed concrete UTC instants to the detector, especially around daylight-saving changes. Duplicating schedule parsing in the worker and monitor creates two clocks that can disagree while both appear locally correct.
Turn detector output into an incident page contract
The replay can now become an executable contract. support-import-0200 is due at 02:00 UTC, has a two-minute start grace, and must produce a result by 02:20. The worker starts at 02:01, records retry 2 at 02:11, and has no terminal result at 02:21. The page should identify the missed result deadline and retry attempt; it should not say the scheduler never ran.
Create events.jsonl with two lines:
{"run_id":"support-import-0200","kind":"started","observed_at":"2026-08-16T02:01:00Z","attempt":1,"records":0}
{"run_id":"support-import-0200","kind":"retry","observed_at":"2026-08-16T02:11:00Z","attempt":2,"records":0}
Then evaluate the fixed instant, without sleeping through a twenty-minute test:
go run main.go -run support-import-0200 -evidence events.jsonl -scheduled 2026-08-16T02:00:00Z -now 2026-08-16T02:21:00Z -start-grace 2m -result-timeout 20m
The expected decision is PAGE result_missing_after_retry attempt=2. Remove both lines and evaluate at 02:03 to test PAGE never_started. Add a result event at 02:18 to test success. These cases separate a missed dispatch, exhausted recovery, and a completed import without relying on log phrasing.
Don't alert on every retry.
A retry notification may be useful as timeline evidence, but paging on it creates noise while automation is still operating inside policy. The catch is that waiting for the whole-run deadline is not suitable when one failed attempt can corrupt data or consume the entire response objective. In that case, page on that explicit consequence and retain the retry as context. The correct threshold comes from the impact the support team can tolerate, not from a generic dashboard default.
I'm not sure a universal number of retained attempts or days of evidence exists. Resolve that locally by asking how long incidents can go undiscovered, what customer identifiers may be retained, and which audit questions the on-call responder must answer. Analytical storage such as ClickHouse can hold immutable historical events for later queries, but the live page should still depend on a small, explicit deadline calculation rather than a percentile over many runs. The Core Web Vitals guidance uses the 75th percentile for a different problem; copying that aggregation into missed-job detection could hide one overdue customer import inside an acceptable fleet distribution.
Deploy with shadow evaluation and fixed-time replays
Before notification delivery is enabled, run the new evaluator in shadow mode against four fixed-time cases: omit every event, leave only started, leave started plus retry, and write an on-time result with zero records. Compare its reason codes with the page contract, not with the color of an existing panel. Then repeat an overdue evaluation and confirm deduplication updates the same incident instead of opening a new page on every detector pass.
Now ask the awkward question: what page fired?
The notification should carry the logical run ID, scheduled time, result deadline, last evidence kind and time, attempt number, and reason code. It should not require a responder to infer the affected import from a graph. Test clock behavior with UTC instants that cross a local daylight-saving boundary, restart the detector while an expectation is pending, and confirm the expectation and evidence survive. If either disappears, the system is demonstrating process uptime, not incident reconstruction.
Test result delivery separately from job execution. A worker can finish its import and fail to persist the terminal evidence; from the detector's point of view, that is still a missing result record, so the runbook must tell the responder where to check the imported data before retrying. Blindly launching another attempt may duplicate support records. The monitor should describe uncertainty precisely rather than turn it into an automatic write.
Price operational ownership before rollout
The purchase price of this example is zero, but that isn't the decision. A self-hosted detector is appropriate when support identifiers must remain inside your boundary, when incident reconstruction requires custom evidence, or when schedule expectations already exist in local control-plane data. Upgrades, storage, backups, notification delivery, and the detector's own health become team responsibilities, so estimate those duties before assigning the page.
Stick with a simple external heartbeat monitor when the only actionable question is “did this command report on time?”, the payload contains no sensitive context, and operating another component would create more risk than it removes. Use a metrics-based rule when fleet-level capacity is the actual incident and no individual run needs a page. Use log search for investigation, not as the only declaration that an expected run exists. These are different jobs, and forcing one signal to do all three usually leaves the 3 a.m. responder with plenty of data but no defensible timeline.
Rollback should disable notification delivery while continuing to record expectations and evidence, then restore the previous evaluator. Do not delete the timeline to silence a noisy rule. After rollback, replay the same fixed cases against the replacement policy and compare reason codes before re-enabling pages.
Top comments (0)