Short answer: for Nodejs cron health checks and missed job detection, a free self-hosted completion ledger is the clearest no-heartbeat monitoring alternative for a nightly B2B SaaS data pipeline; use periodic heartbeats only when operators also need progress inside a long-running job.
Start with the incident question. If the on-call engineer must explain which tenant import ran, how far it got, and which attempt finally completed, a bare up/down check is too small. A completion ledger or searchable structured log gives the reconstruction trail. It can still be free and self-hosted because the detector depends on an event contract, not a hosted callback service.
| Signal | Pick this when | What it proves | Main limitation |
|---|---|---|---|
| Completion event | The job has a known finish window | A named run reached a terminal state | Silence alone cannot show intermediate progress |
| Periodic heartbeat | A single run is long and operators need liveness during execution | The process was alive at a recent instant | Extra writes create noise and still need run identity |
| Scheduler audit | The scheduler is the authority for dispatch | A trigger was created or attempted | Dispatch does not prove business work completed |
| Structured log query | Incident reconstruction matters across stages and retries | The run timeline and last known stage | Detection depends on query freshness and retention |
How should Node.js cron health checks detect missed jobs without heartbeat monitoring?
Treat the nightly pipeline as a small state machine. In words, the diagram is: expected window opens, a run starts, stages finish, the run either succeeds or fails, and the monitor evaluates the record after a grace period. Every event carries the same runId. That one field turns scattered messages into a timeline.
This separates failures that are often collapsed into “cron is down.” No run_started event after the expected window means the job was missed or dispatch was not observed. A start without a terminal event by the deadline means a timeout. A failure followed by another attempt is a retry sequence. A success means the business operation reached the terminal condition defined by the team.
Keep those labels distinct.
They lead to different next steps. A missed dispatch points toward the scheduler, deployment timing, or host availability. A timeout points toward the last completed pipeline stage and its dependency. Retry exhaustion points toward a repeated operation-level failure. If all three become one generic alert, the first several minutes of an incident are spent rediscovering information that the event model could have preserved.
The terminal condition deserves special care. For a multi-tenant export, “the process exited” may be weaker than “every planned tenant partition was committed.” Define success at the business boundary, then emit run_succeeded only there. The monitor is only as honest as that event.
Pick the signal that matches the reconstruction job
Pick a completion event when each run is bounded, the completion deadline is predictable, and one terminal marker answers the health question. This is the least complex option for a nightly batch. One start and one terminal event per attempt are usually enough to detect silence, timeout, failure, and recovery without a stream of heartbeats. Pick periodic heartbeats when the job can run for hours and “still processing” is operationally useful. The catch is that a heartbeat proves recent liveness, not correct progress. Include runId, attempt, and stage or the signal cannot distinguish a healthy retry from an old process that is still emitting. Heartbeat expiry also needs a tolerance larger than ordinary stage jitter, or slow work turns into noisy alerts. Pick the scheduler's audit record when the immediate question is whether a trigger was dispatched. Don't use it as proof that tenant data landed. Scheduling and business completion sit on opposite ends of the execution path. These choices can coexist, but each alert should name the evidence it evaluated; otherwise “scheduler healthy” can be mistaken for “pipeline complete” during the exact incident where the distinction matters.
Pick searchable structured logs when the team needs a before-and-after view during an incident. An analytical store can support this shape of investigation, especially when events are filtered by time, tenant, run, and stage. The ClickHouse documentation describes the database as a column-oriented SQL system for analytical processing; that makes it a relevant storage pattern to evaluate, not a required product choice. A file, relational table, or another queryable event store can satisfy the same contract at smaller scale.
There isn't one universal retention period. I'm not sure what is right without the pipeline's incident frequency, audit obligations, and event volume. Resolve that uncertainty with a simple question: how far back must an engineer reconstruct the last disputed import? Retain complete run timelines for at least that window, then validate the storage cost with production-like event counts.
Implement completion-led detection in TypeScript
The useful abstraction is tiny: append structured events, then load events for the latest expected run window. The storage adapter can write to a local service you operate. The health logic remains plain TypeScript and has no vendor SDK dependency.
type PipelineEventType =
| "run_started"
| "stage_finished"
| "run_failed"
| "run_succeeded";
type PipelineEvent = {
eventType: PipelineEventType;
runId: string;
job: "nightly-tenant-index";
attempt: number;
occurredAt: string;
stage?: "extract" | "normalize" | "index";
tenantCount?: number;
errorCode?: string;
};
interface EventStore {
append(event: PipelineEvent): Promise<void>;
query(job: PipelineEvent["job"], since: Date): Promise<PipelineEvent[]>;
}
const event = (
eventType: PipelineEventType,
runId: string,
attempt: number,
fields: Partial<PipelineEvent> = {},
): PipelineEvent => ({
eventType,
runId,
attempt,
job: "nightly-tenant-index",
occurredAt: new Date().toISOString(),
...fields,
});
Instrument the work at meaningful boundaries. The retry loop owns the attempt number; the outer run owns the stable runId. This matters because changing the identifier on every retry destroys the incident chain.
type PipelineDeps = {
store: EventStore;
extract: () => Promise<number>;
normalize: () => Promise<void>;
index: () => Promise<void>;
};
export async function runNightlyPipeline(
runId: string,
attempt: number,
deps: PipelineDeps,
): Promise<void> {
await deps.store.append(event("run_started", runId, attempt));
try {
const tenantCount = await deps.extract();
await deps.store.append(
event("stage_finished", runId, attempt, { stage: "extract", tenantCount }),
);
await deps.normalize();
await deps.store.append(
event("stage_finished", runId, attempt, { stage: "normalize" }),
);
await deps.index();
await deps.store.append(
event("stage_finished", runId, attempt, { stage: "index" }),
);
await deps.store.append(event("run_succeeded", runId, attempt));
} catch (error) {
const errorCode = error instanceof Error ? error.name : "UNKNOWN_ERROR";
await deps.store.append(event("run_failed", runId, attempt, { errorCode }));
throw error;
}
}
Now classify health after the expected window. The configuration below is an example policy, not a claim about every nightly pipeline: execution is expected to begin at 02:00 UTC, gets a 15-minute start grace period, and has 90 minutes to finish. Use the same UTC clock in the scheduler, event producer, and monitor.
type Health =
| { status: "waiting" }
| { status: "missed" }
| { status: "running"; runId: string; attempt: number }
| { status: "timed_out"; runId: string; lastStage?: PipelineEvent["stage"] }
| { status: "failed"; runId: string; attempts: number; errorCode?: string }
| { status: "healthy"; runId: string; attempt: number };
const minutes = (value: number): number => value * 60_000;
export function evaluateNightlyRun(
events: PipelineEvent[],
expectedStart: Date,
now: Date,
startGraceMs = minutes(15),
runTimeoutMs = minutes(90),
maxAttempts = 3,
): Health {
const ordered = [...events].sort(
(a, b) => Date.parse(a.occurredAt) - Date.parse(b.occurredAt),
);
const starts = ordered.filter((item) => item.eventType === "run_started");
if (starts.length === 0) {
return now.getTime() <= expectedStart.getTime() + startGraceMs
? { status: "waiting" }
: { status: "missed" };
}
const latestStart = starts.at(-1)!;
const runEvents = ordered.filter((item) => item.runId === latestStart.runId);
const success = runEvents.findLast((item) => item.eventType === "run_succeeded");
if (success) {
return { status: "healthy", runId: success.runId, attempt: success.attempt };
}
const failures = runEvents.filter((item) => item.eventType === "run_failed");
if (failures.length >= maxAttempts) {
const latestFailure = failures.at(-1)!;
return {
status: "failed",
runId: latestFailure.runId,
attempts: failures.length,
errorCode: latestFailure.errorCode,
};
}
if (now.getTime() > Date.parse(latestStart.occurredAt) + runTimeoutMs) {
const lastStage = runEvents
.filter((item) => item.eventType === "stage_finished")
.at(-1)?.stage;
return { status: "timed_out", runId: latestStart.runId, lastStage };
}
return {
status: "running",
runId: latestStart.runId,
attempt: latestStart.attempt,
};
}
Alert only on the states that require action: missed, timed_out, and failed. Keep waiting visible in a dashboard but quiet. A retry should update the same incident context, not page a second on-call engineer as if a different pipeline broke.
The crisp before-and-after is the payoff. Before, the alert says “nightly cron missing.” After, it says “run 2026-08-12-nightly, attempt 3, timed out after normalize; no index completion event.” The identifier and stage are illustrative event values, but the shape shows what an operator needs to start reconstruction.
Test the timeline, not just the happy path
Table tests fit this detector because time and event order create most mistakes. Freeze now. Feed the evaluator explicit timelines. Assert the classification.
import assert from "node:assert/strict";
const expectedStart = new Date("2026-08-12T02:00:00.000Z");
const afterGrace = new Date("2026-08-12T02:16:00.000Z");
assert.deepEqual(
evaluateNightlyRun([], expectedStart, afterGrace),
{ status: "missed" },
);
const completed: PipelineEvent[] = [
{
eventType: "run_started",
runId: "2026-08-12-nightly",
job: "nightly-tenant-index",
attempt: 1,
occurredAt: "2026-08-12T02:01:00.000Z",
},
{
eventType: "run_succeeded",
runId: "2026-08-12-nightly",
job: "nightly-tenant-index",
attempt: 1,
occurredAt: "2026-08-12T02:44:00.000Z",
},
];
assert.deepEqual(
evaluateNightlyRun(
completed,
expectedStart,
new Date("2026-08-12T03:45:00.000Z"),
),
{ status: "healthy", runId: "2026-08-12-nightly", attempt: 1 },
);
Add cases for a start just inside the grace boundary, a run just beyond the timeout, two failed attempts followed by success, events arriving out of order, a duplicated terminal event, and a stage event from another runId. Then test the storage adapter with the actual query consistency and clock behavior used in deployment. The pure evaluator can be correct while a stale query still delays detection.
Also test alert recovery. A late success after a timeout should resolve the active alert and preserve the earlier timeout in the timeline. It should not erase history. This is where incident reconstruction becomes more useful than a binary ping.
Limits and switching criteria
Completion-led monitoring is not suitable when a run has no credible deadline, when progress inside the run is the primary operational signal, or when the event store shares the exact failure domain being monitored. Add a heartbeat for long opaque stages. Stick with scheduler audit events when dispatch compliance is the only requirement. Use an independent monitor path when losing the workload and its event storage together would hide the incident.
Retries need restraint too. Retry only operations the application defines as safe to repeat, cap the attempts, and retain one stable run identity. A detector cannot make a non-repeatable write safe. It can only show what happened.
There is another boundary: this design measures pipeline execution, not browser experience. Core Web Vitals use field metrics such as LCP, INP, and CLS to assess user experience, with the 75th percentile used for assessment. Those signals answer a different health question. Keep them in the wider observability program, but don't mix them into the cron completion rule.
For a nightly B2B SaaS pipeline, the practical default is a structured completion ledger plus a deadline evaluator. It gives incident reconstruction without forcing periodic heartbeat traffic. Change the signal when the operating question changes.
Top comments (0)