Choose healthchecks for cron jobs as the primary missed-task detector; use app metrics as the second layer for monitoring stage latency, record counts, and suspicious results. For a one-person Node.js SaaS shipping weekly, that split keeps each alert tied to a specific scheduled logistics run while preserving enough evidence to reconstruct the incident.
| Choice | Detects a missing run | Reconstructs one run | Best role |
|---|---|---|---|
| Deadline heartbeat | Directly | With run metadata | Primary missed-task alarm |
| Application metrics | Indirectly | With correlated logs | Diagnosis and trend context |
Recommendation: put a deadline around each expected completion, then emit a small set of run metrics and structured logs under the same run ID. Choose metrics alone only when the schedule is elastic and absence is not itself a failure.
Should you choose healthchecks or app metrics for cron jobs?
A nightly pipeline can fail before application code starts. The scheduler may never launch it. A container may be rejected. A deployment may remove the schedule. In each case, an in-process counter has no chance to increment. Silence is ambiguous: it can mean no work, no process, no scrape, or no network path.
A deadline heartbeat models a narrower promise: a named job must report completion inside an expected window. The monitor owns the clock outside the worker, so it can notice total absence. Keep the heartbeat at completion, not merely at start; a start signal followed by a stuck import is not success.
Time zones make the promise harder than it looks. For a pipeline serving US and EU logistics records, store the schedule and timestamps in UTC, preserve the source region as data, and define one explicit grace period. Daylight-saving transitions should not silently create a 23-hour or 25-hour assumption. The scheduled instant, start time, completion time, and deadline are separate fields.
Short version: watch the promise from outside the process.
No launch. No metric.
Incident reconstruction is the deciding criterion
The alarm should lead to a run, not a dashboard scavenger hunt. Give every scheduled attempt a unique run_id, and attach it to the heartbeat, logs, and metrics. For a nightly shipment-status import, useful evidence includes the scheduled time, region, stage, input count, accepted count, rejected count, and terminal status. That is enough to distinguish "never launched" from "launched, parsed 84,210 rows, then failed before commit." It is not permission to log shipment addresses or customer tokens.
OWASP recommends excluding or masking sensitive data such as access tokens, passwords, and personal data from logs. Apply that rule before ingestion. A run identifier should be opaque; it should not encode an email, tracking number, or tenant name. Structured logs help search, but a stable schema matters more than a large payload.
The reconstruction path is short:
- Find the deadline event for the job and scheduled instant.
- Read its
run_id, if one exists. No ID suggests execution did not reach instrumentation. - Search structured logs by that ID and order them by timestamp.
- Compare stage counters with the terminal status and durable output.
This is why I would not make a generic last_success_timestamp gauge the only detector. It can answer when the last success occurred, but the alert expression must still encode cadence, grace, scrape health, and label identity. A dedicated deadline makes the expected event explicit. Metrics then earn their keep during diagnosis.
Keep the instrumentation boring
The worker needs one correlation object and a tiny interface. This example uses standard fetch, emits JSON, and reports completion only after the durable write succeeds. It does not treat an accepted HTTP request as proof that the business transaction committed.
import { randomUUID } from "node:crypto";
type Region = "us" | "eu";
interface RunEvent {
runId: string;
job: "shipment-status-import";
region: Region;
scheduledAt: string;
accepted: number;
rejected: number;
}
class CompletionReporter {
constructor(private readonly endpoint: URL) {}
async complete(event: RunEvent): Promise<void> {
const response = await fetch(this.endpoint, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(event),
signal: AbortSignal.timeout(5_000),
});
if (!response.ok) throw new Error(`completion report rejected: ${response.status}`);
}
}
async function runNightlyImport(
region: Region,
scheduledAt: string,
reporter: CompletionReporter,
): Promise<void> {
const runId = randomUUID();
const result = await importAndCommitShipments({ runId, region, scheduledAt });
await reporter.complete({
runId,
job: "shipment-status-import",
region,
scheduledAt,
accepted: result.accepted,
rejected: result.rejected,
});
}
There is a deliberate failure mode here: data may commit while completion reporting fails. Make the import idempotent on run_id or a durable batch key, then let a retry report the same logical run. Never move the completion call ahead of commit just to eliminate that awkward state. That creates a cleaner dashboard and a false success.
The trade-off is extra state at the boundary: the deadline service knows the expected schedule, while the worker knows whether the durable commit happened. A heartbeat cannot prove that every shipment row is correct, and it cannot explain a slow validation stage by itself. It is also a poor fit for irregular jobs with no meaningful due time. Those limitations are why the metrics and structured events remain part of the design rather than optional decoration.
Test four paths before deployment: the scheduler does not launch; the worker starts and hangs; commit fails; commit succeeds but the reporter is temporarily unreachable. Also test that a retry cannot duplicate shipment updates. These tests exercise the boundary that actually pages you.
Two clocks, one ownership rule
Detection latency is controlled by the schedule, deadline, and grace period. Set the grace from observed completion behavior and the business cutoff, not from a round number that looks tidy. A pipeline due before warehouse dispatch needs a deadline anchored to that dispatch decision. One processing archival exports can tolerate more drift.
Metrics answer different questions. Record run duration, input and rejection counts, and perhaps the age of the newest source record. Keep labels bounded: job, region, stage, and terminal status are reasonable dimensions; run_id belongs in logs rather than metric labels because every run creates a new value. Metrics should reveal trends without becoming the source of truth for individual-run identity.
One owner must define each clock. Let the external monitor own "completion is overdue." Let the application own "duration or rejection behavior is abnormal." If both systems page on the same missing completion with slightly different windows, a solo operator gets two alerts and no additional evidence. Time is scarce. Every duplicate page takes time from the weekly ship cycle.
One page is enough.
When are metric alerts the better first choice?
Use metrics first when work is continuous, demand-driven, or intentionally sparse. A queue consumer that should keep lag below a threshold does not have one meaningful nightly deadline. Neither does a compactor triggered only after enough data accumulates. In those systems, throughput, queue age, error rate, or time since the oldest pending item maps more closely to user impact than a synthetic scheduled heartbeat.
Metrics can also lead when the scheduler is already a durable workflow engine with externally visible per-run state and deadline semantics. Adding another heartbeat may duplicate the same contract. Preserve the common run_id and export diagnostic metrics, but rely on one authoritative missed-run detector.
For the nightly logistics pipeline, those exceptions do not apply. There is a promised completion window and a concrete downstream dispatch decision. The lean design is one external completion deadline, a few bounded metrics, and searchable structured events joined by an opaque run ID. It outsources the undifferentiated clock watching while keeping the evidence needed to answer the expensive question: what happened to this run?
References
The sources below define the security, runtime, correlation, and metric-label constraints used in this design.
Sources
- OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- Node.js documentation,
fetch: https://nodejs.org/api/globals.html#fetch - Node.js documentation,
crypto.randomUUID: https://nodejs.org/api/crypto.html#cryptorandomuuidoptions - OpenTelemetry logs data model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
- Prometheus metric and label naming: https://prometheus.io/docs/practices/naming/
Top comments (0)