Short answer: for a Next.js service with a Node.js cron job, treat the heartbeat as a completion record, then use a health check to detect when the expected job result does not arrive; keep metrics for alert timing and structured logs for incident reconstruction. A process check alone cannot tell you that last night's pipeline never finished.
I care about this distinction because a scheduler can be alive while the work is dead. A Next.js or Node.js service may answer requests all night, while its cron callback was skipped, returned early, or delivered the same batch twice. The monitor needs evidence of the job's outcome, not evidence that a runtime still has a PID.
How should a Next.js Node.js cron job use heartbeats, health checks, metrics, and logs to detect a missed run?
Use four signals with different jobs:
- A heartbeat is the durable statement that a particular run reached a known state.
- A health check asks the monitoring system to inspect the age of that statement.
- A metric makes lateness and failure count cheap to aggregate and alert on.
- A structured log preserves the fields needed to explain the incident later.
Do not collapse these into one “healthy” boolean. The health endpoint for the web process can remain green while last_success_at is stale. Conversely, a metric can say a run failed while logs explain which partition and checkpoint caused it.
The monitor should check a freshness rule such as “the latest successful completion is newer than the allowed schedule window.” The window must include the normal runtime, queue delay, and a small operational margin. I would define that number from observed behavior and a written service-level objective; I’m not sure any fixed five-minute threshold is correct for your pipeline.
Here is the core transition in Go. The storage interface is intentionally generic: it can point at a database, object store, or another durable system. The important behavior is that a duplicate terminal write is safe and that an unsuccessful run does not masquerade as a successful heartbeat.
package heartbeat
import (
"context"
"time"
)
type RunState string
const (
StateSucceeded RunState = "succeeded"
StateFailed RunState = "failed"
)
type Completion struct {
RunID string
Job string
Schedule string
State RunState
FinishedAt time.Time
RecordCount int64
}
type Store interface {
PutCompletion(ctx context.Context, completion Completion) error
}
func RecordCompletion(ctx context.Context, store Store, completion Completion) error {
if completion.RunID == "" || completion.Job == "" {
return &ValidationError{Message: "run_id and job are required"}
}
if completion.State != StateSucceeded && completion.State != StateFailed {
return &ValidationError{Message: "state must be terminal"}
}
// PutCompletion must be keyed by job, schedule, and run_id, so retries are idempotent.
return store.PutCompletion(ctx, completion)
}
type ValidationError struct {
Message string
}
func (e *ValidationError) Error() string { return e.Message }
The production contract around PutCompletion matters more than the function name. A retry should return the existing outcome for the same identity, not create a second business result. If the operation is not safe to retry, the alert may be fixed while the data is still duplicated.
The run record that makes a postmortem searchable
A useful log line is an event, not a paragraph. Put the same identifiers in every phase: job, schedule, run_id, started_at, finished_at, state, checkpoint, record_count, and error_code. Keep timestamps in UTC and encode them consistently. Avoid putting customer payloads into the log merely because they are available; that increases exposure without improving the run decision.
The smallest useful sequence looks like this:
package main
import (
"encoding/json"
"log"
"time"
)
type Event struct {
Event string `json:"event"`
Job string `json:"job"`
Schedule string `json:"schedule"`
RunID string `json:"run_id"`
State string `json:"state,omitempty"`
Checkpoint string `json:"checkpoint,omitempty"`
RecordCount int64 `json:"record_count,omitempty"`
ErrorCode string `json:"error_code,omitempty"`
OccurredAt time.Time `json:"occurred_at"`
}
func writeEvent(event Event) {
data, err := json.Marshal(event)
if err != nil {
log.Printf("event_encode_failed error_code=%q", "json_encode")
return
}
log.Print(string(data))
}
The start record helps answer “was it invoked?” The checkpoint fields answer “how far did it get?” The terminal record answers “what does the system believe now?” Those are different questions, and a postmortem becomes much shorter when the fields can be queried without parsing prose.
If you ship logs through an appender-like pipeline, define behavior for write failures, buffering, and shutdown. Logback's appender documentation is a good reminder that output components have lifecycle and error-handling responsibilities; the same principle applies to a Node.js logger or a self-hosted collector. A log transport failure must not silently change a successful business result into an unknown one. The completion heartbeat belongs in durable state, while logs provide the investigative trail.
Turning an old completion into an actionable alert
Use a counter for terminal outcomes and a gauge for freshness. Useful names are cron_runs_total{job,state}, cron_last_success_timestamp_seconds{job}, and cron_run_duration_seconds{job}. A counter tells you that failures are happening. The freshness gauge tells you that nothing happened at all.
The health check can expose a small, non-sensitive result to the monitoring service:
package health
import (
"fmt"
"time"
)
func CheckLastSuccess(lastSuccess time.Time, now time.Time, maxAge time.Duration) error {
if lastSuccess.IsZero() {
return fmt.Errorf("no successful completion recorded")
}
if now.Sub(lastSuccess) > maxAge {
return fmt.Errorf("last successful completion is stale")
}
return nil
}
The endpoint should report the job-specific condition, not dump the entire log stream. Keep authorization and sensitive identifiers out of the response. Have the monitor call it on a schedule that is shorter than the allowed lateness, then alert on sustained failure rather than one transient check. The exact retry count is a policy decision: document it with the pipeline's schedule and expected runtime so the alert is explainable.
There is a trade-off here. A short freshness window catches a missed run quickly but creates noise during normal queue delays. A long window reduces pages and delays detection. Pick the smallest window that the measured runtime distribution and recovery procedure can support, then revisit it after a real incident.
Why a green process can hide a missed run
For a B2B SaaS nightly data pipeline, the useful question is not “is the app up?” It is “did the run for this schedule finish, and can I reconstruct what happened?” That decision axis changes the design.
The bounded failure pattern is familiar from on-call work: the web process remains healthy, the cron invocation emits a start log, and the downstream search index contains no completion marker. A broad uptime check stays green. A missing-run alert built from a completion heartbeat turns the silence into a timestamped signal. During reconstruction, I want to compare the scheduled time, the run_id, the last checkpoint, and the terminal state in one query; otherwise the operator is forced to infer a run from scattered timestamps, retries, and partial logs, which is exactly how a quiet data gap turns into a duplicate delivery later.
The invariant is simple: one logical run gets one stable run_id, one start event, one terminal event, and a heartbeat that records the terminal state. The terminal event must be written only after the pipeline has reached the point your business considers complete. “The function returned” is often too weak a definition.
Silence is a signal.
Duplicate delivery is the other half of the postmortem. A retry can make the same run_id appear twice, so the consumer and the completion writer need idempotent behavior. Store the run identity with the result, reject a second terminal transition, and make alerting group repeated symptoms by a stable fingerprint. Grouping repeated events by a stable fingerprint keeps an alert stream from hiding the one run that matters.
When a basic uptime check is still the right boundary
Start with the failure mode. If the requirement is “the process responds,” an uptime check may be enough. If it is “the nightly pipeline produced a complete, searchable dataset,” use a completion heartbeat plus structured telemetry. If the scheduler itself is unreliable, move scheduling and execution ownership to a system that records attempts and retries separately, while preserving the same run_id contract.
The catch is that this design is not suitable when the job has no durable notion of completion or when the team cannot operate a state store and alert policy. In that case, stick with a simpler external check until those prerequisites exist; a sophisticated dashboard without a trustworthy terminal record only gives the incident more decoration.
Test the contract before deployment. Exercise a normal run, a timeout after the last checkpoint, a failed write, a retry with the same run_id, and two concurrent deliveries. Verify that the expected alerts fire, that a duplicate does not double-count business output, and that an operator can find the relevant logs using one identifier. Include the monitor and the heartbeat store in the recovery runbook. Otherwise the first missed run will become a second incident about the monitoring system.
The decision rule is therefore narrow: use liveness for availability, completion heartbeats for missed-run detection, metrics for alert math, and logs for reconstruction. Keep the signals separate, make the terminal write idempotent, and choose the lateness window from the job's actual operating envelope.
Top comments (0)