Missed Cron Job Monitoring: 4 Heartbeat Signals for Learning Pipeline Failures

The least complex reliable design is to emit four structured checkpoints for every expected nightly cron run: expected, started, failed or completed, and a post-deadline overdue assessment. Store them beside the pipeline's searchable logs, keyed by one stable run ID and one scheduled instant in UTC. That gives an operator enough evidence to distinguish a scheduler miss from a task crash, a slow import, and a notification failure. It works as a healthchecks replacement when the real requirement is searchable incident evidence rather than a bare heartbeat.

Short answer: an error alert alone cannot prove that a scheduled task ran. Treat schedule evaluation as an independent process, record the expected run before execution, and alert from persisted state after a grace period. For an edtech pipeline that imports enrollment and course activity overnight in both the EU and US, this design favors incident reconstruction over an attractive monitoring screen. It also stays portable: the scheduler, log store, and notification transport can change without changing the event contract.

How should heartbeat monitoring detect a missed cron job run?

Start with the question an on-call engineer will ask the next morning: "Did the scheduler fail to dispatch, did the worker fail, or is the job still processing?" A lone success ping answers only one narrow question. Its absence collapses several different states into the same silence, which makes failure alerts hard to trust during reconstruction.

Silence is ambiguous.

The data flow is small. A schedule evaluator writes an expected event for each region and scheduled instant. A worker claims that run ID and writes started, then either completed or failed. After the allowed completion window, a separate evaluator reads the event stream and writes overdue only when no terminal event exists. The alert sender consumes failed and overdue; it is not the source of truth.

Use these four checkpoints because each closes a specific gap:

1. expected proves the schedule existed and supplies the deadline.
2. started proves dispatch reached a worker.
3. failed or completed records the terminal outcome.
4. overdue records the evaluator's decision, including the observation time.

Keep "expected" separate from "started." That distinction is the whole diagnosis when a run never launches.

A small event model you can run

The following Node.js TypeScript uses an in-memory store so the state transition is visible. In production, the EventStore boundary should use durable storage that supports your retention and access requirements. The example doesn't depend on a particular scheduler or alert service.

type Region = "eu" | "us";
type Stage = "expected" | "started" | "completed" | "failed" | "overdue";

type RunEvent = {
  runId: string;
  region: Region;
  scheduledFor: string;
  observedAt: string;
  stage: Stage;
  attempt: number;
  records?: number;
  errorCode?: string;
};

interface EventStore {
  append(event: RunEvent): Promise<void>;
  list(runId: string): Promise<RunEvent[]>;
}

class MemoryEventStore implements EventStore {
  private readonly events: RunEvent[] = [];

  async append(event: RunEvent): Promise<void> {
    this.events.push(structuredClone(event));
  }

  async list(runId: string): Promise<RunEvent[]> {
    return this.events.filter((event) => event.runId === runId);
  }
}

const makeRunId = (region: Region, scheduledFor: string): string =>
  `learning-import:${region}:${scheduledFor}`;

async function assessRun(
  store: EventStore,
  region: Region,
  scheduledFor: string,
  observedAt: string,
  graceMs: number,
): Promise<"pending" | "terminal" | "overdue"> {
  const runId = makeRunId(region, scheduledFor);
  const events = await store.list(runId);
  const terminal = events.some(
    (event) => event.stage === "completed" || event.stage === "failed",
  );

  if (terminal) return "terminal";

  const deadline = Date.parse(scheduledFor) + graceMs;
  if (Date.parse(observedAt) <= deadline) return "pending";

  if (!events.some((event) => event.stage === "overdue")) {
    await store.append({
      runId,
      region,
      scheduledFor,
      observedAt,
      stage: "overdue",
      attempt: 0,
    });
  }
  return "overdue";
}

async function example(): Promise<void> {
  const store = new MemoryEventStore();
  const scheduledFor = "2026-10-12T01:00:00.000Z";
  const runId = makeRunId("eu", scheduledFor);

  await store.append({
    runId,
    region: "eu",
    scheduledFor,
    observedAt: "2026-10-12T00:55:00.000Z",
    stage: "expected",
    attempt: 0,
  });

  const result = await assessRun(
    store,
    "eu",
    scheduledFor,
    "2026-10-12T03:01:00.000Z",
    2 * 60 * 60 * 1000,
  );

  console.log(result, await store.list(runId));
}

void example();

The two-hour grace period is example policy, not a universal recommendation. Derive it from the pipeline's normal completion distribution and the point at which stale learning data becomes operationally harmful. A tighter window detects trouble earlier but pages on legitimate long runs; a wider one reduces noise while delaying investigation. The trade-off is detection speed against false alarms. I would ship with a documented conservative window, collect durations, then revise it from observed data rather than intuition.

Notice that retries keep the same runId and increment attempt. If every retry invents a new identity, a log search can make one scheduled import look like several unrelated jobs. Also make append operations idempotent in durable implementations. Evaluators can run twice, notifications can be retried, and duplicate overdue records shouldn't create duplicate incidents. This is a poor fit for sub-minute tasks where writing four durable events costs more than the work itself; a single counter plus an external schedule evaluator is enough there. It is also unnecessary for best-effort maintenance work that has no deadline and creates no incident when skipped.

Preserve time without trusting local clocks

Regional schedules are where a clean-looking design often becomes ambiguous. "Run at 1:00" is not an incident key. Store the scheduled instant as an ISO 8601 UTC timestamp, retain the region as a separate field, and keep the configured time zone in schedule metadata. That preserves the operator's intent without making daylight-saving transitions part of every log query.

Don't infer a missed EU run from the US run, even if both import the same classes of records. They are separate obligations with separate run IDs. A delayed EU source can coexist with a healthy US source, and combining them behind one heartbeat destroys that fact.

Clock handling needs restraint. scheduledFor comes from the schedule evaluator; observedAt records when each component saw the event. Ordering by observedAt alone can be misleading if hosts disagree. Reconstruct primarily by the stable run ID and stage transition, then use timestamps to inspect latency. The terminal event should include compact counts such as processed records, but student names, email addresses, raw prompts, and model output do not belong in an operational event.

That last boundary matters in education systems. NIST SP 800-66 Revision 2 is written for implementing the HIPAA Security Rule, so it is not a blanket compliance label for an edtech pipeline. Its treatment of risk analysis, access control, audit controls, and incident procedures is still a useful primary reference when regulated health information is actually in scope. Compliance scope must be established by the organization; a logging pattern cannot establish it.

Query the timeline, not the error string

An incident search should begin with region plus scheduledFor, resolve the runId, and return all four stages in observation order. This is cheaper and more dependable than indexing large payloads just in case they become useful. Keep high-cardinality identifiers where they earn their cost: run ID, attempt, region, stage, and a bounded error code.

The shape of the timeline tells you where to look next:

Observed events	Working diagnosis	Next evidence
expected, then overdue	dispatch was not observed	scheduler and queue delivery logs
expected, started, then overdue	worker is slow, stuck, or lost	worker lease, progress, and dependency logs
expected, started, failed	worker reported an error	bounded error code and attempt history
expected, started, completed	pipeline finished	downstream publication or index freshness

These are working diagnoses, not proof of root cause. For example, a worker may finish after the evaluator writes overdue. Preserve both events. Don't rewrite history to make the timeline look tidy; close or downgrade the incident when the late terminal event arrives.

The search experience also has a user-facing cost. If missing nightly data affects pages used by teachers or administrators, correlate pipeline freshness with page performance instead of claiming the pipeline caused a slow page. Core Web Vitals define LCP, CLS, and INP and recommend evaluating the 75th percentile of page loads. Those measurements describe user experience; the run timeline describes data operations. A shared incident view can place them side by side without pretending they are the same signal.

Alert only from durable decisions

Send an immediate notification for a recorded failed event. Send a missed-run notification only after the independent evaluator persists overdue. The persisted decision makes notification retries safe to reason about and leaves evidence when the delivery channel itself is unavailable.

Deduplicate on runId plus alert kind, not on the rendered message. Route EU and US incidents according to ownership, but keep severity tied to business impact: an overdue auxiliary aggregation should not page like a blocked enrollment import. Cost matters here. More log fields, longer retention, and broader full-text indexing all have a bill, while the five compact query keys above usually carry the investigation.

Test silence deliberately. A happy-path test proves too little because the dangerous case is the component that never starts. Cover an absent started, a worker failure, a run that completes just before the deadline, a late completion after overdue, a duplicate evaluator pass, and two regional runs at the same wall-clock time. Inject a notification failure and verify that the decision remains queryable for replay.

Before deployment, read the event contract as an operator rather than its author. Confirm that schedule creation precedes dispatch, timestamps are UTC instants, retries retain identity, terminal writes happen in a finally-aware execution path, and the evaluator cannot depend on the worker it watches. Then run one synthetic schedule per region through the real storage and notification boundaries. The final check is blunt: with the worker disabled, can a person search one run ID and explain the silence without guessing?

Further reading

• https://web.dev/articles/vitals
• https://csrc.nist.gov/pubs/sp/800/66/r2/final