Short answer: when a cron job can miss a run, heartbeat monitoring must happen outside the scheduled worker. Record starts and terminal outcomes separately, then alert after a region-specific deadline plus a small grace window. That choice detects both silence and explicit failures while keeping rollback independent of the code being rolled back.
| Choice | Never started | Failed after start | Rollback coupling |
|---|---|---|---|
| External deadline plus start/outcome events | Yes | Yes | Low |
| Exception alert inside the task | No | Yes | High |
| Success-only ping | Yes | Indirectly | Medium |
Recommendation: use the first row for EU and US delivery-notification schedules. Treat an exception alert as supporting evidence, not proof that the schedule ran. Short answer: the absence detector must live beyond the failure boundary it observes.
How should heartbeat monitoring alert on a missed cron job run?
Code can report only after code executes. A scheduler that does not launch the process produces no caught exception, no retry log, and no final failure event. Silence looks healthy if the alert condition asks only whether failures are greater than zero.
No process. No error.
For this logistics job, define the expected unit before touching an SDK: one notification batch, for one region, for one scheduled window. Give it a deterministic run key such as eu:2026-10-18T08:00Z. The watchdog then asks a blunt question: did that expected key reach a terminal state before its deadline?
Do not let a deployment invent a second identity for the same window. A version field belongs in event metadata, while the run key remains stable across rollback. Otherwise the old and new releases can each appear incomplete, creating two pages for one delivery batch. Config bloat starts exactly this way: one innocent label becomes control flow.
Here's a concrete rollout to reason through. The EU window is due, version a17 starts it, and the deployment is rolled back before delivery finishes. Version a16 then resumes work for the same scheduled window. If version is part of identity, the monitor sees an abandoned a17 run and a separate a16 run; it can page even though the expected batch reaches a terminal state. If identity is the region plus scheduled instant, both releases report against one expectation and version stays diagnostic metadata. The monitor can now distinguish a handoff from a duplicate schedule without special rollback configuration. This is the kind of branch I benchmark for developer effort too: count the fields an application must provide, the states an operator must interpret, and the deployment-specific rules the monitor needs. A design that needs a new rule for every release mechanism is already too coupled.
Rollback safety is the first decision criterion
A safe rollback can change worker code without erasing the monitor's memory of what was due. Keep the schedule definition and observed run state in a control path that survives an application revert. The worker emits facts; the watchdog evaluates deadlines.
Use three states: started, succeeded, and failed. They answer different questions. A missing started event points toward launch or scheduling. A started event without a terminal outcome points toward a crash, timeout, or stalled dependency. An explicit failed outcome carries the error category immediately. No giant taxonomy is required on day one.
The alert payload should include the region, scheduled window, last state, deployment version, and run key. It should not include recipient content. For health-related data, NIST SP 800-66r2 is a useful security-rule implementation reference; even outside healthcare, its existence is a reminder to make data handling an explicit design decision rather than dumping business payloads into telemetry.
Rollback also changes who owns retries. Pick one layer. If the scheduler retries and the worker retries each delivery independently, record both attempt numbers without changing the stable run key. The watchdog evaluates completion of the window, not the number of processes that happened to execute.
EU and US schedules should have separate expected windows even when they run identical code. Store those windows as absolute instants and retain the business timezone as metadata. That keeps the watchdog comparison simple while leaving enough context for an operator to understand why a run was due.
The grace period is an operational choice, not a universal constant. Measure normal start delay and completion duration in your own system, then set the deadline beyond expected jitter but before the notification loses business value. I would benchmark those two distributions before adding knobs. A single global timeout is attractive until one region's queue behavior makes it either noisy or late.
This is also where a generic availability metric can mislead. Core Web Vitals uses a 75th-percentile assessment for user-experience metrics. That threshold is appropriate to its stated web-performance purpose; a missed scheduled batch is a discrete expected event, so its deadline needs per-window evaluation rather than a population percentile copied from another domain.
Different jobs need different deadlines. Keep that variation in data, not scattered configuration files.
A small TypeScript boundary
The worker needs little glue: one typed event sink and a wrapper that always emits a terminal outcome after a start. The code below deliberately leaves transport behind an interface.
type Region = "eu" | "us";
type RunState = "started" | "succeeded" | "failed";
type RunEvent = {
runKey: string;
region: Region;
scheduledFor: string;
version: string;
state: RunState;
observedAt: string;
errorKind?: string;
};
interface EventSink {
emit(event: RunEvent): Promise<void>;
}
async function runNotificationBatch(
base: Omit<RunEvent, "state" | "observedAt" | "errorKind">,
sink: EventSink,
deliver: () => Promise<void>,
): Promise<void> {
await sink.emit({ ...base, state: "started", observedAt: new Date().toISOString() });
try {
await deliver();
await sink.emit({ ...base, state: "succeeded", observedAt: new Date().toISOString() });
} catch (error) {
await sink.emit({
...base,
state: "failed",
observedAt: new Date().toISOString(),
errorKind: error instanceof Error ? error.name : "UnknownError",
});
throw error;
}
}
The watchdog is separate. At each evaluation, it loads expected windows whose deadlines have passed and checks for a terminal event under the stable key. Make alert creation idempotent on that key. A second evaluator pass should update or ignore the existing incident, never wake someone twice.
Test four paths before deployment: no start, explicit failure, start without completion, and success arriving just inside the deadline. Then test the awkward deployment path: launch the new version, create a start event, roll back, and let the old version finish the same run key. The expected result is one completed window and no orphan alert.
When is the runner-up enough?
An exception-only alert is reasonable for a manually triggered repair script where an operator already proves that execution began. It is also useful beside the watchdog because it can deliver the error category before the deadline expires. It remains the runner-up for scheduled notification delivery because it cannot report code that never ran.
A success-only heartbeat is a leaner option when each schedule has one process, completion time is predictable, and diagnosis can wait. It proves eventual completion with less event volume. The trade-off is slower separation of "never launched" from "started and stalled," which matters during rollback when two versions may briefly participate.
My decision rule is plain: if a missed delivery window can outlive a deployment, preserve the expected window and its state outside that deployment. Keep the worker instrumentation tiny. Benchmark the grace period. Page once.
Top comments (0)