Short answer: to catch a missed cron job run, make the heartbeat prove that every regional cohort completed, then alert only after its scheduled window plus a deliberate grace period has closed.
For a media experiment split across EU and US tenants, one green process is weak evidence. The scheduler may be alive while an EU cohort never starts, or a task may start and never publish a usable result. A useful heartbeat therefore represents a business transition: this cohort finished this experiment window. That is the signal. Process noise is everything around it.
Replace the pulse with an expected state transition
The tempting model is tiny: emit a pulse every time the cron callback begins, then alert when pulses stop. It catches a dead scheduler. It does not prove that the intended cohort was selected, the comparison was written, or both regions reached the same stage.
Use a before/after mental model.
Before: scheduler alive -> start pulse -> maybe useful work.
After: expected window -> region and cohort claimed -> work completed -> result accepted -> completion recorded.
That change matters because “no heartbeat” is not the only failure state. The monitor should distinguish a run that has not started, a run currently inside its allowed window, and a run that completed. Keep “failed” separate too. A reported task failure can page immediately; silence must wait until the deadline, because silence is ambiguous before then.
Think of the system as two clocks. The execution clock belongs to the scheduled task. The observation clock belongs to an independent monitor. If both clocks live in the same process, one stopped process removes both the work and the witness. Separate them — even when the first version is just a small monitor launched by a different scheduler — so the absence of work remains observable.
How should a Node.js cron heartbeat catch a missed EU or US scheduled task?
Represent each expectation explicitly. For this experiment, the identity is the tuple (experiment, region, cohort, scheduledFor). A completion for EU control must never satisfy EU treatment, and neither should satisfy US treatment. This is where a single global “last seen” timestamp creates false confidence.
Here is a compact TypeScript model. It deliberately avoids a monitoring vendor or transport. Persist these records in a store shared by workers and the monitor; wire notify to the alert channel your team already owns.
type Region = "eu" | "us";
type Cohort = "control" | "treatment";
type ExpectedRun = {
experiment: string;
region: Region;
cohort: Cohort;
scheduledFor: string;
deadline: string;
};
type Completion = {
key: string;
completedAt: string;
outcome: "accepted" | "failed";
};
const runKey = (run: ExpectedRun): string =>
[run.experiment, run.region, run.cohort, run.scheduledFor].join(":");
function findAlerts(
expected: ExpectedRun[],
completed: Map<string, Completion>,
now: Date,
): string[] {
const alerts: string[] = [];
for (const run of expected) {
const key = runKey(run);
const completion = completed.get(key);
if (completion?.outcome === "failed") {
alerts.push(`failed:${key}`);
continue;
}
if (!completion && now.getTime() > Date.parse(run.deadline)) {
alerts.push(`missed:${key}`);
}
}
return alerts;
}
The important line is not the date comparison. It is the key. Without the full identity, a busy US treatment cohort can continually refresh a shared timestamp and hide a silent EU control cohort. For an experiment comparison, that produces a particularly nasty result: the pipeline appears healthy while the dataset is asymmetrical, so a downstream reader may compare unlike windows.
Generate expectations before execution, not from execution. If the worker creates its own expected record, a worker that never starts leaves no evidence that anything was due. A deployment step or independent scheduling controller should materialize the expected runs for both regions and both cohorts. The task then writes exactly one terminal outcome against the pre-existing key.
This is crisp. Four expected records enter the window. Four terminal records should leave it.
Make the completion heartbeat mean “safe to compare”
A completion signal should sit after the last condition needed by the reader of the experiment. Sending it after records are fetched but before the aggregate is committed monitors activity, not success. For this media workflow, emit completion only after the cohort result is durable and accepted for comparison.
The order can be taught as a diagram in words: claim the run, read the cohort, calculate the experiment aggregate, validate the output, commit it, then record completion. If validation fails, record a failure outcome with the same run key. The alert can now say which experiment, region, cohort, and window failed without guessing from logs.
Retries complicate that sequence. The completion write should be idempotent: repeating an accepted completion for the same key must not create a second logical run. The experiment output needs the same protection. A retry that produces two aggregates is not healthier than a missing run; it merely fails in the opposite direction.
Deployments create another trap. Imagine revision A claims the 10:00 EU control run, then revision B becomes active during the calculation. If both revisions can publish, duplicate work may race. If neither retains ownership, the run may disappear. Put the scheduled window in the idempotency key, make ownership visible, and let a new worker resume or decline based on persisted state rather than process memory. Don't use the deployment timestamp as proof that a scheduled window was handled.
No magic here.
The monitor also needs its own health signal, but do not let that become recursive page theater. A simple external check that the monitor evaluates on schedule is enough to separate “all jobs are missing” from “the observer did not run.” The goal is a short chain of independent evidence, not monitors watching monitors forever.
Tune signal quality before adding more alerts
Start with one actionable page: an expected run passed its deadline without an accepted completion, or it reported a terminal failure. Route lower-confidence states to a dashboard or ticket until their response is clear. An alert that fires while a job is legitimately inside its completion window teaches responders to ignore it.
The grace period should come from the schedule and the job's known operating envelope, not a universal number copied across tasks. I’m not sure a single fixed grace period can be correct for both regions when their cohort sizes and publication dependencies differ; production duration data, broken down by region and cohort, is what resolves that uncertainty. Keep the rule inspectable: scheduled time, deadline, observed terminal state, and evaluation time should all appear in the alert evidence.
| Monitor | What it proves | Main noise risk |
|---|---|---|
| Process pulse | A scheduler process emitted recently | Healthy process, skipped cohort |
| Start event | A specific run was claimed | Work began but never became usable |
| Completion event | A specific result became comparable | Deadline set tighter than normal runtime |
| Explicit failure | The task reached a known failed state | Retried failures page repeatedly without deduplication |
For the experiment itself, pair operational completion with an outcome-quality check. Core Web Vitals offer a useful pattern for cohort reporting: evaluate user experience at the 75th percentile, segmented by mobile and desktop. The transferable lesson is segmentation. A global aggregate can conceal the weak slice, just as one global heartbeat can conceal a missing region. This does not mean every experiment should adopt web-vitals thresholds; it means the operational key and the analysis key should preserve the dimensions that drive the decision.
Alert labels are part of that design. Include stable dimensions such as experiment, region, cohort, and scheduled window. Avoid labels based on error text or tenant IDs if they create an unbounded stream of unique series. Detailed error context belongs in logs linked by the stable run key. Metrics answer “how many and where?” Logs answer “what happened?” The completion record answers “did this exact obligation finish?”
When is heartbeat monitoring the wrong replacement?
Heartbeat monitoring is not suitable when the real requirement is strict workflow orchestration with dependencies, backfills, and operator-controlled replay. In that case, use a workflow system that stores each transition as first-class state, then monitor the workflow's terminal expectations. A thin heartbeat layer should not impersonate a durable state machine.
Stick with a simple scheduler plus completion monitor when tasks are independent, the expected windows are easy to enumerate, and retries can be made idempotent. Move to a queue when execution needs durable delivery and controlled concurrency. Move to a workflow engine when the media experiment has branching steps, cross-region dependencies, manual approvals, or frequent backfills. The catch is operational weight: stronger coordination gives better state visibility, but it also introduces another control plane the team must deploy, secure, observe, and understand.
There is a compliance boundary too. If experiment records contain regulated health information, a generic heartbeat design is not a security program. NIST SP 800-66r2 provides guidance for implementing the HIPAA Security Rule; teams in that scope need a documented risk-management approach around the stores, logs, notifications, access controls, and retention involved. Keep sensitive payloads out of alert text. A stable opaque run key is usually enough for responders to locate authorized detail.
The decision rule is short: page on missing obligations, not missing noise. Define each regional cohort run before it is due, record completion only when its result is safe to compare, and keep the observer independent from the worker. Then a silent EU slice cannot borrow health from a busy US slice.
Top comments (0)