Short answer: for a marketplace cohort experiment, use an externally evaluated completion heartbeat when you only need a missed-job alert; use a durable run ledger when incident reconstruction must show which tenant cohort started, finished, failed, or never ran. A log line from inside the job cannot prove that a job never started. The monitor needs an expected deadline created independently of the worker, plus enough cohort context to explain the gap.
Decision field guide
| Signal | Pick it when | What it can establish | Main trade-off |
|---|---|---|---|
| Completion heartbeat | One scheduled task has one clear deadline | The expected completion signal arrived, arrived late, or is missing | Tiny integration surface, but little evidence about partial cohort work |
| Durable run ledger | An experiment spans many tenant cohorts or can partially complete | Planned, running, succeeded, failed, and overdue states per cohort | Better reconstruction, with storage, retention, and monitor logic to own |
| Queue-native schedule and dead-letter signals | Work is already expressed as durable messages | Enqueue, attempt, retry, and terminal handling events | Strong delivery history, but the queue's semantics become part of the design |
| Structured log query | Logs are already centralized and the alert can tolerate query delay | A start, completion, or explicit error record was observed | Fast to adopt, but absence remains ambiguous unless expectations live elsewhere |
For a single nightly cleanup, a completion heartbeat is often enough. Define the due time outside the process, send one signal only after the useful work commits, and alert after a measured grace window. Don't emit success from a finally block. That turns a thrown error into a healthy-looking run.
A run ledger is the sharper choice for the marketplace case. Suppose an experiment compares control, variant-a, and variant-b across tenant cohorts. One global ping can say that the parent process reached its last line while hiding the fact that variant-b was skipped. A ledger makes the unit of expectation match the unit an incident responder will investigate.
Queue signals fit when each cohort is already a message and the queue durably records attempts. Keep the experiment identifier and cohort key in that message metadata, then map queue outcomes into the same small state model used by dashboards and alerts. Logs still matter for diagnosis. They just shouldn't carry the entire burden of proving non-execution.
Silence is the signal.
How should a Node.js cron heartbeat detect a missed background job?
Create the expectation before the Node.js cron worker is due, then let a separate monitor compare current time with the stored deadline. That separation detects the hardest case: the background job never starts, so it cannot report its own failure. For incident reconstruction, key each expectation by experiment, tenant cohort, and scheduled instant; update it as work starts and reaches a terminal state.
The diagram in words is short: scheduler writes expected cohort runs; worker claims one run; worker records the outcome; monitor reads overdue or failed runs; alert router groups them by experiment and scheduled instant. The worker and monitor must not share one process lifecycle. If they disappear together, silence can look healthy.
Use deadlines.
A completion expected at 02:00 with an empirically chosen grace period expresses the condition the team cares about. A pulse every minute only establishes that some loop was alive. It doesn't establish that every tenant cohort was compared or that the result was committed.
The grace period deserves real data. I'm not sure a fixed ten-minute grace works for your marketplace; cohort size, scheduler jitter, and downstream latency decide that. Start with observed high-percentile duration plus scheduler delay, review false alerts, and keep a hard upper bound that still leaves humans time to act before the experiment's next decision window.
Implement the expected-run ledger
The following TypeScript keeps storage in memory so the state machine is visible and runnable without a package. In production, put the RunStore contract behind durable storage that supports atomic state changes. Register all cohort expectations from the scheduling control plane before starting any cohort worker.
type RunState = "scheduled" | "running" | "succeeded" | "failed";
type ExpectedRun = {
id: string;
experimentId: string;
cohort: string;
plannedAt: Date;
deadlineAt: Date;
state: RunState;
startedAt?: Date;
finishedAt?: Date;
errorCode?: string;
};
interface RunStore {
create(run: ExpectedRun): Promise<void>;
update(id: string, patch: Partial<ExpectedRun>): Promise<void>;
list(): Promise<ExpectedRun[]>;
}
class MemoryRunStore implements RunStore {
private readonly runs = new Map<string, ExpectedRun>();
async create(run: ExpectedRun): Promise<void> {
if (this.runs.has(run.id)) throw new Error(`duplicate run: ${run.id}`);
this.runs.set(run.id, run);
}
async update(id: string, patch: Partial<ExpectedRun>): Promise<void> {
const current = this.runs.get(id);
if (!current) throw new Error(`unknown run: ${id}`);
this.runs.set(id, { ...current, ...patch });
}
async list(): Promise<ExpectedRun[]> {
return [...this.runs.values()];
}
}
const runId = (experimentId: string, cohort: string, plannedAt: Date) =>
`${experimentId}:${cohort}:${plannedAt.toISOString()}`;
async function registerExpectedRuns(
store: RunStore,
experimentId: string,
cohorts: string[],
plannedAt: Date,
graceMs: number,
): Promise<void> {
for (const cohort of cohorts) {
await store.create({
id: runId(experimentId, cohort, plannedAt),
experimentId,
cohort,
plannedAt,
deadlineAt: new Date(plannedAt.getTime() + graceMs),
state: "scheduled",
});
}
}
async function compareCohort(experimentId: string, cohort: string): Promise<void> {
// Replace with the idempotent marketplace comparison and result commit.
console.log(JSON.stringify({ event: "cohort_compared", experimentId, cohort }));
}
async function runCohort(
store: RunStore,
experimentId: string,
cohort: string,
plannedAt: Date,
): Promise<void> {
const id = runId(experimentId, cohort, plannedAt);
await store.update(id, { state: "running", startedAt: new Date() });
try {
await compareCohort(experimentId, cohort);
await store.update(id, { state: "succeeded", finishedAt: new Date() });
} catch (error) {
const errorCode = error instanceof Error ? error.name : "UnknownError";
await store.update(id, { state: "failed", finishedAt: new Date(), errorCode });
throw error;
}
}
type Alert = {
kind: "job_failed" | "job_missed";
experimentId: string;
cohort: string;
plannedAt: string;
state: RunState;
errorCode?: string;
};
async function findAlerts(store: RunStore, now: Date): Promise<Alert[]> {
const alerts: Alert[] = [];
for (const run of await store.list()) {
if (run.state === "failed") {
alerts.push({
kind: "job_failed",
experimentId: run.experimentId,
cohort: run.cohort,
plannedAt: run.plannedAt.toISOString(),
state: run.state,
errorCode: run.errorCode,
});
} else if (run.state !== "succeeded" && now > run.deadlineAt) {
alerts.push({
kind: "job_missed",
experimentId: run.experimentId,
cohort: run.cohort,
plannedAt: run.plannedAt.toISOString(),
state: run.state,
});
}
}
return alerts;
}
async function main(): Promise<void> {
const store = new MemoryRunStore();
const plannedAt = new Date("2026-08-20T02:00:00.000Z");
const cohorts = ["control", "variant-a", "variant-b"];
await registerExpectedRuns(store, "checkout-copy-17", cohorts, plannedAt, 10 * 60_000);
await runCohort(store, "checkout-copy-17", "control", plannedAt);
await runCohort(store, "checkout-copy-17", "variant-a", plannedAt);
const alerts = await findAlerts(store, new Date("2026-08-20T02:11:00.000Z"));
console.log(JSON.stringify(alerts, null, 2));
}
void main();
This example deliberately leaves variant-b in scheduled. At 02:11, the monitor reports a job_missed alert for that cohort while the two completed cohorts remain quiet. That is more actionable than “cron did not ping”: the alert contains the experiment, cohort, planned instant, and last known state, without copying marketplace customer records into the monitoring path.
Keep state transitions idempotent. A scheduler retry may attempt to create the same expectation twice, and a worker retry may repeat a completion update. The in-memory example rejects a duplicate so the behavior is obvious; a durable implementation can use a unique key and an upsert policy whose conflict behavior is tested. Also decide what running past the deadline means. For paging, it is overdue. For replay, it may require a lease check before another worker takes the same cohort.
Alert grouping matters. Page once for an experiment window with the affected cohort count, then attach the individual run keys for investigation. Paging once per tenant creates noise during a shared scheduling failure. Keep the detailed evidence queryable even when the notification is grouped.
Test and deploy the detector
Test the negative space first. Freeze time, register three expected cohorts, complete two, advance beyond the deadline, and assert that exactly one missed alert appears. Then cover an explicit thrown error, a run still inside its grace window, a duplicated schedule event, a late success, and two monitor evaluations of the same failed record. The last case forces an alert-delivery policy: deduplicate by run ID and alert kind, or record notification state separately.
Deploy the monitor on a failure boundary independent from the cron worker. Its clock must be trustworthy enough for the chosen grace window, and its storage query needs an index that follows the overdue predicate. Watch the detector itself with a simple freshness signal such as its last successful evaluation time. Otherwise a silent monitor can suppress every downstream alert. Logs should carry stable fields rather than a prose-only message: experiment_id, cohort, planned_at, run_id, state, and error_code. Metrics can summarize expected, successful, failed, and overdue runs by experiment class, but avoid a tenant ID label in a metrics backend because cohort cardinality grows quickly. The ledger holds per-run evidence; metrics show the shape of the system. Treat identifiers and retention as design choices. GDPR Article 5 requires personal data to be adequate, relevant, and limited to what is necessary, so monitoring records should use a pseudonymous tenant key and omit customer payloads. Article 17 defines a right to erasure subject to its stated grounds and exceptions. Connect the ledger's retention and deletion path to the system that resolves the pseudonymous key; an alert archive should not become an accidental copy of marketplace data.
Ship in two stages. First evaluate and record would-be alerts without paging, comparing them with scheduled-run history to tune deadlines and grouping. Then enable notifications with a runbook that answers three questions: which cohorts are affected, whether work started, and whether replay is safe. Crisp evidence beats a louder alarm.
Limits and decision rule
A heartbeat is not suitable when one ping can conceal partial cohort completion. Use the ledger then. Stick with a completion heartbeat when the job is atomic, the deadline is clear, and responders only need to know that completion is absent; the extra states and retention work of a ledger would add little value.
The ledger also cannot prove that the experiment result is correct. It proves execution state according to the points where the application writes transitions. Business invariants need separate checks, such as expected cohort counts and result completeness. Queue history can replace part of the ledger when messages already represent those invariants, but tying reconstruction to queue retention and redrive semantics is a real operational commitment.
The final decision is practical: choose the smallest signal that can answer the incident question without asking the failed worker to report its own absence. For marketplace experiments, that usually means externally registered expectations at cohort granularity, a completion state written after commit, and an independent deadline monitor.
Top comments (0)