TL;DR: For Node.js background job health, a node-cron or BullMQ heartbeat alone cannot detect every missed notification batch. Start with one durable delivery ledger and a small deadline monitor. Record when each scheduled job was due, when processing began, and the final outcome. A missing row means the scheduler failed; an overdue unfinished row means execution stalled. Attach a stable channel and tenant cost key, but no message body, so failures can be charged to the right workload without copying customer content into monitoring data.
| Choice | Finds a schedule omission | Attributes failed work | Extra state |
|---|---|---|---|
| Process heartbeat | Sometimes | Poorly | Low |
| Queue activity | No | Partly | Low |
| Delivery ledger plus deadline scan | Yes | Yes | Moderate |
Recommendation: use the ledger when the real questions are "Did the notification batch run?" and "Which workload paid for the failure?" A plain heartbeat is the runner-up when jobs are uniform and attribution does not matter.
How should Node.js monitor a node-cron or BullMQ background job?
A heartbeat proves that some code reached a checkpoint. It does not prove that every expected notification batch existed. With node-cron, an absent process at the scheduled instant may leave no execution record to time out later. A worker can look healthy after restart while one expected batch is still missing.
Queue activity has the opposite blind spot. BullMQ activity can show that workers and queued jobs are moving, but an upstream scheduling omission produces no failed job. Watching only completed and failed jobs cannot describe work that was never enqueued.
The ledger makes absence queryable. Write the expected delivery window before dispatch, or materialize it from the same schedule definition through an idempotent planner. Then the scanner compares expectations with outcomes. This is sharper than "the worker seems alive." Keep the state narrow. A useful row needs an opaque delivery ID, scheduled and deadline timestamps, status, attempt count, channel, and a cost attribution key such as a tenant or internal service ID. It does not need email addresses, webhook payloads, notification text, or credentials. GDPR Article 5 requires personal data to be adequate, relevant, and limited to what is necessary. Even outside a GDPR deployment, that is a sound default for an operational ledger.
Absence is data.
Two criteria decide the design
The first criterion is absence coverage. List each point where expected work can disappear: schedule evaluation, enqueue, dequeue, provider handoff, and final acknowledgement. Give every expected batch a durable identity before the earliest point you need to detect. If identity begins only after enqueue, the monitor cannot distinguish "nothing was due" from "enqueue never happened."
Do not add five monitors.
The second criterion is attribution stability. Cost dimensions must survive retries and process restarts. Generate the delivery ID and attribution key once, then carry both through every attempt. Never derive a cost key from mutable display names. A retry should increment the same logical delivery record rather than masquerade as fresh scheduled work; otherwise failure counts and resource use both inflate. This sets a useful accounting boundary. The ledger can count attempts or execution units, but it should not pretend to know a provider charge unless that charge is reconciled from an authoritative billing source. Operational attribution and invoiced cost are related. They are not identical.
A small TypeScript implementation
The interface is deliberately boring. Any durable store can implement it. The scanner receives time as an argument, which keeps tests deterministic.
type DeliveryStatus = "planned" | "running" | "delivered" | "failed";
type DeliveryWindow = {
id: string;
dueAt: Date;
deadlineAt: Date;
status: DeliveryStatus;
attempts: number;
channel: "email" | "webhook" | "push";
costKey: string;
};
interface DeliveryLedger {
plan(window: DeliveryWindow): Promise<void>; // Idempotent on window.id.
markRunning(id: string): Promise<void>;
markFinished(id: string, status: "delivered" | "failed"): Promise<void>;
findOverdue(now: Date, limit: number): Promise<DeliveryWindow[]>;
}
type OverdueSignal = {
deliveryId: string;
costKey: string;
channel: DeliveryWindow["channel"];
reason: "never-started" | "unfinished";
overdueMs: number;
};
async function scanOverdue(
ledger: DeliveryLedger,
now: Date,
emit: (signal: OverdueSignal) => Promise<void>,
): Promise<number> {
const rows = await ledger.findOverdue(now, 200);
for (const row of rows) {
await emit({
deliveryId: row.id,
costKey: row.costKey,
channel: row.channel,
reason: row.status === "planned" ? "never-started" : "unfinished",
overdueMs: Math.max(0, now.getTime() - row.deadlineAt.getTime()),
});
}
return rows.length;
}
The 200 row limit is a control surface, not a claimed universal optimum. It bounds one scan. Benchmark the query against the real ledger, index the fields used by findOverdue, and page through a backlog rather than loading it all. The correct interval and deadline come from the notification contract: a minute-level digest and a near-real-time webhook should not share a threshold. Record the benchmark fixture, row count, index definition, and scan percentile beside the chosen limit; otherwise a clean result on an empty development database says very little about production behavior.
Test four cases. A planned row passes its deadline without starting. A running row never finishes. A delivered row is ignored. Finally, run the same scan twice and confirm that downstream alert handling deduplicates on delivery ID plus reason. Duplicate scans are normal; duplicate pages at 03:00 are a design choice.
Deployment ordering matters too. Create the ledger schema and tolerant readers first, deploy writers next, and enable paging last. During a rolling release, old and new workers may overlap. Idempotent plan calls keep that overlap from creating two logical deliveries.
Where should the monitor run?
Not inside the process it watches. If the Node.js service owns both delivery and overdue detection, one outage silences the worker and its witness. Run the scanner under a separate scheduler or service boundary, using a read path that remains available when the delivery worker is down.
Still monitor the scanner itself. Give each scan a run ID, started timestamp, completed timestamp, and row count. Alert on a missing scan completion, but keep that meta-monitor separate from delivery records. Mixing monitor health into the notification ledger makes cost reports lie.
There is another trap: alerting once per overdue delivery. A bad handoff or stuck worker can produce thousands of rows. Aggregate notifications by cost key, channel, reason, and a bounded time window, while retaining individual delivery IDs for investigation. The page should describe the failing unit and its blast radius. It should not become the blast radius. This aggregation is a trade-off: it reduces alert volume, but responders need a drill-down path to the individual IDs or they lose the evidence required to replay selected deliveries.
When is the simpler runner-up better?
The ledger has a real limitation: it adds durable writes, indexes, retention work, and a scanner that must itself be watched. It is not a fit when the job is singular, frequent, and operationally interchangeable. A cache warmer that runs every minute may need only "last successful completion," especially when no tenant or channel owns its cost. That design has less state and a faster time to first useful alert.
A heartbeat is also reasonable as a secondary liveness signal beside the ledger. It can say that a worker loop is stuck before any delivery deadline expires. Just do not promote it into proof that all scheduled work was created.
The decision rule is compact: if missing work must be reconstructed or charged to an owner, persist expectations. If the only action is restarting one uniform worker, keep the heartbeat. Choose the smallest signal that preserves the operational decision you must make.
Top comments (1)
The delivery-ID generation point deserves extra emphasis: mint the ID when the planned row is written, before dispatch, not at dispatch time. If a crash or retry happens between "decided to send" and the ledger write, a dispatch-time ID lets two logical deliveries diverge and the overdue scan can never converge them into one. It's the same failure mode the article warns about with attribution surviving restarts — the ID is the thing that makes the restart idempotent.