To monitor cron job silent failure, use app logs and an external heartbeat together, then keep a durable run record that survives rollback. The deciding constraint is simple: a nightly developer-tools pipeline can fail before its code emits a single log line. Logs explain work that ran; heartbeat monitoring detects work that never appeared.
TL;DR: Treat the scheduler, the job, and the monitor as three failure domains. Send a start signal, write structured progress logs, commit a success record only after the output is durable, and let an independent deadline watcher flag a missing success. During rollback, preserve the same run ID and monitor contract so the evidence remains comparable across releases.
How should an app monitor cron job silent failure?
A log search answers questions about events that reached the log path. It cannot distinguish “the scheduler never launched the process” from “the process launched but could not initialize logging” when both cases produce no event. Silence is ambiguous.
Dead quiet.
That matters for a nightly pipeline that indexes structured logs for search. Suppose the expected run is 2026-10-13, the deployment is release-184, and the input partition is 2026-10-12. Those three values belong in every event. They let an operator tell a delayed run from a duplicate and compare the release before and after rollback without guessing from timestamps.
The independent heartbeat answers a narrower question: did a named run reach a defined checkpoint before its deadline? It should not pretend to explain the failure. That is the log stream's job. Keeping those responsibilities separate makes the alert useful instead of noisy.
This is the trade-off: two signals require more plumbing and two delivery paths can disagree, but one silent path no longer erases all evidence. For a small SaaS, that is a good revenue-per-hour decision. I choose the extra signal because a missed nightly index can leave yesterday's developer logs looking current, while the implementation stays small enough to test in one release. Build the contract once, keep it boring, and return to shipping features weekly.
The smallest useful run contract is plain data.
The run record needs stable identity, explicit state, and enough deployment context to support rollback. Avoid using a random request ID as the only key. A retry then looks like unrelated work.
type RunState = "started" | "succeeded" | "failed";
type PipelineRun = {
job: "nightly-log-index";
scheduledFor: string;
attempt: number;
release: string;
inputPartition: string;
state: RunState;
observedAt: string;
recordsWritten?: number;
errorCode?: string;
};
const runKey = (run: PipelineRun): string =>
`${run.job}:${run.scheduledFor}`;
The logical key stays fixed across retries. attempt increases. release records which code produced the event. Do not mark success when processing ends in memory; mark it after the searchable output and the durable run record have both committed according to the pipeline's storage contract.
Structured events are a better fit than prose for this path because fields can be filtered and grouped. An analytical store can support that search pattern, but storage choice does not change the signal model. The schema is the asset. Outsource or replace the undifferentiated storage layer when needed without rewriting the job's meaning.
Build the two-signal path
The job emits local evidence throughout execution and reports milestones to a monitor outside the job process. The endpoint below is deliberately generic. Authentication, retry limits, and timeouts belong in the shared client, not scattered through pipeline logic.
type Milestone = "start" | "success" | "failure";
type Heartbeat = {
job: string;
scheduledFor: string;
attempt: number;
release: string;
milestone: Milestone;
observedAt: string;
};
async function sendHeartbeat(event: Heartbeat): Promise<void> {
const response = await fetch(process.env.HEARTBEAT_URL!, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${process.env.HEARTBEAT_TOKEN!}`,
"idempotency-key": `${event.job}:${event.scheduledFor}:${event.milestone}`,
},
body: JSON.stringify(event),
signal: AbortSignal.timeout(5_000),
});
if (!response.ok) {
throw new Error(`heartbeat_rejected:${response.status}`);
}
}
A production caller should decide explicitly whether a failed heartbeat fails the pipeline. There is no universal answer. For the start signal, recording the error and continuing may preserve useful work. For the success signal, silently continuing is dangerous because the watcher will report a missing run even though output exists. A bounded retry plus a durable outbox makes that boundary clearer: the job commits its success and an intent to notify in one storage transaction, then a separate sender delivers the heartbeat. If the receiver returns 429, the sender should honor a valid Retry-After value, add bounded jitter, and retry from that outbox; the idempotency key prevents the same milestone from becoming multiple logical notifications when the receiver supports that contract.
Keep the deadline outside the process. A timer created by the job cannot report that the job was never created. The watcher should evaluate a schedule in a named time zone, allow a documented grace window, and alert on a missing terminal state. UTC removes daylight-saving ambiguity for a globally operated pipeline; if business rules require Europe or US local time, use an IANA time-zone identifier and test both clock transitions.
One alert is enough. Page on a missed deadline, then use fields such as runKey, attempt, release, and errorCode to investigate. Alerting on every intermediate log line converts a simple availability check into an expensive rules engine.
The limitation is real: heartbeat monitoring proves that a checkpoint was reported, not that every output row is correct. It is also a poor fit for continuously running consumers, where lag or freshness is the better signal. For those workloads, monitor the age of the last durable output and queue progress instead of inventing a nightly deadline.
Make rollback preserve the evidence
Rollback safety is mostly schema discipline. The previous release and the new release must agree on the run key, terminal states, and deadline semantics. Add fields compatibly. Do not rename scheduledFor in the same deployment that changes the scheduler, because a failed rollback could leave two versions writing records the other cannot interpret.
Before release, test four cases: the scheduler does not launch the job; startup fails before application logging; processing fails after start; and output commits but heartbeat delivery is delayed. Also test a retry after rollback. The expected result is not merely “an alert fired.” The operator must be able to identify one logical run, see every attempt, and determine whether searchable output committed.
This is where I would spend the extra hour. A polished dashboard can wait. A deterministic failure test protects the one-person on-call rotation every night.
Evidence first.
Use idempotent writes keyed by the logical run and the source record identity. A retry should converge on the same output, not append a second copy. The heartbeat does not provide that guarantee; it only reports progress. Correctness stays in the pipeline and its storage transaction.
What changes at larger scale
At ten nightly jobs, a small schedule registry and one watcher may be enough. At hundreds, centralize ownership, deadline policy, schema versions, and maintenance windows. Keep job teams responsible for idempotency and meaningful error codes.
The signal model still holds. Logs explain. Heartbeats detect absence. Durable run records connect both to deployment and rollback history.
More volume may justify retention tiers, partitioning, and a dedicated analytical store for structured event queries. Measure ingestion delay and query needs before changing storage. Price is not the primary decision; recovery time, operational load, data portability, and the ability to replay evidence across a rollback matter more.
Top comments (0)