DEV Community

ThalynRift3485
ThalynRift3485

Posted on

Silent Failure in SaaS: How Cron Heartbeats Complement Error Tracking and Healthchecks

#observability #monitoring #cron #saas

Short answer: error tracking, uptime monitoring, and healthchecks observe different edges of a system. A cron heartbeat covers the missing edge: whether scheduled work actually finished. For a small SaaS, use all three signals with explicit ownership, because a green HTTP endpoint and an empty error inbox do not prove that a background job ran.

That distinction matters most for silent failure. The process can be alive, the URL can return 200, and the job can still be absent, stuck, or producing no useful side effect. The fix is architectural: define the claim each signal is allowed to make, then alert on the absence of the signal that proves the claim.

What do error tracking, uptime monitoring, cron heartbeats, and healthchecks each prove?

Error tracking is an exception pipeline. Code captures a thrown error, adds context such as a release or request id, and sends an event to a collector. A logging framework can route that event through an appender or handler. This is excellent for finding a failing code path, but it depends on an error being raised and reported. A swallowed exception, a skipped branch, or a job that never started can leave the tracker quiet.

Uptime monitoring is an external pull. A checker requests a public URL and records reachability, status, and usually latency. It can catch DNS failure, a refused connection, or a bad status. It cannot tell whether a successful response contains the right invoice total, or whether last night's queue was consumed.

Healthchecks are process or dependency assertions, usually exposed as a liveness or readiness endpoint. An orchestrator can restart an unhealthy instance or stop routing traffic to it. That is useful for serving traffic. It is not a schedule ledger: a healthy worker can miss a daily run indefinitely.

A cron heartbeat reverses the direction. The job emits a start or completion ping, and an independent monitor alerts when the expected ping is late. Absence becomes data. The heartbeat should be sent only after the side effect it represents is confirmed, not merely after a request was accepted by a queue.

Signal Primary question It does not prove
Error tracking What exception reached the reporting path? That work started or completed
Uptime monitoring Can an outside checker reach this endpoint? That the response is correct
Healthcheck Can this instance serve or accept work? That a scheduled run happened
Cron heartbeat Did this run emit its expected completion? That every result is correct

How should a beginner SaaS model silent failure?

Start with a failure matrix, not a dashboard. For each job, write down its schedule, owner, expected duration, and the observable event that counts as success. “The API returned 200” is rarely the right event for an asynchronous workflow. “The provider acknowledged 412 deliveries” is closer, because it names a side effect. Consider a nightly export that reads rows, publishes one message, and lets a worker write an object to storage. The exporter can report success when the broker accepts the message; the worker can be healthy while listening to the wrong subscription; and storage can accept a zero-byte object while every component returns a normal status. An error tracker sees no exception in that chain. An uptime probe sees no bad response. A liveness check sees a process that is ready. Only a completion signal tied to the object size, row count, or checksum tells you that the promised effect occurred. That is why the definition belongs in the job contract, alongside the retry policy and owner, instead of being inferred later from a generic green dashboard.

No dashboard fixes that.

Then separate four states: never started, started but hung, failed loudly, and finished with an invalid result. Error tracking is strongest in the third state. A start and finish heartbeat cover the first two. Assertions and domain metrics cover the fourth. One signal cannot substitute for the others.

The monitor must live outside the failure domain it observes. If the scheduler, worker, and checker share one host or credential, a single outage can erase the evidence and the alert. Keep the heartbeat payload small and stable: job name, run id, state, duration, and a count of confirmed effects. Do not put customer data in it.

There is a boring operational detail that saves time later: keep the job registry beside the code. A deleted job should delete its alert definition as part of deployment. Hand-maintained checks drift, and stale red rows train a team to ignore real pages.

What does a minimal TypeScript heartbeat wrapper look like?

The wrapper below uses a generic HTTPS endpoint. Its contract is intentionally narrow: work returns confirmed side effects, and the completion ping follows that confirmation.

type Heartbeat = {
  job: string;
  runId: string;
  state: "start" | "ok" | "fail";
  effects?: number;
  durationMs?: number;
};

const heartbeatUrl = process.env.HEARTBEAT_URL;

async function sendHeartbeat(event: Heartbeat): Promise<void> {
  if (!heartbeatUrl) throw new Error("HEARTBEAT_URL is not configured");
  const response = await fetch(heartbeatUrl, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify(event),
  });
  if (!response.ok) throw new Error(`heartbeat rejected: ${response.status}`);
}

export async function runWithHeartbeat(
  job: string,
  work: () => Promise<number>,
): Promise<void> {
  const runId = `${job}-${Date.now()}`;
  const started = Date.now();
  await sendHeartbeat({ job, runId, state: "start" });
  try {
    const effects = await work();
    if (effects < 1) throw new Error("job completed without a confirmed effect");
    await sendHeartbeat({
      job,
      runId,
      state: "ok",
      effects,
      durationMs: Date.now() - started,
    });
  } catch (error) {
    await sendHeartbeat({
      job,
      runId,
      state: "fail",
      durationMs: Date.now() - started,
    });
    throw error;
  }
}
Enter fullscreen mode Exit fullscreen mode

The failure path deliberately rethrows. The heartbeat records the missed completion, while the error tracker can retain the stack and request context. If sending the failure ping is itself unreliable, log that fact locally and let the monitor's missing completion alert remain the backstop; do not claim the run succeeded.

Set the grace period from reality: the schedule interval plus a normal long run and delivery delay. A daily job that normally needs ten minutes should not page at minute eleven if the queue provider has a documented delay. Conversely, a week-long grace period turns a silent failure into a support ticket.

What changes at scale, and where are heartbeats the wrong tool?

At a handful of jobs, a static registry and one alert per job are enough. With dozens, you can't rely on memory: add ownership, escalation, deduplication, and a run identifier that appears in logs, traces, and metrics. Test the instrumentation itself: invoke each job with a fake transport and assert that an ok event includes a non-zero effect count. This catches coverage lost during refactors.

The catch is that a heartbeat measures the wrapper, not business correctness. A job can write the wrong rows and still send a green ping. Add domain assertions, reconciliation queries, and rate-based metrics for that case. Heartbeats are also a poor fit for sub-minute loops; a counter and a rate alert carry more information with less alert noise.

Stick with external uptime checks for user-facing paths, and use healthchecks for instance lifecycle decisions. Use error tracking for diagnosis. Choose a heartbeat when the important fact is temporal: a run was expected, and its absence needs a page.

I'm not sure one universal grace-period formula exists; queue latency, retries, and business deadlines change the answer. Measure those distributions in your own system and document the assumption next to the alert rule.

References

Top comments (0)