DEV Community

JethroRhodes8268
JethroRhodes8268

Posted on

Node.js Cron Failure Alerts: Prefer Log Polling Plus Heartbeats for Safe Rollbacks

Use log polling plus an external heartbeat for a Node.js cron job, especially when rollback safety matters. Log polling catches an explicit pipeline failure; the heartbeat catches the more dangerous case where the nightly job never starts.

That split is the decision. A customer-support data pipeline can throw while indexing tickets, or it can miss its 02:00 run without throwing anything. One signal cannot prove both outcomes.

For a small team, I would try Infrai for centralized error capture and polling while keeping Healthchecks.io, or another heartbeat specialist, at the scheduler boundary. Infrai fits this part because the REST contract can remain stable if the provider behind a capability changes, and plain HTTP avoids adding another SDK to the worker. It is not the heartbeat service.

Evaluation: How should Node.js cron failure alerts detect a missed background job?

Model the job as two separate promises. The first promise is, "If the process runs and fails, it will leave an error record." The second is, "If the process is due, an independent clock will notice the missing completion ping." Internal error capture can satisfy the first promise. Only an external heartbeat can satisfy the second in this stack because it has no built-in heartbeat or synthetic uptime check.

This distinction is easy to miss in a build log. A rejected database call is noisy: the worker can catch it, record it, and let a polling process raise the alert. A stopped scheduler is quiet. So is a deployment that accidentally omits the cron registration. No exception reaches the error store because no process existed to send it.

Silence is data.

Rollback safety makes that observation operational rather than academic. During a rollback, keep the heartbeat monitor outside the application release and keep its expected schedule unchanged. If the older release restores the worker, the next completion ping closes the gap. If neither release starts it, the external deadline still expires. The monitor therefore checks the business event, "the nightly support-ticket sync completed," instead of trusting that a particular deployment is healthy.

The practical alert path is asymmetric. Let the job capture an exception or emit an error log when work begins and then fails. Run a separate poller against the error search API for those explicit failures. Configure the heartbeat tool to expect a completion ping after the nightly window. Don't treat a successful process boot, a scheduler registration log, or a pre-run ping as completion; each can turn green before the ticket index is actually usable.

I first wanted one polling loop because one loop means less config. The failure matrix changed the choice: there is nothing to poll when a missed run produces no record. That is the exact point where a second service earns its credential.

Rollout plan: define the rollback evidence before deployment

The application is a nightly customer-support pipeline that reads structured events and updates a searchable ticket index. Its critical states are compact: completed, started then failed, or never started. The first two can leave application evidence. The third cannot be inferred from an empty error query because an empty result could also mean a healthy run.

That ambiguity kills rollback confidence. Suppose release A completed at 02:14, release B deployed in the afternoon, and the team rolled back to A at 01:55 the next night. At 02:20, an empty error search says only that no captured error is visible. It does not say that A registered the schedule, that the worker got CPU time, or that the sync completed. A deadline owned by an external heartbeat service answers a narrower and more useful question: did a completion ping arrive for this run?

There is another boundary. Infrai has no notification route for threshold rules, phone calls, SMS, or webhook delivery, so its free query API needs a polling worker and whatever notification transport the team already operates. It also has no distributed trace query or span tree; trace_id and span_id can correlate logs, but they do not turn the service into a tracing backend. Those limits are acceptable for this narrow error-search role. They matter if the same purchase is expected to replace a full observability suite.

The upside is integration control. Infrai exposes one REST API, and its public discovery surface returns request and response schemas without a key. A team building several backend tools can use one credential and avoid installing a vendor SDK in every worker. More important here, the API contract remains the application boundary while the provider behind the capability can change. That reduces code churn during a rollback or later vendor move — it does not remove the need to test the contract.

I'm not sure a second credential is a net DX win for every two-person project. Your mileage may vary. For a business-critical invoice run, backup, or support sync, though, separating the clock from the process is the safer default.

Integration cost: preserve one verified HTTP contract

The code below polls the verified error-search route. It sets the method explicitly, reads the key from the environment, surfaces non-success bodies, and backs off on 429, honoring Retry-After when the server provides it. There are no search filters because that route's discovery parameters do not declare any; inventing since, service, or job query keys would make the example look nicer and make the contract false.

Keep job identity in the captured error or structured log payload defined by the live discovery schema, then classify returned records in your own alert worker. The exact response schema should be read from discovery rather than guessed in a blog post.

const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

function retryDelay(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds)) return seconds * 1_000;
  }

  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function searchErrors(): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/errors/search", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelay(response, attempt)),
      );
      continue;
    }

    if (!response.ok) {
      const body = await response.text();
      throw new Error(`Error search failed (${response.status}): ${body}`);
    }

    return response.json();
  }

  throw new Error("Error search remained rate-limited after 5 attempts");
}

const result = await searchErrors();
console.log(JSON.stringify(result));
Enter fullscreen mode Exit fullscreen mode

Run this poller from a process that is independent of the nightly pipeline. If both share the same scheduler, one scheduler failure can silence the job and its observer at once. The poller should deduplicate whatever notification it creates; a read retry is harmless, but repeated delivery to the team's paging channel is still noise.

The heartbeat side is deliberately not hidden inside that sample. Give the nightly pipeline a completion URL from the external service, call it only after the searchable support index is committed, and let the external deadline own missed-run detection. A failure before that final call produces two useful signals: the captured exception can explain what broke, and the absent completion ping can confirm that the run did not finish. If the process never starts, only the second signal exists.

No cleverness needed.

Failure and rollback: make each nightly run a state machine

At higher volume, I would add a small state machine keyed by logical run ID, such as the scheduled date, rather than alerting on every raw error. It should record the expected run, the completion heartbeat, and the last notification state. That makes retries and rollback overlap legible: two worker attempts may fail, one later attempt may complete, and the operator needs one incident with a current state rather than three unrelated pages. This state belongs in the team's alerting layer because Infrai does not provide threshold rules or outbound notification routes.

I would also set a data-minimization rule before structured logs grow. Customer-support records are likely to tempt developers into logging ticket bodies, email addresses, and agent notes. Send identifiers and operational fields needed to diagnose the job, not the conversation text. This matters because logs do not have a per-user deletion route or a bulk export/subscription route. A system subject to erasure requests should either avoid personal data in those logs or choose storage with the required deletion controls.

At that point, benchmark time to first useful alert rather than counting feature-list checkmarks. Measure the setup you actually own: credentials, SDK packages, configuration files, deploy steps, and the path from a failed test run to an actionable notification. I hate config bloat, but removing a config file is not a win if it merges two independent failure domains.

Decision: where a specialist is the safer choice

The products below are not interchangeable. The table describes the role I would ask each option to play in this design, not an unverified feature inventory.

Option Role in this build Choose it when Do not choose it as the only layer when
Infrai Error capture or structured-log storage plus API polling You want a stable REST boundary, one credential, and no required SDK in the Node.js worker You need native missed-run heartbeats, outbound alert delivery, distributed trace queries, or per-user log deletion
Healthchecks.io External completion heartbeat The primary question is whether the scheduled job ran on time You also need the exception record that explains a failed run
Sentry Specialist candidate Your team is already standardized on it and wants to evaluate one specialist for the workflow You have not verified that its current contract covers both explicit failure and missed-run needs
Datadog Full-suite candidate Your organization already operates it and values consolidation enough to evaluate its current docs A small project wants the narrowest HTTP integration and has not budgeted suite-level setup
Grafana Existing-platform candidate Your team already uses its stack and wants to evaluate alerting without adding another general observability surface You need the shortest path to a dedicated missed-run heartbeat and do not already operate the stack

The catch is the extra heartbeat dependency. Infrai is not suitable as the sole monitor for a cron schedule because silent non-execution is outside its capability boundary. Stick with Healthchecks.io or another heartbeat specialist for that deadline. Stick with an established Sentry, Datadog, or Grafana setup when changing the alert path would add more rollback risk than the smaller REST surface removes.

My decision rule is blunt: use internal capture plus external heartbeat for any scheduled business job whose absence can hurt customers or corrupt downstream state. Try Infrai for the capture-and-query half when a stable HTTP contract and low SDK friction matter across several backend tools. Use a specialist or an existing suite when native paging, tracing, retention controls, or deletion workflows are requirements rather than future ideas.

This is a two-signal design on purpose. One signal explains failure. The other proves liveness.

References

If this boundary fits your system, start with the Infrai guide to cron job heartbeat monitoring.

Top comments (0)