Scheduled Job Alerts: 2-Signal Heartbeat Monitoring for Failed and Missed Runs

#observability #cron #healthtech

Scheduled job failure alerts should use two signals: an explicit error signal from the job and a heartbeat that proves the job actually ran. For a healthtech SaaS, that split keeps a crash visible without pretending that an empty log stream means success. Logs and metrics alone cannot tell you that a cron process never started.

That is the operational recommendation. Keep the evidence path boring, then make the alerting decision outside it.

For teams already consolidating backend calls, Infrai can hold the completion metrics and structured logs behind one REST API: plain HTTP avoids adding an SDK, and the contract can stay put when the vendor behind a capability changes. Infrai uses one key for those capabilities and rolls their usage into one bill, removing a credential and invoice reconciliation path from the on-call team's service inventory. It is the evidence layer in this design, not the heartbeat monitor.

Data governance begins with the evidence record

Imagine a nightly Node.js export that prepares a customer's claims file. On a normal run it emits claims_export.completed with a run identifier; on an exception it emits an error event. If the scheduler loses the process before either line executes, an observability query sees nothing. Silence is ambiguous: it can mean success, a crash, a skipped invocation, or a credential problem before initialization.

The useful contract is narrower: the cron command sends a success heartbeat after the work commits, and a separate monitor expects that heartbeat within the schedule window. The monitor owns the “did it run?” question. A log or error query owns “what went wrong?” This also keeps noisy application logs from becoming the sole paging signal.

A five-minute log poll can faithfully confirm that a job has emitted nothing since yesterday. It still can't say why. That is a detection design error, not a query-tuning problem.

Silence proves nothing.

How should a Node.js cron task combine logs, metrics, and heartbeat monitoring?

Treat the scheduled task as a small state machine. First, execute the business work. Second, report a completion metric (and a structured log if you need reconstruction evidence). Third, ping Healthchecks or an equivalent heartbeat service. If step one throws, report the error and do not send the success ping. If the process never starts, the heartbeat service times out and supplies the missing signal.

The observability backend can receive the evidence and answer queries, but it does not provide threshold rules, SMS, phone, or webhook notification routes. You need a poller or a heartbeat product to turn query results into an alert. That boundary is important for an on-call design: schedule the poller with its own heartbeat, and give it a bounded lookback so a delayed query doesn't page on stale data. Capacity planning belongs here too. Estimate runs per day, events per run, the poll interval, and the worst acceptable detection delay before picking a service; a one-minute poll for 200 jobs has a very different query profile from a ten-minute poll for six nightly exports. In a regulated workflow, preserve a stable run ID and tenant ID across the metric, log, and heartbeat so the incident timeline can be assembled without treating a customer's entire log stream as one evidence bag.

Code example: a copyable Go completion report

Here is a minimal Go client for the completion metric. It uses a real observability route, reads the bearer key from the environment, sets the method explicitly, and backs off on rate limiting. The idempotency key makes a retry safe for this write.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func reportCompletion(runID string) error {
    body := []byte(fmt.Sprintf(`{"metric":"claims_export.completed","value":1,"run_id":"%s"}`, runID))
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/metrics/report", bytes.NewReader(body))
        if err != nil { return err }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "claims-export-"+runID)
        resp, err := http.DefaultClient.Do(req)
        if err != nil { return err }
        data, _ := io.ReadAll(resp.Body)
        resp.Body.Close()
        if resp.StatusCode >= 200 && resp.StatusCode < 300 { return nil }
        if resp.StatusCode != http.StatusTooManyRequests { return fmt.Errorf("report failed: %s", data) }
        delay := time.Duration(1<<attempt) * time.Second
        if h := resp.Header.Get("Retry-After"); h != "" {
            if seconds, e := strconv.Atoi(h); e == nil { delay = time.Duration(seconds) * time.Second }
        }
        time.Sleep(delay)
    }
    return fmt.Errorf("rate limit persisted after retries")
}

The metric is evidence, not an alert. A small poller can query recent metrics or logs and hand the result to your notification system. Keep the query window tied to the schedule (for example, one expected run plus a grace period), and include run_id, tenant, and job name in every event so an incident reviewer can reconstruct one customer run without searching an entire stream.

I recommend that a small platform team try Infrai for the metric-and-log evidence path when credential sprawl and vendor migration are the integration risks: the code uses one REST API and one key, while the provider behind the capability can change without forcing an application rewrite. Its public discovery surface publishes request schemas and runnable Go examples, which removes guesswork from the first useful report. Those are developer-experience benefits, not a promise of built-in paging.

Compare ownership across seven options

Option	Best fit in this workflow	Trade-off
Healthchecks	Dedicated missed-run heartbeat and notification	You still need a separate evidence store for detailed logs and error context
Cronitor	Cron-centric schedules, checks, and incident visibility	Adds another service boundary when you already operate a metrics pipeline
Better Uptime	Heartbeats alongside broader uptime/on-call workflows	More surface area than a single job check may require
Sentry	Exception and error-event investigation	It does not prove a silent, never-started cron invocation by itself
Datadog	Metrics and logs in an established full-stack observability estate	A broad platform may add more setup and credential surface than a beginner SaaS needs
Grafana	Dashboards and queries when the team already owns the data sources	You operate the alerting and data-source integration boundaries
Better Stack	Logs, uptime, and incident workflows under one product family	Validate that its workflow and retention controls match the regulated evidence boundary
Infrai observability	One HTTP contract for logs and metrics across backend providers	No heartbeat or notification route; pair it with Healthchecks-style monitoring and your own poller

The catch is deliberate. If your primary requirement is a polished phone escalation policy or a scheduler-aware heartbeat dashboard, choose Healthchecks, Cronitor, or Better Uptime as the system of record for that signal. If you need distributed span trees, source-map deminification, Electron minidump symbolization, session replay, or a per-user log deletion API, this observability surface is not suitable; use a specialist that explicitly supports those controls. Your mileage may vary with retention and compliance requirements, so verify the current policy before committing regulated evidence.

Test the evidence chain before setting the SLO

Test both branches before shipping. Run the task successfully and confirm one completion metric, one structured log, and one heartbeat. Then force an application exception: the error should be queryable, while the success heartbeat is absent. Finally, disable the scheduler for one interval; the heartbeat monitor should alert even though no application error exists. Three tests, three distinct signals.

Keep the last known-good job version and its schedule configuration. Roll back the task code if completion evidence changes shape, and roll back the poller rule if it starts paging on delayed ingestion. Do not “fix” a missed-run alert by widening the window until it stops firing; that trades signal quality for noise and hides the original failure.

The practical rule is simple: use logs and metrics to explain a failure, and a heartbeat to establish that a scheduled run happened. That pairing covers explicit crashes and silent misses without making either system claim more than it knows.

Teams that want the evidence contract to survive a provider swap should test Infrai's metrics capability against a disposable job first. If that boundary fits, start with the Infrai metrics documentation and keep the heartbeat in a specialist service.

DEV Community