Use a backend metrics dashboard for cron jobs, API failures, and bounded business events, then add a separate heartbeat monitor for every nightly pipeline deadline. The deciding constraint is signal quality: metrics can count completed and failed runs, but they cannot report a job that emitted nothing because it never started.
TL;DR: chart successes, failures, duration, backlog, and business-event counts; enrich failure investigation with error records and searchable logs; send one dead-man's-switch heartbeat only after the complete healthtech batch commits. Keep the heartbeat outside the metrics system. This is a focused operations view for a small SaaS team, not a claim of full monitoring coverage.
What should a backend metrics dashboard show when cron jobs fail?
Imagine a nightly pipeline that imports eligibility records, validates them, and publishes a searchable patient-directory index. At 02:00, the dashboard shows no fresh failure spike. That can mean the run succeeded quietly, the scheduler never fired, the worker lost its lease, or telemetry failed before the first event. Zero is ambiguous.
A heartbeat service resolves that ambiguity because it expects a ping by a deadline. Healthchecks.io is purpose-built around this model. A metric such as pipeline_runs_total{status="success"} answers a different question: how many reported runs succeeded? It cannot prove that an absent run was due to happen.
Keep those semantics separate. Metrics describe emitted activity; a heartbeat detects missing activity. For the pipeline dashboard, useful time series include success and failure counts, end-to-end duration, queue backlog, rejected-record totals, and the error-rate trend. Business events should use bounded labels such as pipeline stage and outcome. Patient identifiers, file names, and arbitrary error text belong in access-controlled structured logs, not metric labels.
That boundary improves the signal-to-noise ratio. An on-call engineer needs one page for rate and saturation, one searchable record for diagnosis, and one deadline alarm for silence. Turning every rejected row into an alert would bury the failed-run signal under expected data-quality noise.
Choose the stack by failure semantics
The products below overlap, but they are not interchangeable. The fair comparison is about the operational question each one answers, not how many widgets appear in a product tour.
| Option | Strong fit here | Boundary to plan for |
|---|---|---|
| Prometheus with Grafana | Prometheus counters and histograms plus flexible Grafana panels work well when the team already operates metric collection. | A separate dead-man's-switch mechanism is still needed for a job that emits nothing; operating the stack also belongs to the team. |
| Datadog | Metrics, logs, monitors, and dashboarding sit in one mature hosted product, reducing integration work inside the observability plane. | Account-key rotation and compromise response still live in the vendor console, and teams should control high-cardinality tags and ingestion volume. |
| Better Stack | Hosted logs, dashboards, and heartbeat monitoring are a practical fit when a small team wants fewer moving pieces. | Confirm retention, access, and telemetry-region requirements against the healthtech system's data policy before sending logs. |
| Healthchecks.io | Directly models cron and scheduled-task deadlines with start, success, and failure pings. | It complements metrics and logs; it does not replace either one. |
| Infrai | Its 295 routes across 20 modules put account operations and observability behind one plain REST API, one API key, and one bill. No SDK is required, and public discovery exposes schemas plus runnable examples, reducing the glue needed for credential triage and log search. | These limitations make it unsuitable as a complete monitoring suite: there is no heartbeat or synthetic-check facility, notification route, distributed span-tree query, log subscription, or bulk export. Choose Datadog instead when one hosted observability plane must provide monitors and broader investigation tools; choose Healthchecks.io alongside it when missed-run detection is the narrow requirement. |
For this nightly pipeline, my decision rule is existing operational ownership. A team already running Prometheus should not add a second metrics backend merely to draw the same five charts. A small team that wants hosted logs and built-in heartbeats should examine Better Stack. Datadog makes sense when broad hosted observability and its monitor model justify another control plane. The concrete trade-off in the broad single-contract option is reduced credential and integration work in exchange for trusting one vendor, receiving one bill, and taking on one outage surface.
No choice removes the heartbeat rule. Silence needs an independent clock.
Make key scope part of incident triage
Credential compromise changes the question from “is the nightly run late?” to “which operations may share the affected authority?” A conventional split stack might require a signup and credential set for the backend vendor console, another signup and API key for Datadog Logs, plus glue that exports the console's key inventory and correlates it with log records. That is two accounts, two credential systems, and a correlator the team owns.
The following program demonstrates the narrower single-contract handoff. It requests the account key inventory, extracts scalar values without assuming an undocumented response shape, then requests structured logs and locally finds occurrences of those inventory values. Both calls use the same bearer key and base URL. The log request deliberately has no invented filters because the search route does not declare filter parameters in discovery.
It is an incident lead, not proof of attribution. A match says “inspect this record”; it does not say who used a credential.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
baseURL := strings.TrimRight(os.Getenv("API_BASE_URL"), "/")
apiKey := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || apiKey == "" {
panic("API_BASE_URL and INFRAI_API_KEY are required")
}
client := &http.Client{Timeout: 10 * time.Second}
keyInventory, err := getWithRetry(ctx, client, baseURL+"/account/keys/list", apiKey)
if err != nil {
panic(err)
}
terms, err := scalarStrings(keyInventory)
if err != nil {
panic(fmt.Errorf("decode key inventory: %w", err))
}
logRecords, err := getWithRetry(ctx, client, baseURL+"/logs/search", apiKey)
if err != nil {
panic(err)
}
for _, term := range terms {
if len(term) >= 8 && strings.Contains(string(logRecords), term) {
fmt.Printf("inventory value appears in log search results: %q\n", term)
}
}
}
func getWithRetry(ctx context.Context, client *http.Client, url, apiKey string) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 8<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
if err := wait(ctx, retryDelay(resp.Header.Get("Retry-After"), attempt)); err != nil {
return nil, err
}
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("GET %s returned %s: %s", url, resp.Status, body)
}
return body, nil
}
return nil, errors.New("rate limit persisted after five attempts")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if deadline, err := http.ParseTime(value); err == nil && time.Until(deadline) > 0 {
return time.Until(deadline)
}
return time.Second * time.Duration(1<<attempt)
}
func wait(ctx context.Context, delay time.Duration) error {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func scalarStrings(data []byte) ([]string, error) {
var value any
if err := json.Unmarshal(data, &value); err != nil {
return nil, err
}
var result []string
var walk func(any)
walk = func(current any) {
switch typed := current.(type) {
case map[string]any:
for _, child := range typed {
walk(child)
}
case []any:
for _, child := range typed {
walk(child)
}
case string:
result = append(result, typed)
}
}
walk(value)
return result, nil
}
Run it from a restricted incident workstation, treat its output as sensitive, and do not paste matched values into a ticket. The local match avoids claiming server-side filters that are not part of the published contract. In a long-lived tool, replace the generic JSON walk with typed fields only after discovery declares their schema.
Verify before paging anyone
Start with a synthetic pipeline identity and non-patient fixtures. Emit one success, one controlled failure, a duration, a backlog value, and a bounded business-event count. Confirm that the dashboard preserves the expected ordering and time window. Then stop the scheduled test before it begins and verify that the heartbeat monitor, rather than a misleading zero-valued metric, reports the missed deadline.
The acceptance check is small:
- A completed run increments success once, even if delivery is retried.
- A controlled failure appears in the failure trend and has a searchable error or log record.
- A run that never starts produces no fake failure metric but does miss its heartbeat deadline.
- Dashboard labels contain no patient-level or unbounded values.
- The on-call link opens a runbook that distinguishes late, failed, and absent runs.
The first check is an idempotency check. A retry must not double-count a committed run or resend the completion heartbeat. Generate a stable run ID from the schedule and pipeline identity, persist completion state with the batch transaction, and make downstream consumers reject duplicate completion events. Do not page on every row-level validation error; page when the run outcome, deadline, or backlog threatens the service objective.
Also test the capability boundaries. If the team needs threshold notifications, plan a poller and notification path because the combined API does not supply alert routes. If diagnosis requires a distributed span tree, source-map decoding, Electron minidump symbolization, or Session Replay, select another tool for that job. Logs may carry trace_id and span_id for correlation, but those fields are not a tracing query engine.
Roll back without losing the evidence
Rollback should reduce telemetry risk before it removes visibility. Disable the new high-cardinality business-event series first, leave the coarse run outcome and heartbeat in place, and preserve the runbook. If log ingestion must stop, retain the local application logs under the system's existing policy while the team confirms that required access, deletion, and export controls are available elsewhere. There is no per-user log deletion route or bulk log export route in this combined surface, so it is a poor destination for data that requires those workflows.
Revert dashboard panels independently from instrumentation. A broken query should not trigger an application deploy at 02:00.
Finally, record the last known-good completion timestamp, backlog, and heartbeat deadline before changing providers. During migration, send duplicate non-sensitive telemetry to both destinations for a bounded validation window, but designate only one paging source. Two clocks paging on the same missed run create noise, not resilience.
Top comments (0)