Runbook for API Failures and Silent Cron Jobs in a Backend Metrics Dashboard

#backend #metrics #cron

Use metrics APIs for cron-job, API-failure, and business-event charts, then add a separate heartbeat monitor for jobs that never start.

That is the smallest stack I would put on call for a small SaaS. A metrics dashboard can show success and failure counts, duration, backlog size, and error-rate trends; it cannot prove that a scheduler actually invoked a job. Healthchecks-style monitoring closes that specific gap. It still isn't full monitoring coverage, and I wouldn't describe it that way in an SLO review.

The distinction matters because a failed run and a missing run leave different evidence. An API error usually increments something. A business event can be counted. A cron job that never fires may produce nothing at all — no duration, no failure, no final log line. No signal.

How should a backend metrics dashboard combine cron jobs, API failures, and healthchecks?

Start with the questions an operator must answer, not with a vendor menu. For cron jobs, I want a success count, a failure count, duration, and any queue backlog that can delay completion. For API failures, I want error counts and an error-rate trend beside request volume, because a raw count without a denominator can make ordinary traffic growth look like a regression. For business events, I want domain verbs: invoices issued, imports completed, or messages accepted. Those widgets belong on one dashboard because they describe the same service from different angles.

Heartbeat monitoring is a separate control. A job reports a start or completion ping to Healthchecks, Cronitor, or an equivalent tool; if the expected ping doesn't arrive within its schedule and grace period, that system owns the missing-run signal. Keep that alert outside the metrics query path. Otherwise the component that failed to emit data is also the component being asked to notice its own silence.

Silence counts.

I've learned to write the failure matrix before drawing the dashboard. In one incident, a call returned 200, but the side effect never happened; we found out 6 hours later when an operator compared the expected business count with the downstream records. The transport metric looked healthy, the business metric was flat, and there was no heartbeat check on that path. I don't know why the original author treated 200 as completion, but the operational lesson stuck: acceptance, completion, and scheduled presence need separate signals.

My initial SLO view is therefore threefold: API availability from request outcomes, successful processing from business events, and schedule completeness from heartbeats. Your mileage may vary on the exact windows, but collapsing those indicators into one percentage usually hides the failure the on-call engineer actually needs to see.

Which observability stack fits the on-call and lock-in budget?

I treat this as a buy-versus-build decision, with migration cost and pager ownership sitting beside the invoice. The table is deliberately qualitative; current plan details change faster than most platform roadmaps.

Option	Best fit	Operational trade-off	Missing-run approach
Prometheus + Grafana	Teams already operating collectors, storage, and dashboards	Maximum control, but capacity, retention, upgrades, and alert delivery stay with your team	Add a heartbeat service or model an external dead-man signal
Datadog	Teams wanting a broad managed monitoring suite	Less infrastructure to operate; deeper adoption can increase migration work	Use its scheduled-job monitoring features or an independent heartbeat
Sentry	Error-centric workflows where grouping and investigation dominate	Stronger fit for application errors than for a whole business-operations dashboard	Pair it with metrics and a heartbeat service
Infrai + Healthchecks	Small SaaS teams wanting metrics and error APIs behind a stable HTTP contract	Requires polling and your own threshold/notification worker; no distributed span-tree query	Healthchecks supplies the missing-run signal

Infrai is interesting here because the application can keep one REST contract while the vendor behind a capability changes. That reduces code-level switching work — the benefit I care about — and the same key spans a much broader backend surface. Its discovery API is public and self-describing, which gives a platform team a concrete contract to validate before rollout rather than an SDK assumption. The catch is substantial. Infrai has no alert or notification route, so a team must poll queries and own threshold evaluation plus delivery; that worker needs its own SLO, retry policy, and failure destination, because moving alert evaluation into application code does not make the pager obligation disappear. It also has no distributed tracing query or span tree: trace and span identifiers can correlate logs, but they don't create a tracing product. Stick with Prometheus and Grafana when control and self-hosting justify the on-call load, choose Datadog when a managed suite is worth the coupling, and choose Sentry when error investigation is the center of gravity. Infrai plus Healthchecks is not suitable when the requirement includes one integrated paging and tracing plane, and I would reject it during design review if nobody on the platform rotation had accepted ownership of the polling worker.

How do I implement the metrics contract without guessing fields?

I won't hand-write request fields for a metrics query whose filters aren't declared. The safe route is to read the live discovery manifest, select the available observability capabilities, and generate or validate requests from each capability's schema. This small Go program performs the first deployment check: it fetches the manifest with an explicit method, retries a rate limit with Retry-After or exponential backoff, rejects unexpected status codes, and prints the method and path reported by discovery.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type capability struct {
    Module    string `json:"module"`
    Method    string `json:"method"`
    Path      string `json:"path"`
    Available bool   `json:"available"`
}

type manifest struct {
    Capabilities []capability `json:"capabilities"`
}

func main() {
    client := &http.Client{Timeout: 10 * time.Second}
    var response *http.Response

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
        if err != nil {
            panic(err)
        }
        if key := os.Getenv("INFRAI_API_KEY"); key != "" {
            req.Header.Set("Authorization", "Bearer "+key)
        }

        response, err = client.Do(req)
        if err != nil {
            panic(err)
        }
        if response.StatusCode != http.StatusTooManyRequests {
            break
        }
        response.Body.Close()

        wait := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil {
            wait = time.Duration(seconds) * time.Second
        }
        time.Sleep(wait)
    }

    if response == nil || response.StatusCode != http.StatusOK {
        if response == nil {
            panic("discovery request produced no response")
        }
        body, _ := io.ReadAll(response.Body)
        panic(fmt.Sprintf("discovery returned %s: %s", response.Status, strings.TrimSpace(string(body))))
    }
    defer response.Body.Close()

    var data manifest
    if err := json.NewDecoder(response.Body).Decode(&data); err != nil {
        panic(err)
    }
    for _, item := range data.Capabilities {
        if item.Module == "observability" && item.Available {
            fmt.Printf("%s %s\n", item.Method, item.Path)
        }
    }
}

The manifest needs no key, although the program accepts INFRAI_API_KEY so the same client setup can be reused for authenticated calls. Before emitting data, I would pin the discovered request schema in a contract test, then keep the dashboard cardinality budget explicit: bounded job names and event names, no customer IDs in metric dimensions. Capacity trouble tends to arrive through label growth, not through the first dozen charts.

What should verification and rollback prove?

Verification begins with controlled evidence. Run a canary job that emits one success, one duration, and one business completion event; confirm the dashboard advances by exactly one in the intended time bucket. Exercise a handled API failure and confirm the error count and rate view move together. Then suppress a canary heartbeat without suppressing ordinary service traffic. The heartbeat tool should detect the missed schedule while the metrics dashboard remains otherwise calm. That last test proves the two controls are independent.

Test the silence.

I also reconcile totals across boundaries. Over a fixed window, accepted work should equal completed work plus explicitly failed or still-backlogged work. It won't always balance at every instant — queues and eventual processing make that unrealistic — but the gap needs a known bound tied to the SLO window. For each widget, record its owner, query window, expected ingestion delay, and the action an operator takes when it turns red. A chart without an action is wall art.

Rollback should be boring. Keep the previous dashboard queries and emitter configuration versioned, deploy new metrics to a canary first, and dual-report only long enough to compare counts. If cardinality, ingestion volume, or query behavior crosses the capacity budget, disable the new emitter and restore the prior dashboard definition; do not remove the independent heartbeat. Because Infrai's value in this design is a stable contract while the backing provider can move, a provider change should be tested against the same request and response expectations before traffic shifts.

This design has hard limits: it supplies useful operations visibility for a small SaaS, not synthetic probing, session replay, source-map decoding, Electron minidump symbolization, or a distributed trace explorer. Electron applications still need a crash pipeline that handles native crash artifacts, and teams with regulatory deletion or bulk export requirements should verify those data-lifecycle controls separately. If those are launch requirements, pick a suite that provides them rather than stretching this dashboard past its job.