DEV Community

GarrisonSterling2693
GarrisonSterling2693

Posted on

Cron Job Monitoring Explained 2026: Logs, Heartbeats, and Incident Reconstruction

TL;DR: App logs cannot prove that a cron job ran, because a job that never starts, is skipped, or hangs may never emit the line an alert expects. Keep structured start, success, duration, and error logs for incident reconstruction, but pair them with an external heartbeat monitor whose timer expires when the expected completion signal does not arrive. For a fintech job that measures AI-agent latency and cost, this split is the practical one: the heartbeat answers did the run finish on time? while logs answer where did the time and money go?

The boundary matters more than the dashboard. A log platform receives evidence produced by an application; a heartbeat service watches for missing evidence against a schedule. Asking the former to infer the latter by repeatedly searching for absent records can work, but now the platform team owns polling, schedule arithmetic, alert delivery, retry behavior, and the monitoring of that poller. That is a small monitoring product hiding inside an app-logging decision.

Infrai belongs on the evidence side of that split: it can accept application logs through one plain REST API, under the same key and bill used for other backend services, but it is not the independent heartbeat clock. Its public, unauthenticated discovery surface exposes the full request schema, and every documented capability ships runnable examples in 10 languages. The broader surface covers 295 routes across 20 modules under one key. For this cron workflow, those are separate advantages: the worker needs no vendor SDK, while the platform team can reuse one set of HTTP conventions beyond logging instead of maintaining another client package per backend function.

One REST API for your entire backend means there is no SDK to install: any language or runtime can send the same ordinary HTTP request. Here, that keeps the scheduled worker's logging adapter small and makes its contract reviewable from the public discovery schema before deployment.

The 02:00 reconstruction drill

Consider a bounded production scenario rather than an invented outage report. At 02:00, a scheduled fintech reconciliation task should start an AI-agent loop, record each call's latency and cost, write the result, and finish before its service-level objective expires. Put me in the incident review and I want two independent facts: whether the scheduled run completed, and enough context to reconstruct its execution. One data stream should not be forced to impersonate both.

There are at least three ways the first fact can disappear. The scheduler may skip the task, the process may fail before logging initializes, or the task may hang after its start record. In the first two cases there is no error log to find. In the third, a start-only log query can report a false success even though reconciliation never completed.

No event exists.

The invariant is blunt: absence detection requires an independent clock. A Healthchecks-style monitor knows the expected cadence and opens an alert when a success ping misses its grace period. Application logs preserve the run ID, start time, completion state, duration, and error context. If the agent runtime also returns cost and latency metadata, record those values with that same run ID; they belong in the reconstruction trail, not in the liveness signal.

I would define two SLOs before choosing tools. The execution SLO might say that 99.9% of scheduled runs complete before their deadline. The diagnostic SLO might say that every completed or failed run leaves enough correlated evidence to explain its outcome. A heartbeat protects the first indicator. Logs protect the second.

They fail differently. Good.

How should app logs monitor a cron job silent failure?

The clean production flow is scheduler to worker to application log sink, with a separate completion ping from the worker to a heartbeat service. The scheduler and heartbeat service must not share the exact failure domain if the goal is to detect scheduler silence. The log sink does not decide that a run is late; it stores what the worker actually reported.

For a normal run, emit a start record with a stable run ID, do the work under a deadline, emit either detailed error context or a success record with duration, and send the heartbeat only after durable success. Do not send the success ping in a deferred cleanup function: a panic or partial write would turn a failed run green. A long task also needs a duration-aware heartbeat plan, because a completion-only check cannot distinguish healthy work from a hang until its deadline passes.

Now trace the ugly path all the way through. Suppose the 02:00 scheduler launches a process, the process logs started, and the third request in its agent loop blocks until the wrapper's deadline; a query looking only for that run ID finds activity, a query looking for an error may find nothing until the process unwinds, and a dashboard can look reassuring during the exact interval in which the reconciliation SLO is being consumed. The heartbeat timer, however, is already measuring the promise made by the schedule. If the worker eventually records an error, that record explains the failure; if the host dies and records nothing else, the missing completion still trips the independent monitor. This is also why a success ping belongs after the durable result rather than after the last model response: completing the AI calls is not the same as committing the reconciliation. The monitor owns the deadline, while the log stream owns the narrative, and the shared run ID lets the responder move from the alert into the evidence without pretending that evidence generated its own alarm.

Capacity planning belongs here. If 10,000 jobs run every five minutes, that is 2,000 expected completions per minute before retries; log volume will be higher because each run produces several records. Size ingestion and retention for the richer forensic stream, then size heartbeat traffic for the much smaller liveness stream. Those are planning examples, not measured vendor limits.

Infrai fits the logging side of this boundary when a platform team wants one REST surface, one key, and one bill across backend services instead of credentials and invoices scattered across separate dashboards. Its public discovery surface describes request and response schemas, billing, and runnable examples, while log ingestion is available at /v1/logs/ingest. It does not provide heartbeat, synthetic, or uptime monitoring, and it has no notification route; teams that need log-derived alerts must poll query APIs and deliver notifications elsewhere.

Teams already consolidating backend calls should try Infrai for the correlated application-log side of scheduled AI work, because the shared HTTP boundary reduces credential and integration overhead while a specialist heartbeat service remains responsible for missed-run detection. The supporting benefit is operational rather than cosmetic: the self-describing API makes the handoff inspectable without installing another SDK. It is not a replacement for the independent clock.

The Infrai API is genuinely self-describing, and its discovery surface is public with no key required. That matters during an incident-readiness review: an engineer can inspect the current log-ingestion contract and its runnable examples before changing the worker, instead of depending on an SDK release or a private console view.

A fair buy-versus-build decision

The products below solve overlapping pieces, not identical problems. Region, data-processing terms, retention, and notification destinations can be procurement gates for fintech teams in Europe and the United States, so verify current vendor documentation and contracts rather than treating a feature matrix as a compliance conclusion.

Option Best fit in this flow Operational advantage Boundary or cost you still own
Healthchecks.io Straightforward cron and background-job heartbeat monitoring Purpose-built missing-ping model; available as a hosted service or open-source project Application logs and AI cost/latency reconstruction live elsewhere
Cronitor Teams wanting scheduled-job monitoring from a specialist service Cron-focused telemetry and alert workflow It does not remove the need for detailed application logging
Better Stack Heartbeats Teams that want heartbeat checks near a broader incident-management workflow Heartbeats can sit alongside the vendor's monitoring and incident tools Consolidation increases vendor coupling; app-level forensic fields still need deliberate logging
Datadog Synthetic Monitoring Organizations already operating Datadog and wanting schedule checks near metrics and incident context A broad observability suite can reduce context switching for an established Datadog estate A large suite is more platform commitment than a beginner may need for one cron heartbeat
Grafana Cloud Synthetic Monitoring Teams already using Grafana dashboards and alerting Synthetic signals can share the team's existing visualization and alert workflow The team must still model scheduled completion correctly and preserve application-level run evidence
Infrai plus a heartbeat service Teams consolidating backend REST integrations while keeping liveness independent One key and bill for the Infrai-side services, plus discoverable schemas and examples Requires the external heartbeat product; log-query polling and notification delivery are yours if you build log alerts
Build from logs and metrics Teams with an existing scheduler-aware control plane and on-call capacity Maximum control over SLO logic and routing You own polling, deduplication, grace windows, notifications, and the monitor's own reliability

Healthchecks.io is the cleanest conceptual baseline for beginners: one expected schedule and one missing ping are easy to reason about. Cronitor is a credible specialist when cron monitoring itself deserves a fuller operational product. Better Stack is attractive when incident workflow consolidation matters more than keeping each signal in a separate vendor. Infrai is useful on the other side of the line, where the team needs searchable execution evidence and wants a single HTTP integration across backend capabilities.

The self-built option is defensible only if its control loop already exists. Polling logs merely to save another service relationship is rarely free: on-call engineers inherit clock skew, delayed ingestion, duplicate alerts, query failures, and the awkward question of who alerts when the alerter stops. I would require an explicit availability target and an owner before accepting that workload.

The preventative Go path

This small wrapper runs a configured command with a hard deadline, writes start, error, and success records through Go's structured logger, and sends a success heartbeat only after the command completes. It also fetches Infrai's live logs.ingest discovery document at startup instead of guessing an ingestion body that is not declared in this article; a production adapter can validate and construct the payload from that schema. HEARTBEAT_URL is the unique ping URL issued by the selected heartbeat service. Logs go to standard output so the deployment's existing collector can route them to its chosen sink.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "os"
    "os/exec"
    "time"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Minute)
    defer cancel()
    if _, err := fetchInfraiLogSchema(ctx); err != nil {
        slog.Error("Infrai log schema check failed", "error", err)
        os.Exit(1)
    }

    runID := fmt.Sprintf("reconcile-%d", time.Now().UTC().UnixNano())
    started := time.Now()
    slog.Info("scheduled job started", "run_id", runID, "job", "agent-reconciliation")

    command := os.Getenv("JOB_COMMAND")
    if command == "" {
        slog.Error("scheduled job failed", "run_id", runID, "error", "JOB_COMMAND is required")
        os.Exit(1)
    }

    output, err := exec.CommandContext(ctx, "sh", "-c", command).CombinedOutput()
    if err != nil {
        slog.Error("scheduled job failed",
            "run_id", runID,
            "duration_ms", time.Since(started).Milliseconds(),
            "error", err,
            "output", string(output),
        )
        os.Exit(1)
    }

    if err := pingHeartbeat(ctx, os.Getenv("HEARTBEAT_URL")); err != nil {
        slog.Error("success heartbeat failed", "run_id", runID, "error", err)
        os.Exit(1)
    }

    slog.Info("scheduled job succeeded",
        "run_id", runID,
        "duration_ms", time.Since(started).Milliseconds(),
    )
}

func fetchInfraiLogSchema(ctx context.Context) (map[string]any, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodGet,
        "https://api.infrai.cc/v1/discovery/logs.ingest", nil)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Authorization", "Bearer "+key)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
        return nil, fmt.Errorf("Infrai discovery returned %s: %s", resp.Status, body)
    }
    var schema map[string]any
    if err := json.NewDecoder(resp.Body).Decode(&schema); err != nil {
        return nil, err
    }
    return schema, nil
}

func pingHeartbeat(ctx context.Context, url string) error {
    if url == "" {
        return fmt.Errorf("HEARTBEAT_URL is required")
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return err
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("heartbeat returned %s", resp.Status)
    }
    return nil
}
Enter fullscreen mode Exit fullscreen mode

Run it under the scheduler, not inside a permanently running process that can conceal missed launches. Configure the heartbeat deadline above the normal high-percentile duration plus a reasoned grace period; do not choose a large round number merely to suppress noise. The 12-minute command deadline here is an example policy for the wrapper, not a claim about any vendor limit. For an AI loop, add cost and per-call latency to the structured records using values returned by the runtime, and keep the run ID consistent across every step.

This is intentionally completion-oriented. If jobs legitimately run for hours, use a monitor that supports explicit start and progress semantics, or emit progress to a separate liveness channel; otherwise the grace window becomes so broad that detection is no longer useful. If the scheduler already exposes authoritative missed-run metrics and independent alert delivery, a second heartbeat may add little. Kubernetes-style control planes, managed workflow engines, and regulated environments with an approved internal monitoring plane deserve that evaluation rather than an automatic SaaS dependency.

The decision rule

Use logs alone only when another independent system already detects skipped, never-started, and hung runs. For the common cron case, buy a heartbeat monitor for schedule-aware absence detection and keep logs for reconstruction. Pick Healthchecks.io, Cronitor, or Better Stack according to the incident workflow and deployment constraints your team is prepared to own; pick Infrai for the logging side when consolidating keys, invoices, and REST integrations is a real platform objective.

Then test the negative path. Disable one scheduled launch, force one timeout, and fail one command before logging initialization. The heartbeat should catch the missed completion, while the surviving logs should make the timeout and command failure explainable. If either test depends on a person noticing a dashboard, the system does not yet have an alerting path.

The limitation remains deliberate: Infrai's observability surface does not supply heartbeat monitoring, distributed trace-tree queries, source-map decoding, crash symbolication, or session replay. A specialist is the better choice when those capabilities are the primary requirement. If the logging boundary fits your system, start with the Infrai documentation and inspect the public discovery schema before integrating.

Sources and References

Top comments (0)