DEV Community

SeraphinaLyn7139
SeraphinaLyn7139

Posted on

Missed Cron Job Runs: Heartbeat Monitoring for Safe Import Rollbacks

#observability #sre #monitoring

Short answer: treat every scheduled property import as a leased promise. Send a heartbeat only after the import has produced and committed an acceptable result, set the monitor's grace period from measured runtime plus scheduler delay, and page when that heartbeat is absent. Logs and metrics still explain a failure, but they cannot reliably prove that a process which never started did not run.

Rollback safety decides the shape of this design. The monitor must observe the import's committed outcome; it must never become another writer to the property database, and disabling or reverting the heartbeat wrapper must not undo a successful import. This separation also keeps a monitoring change out of the data rollback path.

How should heartbeat monitoring detect a missed cron job run?

That question exposes the hole in log-only alerting. A process can log a parse error after it starts. It cannot emit an error when the scheduler never launches it, its host is unavailable, or execution stops before the logging client initializes. A query for recent records can approximate absence detection, but then the query loop, its schedule, and its notification path become a second monitoring system that the platform team owns.

For a nightly property-management feed, define success more narrowly than "the process exited." A useful completion event means the expected source was selected, validation passed, and the new result became visible through the application's normal commit boundary. If validation rejects a suspiciously empty feed and retains yesterday's data, that is a failed run for freshness even though rollback behavior worked correctly.

The first SLO should therefore be about result freshness, not cron execution: for example, each scheduled source has one accepted result within its declared delivery window. Choose the actual window from the business commitment and observed runtime distribution. Do not copy an arbitrary five-minute grace period into production. Capacity planning matters here because a queue can cross from healthy to late without any single worker crashing; test the deadline burst at the smallest, expected, and next capacity-planning load, then set the grace period from those observations rather than from intuition. The explicit trade-off is page speed against false positives: a tight grace period detects stale data sooner, while a looser one protects on-call attention when upstream files arrive late.

Silence is a signal.

Put the heartbeat after the commit boundary

The safest implementation is a thin supervisor around the existing importer. It starts the same binary with the same arguments, passes through its output, and sends one completion ping only when the child exits successfully. The monitor URL is configuration, so rollback is just restoring the previous scheduler command; no schema migration or compensating data write is involved.

This runnable Go program deliberately does not retry the import. Retrying a write-capable job at the monitoring layer can duplicate leases, charges, or tenant updates unless the importer already has a tested idempotency contract. Let the scheduler's established policy own execution retries, and let the heartbeat report the final outcome.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "os/exec"
    "strconv"
    "strings"
    "time"
)

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "usage: heartbeat-run <command> [args...]")
        os.Exit(2)
    }

    cmd := exec.Command(os.Args[1], os.Args[2:]...)
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.Stdin = os.Stdin
    if err := cmd.Run(); err != nil {
        fmt.Fprintf(os.Stderr, "import failed; heartbeat withheld: %v\n", err)
        if diagErr := printRecentLogs(); diagErr != nil {
            fmt.Fprintf(os.Stderr, "log lookup also failed: %v\n", diagErr)
        }
        os.Exit(exitCode(err))
    }

    pingURL := os.Getenv("HEARTBEAT_URL")
    if pingURL == "" {
        fmt.Fprintln(os.Stderr, "import committed but HEARTBEAT_URL is unset")
        os.Exit(1)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, pingURL, nil)
    if err != nil {
        fmt.Fprintf(os.Stderr, "build heartbeat request: %v\n", err)
        os.Exit(1)
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Fprintf(os.Stderr, "send heartbeat: %v\n", err)
        os.Exit(1)
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "heartbeat returned %s\n", resp.Status)
        os.Exit(1)
    }
}

func printRecentLogs() error {
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        return errors.New("INFRAI_BASE_URL or INFRAI_API_KEY is unset")
    }

    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/v1/logs/search", nil)
        if err != nil {
            cancel()
            return fmt.Errorf("build log request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            cancel()
            lastErr = err
            continue
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
        resp.Body.Close()
        cancel()
        if readErr != nil {
            return fmt.Errorf("read log response: %w", readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            lastErr = fmt.Errorf("log search rate limited")
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("log search returned %s: %s", resp.Status, body)
        }
        fmt.Fprintf(os.Stderr, "recent logs: %s\n", body)
        return nil
    }
    return fmt.Errorf("log search failed after retries: %w", lastErr)
}

func exitCode(err error) int {
    var exitErr *exec.ExitError
    if errors.As(err, &exitErr) {
        return exitErr.ExitCode()
    }
    return 1
}
Enter fullscreen mode Exit fullscreen mode

There is an uncomfortable edge case: the import can commit and the heartbeat request can fail. The alert is then a false positive, but the data remains correct. That is the conservative failure mode. Do not move the ping before the commit to quiet the monitor; doing so creates false success, which is worse because nobody investigates stale tenant data.

If false positives become frequent, use a durable completion record written in the import's existing transaction and a separate reporter that converts unreported completion records into heartbeats. That is more machinery, so it belongs only where the freshness SLO and paging volume justify another queue, retention policy, and recovery procedure.

Buy or build the absence detector

The practical comparison is operational ownership, not a feature-count contest. Product details and packaging change, so validate notification channels, regions, data retention, and current terms against each vendor's documentation during procurement.

Option Operational fit Rollback and ownership trade-off
Healthchecks.io Purpose-built cron and heartbeat monitoring, with a documented ping model and self-hosted project available Narrow scope makes the wrapper easy to remove; self-hosting transfers upgrades, mail delivery, and availability to your team
Cronitor Cron monitoring with telemetry and job-monitoring workflows Managed operation reduces local machinery, while configuration and alert routing remain vendor dependencies to test
Better Stack Heartbeats Heartbeats integrated with a broader incident-management and observability offering Useful when that alert path is already owned; broader adoption raises migration scope beyond this one job
Dead Man's Snitch A focused dead-man-switch model for scheduled jobs Small integration surface, but teams still need to verify that escalation and regional requirements match their on-call policy
Datadog Cron and job monitoring can sit beside an existing Datadog telemetry estate Consolidation helps established users, while adopting a broad platform solely for one heartbeat expands procurement and migration scope
Grafana Cloud Synthetic Monitoring and alerting fit teams already operating Grafana dashboards and policies Flexible integration comes with more policy surface than a narrowly focused dead-man switch
Sentry Cron Monitoring connects scheduled-task status with application error investigation Strong fit when Sentry already owns error triage; it should not be mistaken for the property's committed-result record
Self-built poller Can query an existing completion record or recent telemetry on a custom schedule Maximum control, plus full ownership of the poller, notifier, deduplication, regional deployment, and its own missed runs

My decision rule is blunt: buy the heartbeat unless custom compliance constraints or an existing, proven control plane make ownership cheaper in on-call hours. A home-grown poller needs at least two failure domains, deduplicated notifications, monitor-of-monitor coverage, clock-skew handling, and capacity tests around the deadline burst. The code that checks a timestamp is trivial. The service surrounding it is not.

Infrai fits a different part of the stack by providing one key and one bill for every backend service through one REST API. That credential covers 295 routes across 20 modules, instead of leaving the platform team to rotate dozens of API keys and reconcile dozens of separate invoices. Its 23 observability routes can centralize start/end logs, metrics, and error capture, but there is no built-in uptime, synthetic, heartbeat, threshold-notification, phone, SMS, or webhook alerting route, so it should supply investigation evidence rather than serve as the absence detector. Polling recent logs or metrics is possible but weaker; the query filters are not declared in discovery parameters, and logs also lack user-delete and bulk export/subscription interfaces, material constraints for compliance-sensitive property data.

Verify the page before trusting it

Deploy to one noncritical import first. Record the old scheduler command and configuration revision, switch only that job to the wrapper, and run four deliberate checks: a normal committed import produces one healthy heartbeat; a nonzero child exit produces no heartbeat; a job that never launches eventually pages; and a committed import followed by blocked network access pages without corrupting or reverting its result.

Then measure. Track accepted-result age, import duration, heartbeat delivery failures, alerts per source, and time to acknowledge. Load-test the deadline burst at the next meaningful scale boundary rather than extrapolating from a handful of tenants. The page should identify the source, expected deadline, last accepted result, owning team, and rollback command, but it should not embed tenant records or credentials.

I would run the alert through at least one full scheduling cycle before making it paging-critical, then test the escalation path with the primary responder unavailable. During that period, compare the heartbeat against the database's committed completion marker. Any disagreement is actionable: either the wrapper is placed on the wrong side of the commit, or the reported SLO is measuring process activity instead of customer-visible freshness.

Keep the rollback boring. Restore the recorded scheduler command, leave the monitor muted rather than deleted until the next successful legacy run, and never roll back imported property data merely because heartbeat delivery failed. The import's own transactional or versioned rollback remains the authority for data recovery.

References

Top comments (0)