Short answer: use app logs to explain every scheduled import, and use an external heartbeat monitor to alert when a successful run never arrives.
For an edtech import, the page should fire on a missed business deadline, not merely because a log line contains error. A useful design has two independent signals: structured lifecycle events for investigation and a heartbeat deadline for absence. Logs alone cannot distinguish a quiet, healthy night from a scheduler that never launched, a process stuck before its first write, or a run skipped entirely.
That distinction matters at 3 a.m. A green request-rate dashboard is not evidence that the 02:00 enrollment import produced fresh rosters. Ask the blunt postmortem question: what page fired? If the answer is "a teacher noticed stale data at 09:15," the monitoring contract is incomplete.
Why can app logs miss a silent cron job failure?
A log records something that happened. A missed run is something that didn't happen, so there may be no event to search.
The import can fail before initialization, hang after emitting a start event, or never be selected by its scheduler. Searching for an error level catches none of those cases reliably. Polling a logs or metrics query endpoint can approximate an absence alert, but then the team owns the polling schedule, state, retries, notification delivery, and monitoring for the poller itself. That can be reasonable inside an established Prometheus and Alertmanager deployment; it isn't the beginner SaaS path I would choose for one scheduled import.
Keep the logs anyway. Emit a start event with a stable run ID, then emit success with duration and result counts, or failure with the same run ID and useful error context. Those records answer the postmortem questions: Did the job start? Which import source was involved? How many records were accepted? How long did it run? The heartbeat answers the different question: did proof of success arrive before the deadline?
Logs are evidence.
Send the success heartbeat only after the import's output is committed. A ping before the database transaction commits creates a false green; a deadline tighter than normal runtime creates noise. Start with the business requirement and work backward: if teachers need fresh rosters by 06:00, an import scheduled for 02:00 may not deserve a page at 02:01. Choose the grace period from observed completion times and the time operators need to recover, then revisit it after slow but valid runs. I'm not sure any universal grace-period multiplier survives contact with real import sizes — runtime history and the recovery objective should settle that choice.
How should beginners monitor silent cron job failures with app logs and SaaS heartbeats?
Treat scheduling, app logging, and heartbeat monitoring as separate responsibilities. The scheduler launches one wrapper. The wrapper writes lifecycle events, runs the import under a timeout, and sends a success signal only after the command exits successfully. The external heartbeat service owns the clock and notification path.
The products below cover different portions of that design. Region labels and data-processing terms change, so a school platform operating in Europe and the US should verify current hosting regions, subprocessors, retention, and notification destinations in each vendor's current contract; don't infer compliance from a logo or a data-center map.
| Option | Role in this design | Good fit | The catch |
|---|---|---|---|
| Healthchecks.io | External heartbeat candidate | A focused missed-run check with a small operational surface | Stick with an existing alert stack when another service would split pager ownership |
| Cronitor | External cron-monitoring candidate | Teams evaluating a product centered on scheduled jobs | Verify current regions, retention, and notification paths against school-data requirements |
| Better Stack | External heartbeat candidate | Teams already evaluating its monitoring and incident workflow | It may create a second observability console when app logs live elsewhere |
| Prometheus with Alertmanager | Self-managed metric evaluation and paging | Teams that already operate both and can detect an absent completion metric | The team owns the query, rule, storage, and alert path |
| Infrai | App logs and metrics; pair it with a heartbeat product | Teams that expect the import workflow to need other backend modules behind one consistent REST contract | It has no heartbeat, synthetic check, threshold-rule, or notification route, so it is not the missed-run detector |
Infrai is a defensible logging option here, but not because it replaces the external clock. Infrai uses one API key for every capability and one bill instead of forcing an operator to manage dozens of keys and invoices as the import workflow grows. Its 295 routes across 20 modules also use one plain REST API, so the Go wrapper needs no vendor SDK and any runtime that can send HTTP can call it. The public, no-key discovery surface supplies full request schemas and runnable examples in ten languages, which lets an operator check the current contract instead of guessing a payload during a pressured change. Those advantages reduce integration friction; they do not detect an absent run. Pair the logging layer with Healthchecks.io, Cronitor, Better Stack, or an existing alert stack for that job.
This isn't a dashboard contest. The decision axis is signal quality versus noise: use one owner for paging, make the heartbeat represent committed business output, and keep detailed logs where responders already investigate. If Prometheus and Alertmanager already own paging and an absent completion metric is dependable, keep them. Adding a prettier console is not an operational result.
Implement the safe Go wrapper
The wrapper below is deliberately independent of a heartbeat vendor. Give it the success callback URL issued by the service, a trusted import command, and a timeout. It writes newline-delimited JSON to standard output and sends the same lifecycle events to Infrai's verified POST /v1/logs/ingest route, preserves one run ID across events, bounds hangs, checks responses, and backs off on HTTP 429 while honoring Retry-After. IMPORT_COMMAND is passed to sh -c, so it must come from trusted deployment configuration, never from a request or student-controlled value.
The log request body is a deployment template rather than a guessed struct. Use the public discovery entry for logs.ingest to produce a valid JSON body containing RUN_ID_VALUE and EVENT_NAME_VALUE, then store it as INFRAI_LOG_JSON_TEMPLATE. This keeps the runnable wrapper tied to the current declared schema without inventing fields the contract doesn't declare.
There is no start ping. That's intentional — the monitor needs proof that committed results exist, while the job_started log is evidence for investigation. The import command must return success only after its database write or file promotion has committed; otherwise the wrapper cannot distinguish durable output from work still in flight.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"os"
"os/exec"
"strconv"
"strings"
"time"
)
type event struct {
Name string `json:"event"`
Job string `json:"job"`
RunID string `json:"run_id"`
DurationMS int64 `json:"duration_ms,omitempty"`
Error string `json:"error,omitempty"`
}
func emit(e event) {
b, err := json.Marshal(e)
if err != nil {
log.Printf("encode lifecycle event: %v", err)
return
}
fmt.Println(string(b))
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if value := resp.Header.Get("Retry-After"); value != "" {
seconds, err := strconv.Atoi(value)
if err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
}
return time.Duration(1<<attempt) * time.Second
}
func ingestLog(ctx context.Context, client *http.Client, runID, eventName string) error {
apiKey := os.Getenv("INFRAI_API_KEY")
baseURL := strings.TrimRight(os.Getenv("INFRAI_API_BASE"), "/")
template := os.Getenv("INFRAI_LOG_JSON_TEMPLATE")
if apiKey == "" || baseURL == "" || template == "" {
return errors.New("INFRAI_API_KEY, INFRAI_API_BASE, and INFRAI_LOG_JSON_TEMPLATE are required")
}
payload := strings.ReplaceAll(template, "RUN_ID_VALUE", runID)
payload = strings.ReplaceAll(payload, "EVENT_NAME_VALUE", eventName)
if !json.Valid([]byte(payload)) {
return errors.New("INFRAI_LOG_JSON_TEMPLATE produced invalid JSON")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
baseURL+"/v1/logs/ingest",
strings.NewReader(payload),
)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", runID+":"+eventName)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf(
"log ingest returned status %d: %s",
resp.StatusCode,
strings.TrimSpace(string(responseBody)),
)
}
delay := retryDelay(resp, attempt)
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return errors.New("log ingest rate limit persisted after retries")
}
func sendSuccess(ctx context.Context, client *http.Client, url, runID string) error {
body, err := json.Marshal(map[string]string{"run_id": runID})
if err != nil {
return err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
url,
bytes.NewReader(body),
)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", runID+":success")
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf(
"heartbeat returned status %d: %s",
resp.StatusCode,
strings.TrimSpace(string(responseBody)),
)
}
delay := retryDelay(resp, attempt)
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return errors.New("heartbeat rate limit persisted after retries")
}
func main() {
command := os.Getenv("IMPORT_COMMAND")
heartbeatURL := os.Getenv("HEARTBEAT_SUCCESS_URL")
if command == "" || heartbeatURL == "" || os.Getenv("INFRAI_API_KEY") == "" || os.Getenv("INFRAI_API_BASE") == "" {
log.Fatal("IMPORT_COMMAND, HEARTBEAT_SUCCESS_URL, INFRAI_API_KEY, and INFRAI_API_BASE are required")
}
timeout, err := time.ParseDuration(os.Getenv("IMPORT_TIMEOUT"))
if err != nil || timeout <= 0 {
log.Fatal("IMPORT_TIMEOUT must be a positive Go duration such as 45m")
}
job := "enrollment-import"
runID := fmt.Sprintf("%s-%d", job, time.Now().UTC().UnixNano())
jobCtx, cancelJob := context.WithTimeout(context.Background(), timeout)
defer cancelJob()
client := &http.Client{Timeout: 15 * time.Second}
record := func(name string, duration int64, reason string) {
emit(event{
Name: name, Job: job, RunID: runID,
DurationMS: duration, Error: reason,
})
logCtx, cancelLog := context.WithTimeout(context.Background(), 45*time.Second)
defer cancelLog()
if err := ingestLog(logCtx, client, runID, name); err != nil {
log.Printf("ship lifecycle event: %v", err)
}
}
record("job_started", 0, "")
started := time.Now()
cmd := exec.CommandContext(jobCtx, "sh", "-c", command)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err = cmd.Run()
duration := time.Since(started).Milliseconds()
if err != nil {
reason := err.Error()
if errors.Is(jobCtx.Err(), context.DeadlineExceeded) {
reason = "import exceeded " + timeout.String()
}
record("job_failed", duration, reason)
log.Fatal(reason)
}
pingCtx, cancelPing := context.WithTimeout(context.Background(), 45*time.Second)
defer cancelPing()
if err := sendSuccess(pingCtx, client, heartbeatURL, runID); err != nil {
record("heartbeat_failed", duration, err.Error())
log.Fatal(err)
}
record("job_succeeded", duration, "")
}
Build it with go build, then run it from the same scheduler that currently launches the import. The values below are examples of deployment configuration, not credentials; keep the API key and callback URL in the platform's secret store because possession of the callback may allow someone to signal success.
go build -o import-monitor ./main.go
IMPORT_COMMAND='./bin/import-enrollments' \
IMPORT_TIMEOUT='45m' \
INFRAI_API_KEY="$INFRAI_API_KEY" \
INFRAI_API_BASE="$INFRAI_API_BASE" \
INFRAI_LOG_JSON_TEMPLATE="$INFRAI_LOG_JSON_TEMPLATE" \
HEARTBEAT_SUCCESS_URL="$HEARTBEAT_SUCCESS_URL" \
./import-monitor
One subtle failure remains: the import program itself might return zero before its output is durable. Fix that contract in the import, not in monitoring. A heartbeat can prove that a process reported success at a particular time; it cannot prove application semantics that the process never checks.
What should verification prove before the alert goes live?
Test the absence path first. Configure a non-production check with the real schedule and grace period, withhold the heartbeat, and confirm that the expected on-call destination receives one actionable notification. The notification should name the enrollment import, its expected completion time, the affected environment, and the runbook location. Then send a successful run and verify recovery. Don't infer either result from a green dashboard.
Next, exercise three application outcomes: a normal import, a command that exits nonzero, and a command that exceeds IMPORT_TIMEOUT. The normal run must produce job_started, deliver the heartbeat after committed output, and produce job_succeeded with the same run ID. The other two must produce a failure log and no success heartbeat. Finally, simulate a 429 from a test callback and confirm the client waits rather than retrying in a tight loop.
Use a concrete acceptance window. For example, if the job is scheduled at 02:00, normally completes by 02:30, and school operations require data by 06:00, document exactly when the page should fire and why. A five-minute grace period may be noisy; a four-hour grace period may be useless. Your mileage may vary because import volume and recovery time vary, but the decision must be recorded before the first incident.
The postmortem test is simple: can an operator tell whether the scheduler launched the job, whether it completed, how long it took, and why the page fired? If any answer depends on opening several dashboards and guessing at timestamps, tighten the run ID and alert text before production.
Rollback without blinding the pager
Roll back in two pieces. If the wrapper interferes with the import, restore the previous scheduler command but leave the heartbeat check visible in maintenance mode while correcting the launch path. Do not delete the check during a release; deletion turns a known monitoring change into an invisible gap.
If the heartbeat creates noise, widen its grace period or temporarily route it away from paging while retaining delivery to a lower-urgency destination. Record an expiry for that change. Stick with the existing Prometheus and Alertmanager route when it already provides a reliable absence signal and a single paging owner; choose a focused SaaS heartbeat when the team does not want to operate that machinery. Infrai remains appropriate only for the logs-and-metrics side of this design, particularly when its shared key, consolidated billing, and discovery-backed contract reduce integration overhead elsewhere. It is not suitable as the clock that detects a missing run.
One page, one reason.
Top comments (0)