TL;DR: Application logs explain what a marketplace notification job did after it started. They cannot reliably detect that the process never started, stayed hung, or was skipped. Keep structured start, success, duration, and error logs, but pair them with an external heartbeat monitor that expects each scheduled run. Alert on the missing heartbeat, then use the logs to diagnose the delivery failure.
This split keeps the page meaningful. A log query can find a recorded error; an expectation monitor can find an absence. Treating those as the same signal creates a blind spot precisely where scheduled work is most fragile.
How should app monitoring detect a silent cron job failure?
After being paged for missed jobs and duplicate deliveries, I stopped treating “no errors in the logs” as evidence that a scheduler was healthy. The production scenario is bounded but common: a marketplace runs a notification job on a schedule, and downstream delivery failures need investigation. The job may fail before its logger initializes, the scheduler may skip it, or the process may hang before writing its final status. In all three cases, searching for an error line can return nothing useful.
Nothing happened. That is the signal.
Use two signals.
The invariant is simple: the observer that decides a run is missing must live outside the scheduled process. Logs remain valuable evidence. Record a stable run ID, job name, start time, terminal status, duration, and error context so an operator can follow one execution without grouping unrelated deliveries together. A trace_id or span_id can help correlate records, but correlation fields are not a distributed trace or a span tree.
For a job scheduled every 15 minutes, define an expected cadence and an explicit grace period based on the job's normal runtime and scheduler jitter. Do not choose the grace period merely to stop noisy alerts. If it is so wide that the next run begins before the first page, the monitor has ceased to protect the delivery objective.
Separate proof of execution from evidence of failure
Model the run as a small state machine: started, succeeded, or failed. Emit logs at the transitions, and send the heartbeat only after the success condition is durable. For a notification dispatcher, “success” should mean the batch state has been committed, not merely that messages were read from a queue.
That ordering matters. Sending the success ping at startup turns a hung process into a healthy-looking run. Sending it before a database commit can acknowledge work that will be repeated. If the monitor supports a start signal, use it to measure runtime, but never let it substitute for the terminal success signal.
Retries need the same discipline. Scheduled and queued work can execute more than once, so use a deterministic operation key such as job-name + scheduled-slot + notification-id at the side-effect boundary. A heartbeat is an availability signal, not a deduplication mechanism. The consumer must remain idempotent even when monitoring is perfect.
The following runnable Go program wraps one job. It emits structured JSON to standard output, reports failure to a separately configured URL when available, and sends the success heartbeat only after runJob returns nil. The HTTP client has a timeout, checks response status, and retries rate limits while honoring Retry-After when it is expressed in seconds.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type event struct {
Job string `json:"job"`
RunID string `json:"run_id"`
Status string `json:"status"`
Duration int64 `json:"duration_ms,omitempty"`
Error string `json:"error,omitempty"`
}
func writeLog(e event) {
_ = json.NewEncoder(os.Stdout).Encode(e)
}
func verifyLogContract(ctx context.Context, client *http.Client) error {
discoveryURL := os.Getenv("INFRAI_DISCOVERY_URL")
if discoveryURL == "" {
return errors.New("INFRAI_DISCOVERY_URL is not configured")
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, discoveryURL, nil)
if err != nil {
return err
}
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is not configured")
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if err != nil {
return err
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("discovery returned %s: %s", resp.Status, body)
}
var contract struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
Params json.RawMessage `json:"params"`
}
if err := json.Unmarshal(body, &contract); err != nil {
return err
}
if contract.ID != "logs.ingest" || contract.Method != http.MethodPost || contract.Path != "/v1/logs/ingest" || !contract.Available || len(contract.Params) == 0 {
return errors.New("unexpected or unavailable log ingestion contract")
}
return nil
}
func ping(ctx context.Context, client *http.Client, url string) error {
if url == "" {
return errors.New("heartbeat URL is not configured")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("heartbeat returned %s", resp.Status)
}
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(delay):
}
}
return errors.New("heartbeat rate limit persisted after retries")
}
func runJob(context.Context, string) error {
// Replace with an idempotent notification batch operation.
return nil
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
if err := verifyLogContract(ctx, client); err != nil {
writeLog(event{Job: "marketplace-notifications", Status: "contract_check_failed", Error: err.Error()})
os.Exit(1)
}
runID := strconv.FormatInt(time.Now().UTC().UnixNano(), 10)
started := time.Now()
writeLog(event{Job: "marketplace-notifications", RunID: runID, Status: "started"})
err := runJob(ctx, runID)
result := event{Job: "marketplace-notifications", RunID: runID, Duration: time.Since(started).Milliseconds()}
if err != nil {
result.Status, result.Error = "failed", err.Error()
writeLog(result)
_ = ping(ctx, client, os.Getenv("HEARTBEAT_FAILURE_URL"))
os.Exit(1)
}
result.Status = "succeeded"
writeLog(result)
if err := ping(ctx, client, os.Getenv("HEARTBEAT_SUCCESS_URL")); err != nil {
writeLog(event{Job: result.Job, RunID: runID, Status: "heartbeat_failed", Error: err.Error()})
os.Exit(1)
}
}
In a real worker, generate the run ID from the scheduled slot rather than wall-clock nanoseconds if the scheduler can redeliver the same slot. The startup check deliberately reads the public, self-describing contract instead of hard-coding log-ingestion fields that may not be declared in the article. The discovery response provides the request JSON Schema, response schema, billing information, and runnable examples; use its path field when wiring the actual log sender. The example keeps the transport logic visible, while the durable idempotency record belongs beside the notification side effect, usually in the same transactional boundary. In production, cache that contract during deployment rather than adding a control-plane request to every scheduled execution.
Choosing a monitor without buying duplicate noise
Healthchecks, Cronitor, Better Stack, and Datadog can all cover scheduled-job monitoring, but they fit different operating models. Evaluate them against the signal path you actually need, not the length of the feature list.
| Option | Practical fit | Boundary to account for |
|---|---|---|
| Healthchecks | Focused dead-man's-switch monitoring with simple ping URLs; a good fit when the main question is whether a job reported on time | Diagnosis still lives in your logs and application state |
| Cronitor | Cron-oriented monitoring with job telemetry and alerting; useful when scheduled work deserves a dedicated operational view | Adds another monitoring surface to operate and route |
| Better Stack | Heartbeats alongside a broader incident-management and observability product | Broader adoption may be unnecessary for one small cron estate |
| Datadog | Fits teams already correlating logs, monitors, and on-call workflows in Datadog | Configuration and signal volume need care to avoid duplicating existing alerts |
| Infrai plus an external heartbeat service | Plain REST log ingestion without an SDK or client-library version to maintain; its public discovery contract makes request schemas inspectable before deployment | It has no built-in heartbeat, synthetic check, threshold rule, or notification route, so missed-run detection and paging must remain external |
This is not a winner-takes-all decision. A focused heartbeat service plus the existing log backend is often the cleanest design. If an organization already standardizes on Datadog or Better Stack, using its native scheduled-job facility may reduce ownership overhead. The REST option in the last row is reasonable for structured application logs when a plain HTTP interface is the priority. Infrai uses one key across 295 routes and 20 modules, with one bill, which can reduce secret rotation and integration bookkeeping when this notification service already needs other backend capabilities. Its API is self-describing, and the public discovery surface exposes full request and response schemas without requiring a key; that lets a deployment check its logging contract before a scheduled run. That breadth is irrelevant to a team buying only cron monitoring, and then a focused product is the better fit.
The limitation is material: Infrai's log and metric query endpoints must be polled by your own alerting component if you want alerts from that stack. Their discovery parameters do not declare filters, so don't design around undocumented query fields. Choose Healthchecks or Cronitor when dedicated missed-run detection is the requirement; choose Datadog or Better Stack when consolidating with an existing observability and incident workflow matters more than keeping the cron monitor narrow.
Set one paging owner. If the heartbeat provider, a log poller, and a queue-depth alert all page on the same missed batch, operators learn to distrust all three. I prefer the heartbeat as the page for “did not complete,” with logs and queue metrics attached to the runbook as investigation signals. A separate error-rate page is justified only when it represents a different action.
One page is enough.
Where this pattern stops helping
Heartbeat monitoring is weak for continuously running consumers because there is no natural scheduled completion. For those, monitor queue age, processing rate, and consumer liveness instead. It is also insufficient when the job completes but produces incorrect business results; add an outcome check, such as an expected count or reconciliation invariant, without placing customer data in the ping URL.
Be careful with privacy requirements. A logging system without per-user deletion or bulk export may be unsuitable for personal data subject to deletion workflows. Keep notification addresses, message bodies, and user identifiers out of operational logs unless retention and deletion controls have been explicitly designed.
Finally, a successful heartbeat proves only that the code reached the ping. It does not prove every notification was delivered. Provider receipts, bounce events, and durable per-notification state answer that question. Page on the missing run; investigate with logs; reconcile delivery from business state. Those are three distinct controls.
Top comments (0)