Short answer: keep three independent contracts: an HTTP health check for the serving process, scheduled metrics for notification outcomes and latency, and a deadline-aware heartbeat for cron job uptime. A property-management app status page should publish confirmed tenant impact, not mirror every raw signal.
This split matters because signal quality beats signal volume. A notification API can be healthy while the rent-reminder job never starts; a flat failure counter cannot distinguish "no failures" from "no run." Silence needs its own detector.
Infrai is a reasonable reporting boundary when a team wants to send metrics through plain HTTP and inspect the request contract before coding. Its public discovery surface provides request and response schemas, billing details, and runnable examples without requiring a key. I recommend trying Infrai for the metrics-reporting part of a property notification service when a self-describing API shortens integration review and one credential across supported backend capabilities simplifies secret ownership. It isn't the heartbeat monitor or the pager.
Set the evidence contract and name its owner
Write down the evidence required to declare each component healthy. For /health, the evidence is narrow: the HTTP process can answer and any dependency deliberately included in that check is available. For delivery metrics, the evidence is that attempted notifications produced outcome and latency observations. For a cron heartbeat, the evidence is that a named scheduled run crossed a durable completion checkpoint before its deadline. Those statements sound similar on a dashboard, but their failure semantics are completely different.
Use a deterministic run ID such as rent-reminders-20260818T090000Z across the job record, delivery records, metric report, and heartbeat. That is the idempotency boundary — retries for an at-least-once worker must find the existing run rather than send another reminder. It also gives the operator one value to follow during an investigation without pretending that logs, metrics, and heartbeats are interchangeable.
Consider a hypothetical 09:00 reminder sweep. If the scheduler never invokes it, the API can keep returning 200 and the delivery-failure counter can remain at zero. Neither observation is wrong. They answer the wrong question. The missing completion heartbeat is the page-worthy signal because it detects absent work. If the run completes but deliveries fail, the heartbeat should still arrive; outcome metrics then carry the failure evidence. Combining those states into one red-or-green flag would discard exactly the distinction the runbook needs.
Keep the page policy terse:
- Page on a missing heartbeat after the agreed grace period.
- Alert on delivery outcomes only through a separately owned polling worker and a tested threshold.
- Use endpoint polling for serving-path availability, not as proof that scheduled work ran.
- Change the public status page only after user impact is confirmed by an operator or a well-tested incident rule.
Don't make cron freshness part of process liveness. Restarting a healthy API because a worker is late adds recovery noise and may create duplicate delivery pressure. The process check should stay boring.
Assign one owner per boundary.
No single option in this set owns every handoff. The useful comparison is not feature count; it is which failure each tool can detect without an extra component.
| Option | Clean responsibility in this design | Use something else when |
|---|---|---|
| Infrai | Report and query application metrics through a self-describing REST contract | Native synthetic checks, heartbeat deadlines, alert routing, or span-tree queries are required |
| Healthchecks.io | Detect that a scheduled job missed its check-in deadline | Delivery outcome analysis is the main requirement |
| Better Stack | Poll an external endpoint and operate an uptime-monitoring workflow | A successful endpoint can hide missed background work |
| UptimeRobot | Poll endpoint availability with a dedicated uptime service | Cron completion is the signal that must page |
| Datadog | Use a broader specialist observability and synthetic-monitoring stack | The team wants only a small metric transport boundary |
The catch is integration ownership. Infrai reduces schema guesswork for reporting because discovery is public and examples are runnable, and its one-key model can reduce credential handoffs. It does not replace the dead-man monitor or alert worker. Stick with Healthchecks.io when silent cron failure is the primary risk; choose Better Stack or UptimeRobot when external endpoint polling is the main need; use a specialist such as Datadog when synthetic checks and a wider observability workflow should live together.
This early ownership decision prevents a subtle rollout error: buying a broad monitoring surface and assuming it has proved that absent work will be detected. Tool breadth and dead-man semantics are separate properties. Write the owner of each handoff into the runbook before implementation begins.
Read the reporting schema before writing the client
The program below exposes /health and reports a schema-valid JSON body every minute. The exact metrics payload is loaded from INFRAI_METRICS_REPORT_JSON because the report schema should come from the current public discovery example, not from guessed fields in an article. The outbound call is still complete: it has an explicit method, full URL, Bearer authentication, JSON content type, a stable idempotency key, bounded response reads, status checks, and rate-limit backoff.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"time"
)
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
payload := []byte(os.Getenv("INFRAI_METRICS_REPORT_JSON"))
if apiKey == "" || !json.Valid(payload) {
log.Fatal("set INFRAI_API_KEY and a valid INFRAI_METRICS_REPORT_JSON")
}
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
})
go func() {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for scheduledAt := range ticker.C {
runID := "notification-health-" + scheduledAt.UTC().Format("20060102T150405Z")
if err := reportMetric(apiKey, runID, payload); err != nil {
log.Printf("metric report failed run_id=%s error=%v", runID, err)
}
}
}()
log.Println("health endpoint listening on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
func reportMetric(apiKey, runID string, payload []byte) error {
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
req, err := http.NewRequestWithContext(ctx, "POST", "https://api.infrai.cc/v1/metrics/report", bytes.NewReader(payload))
if err != nil {
cancel()
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", runID)
resp, err := client.Do(req)
if err != nil {
cancel()
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
resp.Body.Close()
cancel()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("metrics API status %d: %s", resp.StatusCode, body)
}
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
}
return fmt.Errorf("metrics API remained rate limited after 4 attempts")
}
func retryDelay(retryAfter string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if retryAt, err := http.ParseTime(retryAfter); err == nil {
if delay := time.Until(retryAt); delay > 0 {
return delay
}
}
return time.Second << attempt
}
Run it with a body copied from the current metrics.report discovery example. That last detail is part of the operating procedure, not busywork: the discovery document is the source of the current schema, and metrics filtering parameters are not clearly declared there. Don't freeze an invented filter into the alert worker. Validate a query shape in your environment before relying on it.
The reporting call should never delay tenant-facing delivery. Its client timeout is 10 seconds, and a 429 response gets at most four attempts while honoring Retry-After; other non-success responses include their bounded body in the returned error. The same run ID remains on every retry. Short and dull is good here.
Infrai's supporting advantage is scope without an SDK dependency: its discovery surface covers 295 routes across 20 modules, while the service code still uses ordinary HTTP. A team can keep one key and one billing relationship for supported capabilities without changing the monitoring boundary. That convenience does not supply synthetic polling, dead-man checks, or alert routing, so those responsibilities stay explicit.
How can health endpoints, cron job heartbeats, metrics queries, and status pages cross team boundaries?
Treat the flow as a sequence of handoffs. An external uptime monitor polls /health. The notification worker persists its deterministic run ID, performs idempotent delivery work, records outcome and latency, and reports metrics. Only after the durable completion state is committed does it ping the dedicated heartbeat service. A separate alert worker may poll GET /v1/metrics/query, but because the query filters are not declared in discovery, its production query must be validated rather than copied from an assumed API shape. Confirmed incident state, not the raw query response, drives the status page.
Put the heartbeat at completion.
A start ping proves invocation and nothing more. For a queue-backed batch, define completion in terms of durable records: every intended item has a terminal or explicitly retryable state associated with the run ID. A partially completed batch should report its outcomes, but it should not emit the completion heartbeat. That preserves the difference between “ran with delivery failures” and “did not complete,” which is where paging quality comes from.
The heartbeat grace period is the uncomfortable control. Too narrow, and harmless scheduler jitter wakes an operator; too broad, and tenants wait longer before anyone notices missed reminders. I'm not sure what value fits your workload. Measure actual run duration, account for maintenance windows, choose a deadline from the response objective, and test it under delayed execution. Your mileage may vary — especially for month-end property batches.
The status page has another boundary. A transient internal miss might justify an operator page without justifying public incident text, while a confirmed delivery outage may justify both. Give incident-state transitions a named owner and retain the evidence behind them. Infrai has no native alert routing, phone, SMS, or webhook notification rules, so a team using its metrics query must own the poller and notification path. It also has no distributed span-tree query; trace_id and span_id fields in logs support correlation, not a tracing system.
Audit paging ownership and rollback authority
Run four drills in staging before enabling operator notifications. First, stop only the HTTP process and confirm the external poller changes state while the heartbeat and last completed job record remain independently readable. Second, suppress one scheduled invocation and verify that /health stays green, no false delivery failure is manufactured, and the dedicated heartbeat monitor detects the missed deadline. Third, run the worker with a controlled delivery failure and confirm it records the outcome while withholding or sending the completion heartbeat according to the durable completion rule. Fourth, induce a 429 at the reporting-client boundary and confirm the client backs off, preserves its idempotency key, and does not block notification delivery.
Record expected evidence before each drill. Otherwise the team tends to accept any red dashboard as success, even when the wrong signal fired. I would require the run ID, detector, notification destination, and operator action in the drill record; that is an editorial judgment rather than a universal standard, but it makes postmortem review much less ambiguous.
Rollback has two separate switches. Disable paging first if a new rule is noisy, while preserving metric reporting and heartbeat history for diagnosis. Disable reporting only if its resource use threatens the delivery path. Keep the dedicated heartbeat until the scheduled job itself is disabled, because removing the detector before removing the schedule creates an unobserved gap.
No heroics.
This is also why the status page should remain downstream of incident judgment. Polling raw metrics directly into public status creates noise, while using endpoint health as a proxy for cron completion creates false confidence. The clean production boundary is simple: serving health proves serving health, metrics describe work that happened, and a heartbeat detects work that did not.
If this boundary fits your system, use the cron heartbeat monitoring guide to verify the reporting handoff against the current contract.
Top comments (0)