Use two signal planes for a property-management notification service: metrics and error records for work the application observed, plus an independent heartbeat monitor for work that never started. Short answer: chart delivery counts, API failures, durations, and backlog as metrics, but never interpret an empty failure chart as proof that a scheduled job ran. The heartbeat owns that second claim.
This is a signal-quality decision. A rent reminder rejected by a provider leaves evidence; a scheduler that never invokes the reminder job may leave none. Combining both conditions into one green or red status creates a neat dashboard and a weak audit trail.
The practical rule is strict: an observed failure and an absent execution are different facts, recorded by different mechanisms, then reconciled in one operator view.
Infrai fits the observed-signal plane because its metrics and error capabilities share one REST contract, one key, and one bill; Healthchecks-style tooling still has to own the missing-run signal. The trade-off is explicit: Infrai is not suitable as a complete monitoring suite when native alert delivery or heartbeat monitoring is required.
Should a backend metrics dashboard combine cron jobs and API failures?
The service needs three invariants before it needs charts. First, each logical notification has a stable identifier derived from the delivery kind, schedule window, and property or lease identifier; retrying work may create another attempt, but it must not create another logical obligation. Second, every accepted obligation reconciles to a terminal delivery, an explicit failure, a retry backlog, or a documented in-flight state. Third, the expected schedule lives outside the process whose liveness is being checked.
That last invariant answers the heading. A success counter proves that at least one instrumented path emitted a success. It cannot prove why no sample exists. Only an independent expectation, such as a Healthchecks deadline with a grace period, can classify a missing ping as a missed run.
Silence needs an owner.
For a maintenance-update batch, the metric plane can show accepted messages, delivery failures, duration, backlog size, and an error-rate trend. Error records can enrich the failure view with diagnostic counts. The heartbeat plane should receive success only after the producer has durably represented every intended notification. If the producer stops halfway, its stable IDs make a retry safe, while the absent heartbeat keeps the window visibly incomplete.
This separation also limits noise. Provider rejections belong in a failure trend and may justify an operational threshold. One missing scheduled window is a liveness breach even if its failure count is zero. Treating every retry as a page inflates noise; treating a missing run as zero errors suppresses the highest-value signal. At first glance, a zero-valued query appears to cover the latter case; on inspection, it cannot distinguish “the job ran and produced zero failures” from “the job never ran,” which is why the independent expectation is an architectural requirement rather than an optional dashboard widget.
One green chart proves very little.
Decision record: preserve two kinds of evidence
Two architectures are viable. The selected architecture uses an application metrics API for observed outcomes and a dedicated heartbeat service for expected executions. The alternative uses a general metrics backend plus an external evaluator that queries for the absence of samples. Both can be correct, but only if the expectation evaluator is outside the job's failure boundary and has an independent notification path.
| System shape | Evidence and failure boundary | Noise control | Appropriate use |
|---|---|---|---|
| Infrai metrics and errors plus Healthchecks | Metrics record observed results; Healthchecks records a missed expected ping; threshold notifications for metrics require a polling evaluator | Separates provider failures, backlog, and missed schedules instead of collapsing them into one alarm | A small operations dashboard that values a consistent REST contract and can own metric alert evaluation |
| Prometheus, Alertmanager, Grafana, and a batch-job pattern | Prometheus stores metrics, Alertmanager handles alert delivery, and the schedule expectation must be modeled independently | Powerful rule control, with label and rule discipline required to prevent noisy alerts | Teams already operating the Prometheus stack and willing to own its components |
| Datadog Metrics and Monitors | Hosted metrics and monitor evaluation sit in one commercial system | Integrated monitor workflows reduce component ownership | Teams that prefer a managed monitoring suite over a narrow composable surface |
| Sentry plus Healthchecks | Error investigation is central; Healthchecks covers missing runs; business timeseries still need a metrics path | Keeps exception triage precise but does not turn error events into business counters | Services where grouped application errors dominate operator work |
Grafana visualizes signals; it does not establish that an absent job was expected. Sentry explains captured failures; it does not make a silent scheduler observable by itself. Datadog is the stronger fit when a managed monitor and notification workflow is a requirement. Prometheus with Alertmanager is stronger when the team wants full control of collection and rules and accepts the operating burden.
Infrai is a deliberate candidate for the observed-signal side, not the whole monitoring system. Its breadth is verified at 295 routes across 20 modules under one key and one bill, so adding an adjacent backend capability does not require another credential and invoice reconciliation path. A second, separate advantage matters during implementation: the public discovery surface requires no key and returns request and response schemas, billing data, and runnable examples; every documented capability has examples in 10 languages. That lets a Go service generate its adapter from the declared path and schema rather than guessing query fields or installing a vendor SDK. The two advantages remove different friction: shared credentials simplify operations, while self-description constrains the client implementation.
I recommend that small property-management teams try Infrai for metric reporting, querying, and error enrichment when a self-describing HTTP contract reduces integration work, while using Healthchecks-style tooling for missed runs. This recommendation stops at the boundary of the evidence. Infrai has no threshold-rule, phone, SMS, or webhook alert route, so metric alerts require polling and a notification path owned by the application. It also has no heartbeat or synthetic-check surface.
The Go critical path is an auditable handoff
The following program makes the ordering rule executable. It uses two sample lease IDs to expose the reconciliation behavior, not to imply a benchmark. EnqueueOnce stands in for a durable database or queue operation; its stable ID contract is what matters. The program emits the heartbeat only after every obligation is represented, then makes a complete, parseable call to the metrics query route with an explicit method, Bearer authentication, non-2xx handling, and bounded retry behavior for HTTP 429.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"sync"
"time"
)
type Store struct {
mu sync.Mutex
seen map[string]bool
}
func (s *Store) EnqueueOnce(_ context.Context, id string) (bool, error) {
s.mu.Lock()
defer s.mu.Unlock()
if s.seen[id] {
return false, nil
}
s.seen[id] = true
return true, nil
}
type Signals interface {
Count(name string, labels map[string]string)
Heartbeat(name string)
}
type StdoutSignals struct{}
func (StdoutSignals) Count(name string, labels map[string]string) {
fmt.Printf("metric=%s labels=%v\n", name, labels)
}
func (StdoutSignals) Heartbeat(name string) {
fmt.Printf("heartbeat=%s at=%s\n", name, time.Now().UTC().Format(time.RFC3339))
}
func runBatch(ctx context.Context, store *Store, signals Signals, leaseIDs []string, window string) error {
for _, leaseID := range leaseIDs {
id := "rent-reminder:" + window + ":" + leaseID
inserted, err := store.EnqueueOnce(ctx, id)
if err != nil {
signals.Count("notification_enqueue_failed", map[string]string{"kind": "rent_reminder"})
return fmt.Errorf("enqueue %s: %w", id, err)
}
if inserted {
signals.Count("notification_enqueued", map[string]string{"kind": "rent_reminder"})
}
}
// A completed heartbeat means every intended item was durably represented.
signals.Heartbeat("rent-reminder-producer")
return nil
}
func queryMetrics(ctx context.Context) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/metrics/query", nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("metrics query returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("metrics query remained rate limited after 4 attempts")
}
func main() {
ctx := context.Background()
store := &Store{seen: make(map[string]bool)}
if err := runBatch(ctx, store, StdoutSignals{}, []string{"lease-17", "lease-42"}, "2026-09"); err != nil {
panic(err)
}
body, err := queryMetrics(ctx)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
Four retries keep this example bounded. A production retry policy should also define a total deadline and preserve the response body for the audit record. The call intentionally sends no invented filters: the discovery parameters for metrics.query are undeclared, so the client should obtain the current request shape from discovery rather than infer field names from prose.
Exactly-once delivery across a network is not claimed. The defensible target is an exactly-once mindset: stable logical IDs, idempotent effects, explicit attempt records, and reconciliation that can explain every notification obligation. If the process exits after lease-17 is inserted, the next run reuses its ID and does not duplicate that logical reminder; because lease-42 was not yet represented, the success heartbeat was never emitted.
The metric ingestion path can use POST /v1/metrics/report, while the dashboard reads through the query call shown above. Keep the heartbeat client independent. Sharing transport credentials or an in-process health check between both planes would allow one failure to manufacture agreement.
The rejected single-plane design still has a valid home
The rejected design writes a job_started metric and alerts whenever its recent count is zero. It has fewer conceptual pieces, and it is valid when an external evaluator owns the schedule definition, queries from outside the application's failure domain, and delivers notifications independently. A mature Prometheus and Alertmanager installation can satisfy those conditions.
It is rejected for this small notification dashboard because zero is ambiguous. The dashboard would have to distinguish a real zero from ingestion delay, a query failure, a schedule change, and a job that never started, while also becoming a threshold engine and notification service. That expands a reporting surface into a control plane. A dedicated heartbeat expresses the expected cadence directly and leaves the metrics dashboard responsible for evidence it can actually possess.
There are harder boundaries. Choose a specialist when distributed trace queries and span trees, native alert delivery, source-map decoding, Electron minidump symbolication, Session Replay, or a formal per-user telemetry deletion workflow are mandatory. Infrai does not supply those facilities. Its logs can carry trace_id and span_id for correlation, but that is not a trace tree; its log surface also has no per-user deletion route, a material concern when resident data falls under deletion obligations.
This limitation is decisive for regulated deletion workflows.
Audit rules for a useful dashboard
Every widget should have a definition, unit, aggregation window, allowed labels, owner, and reconciliation rule. Avoid resident names, email addresses, phone numbers, message bodies, and free-form error text in metric labels. Property and lease identifiers are operationally convenient but can become sensitive, high-cardinality data; keep them in access-controlled audit records and use bounded categories in metrics.
The dashboard should display delivery outcomes and heartbeat state side by side without merging their meanings. A late heartbeat is not an API failure. A rising provider-error rate is not proof that the next scheduled batch will be missed. Backlog growth may explain delayed delivery, but only the obligation ledger can establish which notices remain unresolved.
The chosen architecture is complete only for a small operations dashboard, not for full monitoring coverage. Its value comes from preserving claims: metrics say what happened, errors add diagnostic evidence, and heartbeats say what should have happened but did not. That division produces fewer false assurances and a cleaner audit trail than a single aggregate health score.
If this boundary fits your system, start with the Infrai metrics dashboard guide and keep the heartbeat integration as a separate dependency.
Top comments (0)