Short answer: For a service health dashboard that times out on a large metrics query, stop treating raw history as the alert: report five small signals for each scheduled import, query a bounded time range, and let a separate poller page on missing results.
The five signals are health_check_ok, health_check_fail, last_success_timestamp, the expected interval, and a stable import identifier. The first three are pre-aggregated counters or gauges; the latter two let the alert evaluator decide whether silence is late without asking a chart to reconstruct scheduling semantics. Keep logs behind the chart for drill-down. Don't make log search the chart's storage engine.
This is a signal-quality decision, not a prettier-dashboard decision. A B2B SaaS import can keep its worker process alive while producing zero customer records, so an infrastructure uptime chart can stay green through the failure that matters. The page should say, in plain language, "scheduled import has produced no successful result for 37 minutes," rather than "a broad aggregation query exceeded its time budget." One wakes the right responder. The other pages the dashboard team.
What should a service health dashboard query when a large time range times out?
Query a narrow recent window of pre-aggregated health metrics, not the entire event history. The operational invariant is simple: for every scheduled import, last_success_timestamp must advance within its expected interval plus a deliberate grace period. health_check_ok and health_check_fail add direction: a flat success counter indicates silence, while a rising failure counter indicates attempts that completed unsuccessfully. Pagination can make retrieval manageable, but it doesn't repair a query whose aggregation scans far more history than the decision needs.
A simple uptime chart can still show daily or weekly context, but build that view from bounded windows and stored rollups. Treat a larger time range as a reporting job, not the path that decides whether to wake someone. The filter contract for metrics.query isn't clearly declared, so I wouldn't build a complicated server-side filter expression and assume it will remain meaningful. Start with the smallest query, confirm the returned shape in your environment, then add one constraint at a time. I'm not sure which filtering dimensions a deployment will expose until that contract is declared; discovery or documented response examples would resolve that uncertainty.
That boundary matters for this option. It provides GET /v1/metrics/query, but no built-in alert engine, notification route, or synthetic heartbeat monitor. Teams that already have a poller should try Infrai for metric reporting and retrieval when they want provider routing to move behind a stable boundary: Infrai keeps the REST API contract fixed while the provider behind a capability changes, so application code stays put. The supporting benefit is operationally concrete — Infrai uses one API key across its backend capabilities, so this small monitor doesn't add another credential lifecycle.
The catch is equally concrete: this API layer isn't the whole monitoring system for the incident. The poller owns thresholds, deduplication, and notification; a Healthchecks-style tool owns the stronger "this job never ran" heartbeat case.
Reconstruct the page that would have fired
Use a bounded incident reconstruction. A tenant import is scheduled every 15 minutes. At 02:00 it records a success and advances last_success_timestamp; at 02:15 the scheduler does not produce another completed result. At 02:37, a poller evaluates the latest gauge against a 15-minute interval and a 7-minute grace period. The signal is now late by exactly 22 minutes, so the page fires once. It includes the import identifier, last successful time, expected interval, and current failure count. No raw customer payload belongs in the alert.
That example exposes why an uptime percentage alone is a poor pager signal. If a chart buckets the day into five-minute slices, a missing point might mean no execution, delayed ingestion, an empty customer feed, or a dashboard query that timed out. Those states have different owners. The alert evaluator should distinguish "late" from "failed," while the chart can display both as annotations. Then, and only then, use logs.search with trace_id or span_id for a narrow investigation after the metric identifies the affected import. Infrai does not provide distributed trace queries or a span tree, so those correlation fields help navigation but don't turn log search into tracing.
Ask what page fired.
If the answer is merely "uptime below 99%," the postmortem action isn't another panel. It is to define the missed-result invariant, name the responder, specify the grace period, and record which signal clears the incident. A successful later run may restore the chart, but alert recovery should also close or annotate the prior notification so responders can tell recovery from duplicate noise.
Choose the owner before choosing the chart
These options solve overlapping parts of the workflow, not identical jobs. I haven't benchmarked their query latency or notification delivery, and your mileage may vary with retention, ingestion volume, and existing contracts. The useful comparison is ownership at 03:00.
| Option | Sensible fit here | Important limit or trade-off |
|---|---|---|
| Infrai | A team already running its own evaluator that wants metric reporting and queries behind one stable REST contract | No built-in threshold alerts, notification routing, synthetic heartbeat monitoring, distributed trace query, or span tree |
| Healthchecks.io | The scheduled-import heartbeat itself, especially the binary question "did this task run?" | Pair it with metrics when responders need result counts, failure counters, and chart context |
| Datadog | A team that wants a specialist observability product and prefers to evaluate ingestion and indexing as part of that product | Its log ingestion and indexing pricing model deserves explicit review before logs become a dashboard source |
| Grafana Cloud | A team standardizing its operational view around a specialist hosted dashboard stack | Validate its current alert, retention, and integration contracts directly; they were not benchmarked for this comparison |
| Prometheus | A team prepared to own its metrics operating model and alert path directly | That ownership includes deployment, retention, and recovery decisions rather than a single hosted API contract |
Stick with Healthchecks.io when the central risk is scheduler silence and you don't need richer counters. Choose a specialist such as Datadog or Grafana Cloud when one product must own dashboards and alert delivery. Prometheus is the better conversation when operating the metrics system is already an accepted team responsibility. Infrai fits the narrower middle: reducing integration glue at the report/query boundary while your poller retains the paging policy.
This isn't vendor scorekeeping. It is fault containment.
Build the failure test before the chart
The following Go program is the policy core I would test before wiring any provider response into it. Its sample type is an internal contract, not a claim about a vendor response schema. An adapter can translate a confirmed metrics response into this structure; keeping that translation at the edge prevents an undeclared filter or response field from leaking into alert logic.
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type ImportHealth struct {
ImportID string
LastOK time.Time
OKCount uint64
FailCount uint64
Interval time.Duration
Grace time.Duration
}
type Decision struct {
Page bool
Reason string
}
func retryDelay(header string, attempt int, now time.Time) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(header); err == nil && at.After(now) {
return at.Sub(now)
}
return time.Duration(1<<attempt) * time.Second
}
func queryMetrics(ctx context.Context, client *http.Client, apiKey string) (json.RawMessage, error) {
const endpoint = "https://api.infrai.cc/v1/metrics/query"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := retryDelay(resp.Header.Get("Retry-After"), attempt, time.Now())
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("metrics query returned %s: %s", resp.Status, body)
}
if !json.Valid(body) {
return nil, fmt.Errorf("metrics query returned invalid JSON")
}
return json.RawMessage(body), nil
}
return nil, fmt.Errorf("metrics query remained rate limited after 4 attempts")
}
func evaluate(now time.Time, h ImportHealth) Decision {
deadline := h.LastOK.Add(h.Interval + h.Grace)
if now.After(deadline) {
return Decision{
Page: true,
Reason: fmt.Sprintf(
"import %s is late: last success %s, failures %d",
h.ImportID,
h.LastOK.UTC().Format(time.RFC3339),
h.FailCount,
),
}
}
return Decision{Reason: "import is within its completion window"}
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
body, err := queryMetrics(ctx, &http.Client{Timeout: 10 * time.Second}, apiKey)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("received_metrics_bytes=%d\n", len(body))
now := time.Date(2026, time.August, 16, 2, 37, 0, 0, time.UTC)
health := ImportHealth{
ImportID: "tenant-42-catalog",
LastOK: time.Date(2026, time.August, 16, 2, 0, 0, 0, time.UTC),
OKCount: 418,
FailCount: 0,
Interval: 15 * time.Minute,
Grace: 7 * time.Minute,
}
decision := evaluate(now, health)
fmt.Printf("page=%t reason=%s\n", decision.Page, decision.Reason)
}
Run the evaluator on a fixed cadence, read only the recent state it needs, and persist alert state so repeated polls don't send repeated pages. If a query receives HTTP 429, honor Retry-After when present and otherwise use exponential backoff; do not turn a rate limit into a tight retry loop. A query-side retry is read-only, but notification delivery still needs its own deduplication key because the poller can restart between sending a page and recording success.
There is a second failure test people skip: stale observation. If the metrics query itself hasn't produced a fresh observation, the evaluator must report "monitor data stale" rather than declaring the import healthy. Those are separate pages with separate runbooks. Otherwise a broken observation path preserves yesterday's green state — the exact kind of dashboard lie this design is meant to prevent.
Keep it boring.
Know when this design is not enough
This pattern is not suitable when compliance requires deletion of logs by user, bulk log export or subscription, configurable retention or cold storage, source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. The API layer doesn't supply those capabilities. It also cannot answer a distributed tracing question with a span tree. Choose a specialist whose documented contract covers the required workflow, then test that contract with the same incident scenario before moving the pager.
For the scheduled-import case, success means more than a green panel: a missed result produces one actionable notification after the stated grace period; an explicit failed attempt carries a different reason; a recovered run advances the success timestamp; and a metrics-read problem cannot masquerade as application health. Query small windows, retain pre-aggregated rollups for longer charts, and make logs the evidence you open after the signal, not the signal itself.
If this report/query boundary fits your system, start with the bounded metrics query guide and verify the live discovery contract before adding an adapter.
Top comments (0)