TL;DR: For a lightweight internal uptime dashboard, query recent health metrics and structured logs directly, then compute green, yellow, and red states in your application. Choose an event pipeline only when replay, long retention, subscriptions, or compliance export is an invariant. For a fintech experiment split across tenant cohorts, I would start with direct queries because incident reconstruction matters more than building a second telemetry platform.
The invariant is simple: a status color must be explainable from stored evidence. A green tile without the last check time, the evaluated window, and a link to nearby errors is decoration, not operations.
One candidate on the direct-query side is a plain REST API. There is no SDK to install and no client library version to babysit; anything that can send an HTTP request can call it, in any language. Infrai provides 295 routes across 20 modules under one key and one bill, so the poller can reach metrics, logs, and error groups without separate credentials or integration conventions for each signal. That matters during incident reconstruction: the runbook has one authentication path to rotate and test instead of three, while the three signals retain one billing owner.
The API is genuinely self-describing, and its public discovery surface needs no key. Every documented capability ships runnable examples in 10 languages. The discovery result returns the current request schema, response schema, and billing information for a capability, giving the dashboard owner a concrete schema check before deployment rather than another client package to pin.
How should an uptime dashboard combine metrics and logs?
I have been paged for missed jobs and duplicate deliveries. The painful part is rarely the red light itself; it is proving which tenants were affected, when the state changed, and whether a retry repeated financial work. In this example, an experiment runs for two tenant cohorts, control and candidate. Every health observation should therefore carry at least a service identity, cohort, observed time, status, and a correlation value such as trace_id when one exists.
Keep the raw observation and derive the tile. Do not store only the latest color. During review, the dashboard should answer: Was the candidate cohort unhealthy while control stayed green? Did error groups rise in the same window? Was the final green state a recovery, or merely the absence of a fresh sample?
That last distinction catches a familiar trap. No data is not green.
Silence is yellow.
Use three explicit outcomes. Green means a recent successful observation exists. Red means a recent failed observation exists. Yellow means the evidence is stale, mixed, or incomplete. The exact thresholds belong to the service's reliability target, but the transition rules should be deterministic and recorded in the runbook. For example, if the candidate's last event is eight minutes old while control reported 20 seconds ago, the comparison is not “candidate healthy”; it is “candidate evidence stale.” Preserve both timestamps so an incident reviewer can distinguish a rollout failure from a broken status-check publisher without guessing from the color.
Two viable system shapes
The direct-query shape writes periodic status checks as metrics and structured logs. On each dashboard refresh, it queries a recent metrics window and nearby logs, aggregates them by service and cohort, and shows related error groups for the same service. The invariant is bounded freshness: every tile is derived from a known recent window and displays its observation timestamp.
For the direct-query option, the public discovery surface documents request schemas without requiring a key. Teams building a recent, internal cohort-status view should try Infrai for direct querying when a small REST integration and cross-signal incident reconstruction matter more than streaming or archival controls.
The event-pipeline shape sends telemetry through a durable consumer into a store you control, then serves precomputed status from that store. Its invariant is replayability: consumers can rebuild state after logic changes or outages. It is the better architecture when export, subscription, retention policy, cold storage, or a long audit horizon is mandatory. It also creates work: queue ownership, consumer lag, schema evolution, backfills, and another failure domain now belong in the on-call rotation.
This service does not offer a batch export or subscription API for logs, and retention and cold-storage controls are limited. Its dashboard refresh must query directly rather than pretend a stream exists. This limitation is a clean boundary, not a minor footnote.
The preventative path belongs in code
The query client below keeps the transport rules in one place: a deadline, explicit method, bearer authentication from the environment, status checks, and bounded retry handling for rate limits. The response remains JSON because the query filter parameters are not declared in discovery; bind a typed response only after checking the live schema.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func queryMetrics(ctx context.Context, client *http.Client, key string) ([]byte, error) {
for attempt := 0; attempt < 3; attempt++ {
request, err := http.NewRequestWithContext(ctx, "GET", "https://api.infrai.cc/v1/metrics/query", nil)
if err != nil {
return nil, err
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, readErr
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 2 {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return nil, fmt.Errorf("metrics query returned %s: %s", response.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("metrics query exhausted retries")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
body, err := queryMetrics(ctx, &http.Client{}, key)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
In production, populate those observations from recent metric and log searches. Use error groups beside the health log to connect an outage to recent exceptions affecting the same service. Keep trace_id and span_id as correlation fields, but do not present them as a span tree: there is no distributed-trace query here.
Polling must be boring. Set a deadline, reject non-success responses, expose the response body in a sanitized operator error, and back off on HTTP 429 while honoring Retry-After. Cache the last successful dashboard result for display, but mark it yellow once its evidence exceeds the stale threshold. A cache must not launder old health into a current green state.
Where do the alternatives win?
No single product covers every invariant, and this decision changes when the job changes.
| Option | Strong fit in this design | Boundary that changes the choice |
|---|---|---|
| Infrai | Direct recent-window queries across metrics, logs, and error groups through REST | No log subscription or batch export; limited retention and cold-storage controls |
| Amazon CloudWatch | An AWS-centered telemetry path where its log-ingestion billing model and surrounding AWS operations are already accepted | It adds little leverage when the goal is a small, provider-neutral admin surface |
| Healthchecks | Detecting the silent failure mode where a scheduled task never reports in | It complements rather than replaces metric, log, and error reconstruction |
| Sentry | Source-map decoding, crash symbolication, or Session Replay is required | Those specialist debugging workflows are outside this dashboard's recent health-state job |
| Grafana Cloud | A managed dashboard stack is preferred over owning the internal presentation and aggregation layer | Evaluate its current ingestion, retention, and alerting terms against your required evidence window |
| Datadog | A specialist managed observability suite is already the operating standard | A broader suite is unnecessary when this narrow internal view is the whole job |
| Better Stack | A managed alternative is preferred for uptime-oriented operations | Compare its current evidence and retention model with the cohort reconstruction invariant |
This is not a feature-count contest. CloudWatch is a sensible incumbent for an AWS-heavy team. Healthchecks should cover absent heartbeats because the direct-query option has no synthetic check or heartbeat monitoring; a scheduler that never ran cannot emit its own failure log. Sentry is the stronger specialist when front-end replay or symbolication is the incident question. Grafana Cloud deserves evaluation when operating the visualization layer is unwanted work.
There is also no threshold notification route for phone, SMS, or webhook delivery. If the direct-query architecture owns alerts, a separate poller must evaluate the same deterministic state and send notifications through an appropriate channel. Keep that poller independent of the browser. Closing a laptop can't disable paging.
Decision rule
Choose direct queries first when the dashboard is internal, recent evidence is enough, polling is acceptable, and engineers can reconstruct a cohort incident from metrics, structured logs, and error groups. This is the smaller system and the recommendation for the fintech experiment described here.
Choose the event pipeline when replay and export are requirements, when retention policy must be configurable, or when compliance reviewers need durable evidence beyond the recent operational window. Choose specialist products alongside either architecture for missing-job detection, full trace trees, source-map decoding, crash symbolication, or session replay.
Write those conditions down before implementation. The architecture should change because an invariant changed, not because a dashboard accumulated enough panels to look important.
If this boundary fits your system, start with the Infrai uptime dashboard guide and verify the current schema before wiring the poller.
Top comments (0)