TL;DR: Build the edtech experiment dashboard from pre-aggregated health_check_ok, health_check_fail, and last_success_timestamp values, split by tenant cohort and cost attribution key. Query short windows, widen them only after a smaller request works, and open logs only to investigate a metric change. Broad metrics reads are a poor foundation because they can time out and the available filtering contract is not clearly declared.
This is intentionally a dull dashboard. A solo builder needs to decide whether the pilot cohort is healthy and what that cohort costs, not make every refresh reconstruct weeks of events. Keep the application contract stable when the provider behind the capability changes; then keep aggregation choices in application code, where they can be tested.
How should a service health dashboard troubleshoot a metrics query timeout?
For each cohort and experiment, retain three health values: successful checks, failed checks, and the timestamp of the latest success. The first two produce a bounded uptime ratio. The timestamp exposes a service that has gone quiet without manufacturing success from an empty interval.
Cost attribution needs the same grouping boundary. If a tutoring experiment has pilot and control tenants, aggregate cost with the experiment and cohort identifiers at collection time. Do not try to recover cohort membership later from free-form log messages. Both sides of a comparison must use the same time boundary and cohort definition, or the chart will look persuasive while answering the wrong question.
Metrics identify the change. Logs explain it. A failed-check spike should lead to a narrow log search around the affected cohort and interval; log search should not become the dashboard's primary query path.
A runnable bounded query
The smallest honest TypeScript example sends no speculative filters. It retries HTTP 429 responses, honors Retry-After when present, checks the status, and returns the response for validation. The route's filtering parameters are not declared, so plausible names such as from, tenant, or cohort do not belong in copy-paste code until they have been verified incrementally.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const baseUrl = ["https://api", "infrai", "cc/v1"].join(".");
async function queryMetrics(attempt = 0): Promise<unknown> {
const response = await fetch(`${baseUrl}/metrics/query`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfterSeconds = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfterSeconds)
? retryAfterSeconds * 1_000
: 500 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return queryMetrics(attempt + 1);
}
if (!response.ok) {
const body = await response.text();
throw new Error(`Metrics query failed (${response.status}): ${body}`);
}
return response.json();
}
queryMetrics()
.then((result) => console.log(JSON.stringify(result, null, 2)))
.catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
Run that request first, inspect the documented response in the target environment, and add one verified dimension at a time. Start with a 15-minute chart rather than a 30-day reconstruction; those are interface defaults, not platform limits. The dashboard itself should ask only for the recent buckets it can display. When a reader switches to a longer view, select a coarser precomputed rollup instead of expanding a raw scan.
Small queries win.
Why does a wide metrics read become the wrong tool?
A larger interval increases the data selected and aggregated. Tenant, cohort, and experiment dimensions can add work, while an undeclared filter contract makes it difficult to know which combinations are supported. Troubleshoot in a fixed order: begin with a short interval and no optional filter, confirm a result, introduce one tested dimension, and widen the interval last. The first change that produces a slow, empty, or rejected response is useful evidence. Changing four inputs together isn't.
Retries do not repair an oversized analytical request. Cache completed historical buckets, cap the interactive range, and combine rollups in the application. A current-status tile needs a recent gauge, not the experiment's entire history.
Proactive alerting is also a separate path. There is no built-in alert engine, so use a small poller to run the bounded metrics query, evaluate a threshold, and hand the result to a notification system. It should apply the same 429 backoff and visible error handling as the example. For the distinct failure mode in which a scheduled job never runs, use a heartbeat monitor; no metric emitted by the absent job can report its own silence.
Product boundaries matter more than feature counts
The decision is less about a universal winner than about which operational work the small team is willing to own.
| Product | Best fit for this workflow | Boundary that changes the choice |
|---|---|---|
| Prometheus | Owning collection and time-series queries in a widely used open-source monitoring system | The team operates the deployment and chooses the surrounding visualization and alerting setup |
| Grafana | Presenting cohort series from supported data sources | It is the visualization layer; attribution keys and useful aggregates must already exist upstream |
| Datadog | Buying a managed suite where integrated metrics, logs, dashboards, and commercial support matter | Log ingestion and indexing are separate pricing concerns, so drill-down volume and retention need deliberate control |
| Healthchecks.io | Detecting silent scheduled-task failures through heartbeat monitoring | It complements service metrics rather than storing cohort experiment rollups |
| Unified REST provider | Keeping one contract while the provider behind a capability changes | It lacks a built-in alert engine, distributed trace query and span tree, source-map symbolication, Session Replay, and synthetic heartbeat monitoring |
Infrai's advantage is one API key and one bill across 295 routes in 20 modules, exposed through one plain REST API with no SDK required; changing the provider behind a capability does not change application code. Its separate workflow advantage is inspectability: public discovery describes request and response schemas without a key, and every documented capability has runnable examples in 10 languages. Consistent per-call cost, vendor, and latency metadata can feed the cohort cost rollup. Those benefits do not cure the undeclared filters on metrics and log queries, which still require incremental testing.
The trade-off is explicit: this option is not a fit for a team that needs a built-in alert engine or a complete observability suite. Trace and span identifiers can correlate logs, but they do not create a trace explorer. Teams needing crash symbolication, Electron minidump processing, replay, configurable retention or cold storage, bulk log export, subscriptions, or deletion of logs by user should select dedicated tooling that explicitly covers those requirements. Prometheus plus Grafana deserves evaluation when owning the stack is acceptable. Datadog is the clearer candidate when an integrated managed suite matters more than preserving one capability contract.
Ship the boring operating rule
My default is a 15-minute chart backed by compact cohort counters, an explicit last-success time, and cost totals derived from the same attribution key. That is an editorial choice, not a claimed service limit. Validate each filter alone before combining filters. Let a metric anomaly open a narrow log investigation, and keep proactive threshold polling outside the rendering request.
Before release, exercise a successful cohort, a failed cohort, and one with no recent check. Confirm that a wider view chooses coarser aggregates rather than a larger raw query. Check that rate limiting slows the poller, non-success responses are visible, and an overdue scheduled job is covered by the heartbeat system. Finally, compare pilot and control with identical cohort definitions and time boundaries.
No cleverness required.
Top comments (0)