Why a service health dashboard's metrics query times out on large time ranges

robertmiller4179 — Tue, 04 Aug 2026 10:18:46 +0000

If you just want the recommendation: keep the uptime dashboard down to three or four pre-aggregated counters, query them over the shortest time range that actually answers the question, and treat metrics as the index rather than the evidence. A broad metrics query over a large time range is where a service health dashboard starts to crawl and then hit a timeout, and the fix is almost always upstream of the query — in what you record, not in how you chart it.

I run cron and queue infrastructure. Dashboards are how I find out whether last night went badly.

The tile was green for 9 hours and the job had already stopped writing

Last spring I moved a nightly reconciliation from a Node.js cron container to a Go worker behind a queue. The scheduler fired on time, the handler returned 200, and the health tile stayed a calm green the whole night. What actually happened is that the worker accepted the job, kicked off a goroutine, and lost its context the moment the HTTP handler returned — so the reconciliation itself wrote nothing at all. My chart was drawn from a counter that incremented when the dispatch succeeded, not when the work finished. I assumed the 200 meant the side effect had happened. 9 hours later a support ticket told me what the uptime chart couldn't: about 40,000 rows had gone unreconciled, and the graph had been flat green through every one of them.

The metric was honest about the thing it measured. I had measured the wrong thing.

That's the invariant I keep coming back to in every postmortem since: a health signal must be emitted by the code path that produces the side effect, after the side effect, and it must carry the timestamp of the side effect rather than the timestamp of the request. Everything else in this article is downstream of that one rule. If you get it wrong, no amount of query tuning saves you, because the dashboard will keep answering a question you didn't ask — and it will answer it fast, which makes it worse. A slow dashboard makes you suspicious. A fast, confidently wrong one doesn't.

How should I query metrics for a large time range without hitting a timeout?

Short version: don't ask the store to do at read time what you could have done at write time. A query that scans a large time range, groups by three labels and computes an aggregation on the fly is doing per-request work proportional to your retention window. Ninety days of raw points behind a 30-second refresh is a self-inflicted load test.

Three things have kept my panels under a second.

Query narrow windows and stack them. A day of data for the "is it up right now" panel, and a separate, slower panel for the 30-day trend that nobody stares at during an incident. Most dashboard timeouts I've troubleshot came from one greedy panel dragging the whole page down with it, not from the store being slow in general.

Second, keep the series count small and boring. health_check_ok, health_check_fail, last_success_timestamp — three names, low cardinality, one increment per run. A gauge holding the epoch seconds of the last successful run is the single most useful thing on my wall, because now() - last_success_timestamp is the number that pages me, and it costs one point per job to maintain.

Third, don't build a chart out of paginated log lines. Log search is a drill-down tool: the metric tells you when something went sideways, then you page through logs for that ten-minute slice to find out what. Rebuilding an uptime percentage by paginating raw lines and aggregating them client-side works for a week and then quietly becomes the slowest thing you own. Pagination is for reading; aggregation belongs to the write path.

One practical note on filters. On some hosted APIs — including Infrai's metrics query surface — the filter parameters aren't spelled out in the machine-readable schema, so I add one parameter at a time and look at what comes back instead of assuming a syntax borrowed from another vendor. Tedious, but it takes ten minutes and it saves you from a dashboard that silently filters nothing.

Pre-aggregate at write time, not at chart time

The reporter below is the shape I use for a Go job. It's deliberately dull: explicit method, key from the environment, an idempotency key derived from the run so a retry after a network blip never double-counts, backoff that honours Retry-After, and a status check that surfaces the response body instead of assuming success.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type sample struct {
    Name  string  `json:"name"`
    Value float64 `json:"value"`
}

// report emits one pre-aggregated point. runID makes the retry idempotent.
func report(ctx context.Context, c *http.Client, s sample, runID string) error {
    body, err := json.Marshal(s)
    if err != nil {
        return err
    }
    backoff := 500 * time.Millisecond
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost,
            "https://api.infrai.cc/v1/metrics/report", bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", runID+":"+s.Name)

        resp, err := c.Do(req)
        if err != nil {
            return err
        }
        raw, _ := io.ReadAll(resp.Body)
        resp.Body.Close()

        if resp.StatusCode < 300 {
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("metrics report %s: %s", resp.Status, string(raw))
        }
        wait := backoff
        if ra, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            wait = time.Duration(ra) * time.Second
        }
        time.Sleep(wait)
        backoff *= 2
    }
    return fmt.Errorf("metrics report: exhausted retries for %s", s.Name)
}

Wiring it into the job is the part people skip, so here it is in full. Note where the counters live: inside the branch that knows whether the work landed.

func reconcile(ctx context.Context) (int, error) {
    rows := 0
    for _, batch := range []int{100, 100, 40} {
        if ctx.Err() != nil {
            return rows, ctx.Err()
        }
        rows += batch // your real write goes here, synchronously
    }
    return rows, nil
}

func main() {
    ctx := context.Background()
    c := &http.Client{Timeout: 10 * time.Second}
    runID := time.Now().UTC().Format("2006-01-02T15")

    rows, err := reconcile(ctx)
    if err != nil {
        _ = report(ctx, c, sample{Name: "health_check_fail", Value: 1}, runID)
        fmt.Fprintln(os.Stderr, "reconcile:", err)
        os.Exit(1)
    }
    _ = report(ctx, c, sample{Name: "health_check_ok", Value: 1}, runID)
    _ = report(ctx, c, sample{Name: "last_success_timestamp",
        Value: float64(time.Now().Unix())}, runID)
    fmt.Println("reconciled", rows, "rows")
}

The dashboard then reads those three names back with a GET /v1/metrics/query and draws a chart with no aggregation of its own. What I like about doing it over plain HTTP is the operational boredom of it: no SDK to pin in my go.mod, one key, and the contract stays where it is while the thing behind it moves — swap the vendor behind a capability later and this file doesn't change. For an SRE that matters more than any feature list, because the reporting path is the one piece of code that has to keep working while everything around it is on fire.

Where each of these tools actually fits

None of this is an argument for one product. The honest split, from running several of these in anger:

Approach	How data gets in	Query surface	Where it fits	Main limit
Prometheus + Grafana	scrape a `/metrics` endpoint	PromQL	you already run the infra and want full control	you operate, shard and back it up yourself
Datadog	agent plus API	tag-based builder	big fleets, host and APM depth in one place	volume-based ingest and indexing pricing
Better Stack	hosted probes plus log ingest	hosted search	uptime probes, status pages, on-call rotation	less natural for custom in-app counters
Infrai	one REST call per counter	metrics query API	small services that want counters without an agent	lacks an alert engine and synthetic probes

If you already run Prometheus, stick with it — a counter and a last_success_timestamp gauge scraped by something you control is hard to beat, and Grafana will render the chart without you writing a line of dashboard code. Datadog earns its keep once you need host metrics, traces and logs correlated by the same tags, though the bill tracks ingest volume closely enough that teams end up sampling. Better Stack is the one I'd reach for if the actual requirement is a public status page.

When this advice doesn't apply

The catch is that "small and pre-aggregated" is a deliberate trade. You lose the ability to ask a question you didn't think of in advance. If your troubleshooting workflow is genuinely exploratory — slice by customer, then by region, then by build SHA — you want a high-cardinality store and you should pay for one rather than pretending three counters will cover it.

A few other places where I'd not follow my own advice. If you need distributed traces with a span tree, none of the counter-based setups here help; go OpenTelemetry plus a backend that ingests spans. If you need proactive alerting and your platform lacks a rules engine, run a tiny poller — a cron job that reads the metrics query API every minute and posts to your on-call webhook is fifty lines and you control the thresholds. And a counter can only tell you a job ran, never that a job should have run and didn't; for that you want a dead man's switch like Healthchecks pinging in from the job itself.

I'm not sure why the "should have run" case stays so underserved across the whole category. In my case it was the exact failure mode that burned me, and I still assemble it out of parts.

Start with the invariant, not the chart. Emit the signal from the code that did the work, keep the query small, and let the graph be boring.

References

Prometheus instrumentation practices — https://prometheus.io/docs/practices/instrumentation/
Grafana panels and visualisations documentation — https://grafana.com/docs/grafana/latest/panels-visualizations/
Datadog pricing (log ingestion and indexing model) — https://www.datadoghq.com/pricing/
RFC 5424, The Syslog Protocol (log level semantics) — https://datatracker.ietf.org/doc/html/rfc5424
Infrai API documentation — https://docs.infrai.cc