Incident Evidence: Self-Hosted Metabase, Redash, Supabase Charts or Metrics API

#observability #saas #metrics

Short answer: the cheapest way to show SaaS KPIs in an app is the path that can reproduce a disputed number from retained evidence, with tenant boundaries and metric definitions intact. Start with a small, app-owned contract and test it against an incident; choose self-hosted charts for investigation-heavy work, and a managed metrics API for a bounded set of product questions.

A chart is a claim about the past. Treat it like one.

Start with the incident, not the chart

For a B2B SaaS product, the useful question is not whether a dashboard can draw a line. It is whether support can explain that line after a customer says, “Your numbers changed.” The first design artifact should therefore be a reconstruction test: given one tenant, one disputed interval, and one KPI, can an engineer trace the displayed value back to source records and explain every correction?

That test exposes the expensive failures early. A dashboard can be fresh and still be wrong because the event and its aggregate took different paths. It can be accurate globally and still leak another tenant's data. It can render a plausible count after someone changes the meaning of “active account.” None of these problems is fixed by adding a better chart component.

Use a narrow test case. Suppose the product reports completed jobs per day. The test should include a duplicate delivery, a late completion, a retry, and a tenant with no events. The expected result is not only a number: it includes the source event, the counting rule, the correction history, the tenant scope, and an explicit unknown state when the evidence is unavailable.

Three words matter: explain the number.

I would record the acceptance criteria before selecting a dashboard tool. They make a useful boundary between a customer-facing view and an investigation workspace, which often have different query shapes and different tolerance for freshness.

Should self-hosted Metabase, Redash, or Supabase charts serve incident evidence?

Reliability here is a property of the evidence path, not a promise about uptime. Check five relationships.

First, the source event must have a defined point in time. “Completed” could mean the worker acknowledged a job, the transaction committed, or the reporting pipeline observed the record. Those timestamps answer different incident questions. Store the relevant event time and ingestion time when late data matters.

Second, aggregation must be reproducible. A metric specification needs its unit, interval, timezone, population, and treatment of retries. For latency, the percentile policy is part of the value's meaning; Core Web Vitals, for example, describes LCP, CLS, and INP using a p75 threshold rather than treating an arbitrary average as the whole story.

Third, scope must be enforced before chart formatting. Pass tenant identity into the read policy, test adjacent tenant IDs, and make an authorization failure different from an empty series. A zero says “nothing happened.” An unknown says “the system cannot establish what happened.” Those states should not be collapsed.

Fourth, late and duplicate records need an explicit correction rule. A repair that silently overwrites yesterday's point destroys the very evidence needed during a customer incident. Keep the original event identifiers, the correction time, and the metric definition used for the recalculation. I would treat a 429 during a write as a reconciliation event, not as proof that the business operation failed.

Fifth, the dashboard must carry enough context to be challenged. Show the interval, freshness or completeness state, and metric version beside the value. This is less attractive than a large single number, but it prevents a support conversation from starting with an unanswerable screenshot.

The small catalog below is an application boundary, not a vendor integration. Its job is to make metric meaning reviewable before storage and presentation are selected.

from dataclasses import dataclass
from typing import Literal


@dataclass(frozen=True)
class MetricSpec:
    name: str
    unit: str
    window: Literal["hour", "day"]
    source: str
    definition_version: int


CATALOG = {
    "completed_jobs": MetricSpec(
        name="completed_jobs",
        unit="jobs",
        window="day",
        source="job.completed",
        definition_version=1,
    ),
    "request_latency_p75": MetricSpec(
        name="request_latency_p75",
        unit="milliseconds",
        window="hour",
        source="request.observed",
        definition_version=1,
    ),
}


def metric_request(tenant_id: str, metric_name: str) -> dict[str, str | int]:
    if not tenant_id:
        raise ValueError("tenant_id is required")
    spec = CATALOG[metric_name]
    return {
        "tenant_id": tenant_id,
        "metric": spec.name,
        "window": spec.window,
        "source": spec.source,
        "definition_version": spec.definition_version,
    }

This contract still needs a storage policy, an authorization layer, and a reconciliation process. That is the point. A chart library should not quietly become the owner of those decisions.

Choosing the query shape after the test passes

Once the reconstruction test is concrete, the options stop looking interchangeable. The choice follows the kind of question the application must answer.

Path	Best fit	Evidence concern	Poor fit
Managed metrics API	A stable catalog of KPIs with known filters and windows	The metric service may not retain the raw records needed to explain a correction	Analysts need arbitrary joins or raw-event exploration
Self-hosted charts over analytical data	Frequent investigation, changing joins, and team-owned SQL	The team owns backups, upgrades, permissions, query capacity, and retention	The product has a small fixed catalog and little analytical work
Direct queries over operational data	A small dataset and tightly bounded internal traffic	Dashboard reads can compete with transactions and schema changes	Customer traffic can create unbounded or expensive queries
Precomputed analytical tables	Durable history, repeatable joins, and controlled read latency	Pipelines, freshness, lineage, and backfills become explicit work	The question needs immediate results from a tiny stable dataset

This is a reliability decision before it is a cost decision. Managed infrastructure can reduce the operating work the product team carries, while self-hosting can preserve control over query behavior and retention; the invoice is only one line in that comparison. A managed log or metrics service may also separate ingestion from indexing, as the Datadog pricing model illustrates, so a cost estimate has to describe what data is accepted, indexed, retained, and queried.

The catch is that neither a managed metrics API nor self-hosted charts is an evidence guarantee. A metrics API is not suitable when incident responders must invent new joins at 02:00. Self-hosted analytics is not suitable when arbitrary customer-authored queries would cross tenant boundaries or compete with production capacity. Stick with the bounded API for a small, governed product catalog; select an analytical path when investigation is a first-class requirement, and put access controls around it.

Governance is part of the dashboard surface

Assign an owner to each metric definition, source event, retention rule, and correction process. Review changes as data-contract changes, not as harmless label edits. A renamed series can be more damaging than a broken chart if historical values remain visually comparable while their populations differ.

The write path should make duplicates and late arrivals visible to reconciliation. The read path should include tenant scope and a completeness state. The incident path should preserve the source identifiers needed to move from a KPI point to an event window. These responsibilities can live in different services, but someone must own the joins between them.

Your mileage may vary on freshness. A five-minute lag may be acceptable for a product trend and unacceptable for a queue incident. Write the service-level expectation next to the metric definition — where it can be reviewed — instead of smuggling it into a refresh interval that nobody reviews.

Keep raw evidence under a retention policy that matches the incident window. If the aggregate lasts ninety days but its supporting events last seven, the dashboard may be fast while reconstruction is impossible. That can be a valid business choice; it must be a visible one.

A rollout that can survive a disputed number

Run one KPI through the reconstruction test in a non-production environment. Inject duplicate, late, and missing records. Verify that tenant A cannot retrieve tenant B's evidence, then compare the in-app chart with a separately calculated result from source records. Capture the metric definition version in the test output.

Promote the path only after the failure cases have named owners. During the first production rollout, keep the old calculation available for comparison, sample disputed points, and make corrections observable to support rather than silently repainting history. I don't know a universal cheapest option, because volume, retention, analyst workload, and on-call capacity change the answer; the test tells you which cost is unacceptable.

The final decision rule is compact: if the KPI is a stable product contract, a managed metrics API can be the smaller operating surface; if reconstructing an incident requires open-ended joins and raw-event inspection, self-hosted analytical charts are the better fit. In either case, preserve the evidence separately from the picture and make tenant scope, aggregation, freshness, and correction behavior inspectable.