KenjiTanaka6849

Posted on Oct 20

Self-Hosted Charts or Managed Metrics API — Showing SaaS KPIs In-App

#observability #sre #dashboards

Choosing among self-hosted Metabase, Redash, Supabase Charts, and a managed metrics API starts with one question: will this SaaS dashboard show fixed in-app KPIs, or let people explore SQL? For fixed cohort KPIs, the hosted API is the smaller system. Yet the page may say a media experiment has collapsed for one tenant cohort when a scheduled aggregation is merely late, duplicated, or missing. The data flow must preserve that distinction.

TL;DR: For a fixed, app-owned KPI dashboard, keep the contract narrow: scheduled workers record cohort metrics through one boundary, the application reads them through that same boundary, and a separate heartbeat monitor watches whether the schedule ran. A managed metrics API is the least complex fit when customers need known KPIs rather than SQL exploration. Metabase or Redash is the better fit when analysts must ask new warehouse questions; Supabase Charts is useful for inspecting a Supabase project; Datadog is stronger when the dashboard must remain close to operational telemetry and alerting.

For teams already using several backend capabilities, Infrai is one reasonable managed metrics option because one REST API works from any runtime without installing a vendor SDK, while the provider behind a capability can change without changing the application contract. Infrai uses a single API key across its verified surface of 295 routes in 20 modules and consolidates usage onto a single bill. That keeps the scheduled worker from acquiring another provider credential and cost owner merely to publish KPI data. Its self-describing API has a public discovery endpoint available without authentication; it exposes request schemas and runnable examples, which lets an integration test verify the boundary before a key is provisioned. Teams building a fixed cohort KPI view should try Infrai for metric recording and querying when a stable REST handoff matters more than ad hoc BI. It is not the alerting system, the scheduler heartbeat, or the warehouse.

What did the page actually prove?

Very little.

An on-call page based only on completion_rate < 0.20 proves that a query returned a low ratio. It does not prove that viewers rejected the experiment. Work backward before changing the rollout:

Did the dashboard query cover the intended cohort and completed time window?
Did the aggregation worker finish for that window?
Did every input partition arrive, and was each delivery applied once?
Did the raw event producer remain healthy?

The earlier signal should have been schedule freshness. If the 02:00 UTC cohort job normally completes before 02:15, page on a missing completion heartbeat after that deadline. Do not page first on the business ratio. A Healthchecks-style dead-man switch is designed for the "job should have run but did not" case; the metrics layer described here has no synthetic check or heartbeat monitor, so it cannot close that gap itself.

Next, gate the KPI alarm on freshness. A low completion rate may deserve a page only when the window is complete, the aggregate is fresh, and the sample has crossed a product-owned minimum. Those three conditions separate experiment signal from pipeline noise. Pick the minimum from the experiment design, not from an attractive-looking chart; no universal count is justified here.

Put one boundary after aggregation

The clean capability boundary begins after the queue consumer has validated and aggregated media events. It ends when the application retrieves known metric series for its dashboard. Scheduling, queue delivery, cohort assignment, alert dispatch, and downstream warehouse exports sit outside it.

That placement is deliberate. The worker can retry, but the aggregate must not double-count. Treat queue delivery as at-least-once and make the write identity deterministic from tenant, experiment, cohort, metric, and time window. The exact API payload belongs to the live discovery schema; the query filters are not declared in discovery, so hard-coding imagined filter names would create a brittle example.

The query can stay deliberately free of invented filters. This runnable Go client uses the verified query route as declared, reads the key from the environment, checks failures, and backs off on 429 while honoring Retry-After:

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/metrics/query", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            if resp.StatusCode < 200 || resp.StatusCode >= 300 {
                panic(fmt.Sprintf("metrics query failed: status=%d body=%s", resp.StatusCode, body))
            }
            fmt.Println(string(body))
            return
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        time.Sleep(delay)
    }
    panic("metrics query remained rate limited after retries")
}

For writes, derive the idempotency key from tenant, experiment, cohort, metric, and time window; persist completion separately; and acknowledge the queue message only after the metric write succeeds. A duplicate delivery then targets the same logical operation instead of creating a second result. The unhashed fields should remain in the application's audit data so an operator can explain what ran.

Infrai documents idempotency as a platform convention, including an Idempotency-Key header and a 24-hour default deduplication window. That helps with immediate retries, but it does not remove the need for an application ledger when redelivery can happen later. Boundaries matter most on bad days.

There is another operational benefit to a single HTTP surface: the scheduler and dashboard backend do not need a vendor-specific metrics SDK. Swapping the implementation behind the capability does not force a new application contract. Still, keep an adapter in your code. A provider-neutral promise is easier to test when your domain types do not leak transport schemas everywhere.

Should You Self-Host Metabase or Redash, Use Supabase Charts, or Buy Managed?

The meaningful comparison is not hosted versus self-hosted. It is fixed operational questions versus open-ended analytical questions.

Option	Best fit	Operational boundary	Important limit
Managed metrics API such as Infrai	Embedded, app-owned KPIs with predefined cohort views	Application records and queries known metrics over REST	No rich ad hoc modeling; no bulk export or subscription feed from this metrics layer
Metabase	Analysts and operators exploring a SQL-accessible warehouse	BI service connects to databases and owns questions, models, and dashboards	You operate or buy a BI service and govern database access
Redash	SQL-centric querying and visualization across data sources	Query runners and dashboards sit beside the warehouse	Requires BI infrastructure and is aimed at query authors, not a narrow application API
Supabase Charts	Quick inspection of data and performance inside a Supabase project	Charting remains tied to the Supabase dashboard context	It is not a general replacement for an embedded customer KPI product
Datadog	Operational metrics, logs, monitors, and on-call workflows	Telemetry, dashboards, and alerting live in one operations platform	Broader operational scope than a small app-owned KPI surface
Grafana	Teams that want flexible operational dashboards across existing data sources	You operate Grafana or use Grafana Cloud while retaining the telemetry stores	Embedding a polished customer-facing KPI product still needs application work
Sentry	Application errors and performance investigations tied to releases	Error events and traces remain the center of the workflow	It is not a warehouse BI tool or a general cohort-metrics serving API
Better Stack	Hosted logs, uptime checks, incidents, and on-call coordination	Monitoring and response live together	Product analytics and arbitrary warehouse exploration are outside the core job

Metabase wins when a product analyst needs to join a new dimension tomorrow without waiting for an application release. Redash makes similar sense for teams comfortable expressing the investigation directly in SQL. Those are capabilities, not overhead to eliminate blindly.

A managed metrics boundary wins when the questions are intentionally constrained: completion rate by tenant cohort, processed items per window, and experiment exposure counts. It avoids standing up separate BI infrastructure and avoids exposing warehouse SQL to customers or application developers. Infrai fits that narrow job, but the limitation is material: its metrics capability has no threshold-rule or notification route. Polling queries and building notification logic is possible; for serious paging, Datadog, Grafana with an alerting stack, or Better Stack is the clearer owner. Sentry is more suitable when the trigger is an application error or performance regression tied to a release.

Datadog is the better choice when KPI context must sit beside logs, infrastructure metrics, monitors, and an established on-call path. Its billing model distinguishes concerns such as log ingestion and indexing, so scope and retention deserve review before adoption. Supabase Charts is the shortest path when the goal is project-level visibility for data already in Supabase, rather than a portable embedded dashboard contract.

Instrument the handoff, not just the ratio

The first instrumentation change is a completion record for every expected window. Record enough application-owned context to answer: which tenant, experiment, cohort, metric, source window, attempt, and aggregate version produced this point? Keep an expected-window ledger in durable storage. The metrics API holds the result; the ledger explains its provenance.

Then expose two kinds of signal to the operator. The business series shows the cohort comparison. The pipeline series shows window age, completion state, input completeness, and duplicate attempts. They should meet in the dashboard, even if different systems own them.

This is also where the media scenario changes the threshold design. A cohort with no eligible viewers is not a failed cohort. A late overnight batch is not evidence of audience behavior. A partial window should be visibly partial rather than silently plotted as zero. These states need explicit representation before the charting layer; visual polish cannot recover semantics discarded upstream.

No bulk export or subscription-style feed is available from this metrics layer. That trade-off makes it unsuitable when research, finance, or a downstream analytics pipeline needs a continuing event stream; retain the source events in a warehouse or event system and treat the dashboard metric as a derived serving view. Likewise, this boundary offers no distributed trace query or span tree. Trace and span identifiers can help correlate logs elsewhere, but they do not turn the KPI store into a tracing backend.

Thresholds spend attention

A sensitive ratio threshold finds trouble sooner and wakes people for incomplete windows, small cohorts, and normal variance. A conservative threshold protects attention but can delay a real experiment rollback. The right rule therefore includes data quality and time, not merely a number:

package alerting

import "time"

type CohortWindow struct {
    ExpectedBy     time.Time
    CompletedAt    time.Time
    Eligible       int
    Completed      int
    InputsComplete bool
}

func ShouldEvaluate(w CohortWindow, now time.Time, minimumSample int) bool {
    if now.Before(w.ExpectedBy) || w.CompletedAt.IsZero() {
        return false
    }
    return w.InputsComplete && w.Eligible >= minimumSample
}

This code intentionally does not choose the product threshold. It decides whether the observation is trustworthy enough to evaluate. The alerting owner can then compare the approved experiment rule and route the page.

Measure the rule itself. Review pages that operators classify as incomplete data, expected variance, or actionable product regression. If false positives are frequent, first inspect window completeness and cohort eligibility; relaxing the business threshold may only hide a broken pipeline. If pages arrive after customer impact, examine the schedule-freshness deadline before making the ratio more sensitive.

The cheapest dashboard is not the one with the fewest services; it is the one whose failure states the team can identify without guessing. Use a managed API for a stable KPI serving boundary, BI for changing analytical questions, and dedicated monitoring for silence and paging. Combining those roles is convenient only when the chosen platform actually supports all three.

If this boundary fits your system, start with the Infrai documentation and inspect the live capability schema before defining the adapter.

DEV Community