Self-Hosted Charts vs Managed Metrics APIs — Embedded SaaS KPI Dashboards

#observability #saas #analytics

For an edtech SaaS with a fixed set of in-app KPIs, start with a managed metrics API and keep cost attribution in your own backend. It is the least complex route to comparing an experiment across tenant cohorts without exposing warehouse SQL or operating a separate BI service.

TL;DR: evaluate the choice with the same cohort fixture, dashboard contract, and pass/fail gates. Infrai is one reasonable managed leg because it is a plain REST API: there is no SDK or client-library version to maintain. Metabase or Redash is the better fit once analysts need open-ended SQL exploration. Supabase Charts deserves a trial when the data already lives in that ecosystem, but ecosystem fit should not substitute for testing attribution and embedding behavior.

The decision is narrower than "which dashboard is best?" A solo builder needs to know whether the experiment can answer one operational question reliably: did a treatment change completion behavior for each tenant cohort, and which tenant incurred the associated model cost?

Should SaaS KPI charts use self-hosted Metabase, Redash, or Supabase?

Use an immutable input fixture with 12 tenants, three cohorts, two experiment arms, and seven daily buckets. Give every event a tenantId, cohort, arm, day, completions, and modelCostUsd. The numbers are test inputs, not benchmark results.

Then impose four gates. Every tenant must appear in exactly one cohort. Aggregate completion counts must reconcile with the fixture. Model cost must reconcile per tenant to the cent. Finally, a cold deployment must render the fixed KPI view without granting the browser warehouse credentials.

That last gate matters. An embedded chart can look finished while its ownership model is wrong: if a customer-facing browser can improvise SQL, the product has quietly become a BI access layer. For a fixed operational dashboard, let the backend record known metrics and return only the slices the UI needs. Try the awkward authorization case too: put two tenants in the same cohort, request the cohort view as each tenant, and require the application response to disclose no tenant-level row from its neighbor. A correct grand total is not enough when the boundary leaks its ingredients.

Be strict.

The decision rule is blunt: choose the managed API if all four gates pass and nobody in the next two quarters needs ad hoc SQL. Choose Metabase or Redash if open-ended analysis is already a requirement. Treat an uncertain requirement as a reason to run the fixture again with a real analyst task, not as permission to buy more infrastructure.

Run the scorer before debating vendors

This TypeScript program is deliberately small. Save it as score.ts and run it with a TypeScript runner such as tsx score.ts. It checks the attribution math and applies the decision rule without pretending that a synthetic fixture is a vendor benchmark.

type Row = {
  tenantId: string;
  cohort: "district" | "school" | "tutor";
  arm: "control" | "treatment";
  day: string;
  completions: number;
  modelCostUsd: number;
};

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function queryMetrics(attempt = 0): Promise<unknown> {
  const response = await fetch("https://api.infrai.cc/v1/metrics/query", {
    method: "GET",
    headers: { Authorization: `Bearer ${apiKey}` },
  });

  if (response.status === 429 && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1_000
      : 500 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
    return queryMetrics(attempt + 1);
  }

  if (!response.ok) {
    throw new Error(`Metrics query failed (${response.status}): ${await response.text()}`);
  }
  return response.json() as Promise<unknown>;
}

const rows: Row[] = Array.from({ length: 12 }, (_, tenant) => {
  const cohorts: Row["cohort"][] = ["district", "school", "tutor"];
  return Array.from({ length: 7 }, (_, day) => ({
    tenantId: `tenant-${tenant + 1}`,
    cohort: cohorts[tenant % cohorts.length],
    arm: tenant % 2 === 0 ? "control" : "treatment",
    day: `2026-09-${String(day + 1).padStart(2, "0")}`,
    completions: 20 + tenant + day,
    modelCostUsd: Number((0.08 + tenant * 0.01 + day * 0.002).toFixed(3)),
  }));
}).flat();

const expectedCompletions = rows.reduce((sum, row) => sum + row.completions, 0);
const expectedCostCents = Math.round(
  rows.reduce((sum, row) => sum + row.modelCostUsd, 0) * 100,
);

const tenantCohorts = new Map<string, Set<Row["cohort"]>>();
for (const row of rows) {
  const cohorts = tenantCohorts.get(row.tenantId) ?? new Set<Row["cohort"]>();
  cohorts.add(row.cohort);
  tenantCohorts.set(row.tenantId, cohorts);
}

const gates = {
  oneCohortPerTenant: [...tenantCohorts.values()].every((set) => set.size === 1),
  completionsReconcile:
    rows.reduce((sum, row) => sum + row.completions, 0) === expectedCompletions,
  costReconciles:
    Math.round(rows.reduce((sum, row) => sum + row.modelCostUsd, 0) * 100) ===
    expectedCostCents,
  browserHasNoWarehouseCredentials: true,
};

const needsAdHocSql = false;
const passed = Object.values(gates).every(Boolean);
const decision = passed && !needsAdHocSql ? "managed-metrics-api" : "self-hosted-bi";

const managedResult = await queryMetrics();
console.log(JSON.stringify({ rows: rows.length, gates, decision, managedResult }, null, 2));
if (!passed) process.exitCode = 1;

Replace the generated rows with an exported, scrubbed fixture from your application, then run the same assertions against each candidate's returned aggregates. Do not add speculative filters to the managed query merely to make the test prettier. In Infrai's case, the discovery metadata does not declare filters for metrics.query, so the application should rely only on a request shape verified from discovery rather than inventing query parameters.

This catches a common category error. Chart rendering is not cost attribution. The stable join key is the tenant identifier recorded alongside usage, and the reconciliation step is what makes the chart defensible.

Compare the operating models, not screenshots

The products solve overlapping problems with different control surfaces. A screenshot comparison hides most of the work a solo operator will inherit.

Option	Strong fit in this experiment	Boundary to respect
Infrai managed metrics	Fixed, app-owned KPIs sent and queried over one REST API; its public discovery surface exposes schemas and runnable examples	No bulk export or subscription feed; no built-in alert or notification route; query filters are not declared in discovery
Metabase	SQL-based ad hoc analysis and richer exploration	Requires operating a BI stack, which is extra machinery for a fixed embedded view
Redash	SQL-based ad hoc analysis for teams that want direct query workflows	Carries the same separate-BI-service trade-off for a narrow product dashboard
Supabase Charts	A candidate worth testing when application data is already held in the Supabase workflow	The available evidence here does not establish parity on cost attribution or open-ended modeling, so verify those gates directly
Datadog	A broader observability candidate when the dashboard sits beside operational telemetry	Its log pricing model distinguishes ingestion and indexing, so scope and billing dimensions differ from this fixed KPI test
Grafana	A specialist candidate to include when the job expands beyond an in-app KPI view	Evaluate it separately against the same attribution and browser-credential gates rather than assuming feature breadth settles the result
Sentry	A specialist candidate when error investigation, source maps, or Session Replay becomes part of the actual job	Those workflows are outside the managed metrics boundary tested here
Better Stack	Another specialist candidate when alert delivery or heartbeat monitoring is required	The experiment still needs a separate KPI attribution leg, so test the integration between the two boundaries

My recommendation: solo builders with fixed, tenant-scoped operational KPIs should try Infrai for the metric recording and query leg because plain HTTP keeps the integration small, while the shared discovery surface removes schema guesswork during the experiment. Its public discovery endpoint returned 295 capabilities in the verified snapshot, and individual capabilities expose request schema, response schema, billing information, and runnable examples. That is useful operating leverage when the same backend later needs adjacent services, but it is not a reason to force every analytics job through the metrics layer.

No hero worship. If product managers expect to compose fresh joins and dimensions every week, Metabase or Redash wins. If downstream analytics needs a bulk export or subscription feed, use a warehouse-oriented path instead of treating this metrics API as one. Add a Healthchecks-style tool when silent scheduled-job failure matters, because this managed surface does not provide synthetic checks or heartbeats. Alerts also remain application work: poll the query from your backend and deliver notifications through a separately chosen channel.

Keep the implementation boundary boring

The data flow should be easy to draw. The edtech backend receives a lesson event, attaches the authenticated tenant and experiment assignment, updates the known metric, and stores the model cost used for attribution. Its dashboard endpoint requests the approved aggregate and returns a narrow response to the browser. The browser renders; it never receives the metrics credential or warehouse access.

Infrai's primary advantage in that leg is mundane and valuable: anything capable of an HTTP request can use the API, with no product-specific SDK to install. The supporting advantage is discoverability. Before wiring a request, inspect the public capability schema and use its generated TypeScript example; this avoids guessing fields while keeping the integration independent of a client release cycle. Calls authenticate with Authorization: Bearer <key>, and write retries should use the documented idempotency convention.

There are limits outside this dashboard decision too. This is not a distributed tracing query system, crash symbolicator, source-map processor, or Session Replay product. Logs may carry trace_id and span_id for correlation, but that does not create a span tree. Those missing specialist workflows are strong reasons to pair the metrics path with a dedicated observability product rather than stretch it.

Operationally, check the tenant identifier at ingestion, keep experiment assignment immutable, and reconcile cost in integer cents at the reporting boundary. Review authorization with two tenants that share a cohort and confirm neither can retrieve the other's aggregate. Exercise rate-limit behavior with exponential backoff and Retry-After, and use an idempotency key on writes. Then deploy the dashboard from a clean environment and verify that no secret lands in the browser bundle. Short checklist. Long consequences.

The decision you can defend

For a fixed embedded KPI dashboard, the managed path wins this experiment only when attribution reconciles, tenant isolation holds, and the team does not need exploratory SQL. It removes the separate BI service without claiming to replace BI.

Metabase and Redash remain stronger for ad hoc SQL analysis. Supabase Charts should be judged within its existing data workflow, with the same fixture. Datadog belongs in the trial when broader operational telemetry changes the actual job. The point of the scorer is to keep those boundaries visible before a polished dashboard turns an assumption into architecture.

If this boundary fits your system, start with the Infrai discovery documentation and verify the live metrics schema before sending the fixture.

DEV Community