Short answer: choose the delivery model that can preserve cohort definitions, expose stale or missing data, and execute a predetermined rollback rule; hosting cost is secondary until those invariants survive a failed experiment. For a healthtech SaaS comparing treatment and control tenants, I would keep cohort computation behind one narrow metrics contract, then let either a self-hosted chart layer or a managed metrics API consume that contract. This makes the display replaceable without making the rollback decision ambiguous.
The cheap-looking option can become expensive when every dashboard refresh scans raw events, while the expensive-looking option can be wasteful when it retains high-cardinality labels that nobody uses. Don't compare subscription lines alone. Count points, series, queries, retained bytes, and the engineering hours attached to the failure boundary.
Decision record: preserve the rollback evidence
The decision is to separate metric production from chart delivery. A Node.js service emits or derives cohort aggregates; a metrics boundary returns a small, versioned result; the embedded UI renders it. The chart must never reconstruct cohort membership from mutable application rows. That separation is the architectural choice. Self-hosted versus managed is a deployment choice behind it.
Three invariants govern the design. First, every value carries the experiment version, tenant cohort, window start, window end, and denominator. Second, a response reports freshness and completeness rather than turning absent samples into zero. Third, the rollback evaluator reads the same aggregate contract as the chart. If the dashboard and automation calculate independently, their numbers can disagree exactly when an operator needs a fast decision.
The failure boundary is deliberately narrow: ingestion can lag, aggregation can miss a window, or chart delivery can be unavailable, but none of those states may silently authorize continuation. The evaluator should move to an explicit insufficient_data state. It shouldn't infer safety. For example, a policy might require at least 5,000 requests in each cohort and ten complete one-minute windows, then roll back when the treatment error ratio exceeds control by 0.5 percentage points throughout that interval. Those figures are an example policy, not a clinical or statistical universal; traffic shape and risk tolerance determine the real thresholds.
This matters in healthtech because a tenant identifier is operationally useful and dangerously tempting as a label. Keep patient identifiers and request-level attributes out of aggregate labels and dashboard payloads. Use an opaque tenant key only where cohort comparison requires it, and enforce access before the metrics query runs.
Stop there for a moment.
How should a healthtech SaaS compare self-hosted KPI charts with a managed metrics API?
Run the same acceptance test against every option. Metabase, Redash, and Supabase-based charts can be evaluated as self-hosted candidates; a managed metrics API is the fourth candidate class. Their names don't settle the decision. The evidence comes from whether the deployed system meets the rollback contract, how much telemetry it retains, and which operational duties remain with the team.
| Candidate | What to prove in a trial | Cost surface to count | Valid reason to reject it |
|---|---|---|---|
| Self-hosted Metabase | The embedded view preserves cohort filters and freshness metadata | Compute, database scans, storage, upgrades, access control, and on-call time | Reject when operating the full path exceeds the team's ownership budget |
| Self-hosted Redash | The query and visualization use the same versioned aggregate | Query load, cache behavior, storage, maintenance, and incident response | Reject when rollback evidence depends on ad hoc dashboard queries |
| Supabase-based charts | The application boundary prevents direct, over-broad data access | Database load, egress, cache misses, policy maintenance, and UI work | Reject when the chart must couple directly to mutable application tables |
| Managed metrics API | The service exports denominators, freshness, and missing-window state | Ingested volume, retained volume, active series, query volume, and integration work | Reject when data residency or control requirements cannot be met |
This is intentionally not a feature checklist. Product capabilities and commercial terms change, and I'm not sure a public price page can represent a particular tenant distribution or support agreement. Resolve that uncertainty with a 30-day trace replay, an architecture review, and a written quote. The Datadog pricing page is useful evidence that log ingestion and indexing can be separate cost dimensions; it isn't proof that any single product will be cheapest for this workload.
The catch is that self-hosting is not suitable when the team cannot patch, back up, monitor, and restore the entire query path within its recovery objective. Choose a managed boundary in that case, provided it satisfies access and residency constraints. A managed service is not suitable when those constraints require infrastructure-level control or when predictable, already-owned capacity makes its metered dimensions a poor fit. Stick with a self-hosted candidate then, but budget the operator time explicitly.
Count cardinality and retention before comparing prices
Start with units. Suppose the design keeps 12 KPIs for 80 tenants, two cohorts, and two environments. If every label combination exists, the base is 80 × 2 × 2 = 320 series per KPI, or 3,840 series. Adding 15 endpoints yields 57,600 possible series. Adding 50 experiment identifiers after that yields 2,880,000. The last label looked harmless in a schema review; in the cost model it multiplied the largest dimension by fifty. That's the bill-shaped part of observability. Retention changes the second axis. At one point per minute, 80 tenants × 2 cohorts × 12 KPIs × 43,200 minutes produces 82,944,000 points in a 30-day model. Five-minute aggregates reduce that model to 16,588,800 points. These are planning calculations, not benchmark results: compression, indexes, replicas, metadata, and billing units can all change stored or charged volume. Write those implementation-specific multipliers beside the estimate instead of burying them in a monthly total. Logs need the same treatment. A hypothetical 1 KB event at 20 events per second is 1.728 GB per day in decimal units, or 51.84 GB over 30 days before indexing and replication overhead. If only the rollback KPIs matter, retaining every event for the full dashboard horizon is difficult to justify. Keep a short diagnostic window for raw events, derive low-cardinality cohort aggregates, and retain those aggregates for the experiment review period. Sampling can lower event volume, but naive sampling can erase rare failures or distort denominators. Preserve total counts and sampled counts, stratify by the dimensions used in the decision, and test the estimator against an unsampled window.
Cheap is a model result.
A practical worksheet has five rows: ingest bytes, retained bytes or points, active series, query executions, and operator hours. Apply each candidate's current billing unit only after measuring those rows. Do not convert a free allowance or a low storage rate into a recommendation; a query-heavy embedded dashboard and an ingest-heavy forensic system have different dominant terms.
Put one auditable contract on the critical path
The contract below is proposed application architecture, not a claim about a vendor endpoint. It asks an internal metrics boundary for the exact experiment version and closed time window, fails on an unsuccessful transfer, and limits the time an interactive request can occupy a worker. The opaque bearer token belongs in a secret manager, not source control.
curl --fail-with-body --silent --show-error \
--connect-timeout 2 \
--max-time 5 \
--get 'https://metrics.example.test/cohort-kpis' \
--header "Authorization: Bearer ${METRICS_TOKEN}" \
--data-urlencode 'experiment=medication-reminder-v7' \
--data-urlencode 'cohorts=control,treatment' \
--data-urlencode 'metrics=request_count,error_count,p75_latency_ms' \
--data-urlencode 'window_start=2026-08-16T02:00:00Z' \
--data-urlencode 'window_end=2026-08-16T02:10:00Z'
The response contract should contain numerator, denominator, unit, cohort, experiment version, closed window, generated timestamp, and a completeness state. Test four cases before embedding a chart: complete windows, one missing treatment window, a stale aggregate, and a cohort-definition version mismatch. The last three must visibly block a go-forward decision. A cached chart may still be useful for reading history, but its stale state must be obvious and the rollback evaluator must not treat it as current evidence.
Use a separate browser-facing endpoint that returns only authorized aggregates. Don't place a warehouse credential or a general metrics token in client-side JavaScript. Cache by tenant authorization scope, experiment version, window, and metric set; omitting any of those fields can return a technically valid but decisionally wrong result.
Dashboard performance deserves a measurable gate too. Core Web Vitals defines user-facing measures including LCP, CLS, and INP and evaluates them at the 75th percentile. That framework can test the embedded view's delivery quality, but it does not validate the KPI mathematics. Keep the two judgments separate: a fast chart can still show incomplete evidence, while a correct aggregate can still arrive too slowly for an incident workflow.
Rejected option and the conditions that reverse the decision
The rejected design is direct chart access to raw application events. It initially removes an aggregation service and can be appropriate for a small internal exploratory dashboard where queries are infrequent, users are trusted, cohort definitions are still changing, and no automated rollback consumes the result. Its flexibility is real.
It is rejected for the in-app healthtech experiment because it expands the access boundary, repeats expensive scans, and permits the visualization query to drift away from the rollback query. More retention then feels like safety, yet extra raw data doesn't repair a mismatched denominator or an unversioned cohort. The narrow aggregate contract costs engineering time up front, but it buys a testable equivalence: the number an operator sees is the number the policy evaluates.
The decision can reverse. If the dashboard remains internal, the event volume is bounded, rollback is manual, and an existing self-hosted query stack already meets backup and access objectives, direct queries may be the lower-total-cost choice. If embedded usage grows across many tenants, refreshes become frequent, or rollback becomes automatic, precomputed aggregates behind a managed or self-hosted metrics boundary become easier to reason about. Recalculate after measuring. Your mileage may vary because cardinality distribution, not the product label, usually controls the answer.
Top comments (0)