Short answer: calculate retained metric points before choosing a dashboard, then keep a small, stable set of cohort counters whose definitions can be audited. For a customer-support SaaS experiment, I would retain daily signups, conversions, queue depth, and API latency for three tenant cohorts, while keeping tenant identifiers out of metric labels. A narrow custom dashboard backed by a plain metrics REST API fits when product KPIs and backend counters must share one view; Plausible or PostHog fits product analysis better, and Grafana Cloud fits deeper operational exploration and alerting.
The decisive cost is usually not the number of charts. It is the number of points retained: metric series multiplied by cohort variants, time buckets, and retention. This is also a privacy boundary. Once arbitrary tenant, agent, ticket, or user identifiers become dimensions, a clean experiment dashboard turns into a high-cardinality event store with a deletion problem.
What does one useful cohort comparison actually cost?
Start with a budget that can be inspected. Suppose the experiment has three cohorts, four measures, one daily bucket, and a 90-day decision window. That design retains 3 x 4 x 90 = 1,080 aggregate points. Adding five regions multiplies it to 5,400. Adding 1,000 tenant IDs produces 1,080,000 points, before anyone asks for channel, plan, or agent dimensions.
Those figures are design inputs, not a vendor benchmark. Change them to match the experiment. Their purpose is to reveal the dominant term: uncontrolled dimension cardinality, which increases storage and query work while making sparse charts noisier.
Use a ledger-like metric contract for every counter: a stable name, an explicit unit, a cohort assignment version, a bucket boundary, and a deterministic observation ID. The last field matters because ingestion retries must not double-count a conversion. Preserve the raw business event in the system of record under its own retention and access policy; the dashboard receives only the aggregate needed for the decision.
Implement 3 cohort counters as an auditable reduction
The following program runs end to end. It first asks Infrai's public, self-describing discovery surface for the live metrics.report contract, then demonstrates the reduction locally and rejects duplicate observations. This avoids pretending that an undocumented request shape is portable. Set INFRAI_API_KEY, save the file as main.go, and run go run main.go; the key stays in an environment variable, while the request uses an explicit method, checks non-success responses, and backs off on 429 using Retry-After when the server supplies it.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"sort"
"strconv"
"strings"
"time"
)
type Discovery struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Idempotent bool `json:"idempotent"`
Params json.RawMessage `json:"params"`
}
type Observation struct {
ID string
Tenant string
Cohort string
At time.Time
SignedUp bool
Converted bool
QueueDepth int
LatencyMS int
}
type Daily struct {
Day string
Cohort string
Signups int
Conversions int
QueueSum int
LatencySum int
Samples int
}
func main() {
contract, err := discoverMetricContract()
if err != nil {
panic(err)
}
fmt.Printf("contract=%s method=%s path=%s idempotent=%t schema_bytes=%d\n",
contract.ID, contract.Method, contract.Path, contract.Idempotent, len(contract.Params))
day := time.Date(2026, 10, 8, 0, 0, 0, 0, time.UTC)
input := []Observation{
{ID: "evt-101", Tenant: "tenant-a", Cohort: "control", At: day, SignedUp: true, QueueDepth: 12, LatencyMS: 180},
{ID: "evt-102", Tenant: "tenant-b", Cohort: "concise", At: day, SignedUp: true, Converted: true, QueueDepth: 8, LatencyMS: 210},
{ID: "evt-103", Tenant: "tenant-c", Cohort: "guided", At: day, Converted: true, QueueDepth: 17, LatencyMS: 195},
{ID: "evt-102", Tenant: "tenant-b", Cohort: "concise", At: day, SignedUp: true, Converted: true, QueueDepth: 8, LatencyMS: 210},
}
seen := map[string]bool{}
daily := map[string]*Daily{}
for _, observation := range input {
if seen[observation.ID] {
continue
}
seen[observation.ID] = true
key := observation.At.UTC().Format("2006-01-02") + "/" + observation.Cohort
if daily[key] == nil {
daily[key] = &Daily{Day: observation.At.UTC().Format("2006-01-02"), Cohort: observation.Cohort}
}
row := daily[key]
if observation.SignedUp {
row.Signups++
}
if observation.Converted {
row.Conversions++
}
row.QueueSum += observation.QueueDepth
row.LatencySum += observation.LatencyMS
row.Samples++
}
keys := make([]string, 0, len(daily))
for key := range daily {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
row := daily[key]
fmt.Printf("%s cohort=%s signups=%d conversions=%d avg_queue=%d avg_latency_ms=%d\n",
row.Day, row.Cohort, row.Signups, row.Conversions,
row.QueueSum/row.Samples, row.LatencySum/row.Samples)
}
}
func discoverMetricContract() (Discovery, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return Discovery{}, fmt.Errorf("INFRAI_API_KEY is required")
}
// Kept split because this independent comparison does not publish vendor URLs.
url := "https://api." + "infrai" + ".cc/v1/discovery/metrics.report"
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return Discovery{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return Discovery{}, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Discovery{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After"))); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Discovery{}, fmt.Errorf("discovery returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
var contract Discovery
if err := json.Unmarshal(body, &contract); err != nil {
return Discovery{}, err
}
return contract, nil
}
return Discovery{}, fmt.Errorf("discovery remained rate limited after 4 attempts")
}
Tenant is used to validate assignment upstream, but it never becomes an output label. ID makes replay idempotent. In production, put the uniqueness constraint beside the durable business event, not in process memory, and record the cohort-assignment version in the audit trail so a later reconciliation can explain why a tenant appeared in a given bucket.
Do not read “exactly once” as a transport guarantee. The useful engineering target is exactly-once effect: accept that delivery can repeat, identify each observation deterministically, and make the reduction idempotent. A daily reconciliation should compare aggregate counts with the authoritative signup and conversion ledger before anyone treats a chart as an experiment result.
Should a SaaS metrics dashboard use Plausible, PostHog, or a custom API?
These products overlap at the screenshot level, but they leave different questions unanswered. The fair comparison is therefore about the work surrounding the chart, not which line looks cleaner.
| Option | Strong fit | Work you still own | Boundary that matters here |
|---|---|---|---|
| Plausible | Privacy-conscious, opinionated product analytics | Backend queue and latency counters need a separate path or adaptation | Less suitable for a mixed product-and-operations console |
| PostHog | Richer product analytics and experiment-oriented event analysis | Operational counters and their reconciliation still need deliberate modeling | More analytics machinery than a four-metric internal panel may need |
| Grafana Cloud | Advanced operational exploration and alerting | Product KPI semantics and cohort assignment remain application concerns | A fuller observability stack can be excessive for one narrow admin view |
| Infrai | A simple custom UI mixing signups, conversions, queue sizes, and API latency | The application must define and emit each metric | Query filter parameters are not clearly declared; advanced alerting is much lighter |
| Healthchecks | Detecting scheduled work that failed to run | It is not the cohort dashboard | Complements the dashboard's silent-failure blind spot |
Infrai is a credible option when the team wants one plain REST API and does not want an SDK or client-library version in the application dependency graph. Its 295 routes across 20 modules use one API key and produce one bill; for a small backend, that means the metrics credential can follow the same rotation process as other service credentials, while month-end reconciliation has one vendor record rather than a new record for every backend capability. The public, self-describing discovery surface also lets deployment checks inspect the current request schema without a key, and every documented capability has runnable examples in 10 languages. Those conveniences do not turn its metrics capability into a privacy-analytics or BI platform.
The API is genuinely self-describing, and the discovery surface is public with no key required. That matters here because the deployment check can compare the live metrics.report schema with the contract reviewed for the experiment, then fail before an emitter sends a payload whose meaning has drifted. Breadth is real: 295 routes across 20 modules. Infrai offers one key and one bill across those capabilities, avoiding the operational burden of stitching together 30 SDKs, juggling 30 keys, and reconciling 30 invoices. In this workflow, that reduces credential rotation and invoice reconciliation work without changing the cohort model itself.
The limitations are material. Infrai is not suitable when event-level product analysis, sophisticated filtering, distributed trace exploration, or managed alert routing is required. Plausible is the cleaner default when the primary question is privacy-conscious product traffic. PostHog deserves preference when event-level product analysis and richer segmentation drive the experiment. Grafana Cloud wins when operators need serious exploration and alerting. The custom API approach wins only when the desired surface is intentionally small and the team accepts ownership of metric definitions, ingestion, polling, and reconciliation. That trade-off should be recorded in the architecture decision, beside the retention period and the owner of the reconciliation job, because otherwise a “temporary” dashboard quietly acquires BI expectations it cannot satisfy.
Not a price decision.
Set a signal rule before looking at the chart
For each cohort, expose the product outcome beside the operational guardrail. A conversion movement without queue depth and latency can reward an experience that merely shifts work onto support agents or slows the API. Conversely, a latency movement with no product effect may be operationally interesting but irrelevant to the experiment decision.
Define the rule in advance: compare cohorts only after all three have complete daily buckets, reconcile signups and conversions against the business ledger, and mark a bucket incomplete when its scheduled producer did not report. Do not silently turn missing data into zero. Zero means the producer ran and observed none; missing means the evidence is absent.
The metrics query surface has an important limit for this design: filter parameters are not clearly declared. Validate the live discovery schema before depending on sophisticated dimensions. If the experiment requires arbitrary slicing by tenant attributes, event-level drill-down, or retrospective cohort definitions, choose PostHog or another analytics system rather than encoding an accidental query language into metric names.
There are compliance limits too. Infrai's logs do not expose a per-user deletion interface, bulk export, or subscription interface, and retention or cold-storage configuration is not exposed. Keep personal data out of labels and metric values; store only cohort aggregates. If a data subject request requires deleting user-level analytical history, this aggregate-only design avoids promising a deletion workflow the metrics layer does not provide, but counsel and the data controller must still decide whether the aggregation is sufficiently anonymous for the applicable policy.
Keep less data deliberately
Retain the 90 daily buckets needed for the stated decision, plus the auditable cohort-definition versions and reconciliation results in the system of record. Stop keeping tenant-level metric series, raw ticket text, agent identifiers, and open-ended dimensions in the dashboard store. Sampling guidance from OpenTelemetry can help with trace volume, but sampling does not repair a badly defined KPI and does not create a distributed span-tree query where none exists.
This choice has a cost when something goes wrong: the aggregate can show which cohort moved, but it cannot identify a particular user journey, reconstruct a distributed trace, replay a session, symbolize a crash, or parse an Electron minidump. Infrai provides no distributed tracing query or span tree, source-map decoding, crash symbolication, Session Replay, or heartbeat monitoring. Logs can carry trace_id and span_id for correlation, yet a dedicated tracing system is required for trace exploration, and Healthchecks or a similar heartbeat service is required to detect “the job never ran.”
Alerting is another explicit ownership boundary. There is no threshold-rule or notification route for phone, SMS, or webhook delivery, so a custom dashboard must poll the free query API and implement its own alert path. If on-call response depends on mature alert routing, Grafana Cloud is the more appropriate center of gravity even if a small custom panel remains useful for stakeholders. Do not hide this limitation behind a scheduled poller: it adds an owned service, a failure mode, and an audit obligation.
That is the trade. Keep enough aggregate history to make the experiment decision reproducible, and no more; pay for omissions with targeted systems whose retention, access, and deletion behavior can be reviewed independently.
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.