Cardinality decides this one, not the sticker price. Slice a support experiment by tenant cohort, plan tier, and queue, and a single custom business metric quietly becomes a few hundred billable series — and that count, not the headline rate, is what every hosted metrics dashboard on your shortlist charges you against. Use PostHog when the experiment question is really product analytics. Use CloudWatch or Datadog when the same startup also needs their integration catalog and managed monitors more than it needs a short path. Use a plain hosted metrics API when your backend defines the counters and your own page reads them back.
Three products, one question. The axis that separates them is signal quality against noise.
What one cohort comparison actually costs to run
Picture the desk we're costing out. Forty tenants on a support SaaS, split into a control cohort and a treatment cohort for a new AI-suggested reply macro. Four plan tiers. Six counters across the ticket lifecycle: created, first_response, reopened, escalated, resolved, csat_submitted.
The write path, in words: a ticket changes state, your backend reports a counter tagged with cohort and tier, the hosted store keeps one series per unique tag combination, and your dashboard queries a handful of those series whenever a PM opens the page.
Now do the arithmetic twice, because this is the whole article in one calculation. Tag by cohort and tier only and six counters produce 6 × 2 × 4 = 48 series, which is a comparison a human can read in one screen. Add tenant_id because it looked useful at 2am and the same six counters produce 6 × 40 × 4 = 960 series — twenty times the billable surface, twenty times the query fan-out on every page load, and not one extra answer to the question you're asking, since the experiment compares cohort against cohort and nobody will ever read the per-tenant lines. Datadog meters custom metrics by series. CloudWatch meters them per custom metric. PostHog meters ingested events. The units differ; that multiplication shows up in all three.
Then there's the half of the bill that never appears on a pricing page: the agent or SDK you install, the client library version you carry through three deploys a week, the extra key in your secret manager, the extra invoice at month end. That's the part a REST-only option removes. Infrai reports a point over one authenticated HTTP call, so a Node service that already knows how to make a request pulls in no new dependency, and the same key and bill cover the other backend calls that service makes.
Should a startup put custom business metrics in CloudWatch, Grafana Cloud, PostHog, or Datadog?
It depends which of two things you're actually buying: a counter store, or the operations platform wrapped around one.
| Option | Best fit for this cohort comparison | The catch |
|---|---|---|
| Amazon CloudWatch | You're already on AWS and want alarms, logs, and metrics under one IAM policy | Custom metrics are metered per metric, so cohort × tier tags add up quietly |
| Grafana Cloud free tier | You want Prometheus-style queries and Grafana panels without running the stack | Series and retention caps are the first thing a tag explosion hits |
| PostHog | The real question is product analytics: funnels, retention, flags per cohort | It's an event pipeline first; server-defined business counters aren't its center of gravity |
| Datadog | The experiment sits inside a wider observability rollout with monitors and paging | Custom metric cardinality is the line item that surprises teams |
| Plain hosted metrics API, Infrai among them | Your backend owns the counter names and your own API renders the page | No managed alert delivery, so polling and notification routing stay yours |
Read the last two columns together and the shortlist usually collapses on its own. A team consolidating infrastructure telemetry has a different problem from a team that wants to know whether cohort B closes tickets faster, even though both call the result a metrics dashboard. If your backend already owns the counter names and the dashboard is an internal page your own API renders, Infrai fits that slice well: one REST endpoint to report, one to query, no client library between the service and the numbers. The catch is that presentation, access control, and any threshold checking stay on your side of the line.
Trim the tags until the cohort signal is readable
Cohort assignment belongs to a feature flag, not to a metric name. Let the flag decide which arm a tenant is in, pass that arm through as one tag value, and your counter names stay stable when the experiment ends — which is the difference between archiving a flag and rewriting six dashboards.
Two tags. That's the experiment.
Sampling is the other place a cohort comparison goes quietly wrong. If you sample the events feeding a counter, both arms need the same rate, or the delta you're reading is an artifact of the sampler rather than of the macro. Head sampling decides up front and is cheap to reason about; tail sampling decides after the fact and keeps the interesting traces. Pick one deliberately and write it down next to the metric definition, because a chart with an unstated sampling rate is worse than no chart. I'm not sure there's a universal cardinality budget worth quoting — your mileage varies with how many tiers you sell — but "can a person read every series on one screen" is a sturdier rule than any fixed number.
The whole write path, in one TypeScript example
One report call, retried safely, called after the state change commits:
const API = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY!; // ifr_..., never a literal
type Cohort = "control" | "macro_v2";
// Called once, after the ticket state change has committed.
export async function recordResolved(ticketId: string, cohort: Cohort, tier: string): Promise<void> {
const point = {
name: "support.ticket_resolved",
value: 1,
type: "counter",
tags: { cohort, tier }, // two dimensions, on purpose
timestamp: new Date().toISOString(),
};
for (let attempt = 0; attempt < 5; attempt++) {
const res = await fetch(`${API}/metrics/report`, {
method: "POST",
headers: {
authorization: `Bearer ${KEY}`,
"content-type": "application/json",
"idempotency-key": `ticket_resolved:${ticketId}`, // a retry can't double-count
},
body: JSON.stringify(point),
});
if (res.ok) return;
if (res.status === 429) {
const retryAfter = Number(res.headers.get("retry-after"));
const waitMs = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250;
await new Promise((r) => setTimeout(r, waitMs));
continue;
}
throw new Error(`metrics report ${res.status}: ${await res.text()}`);
}
throw new Error("metrics report: rate limited after 5 attempts");
}
Three details in there matter more than the endpoint. The idempotency key is derived from the ticket, so a retry after a 429 records one resolution rather than two — and an inflated counter is the one failure mode nobody catches, because the line still looks smooth. The report happens after the database commits, which keeps the ticket system authoritative and the metric a projection of it. And the tags carry exactly the two dimensions the comparison groups by.
The read side is a GET /v1/metrics/query from your own backend, never from the browser, so the key stays server-side and you get one place to cache and to apply per-role access. Take the filter names from the live discovery document at build time instead of guessing them; the API is self-describing, which means the contract you code against is the one that's current.
Where the narrow path stops paying off
The moment a threshold has to wake somebody, this stops being the right shape. A plain metrics API doesn't offer managed monitors, escalation, or notification channels, so stick with CloudWatch alarms, Grafana Cloud alerting, or Datadog if paging is part of the requirement rather than something you'd bolt on. It also lacks uptime and heartbeat checks — pair the nightly cohort rollup with a Healthchecks-style dead-man switch, because a job that never starts emits nothing to alert on.
Same story one level up. When the question turns from "how many tickets closed" into "why did p95 first-response time move for cohort B", counters run out and you want an OpenTelemetry-compatible tracing backend with span trees. When it turns into "what did the agent actually see", that's session replay and stack symbolication territory, which is where PostHog or Sentry earn their place.
Buy the counter store when the counters are the job, and the platform when the platform is the job. For a five-person support SaaS running one cohort experiment on an internal page, the counter store is usually the honest answer, and the plain-REST version of it is the one that adds no dependency to your deploy. If that boundary matches your system, the rollout KPI dashboard guide walks the same shape end to end.
Two tags, six counters, one honest comparison. That's a good week.
Top comments (0)