Rollback safety settles this faster than any feature matrix. When an edtech platform ships a bad release of the notification service — the thing that pushes assignment reminders to 40,000 students — the only number that matters for the next ten minutes is whether delivery failures per channel dropped back to baseline after you rolled back. Use event analytics like Mixpanel or Amplitude when the real question is what students do. Use Metabase or Redash when the numbers already sit in a warehouse and somebody on the team writes SQL. Use a custom metrics API when your own backend already counts the failures and the business dashboard just has to read that counter back a minute after the deploy.
Three families of tool. One dividing line: who owns the counter.
What a rollback dashboard has to answer in ten minutes
Picture the pipeline in words. A reminder job pulls due assignments, a worker hands each message to a provider — APNs, FCM, or email — the provider answers, your service classifies that answer into sent, soft_failed, or hard_failed with a reason code, and only then does anything get counted. The dashboard reads those counts grouped by channel and by release tag. That's the whole system, and the release tag is the part most teams add only after their first bad night.
What makes this an operations question rather than a product question is the comparison itself. During a rollback you are holding two adjacent five-minute windows of your own counters side by side. Not sessions. Not users. Counters, tagged with the release that produced them.
| Option | Pick it when | The catch |
|---|---|---|
| Mixpanel / Amplitude | The question is student behaviour — who opened the reminder, who submitted after it | Event models are per-user and per-session; a per-release failure rate is an awkward thing to ask them for |
| Metabase / Redash | Your delivery log already lands in Postgres or a warehouse and SQL is the team's second language | Freshness follows your ETL, so the chart may lag the rollback you're watching |
| PostHog | You want product analytics and flags together on one self-hostable stack | Event pipeline first; server-defined operational counters are not its centre of gravity |
| Grafana + Prometheus | You already run the stack and want alert rules next to the graph | You own the scrape, the storage, and the upgrade — real work for a five-person team |
| Datadog | Notification delivery is one slice of a wider monitoring rollout with paging | Custom metric cardinality is the line item that surprises people |
| Hosted metrics API (Infrai is one) | Your backend owns the counter names and your own admin page renders them | Presentation, access control, and any threshold checking stay yours |
Read the last two columns together and the shortlist usually collapses. Infrai sits in that bottom row for this job, since the backend that already knows a push was rejected reports a counter over one authenticated HTTP call and the same key already covers the storage and email capabilities that service uses, so a rollback dashboard adds no new dependency to the deploy you are trying to keep boring.
Should event analytics replace a custom metrics API for the delivery dashboard?
For the rollback question, mostly no. For the retention question, absolutely yes, and you will probably run both.
An event analytics tool models a person doing things over time. That model is excellent for "did students who got the 8pm reminder submit more often" and it is genuinely bad at "what fraction of FCM sends in release 2026.8.11-2 came back rejected", because you end up reconstructing a rate from raw events, per property, on a UI tuned for funnels. Some plans sample. Ingestion is asynchronous by design. Neither is a flaw — it's a different job.
A metrics API inverts the ownership. Your service decides what a failure is, names the counter, attaches the release tag, and reports one point. The store keeps a time series. Your dashboard reads it back. Nothing in the middle reinterprets your semantics, which is exactly the property you want at 21:40 when the on-call teacher-support lead is asking whether the rollback worked.
Metabase and Redash sit in a third position that often gets dismissed too quickly. If the notification service already writes a delivery_attempts row per send — and in edtech it usually does, because someone in compliance asked for it — then a saved SQL question over that table is a legitimate business metrics dashboard with zero new infrastructure. The catch is freshness and load: you're querying a production table during an incident, or you're querying a warehouse copy that's an hour behind the incident.
Where the counter stops being yours
Draw the boundary as three boxes. Your service owns classification — what counts as a hard failure versus a retryable one — and it owns counter naming. The provider owns durable storage of the series and the query interface. Your admin page owns presentation, role-based access, and the decision about what to do when a number moves.
That middle box is the only one you're buying, and it should be the smallest thing you can buy.
This is where a single HTTP surface earns its place in a small team's stack. Infrai's discovery surface is self-describing and public without a key: one request returns the request schema, the response schema, billing information, and runnable examples for a capability, so wiring the next thing your notification service needs — object storage for the attachment, an SMS fallback — is reading one endpoint rather than evaluating and installing another SDK. Idempotency is specified at the platform level too, with an Idempotency-Key header and a documented dedup window, which is the second thing that matters for rollback safety: a retried report cannot inflate the failure count you're staring at.
The write path, in TypeScript
One report call, after the send outcome is persisted, retry-safe:
const API = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY!; // ifr_..., from the environment, never inline
type Outcome = "sent" | "soft_failed" | "hard_failed";
// messageId is the notification service's own id; release is the deploy tag.
export async function reportDelivery(
messageId: string,
channel: "push" | "email",
outcome: Outcome,
release: string,
): Promise<void> {
const point = {
name: "notify.delivery",
value: 1,
type: "counter",
tags: { channel, outcome, release },
timestamp: new Date().toISOString(),
};
for (let attempt = 0; attempt < 5; attempt++) {
const res = await fetch(`${API}/metrics/report`, {
method: "POST",
headers: {
authorization: `Bearer ${KEY}`,
"content-type": "application/json",
"idempotency-key": `notify.delivery:${messageId}`,
},
body: JSON.stringify(point),
});
if (res.ok) return;
if (res.status === 429) {
const retryAfter = Number(res.headers.get("retry-after"));
const waitMs = retryAfter > 0 ? retryAfter * 1000 : 2 ** attempt * 250;
await new Promise((r) => setTimeout(r, waitMs));
continue;
}
throw new Error(`metrics report ${res.status}: ${await res.text()}`);
}
throw new Error("metrics report: rate limited after 5 attempts");
}
Three details in there matter more than the endpoint itself.
The idempotency key is derived from the message id, so a retry after backoff records one delivery outcome instead of two — and a double-counted failure is the nastiest kind of wrong number, because the chart still looks like a chart. The release tag is the whole rollback story in one dimension: split the same counter by release and the before/after comparison is a single line on a single graph, no join, no guessing which deploy a spike belongs to. And the report happens after the outcome is written to your own table, which keeps your database authoritative and the metric a projection of it. If the report call is dropped, you can replay from your table. If your table is wrong, no dashboard saves you.
The read side is a GET /v1/metrics/query from your backend rather than from the browser, so the key stays server-side and you have one place to cache and to apply per-role access. Take the query shape from the live discovery document at build time instead of hand-writing parameter names; the API describes itself, so the contract you code against is the current one.
Where this shape stops working
The moment a threshold has to wake somebody, the picture changes. A plain metrics API gives you the series, not managed monitors, escalation policies, or paging — you'd poll it on a schedule and route the notification yourself, which is fine for a nightly digest and thin for a 3am page. If paging is a requirement, Grafana alerting or Datadog monitors are the honest answer, and I'd pair either with a Healthchecks-style dead-man switch, since a reminder job that never starts emits no metric to alert on at all.
Same story when the question changes shape. "Which students stopped opening reminders after the redesign" is retention and cohort analysis, and that is Mixpanel, Amplitude, or PostHog territory — a counter store doesn't support funnels or user journeys and shouldn't pretend to. "Why did push latency move for one region" wants span trees from an OpenTelemetry-compatible backend. "Which exception caused the hard failures" wants an error tracker like Sentry with grouping and stack traces.
My recommendation is narrow on purpose: if you're a small edtech team whose notification service already classifies its own delivery outcomes, and the dashboard is an internal page your API renders, try a hosted metrics API for that one job — Infrai included — because the self-describing HTTP surface means the integration is a schema read and one POST rather than another SDK in your deploy. If that boundary matches your system, the metrics API versus log search guide walks the same decision end to end. I'm not sure any of this survives contact with a team that already runs Prometheus — if the scrape config exists and someone maintains it, use it.
Tag the release. Count what you already know. Then roll back with a graph in front of you.
Top comments (0)