Short answer: build the dashboard from backend delivery events, then attach feature-flag state as release context. For a property-management notification service, flag statistics can explain which configuration a request encountered, but they cannot tell an operator whether a rent reminder, maintenance update, or access notice was accepted, retried, exhausted, or suppressed. The useful trade-off is signal quality versus noise: record one compact event for each meaningful delivery transition, keep flag evaluation out of the outcome counters, and join the two through low-cardinality dimensions.
This gives a solo builder an operational screen that answers a concrete question: which tenant notifications need attention? It also avoids turning every SDK evaluation, provider callback, and retry attempt into three competing versions of the truth.
Should a product analytics dashboard use backend metrics or flag stats?
Start with decisions, not charts. A property manager needs to know whether a notification reached a terminal state, which channel and message class were involved, and whether failures cluster around a rollout cohort. The dashboard does not need every payload field to answer those questions. Payload detail raises the chance of storing resident data that nobody needs for aggregate analysis.
Define the primary unit as a logical notification, identified by an opaque ID. Attempts are children of that unit. A timeout followed by a successful retry is one delivered notification with two attempts, not one failure plus one success in the headline failure rate.
A small metric set is enough:
- terminal notifications by outcome:
delivered,failed, orsuppressed; - attempts per logical notification, shown as a distribution;
- time from enqueue to terminal state;
- a bounded failure reason family;
- release cohort, captured as context rather than an outcome.
Do not group by resident ID, street address, raw error message, or notification ID. Those values create unbounded series and weak summaries. Keep them in short-lived diagnostic records only when support genuinely requires them. If an opaque analytics identifier can still be linked to a resident, deleting the application row alone may leave event data behind. GDPR Article 17 describes a right to erasure and its exceptions, so maintain a deletion index, set retention periods, and document which aggregates are no longer attributable to a person.
Emit one outcome, not a cloud of counters
The worker creates a logical delivery record, records attempts while contacting a channel provider, and emits one terminal event when processing ends. Ingestion validates the event and stores a narrow append-only record. Aggregation produces tenant, channel, message-class, and cohort slices. Flag evaluation happens before delivery; its bounded result is copied onto delivery context.
Here is a runnable in-memory sketch. It models attempts separately and prevents a retry from inflating terminal failure counts.
type Outcome = "delivered" | "failed" | "suppressed";
type Channel = "email" | "sms" | "push";
type MessageClass = "rent_reminder" | "maintenance_update" | "access_notice";
type DeliveryEvent = {
schemaVersion: 1;
notificationId: string;
tenantKey: string;
channel: Channel;
messageClass: MessageClass;
outcome: Outcome;
attempts: number;
latencyMs: number;
reasonFamily?: "invalid_destination" | "provider_rejected" | "retry_exhausted";
releaseCohort: "control" | "candidate";
};
type Bucket = { notifications: number; failed: number; attempts: number };
function aggregate(events: DeliveryEvent[]): Map<string, Bucket> {
const buckets = new Map<string, Bucket>();
for (const event of events) {
const key = [event.tenantKey, event.channel, event.messageClass, event.releaseCohort].join("|");
const bucket = buckets.get(key) ?? { notifications: 0, failed: 0, attempts: 0 };
bucket.notifications += 1;
bucket.failed += event.outcome === "failed" ? 1 : 0;
bucket.attempts += event.attempts;
buckets.set(key, bucket);
}
return buckets;
}
const events: DeliveryEvent[] = [
{
schemaVersion: 1,
notificationId: "n_01",
tenantKey: "tenant_a7",
channel: "sms",
messageClass: "maintenance_update",
outcome: "delivered",
attempts: 2,
latencyMs: 1840,
releaseCohort: "candidate"
},
{
schemaVersion: 1,
notificationId: "n_02",
tenantKey: "tenant_a7",
channel: "sms",
messageClass: "maintenance_update",
outcome: "failed",
attempts: 3,
latencyMs: 6200,
reasonFamily: "retry_exhausted",
releaseCohort: "candidate"
}
];
console.log([...aggregate(events)]);
The numbers are fixture data, not a benchmark. More important, the type makes ambiguity visible. attempts cannot masquerade as notifications, and reasonFamily stays bounded. Production ingestion still needs runtime validation because TypeScript types disappear at runtime.
One trap deserves emphasis. Do not increment a custom counter when work is enqueued and call it delivery volume. Queued work can be cancelled, suppressed, retried, or abandoned. Emit the terminal event from the component that owns final state, and reconcile stale nonterminal records separately.
Where flag statistics help, and where they add noise
Feature-flag statistics answer rollout questions: which variation was evaluated, for which bounded cohort, and during which release window? That evidence is useful beside delivery outcomes. Evaluation does not prove that a job entered the queue, that a provider accepted it, or that retries ended successfully.
Keep the join modest. Copy a stable cohort label or configuration revision onto the delivery event at decision time. Do not join dashboard queries against a mutable current flag definition; it may no longer describe what happened. Avoid using every flag key as a metric dimension. Most flags have no causal relationship to delivery and merely multiply series.
If the candidate cohort has more exhausted retries, the dashboard has supplied a lead, not a verdict. Check assignment balance, traffic mix, message class, and delivery logs before attributing cause.
Correlation is cheap. Rollbacks are not.
Build the screen around triage
A dense dashboard fits this job. Put total terminal notifications, failure ratio, suppressed count, and a latency distribution at the top. Below that, show a time series split by outcome and a table grouped by tenant, channel, message class, reason family, and release cohort. Operators should narrow the window and inspect a bucket without exposing message bodies.
Noise control needs explicit rules. Page on sustained terminal failures for a service-level slice, not on one provider rejection. Route invalid destinations to data-quality work instead of waking a delivery operator. Show retry pressure even when final delivery succeeds, because it can reveal degradation before terminal failures rise, but keep it out of the failure numerator.
No universal threshold can be stated honestly. Traffic volume, urgency, channel behavior, and the cost of a missed access notice all matter. Establish a baseline from representative traffic, choose a minimum sample size, and test the alert with injected terminal outcomes before enabling paging. The threshold should encode an action: pause a rollout, inspect a channel, or contact a tenant administrator. If nobody can name the action, the alert is decoration.
Cardinality is the quiet budget killer. Estimate the cross-product before shipping a dimension: tenants times channels times message classes times cohorts times reason families. Store validated events, precompute common low-cardinality views, and run uncommon tenant investigations against a bounded window. This keeps the default screen fast without pretending storage and query work are free.
Ship it without losing the evidence
Unit tests should prove that retries produce one terminal notification and that suppressed work never appears as delivered. Integration tests should send valid and invalid schema versions through ingestion. A deployment test should assign synthetic notifications to both cohorts, force one known terminal outcome, and verify that the dashboard receives the expected bucket. Delete those fixtures after the check.
Deploy schema changes additively. For a new optional dimension, first teach ingestion and aggregation to accept it, then start producing it, and only later make the dashboard depend on it. This order is boring. Good.
After each notification-path release, confirm that ingestion is current, terminal totals reconcile with the delivery store for a sampled window, unknown reason families remain empty, and cohort volume matches intended exposure. Check retention and erasure jobs with the same seriousness as delivery counters. Prune panels that have not driven a decision; every unused slice still consumes storage, query time, and attention.
The decision rule is straightforward: backend terminal events own delivery truth; release cohorts explain controlled differences; attempt data warns about pressure. Keeping those roles separate guides action without converting routine retries and flag evaluations into false incidents.
Top comments (0)