Short answer: For a media notification service, use bounded-label metrics for fresh aggregates and a replayable event stream for attempt-level cost attribution; choose the storage and dashboard layers by retention, cardinality, query latency, and ownership boundaries.
A media notification service can report a healthy request rate while quietly paying for retries, provider rejections, and duplicate deliveries. The operational constraint changes the dashboard choice: every metric needs a stable delivery identity and a cost-bearing outcome.\n\nShort answer: model delivery attempts as an event stream, attach a bounded failure reason and route to each event, then aggregate those events into a dashboard with a freshness indicator. Choose storage and visualization components by retention, cardinality, query latency, and ownership boundaries. A bright chart without those fields is incident decoration.
The incident lesson: a green chart can still hide a bill
I have been paged by missed jobs and duplicate deliveries. In the media case, the useful question is not “How many notifications did we send?” It is “Which delivery outcomes created work or cost, and can we attribute them to one attempt?”
The invariant is an attempt ID. One logical notification may have several attempts, so the dashboard must distinguish notification_id from attempt_id. A retry should increment attempt count, not pretend that the original attempt succeeded. The event should also carry a route such as email, push, or webhook, a bounded failure_class, and a timestamp from the service that made the attempt.
I initially treated provider response codes as the main dimension. That made the table look precise and the operating decision worse. Raw codes changed across providers, while the action stayed stable: retryable, terminal, throttled, or unknown. Normalize the action-facing class at the edge and retain the raw code only in logs or trace context.
This approach does not apply when the business event has no stable identity or when the dashboard is only a short-lived developer preview. For production scheduling and notifications, losing identity is the more expensive trade-off.
What should the dashboard answer during an outage?
A useful dashboard answers four questions in under a minute:
- Are delivery attempts arriving now, and is the data fresh?
- Which failure class is growing?
- Which route, campaign, or region owns the growth?
- Are retries producing eventual success or duplicate delivery?
Keep dimensions bounded. route, failure_class, region, and a coarse campaign_type are defensible starting points. Do not put recipient address, notification ID, or arbitrary provider text into a metric label. Those values create high cardinality and can also turn a dashboard into a data-leak surface. Put them in structured logs correlated by attempt_id.
A small event model can make the contract explicit:
package delivery
import "time"
type Outcome string
const (
\tOutcomeDelivered Outcome = "delivered"
\tOutcomeRetryable Outcome = "retryable"
\tOutcomeTerminal Outcome = "terminal"
\tOutcomeThrottled Outcome = "throttled"
\tOutcomeUnknown Outcome = "unknown"
)
type Attempt struct {
\tNotificationID string
\tAttemptID string
\tRoute string
\tRegion string
\tCampaignType string
\tOutcome Outcome
\tRawCode string
\tObservedAt time.Time
}
The dashboard should display the event age beside every current-rate panel. A zero count can mean no traffic, a broken exporter, or an empty query window. Freshness is a measurement, not a cosmetic badge.
How do you attribute cost without turning metrics into billing records?
Start with a cost attribution key, not a price. The key can be joined to a daily finance table later: route, region, campaign type, outcome, and attempt count. This keeps the real-time path stable when provider rates or internal chargeback rules change. Pricing is a separate, versioned input.
For example, a counter named notification_delivery_attempts_total can use only bounded labels. A second counter, notification_delivery_outcomes_total, can separate delivered, retryable, terminal, and throttled. The service emits a structured event for each attempt, and a batch job joins aggregated counts to the applicable rate card. The live dashboard shows exposure and failure mix; the finance report computes the final amount.
That separation prevents a familiar failure mode: someone embeds a mutable monetary value in a metric label or rewrites historical dashboards after a rate change. It also gives incident response a clean decision rule. If retryable attempts rise but eventual delivery remains flat, inspect the retry policy and queue age. If terminal failures rise for one route, inspect the route contract. If all routes go quiet, inspect ingestion freshness before blaming the sender.
Here is the boundary in Go. The recorder rejects unbounded dimensions before they reach the metrics backend; the event sink still receives the detailed record for investigation.
package delivery
import "errors"
var ErrInvalidDimension = errors.New("invalid metric dimension")
func RecordAttempt(metrics Metrics, sink EventSink, attempt Attempt) error {
\tif attempt.Route == "" || attempt.Region == "" || attempt.CampaignType == "" {
\t\treturn ErrInvalidDimension
\t}
\tmetrics.Inc("notification_delivery_attempts_total", map[string]string{
\t\t"route": attempt.Route,
\t\t"region": attempt.Region,
\t\t"campaign_type": attempt.CampaignType,
\t})
\tmetrics.Inc("notification_delivery_outcomes_total", map[string]string{
\t\t"route": attempt.Route,
\t\t"region": attempt.Region,
\t\t"campaign_type": attempt.CampaignType,
\t\t"outcome": string(attempt.Outcome),
\t})
\treturn sink.Append(attempt)
}
type Metrics interface {
\tInc(name string, labels map[string]string)
}
type EventSink interface {
\tAppend(attempt Attempt) error
}
The exact interface will differ in your service. The important behavior is ordering and idempotency. If the sink write fails after the counter increments, the dashboard can overstate the event stream. If the counter write fails after the sink succeeds, the dashboard can understate it. Use an outbox or replayable event log when that discrepancy matters, and make aggregation idempotent on attempt_id. Do not promise exactly-once delivery from a pair of independent writes.
Which tools should I use when building a real-time metrics dashboard?
There is no universal best tool. Prometheus is a strong fit for pull-oriented, bounded-label time series and local alert evaluation, but long retention and very high cardinality require an additional storage design. Grafana is a visualization and dashboard layer; it can query several backends, so it does not by itself define metric durability or event identity. Datadog combines collection, storage, and visualization in a hosted service, which can reduce platform ownership while making tagging discipline and data-retention decisions important. OpenTelemetry provides instrumentation and transport conventions, but it is not a complete dashboard database.
Those are different boundaries, not a ranking. This comparison is deliberately about failure ownership, not a shopping list.
| Option | Collection boundary | Best fit | Main limitation |
|---|---|---|---|
| Prometheus | Pull-oriented metrics | Bounded-label time series and local alerting | Long retention and high cardinality need additional storage design |
| Grafana | Dashboard and query layer | A shared view over several backends | It does not define metric durability or event identity |
| Datadog | Hosted collection, storage, and visualization | Teams reducing platform ownership | Tagging and retention choices still govern signal quality |
| OpenTelemetry | Instrumentation and transport conventions | Consistent telemetry emitted by many services | It is not a complete dashboard database |
For this notification service, I would test each candidate against the same acceptance set: a one-minute freshness target, a replay of duplicate attempts, a route-level failure slice, a cardinality budget, and a query for the last 24 hours by normalized outcome. Record query latency and missing-data behavior. The winner is the system whose failure mode the on-call team can diagnose.
A useful comparison sheet has four columns: collection ownership, durable storage, query semantics, and operational cost attribution. Fill it with observed behavior from a small production-shaped dataset. Documentation can describe a feature; it cannot tell you how the dashboard behaves after an exporter restart at 02:00.
Make the runbook part of the dashboard
Every panel needs an owner, a freshness signal, and a next action. “Delivery failures” is a title. “Retryable failures by route, last 15 minutes; inspect queue age when fresh” is an operating instrument. Keep the raw event link one click away, but avoid putting sensitive identifiers in the chart itself.
Test the dashboard in deployment pipelines with fixed fixtures: no events, one event, a burst of retryable outcomes, duplicate attempt_id values, and a delayed exporter. Alert tests should assert both the threshold and the absence of fresh data. A panel that looks correct only with normal traffic is unfinished.
The Twelve-Factor App recommends treating logs as event streams rather than files. That principle fits this design: metrics provide bounded, fast aggregates; the event stream preserves the detail needed to explain a failed delivery. Keep the two correlated, then make replay and deduplication normal operations instead of emergency inventions.
The practical decision is therefore narrow. Pick the components that let the team see fresh bounded aggregates, retain enough event detail to explain them, and assign every retry to an accountable route. A dashboard that cannot answer who owns the failure is not real-time observability; it is a delayed count.
Top comments (0)