A notification metric is cheap only when it can answer who created the work, which delivery stage failed, and who owns the retry. Choose the backend after fixing that attribution model. For a customer-support notification service, I would compare four paths with the same bounded dataset: CloudWatch, Grafana Cloud's free offering, PostHog, and a small self-hosted metrics API. The winner is the path whose total operational burden fits the team and whose labels preserve responsibility without exposing customer data.
Short answer: do not select on an advertised free tier or ingestion price. First cap label cardinality, separate attempts from logical deliveries, test duplicate handling, and document the European data boundary. Then price the exact retained series, queries, dashboards, alerts, egress, and operator time your design creates.
What did the page fail to tell us?
I run cron and queue infrastructure in production, and I have been paged by both missed jobs and duplicate deliveries. That experience makes one distinction non-negotiable: a retry attempt is not a second customer intent. If the dashboard counts attempts as notifications, a struggling provider can make throughput appear to rise while customers receive late or duplicate messages.
Retries distort both.
The invariant is compact: one logical notification has one stable ID; every attempt has its own ID; each terminal outcome is recorded once per attempt; cost is assigned from low-cardinality operational dimensions. For customer support, useful dimensions can include channel, queue, message class, tenant cost bucket, provider class, region, and outcome. Customer email addresses, ticket IDs, and notification IDs belong in controlled logs or traces, not metric labels.
This changes the backend comparison. A polished chart cannot repair an ambiguous counter. A low ingestion bill cannot repair an unbounded tenant label. And a European endpoint, by itself, does not settle what personal data is sent, where subprocessors handle it, how long it remains, or who can delete it. GDPR assessment is a data-flow exercise, not a checkbox attached to a hosting region.
Build the accounting boundary before the dashboard
Start with three event meanings: accepted, attempted, and terminal. accepted records that the application took responsibility. attempted records a provider call. terminal records either delivered, permanently failed, or exhausted according to the service's policy. A retry increments attempts, never accepted work.
For cost attribution, resist the tempting label that answers every future question. Per-ticket and per-customer labels create a series for each distinct combination and make retention harder to reason about. Put a stable, coarse cost_bucket in metrics, such as a contracted support tier or internal business unit, and keep the mapping from a customer to that bucket in the source system. This loses instant per-ticket slicing. It buys a metric set whose size can be estimated before deployment and whose contents are less likely to identify a person.
A practical upper bound is a multiplication, not a guess. If the chosen schema has 3 channels, 4 outcomes, 2 regions, 6 cost buckets, and 2 provider classes, the full combination is 288 series per instrument before replicas or other dimensions. Measure the real populated combinations during a shadow period; do not assume all 288 exist. The point is to make the budget reviewable.
Syslog's severity model is useful for logs, but severity is not delivery state. RFC 5424 defines numerical severity values from Emergency through Debug. Mapping every permanent notification failure to error may help routing; it still does not tell finance which cost bucket created provider work or tell an operator whether a retry is safe. Keep severity and outcome separate.
How should a custom metrics dashboard backend compare cloud and self-hosted APIs?
The comparison should be a procurement test, not a feature-list contest. Feed each candidate an identical synthetic workload with no personal data, then record the evidence below. Product plans and boundaries change, so verify current terms directly rather than copying a price from an article.
| Path | Question that can disqualify it | Cost evidence to collect | Operational burden to count |
|---|---|---|---|
| CloudWatch | Does the planned region and account boundary satisfy the documented data flow? | Retained metric identities, reads, dashboards, alarms, and transfer generated by the test | IAM, cross-account access, export, and incident query practice |
| Grafana Cloud free offering | Do current limits cover the measured series, samples, users, and retention? | Usage report at steady state and after a retry burst | Limit monitoring, alert routing, and an exit export rehearsal |
| PostHog | Does an event-oriented model answer rate and alert questions without duplicating telemetry? | Events retained, query workload, and any required export | Schema governance, deletion workflow, and on-call usability |
| Self-hosted metrics API | Can the team own storage, compaction, backup, restore, upgrades, and response? | Compute, storage, replication, transfer, backup, and engineering time | The whole service lifecycle, including failed restores |
These rows do not rank the products. They expose different failure surfaces. Cloud attachment may simplify identity and network boundaries for a team already operating there, while increasing coupling to that account design. A hosted free plan can make a bounded trial easy, but a limit is useful only after series and sample counts are measured. Event analytics may suit product questions yet require proof that alerting and aggregation semantics fit operations. Self-hosting offers direct control over the data path and also transfers pager ownership to the team.
No backend removes that trade-off.
Use two workloads. The steady workload verifies ordinary cardinality and query behavior. The retry-storm workload replays the same logical notifications through multiple attempts, which tests whether cost and delivery counts stay honest. Record missing data, duplicate terminal events, query latency, delayed ingestion, and the work required to explain an alert. Do not convert a short trial into a universal performance claim.
Make duplicate recording harmless
Metrics transports can retry, processes can restart, and queue messages can be delivered again. The preventative path needs an idempotency boundary before it increments terminal business counters. This Go example keeps storage interfaces generic and makes the ordering visible:
package delivery
import (
"context"
"errors"
)
type TerminalEvent struct {
AttemptID, Channel, Region, CostBucket, ProviderKind, Outcome string
}
type Deduper interface {
// Claim returns true exactly once for a given key.
Claim(context.Context, string) (bool, error)
}
type Counter interface {
Add(context.Context, string, map[string]string, uint64) error
}
func RecordTerminal(ctx context.Context, d Deduper, c Counter, e TerminalEvent) error {
if e.AttemptID == "" {
return errors.New("attempt ID is required")
}
claimed, err := d.Claim(ctx, "terminal:"+e.AttemptID)
if err != nil || !claimed {
return err
}
labels := map[string]string{
"channel": e.Channel, "region": e.Region, "cost_bucket": e.CostBucket,
"provider_kind": e.ProviderKind, "outcome": e.Outcome,
}
return c.Add(ctx, "notification_delivery_terminal_total", labels, 1)
}
The interface alone does not guarantee atomicity. In production, the claim and durable recording need a failure model: if the process claims the key and dies before incrementing, the counter can be low; if it increments first and dies before the claim, it can be high. Resolve that gap with an outbox or another transactional handoff supported by the system of record, then let the telemetry exporter retry. Test a crash at every boundary.
This advice does not apply unchanged to approximate infrastructure signals such as CPU utilization, where a lost sample and a duplicate business outcome have different consequences. It also does not make a counter a billing ledger. Financial reconciliation should use durable business records; metrics show trends, ratios, and alert conditions.
The limitation matters.
A decision rule the on-call team can defend
Keep a candidate only if it passes all four gates: the retry-storm dashboard preserves logical delivery counts; the cardinality budget survives the measured workload; the documented data flow and deletion process satisfy the organization's GDPR review; and the full operating cost is acceptable to the team that will carry the pager. Weight those gates before seeing vendor results. Otherwise the easiest demo quietly becomes the decision.
A hosted path is not suitable when policy requires direct infrastructure control. A self-hosted API is a poor choice when nobody can own restores and upgrades. Choose the existing cloud boundary only when its coupling, access model, and measured costs are acceptable; choose event analytics only when its semantics pass the operational alert test. These are limitations, not footnotes, because each one changes who gets paged and who can repair the system.
That's the trap.
Feature toggles can reduce rollout risk when introducing the exporter or changing metric schemas. Martin Fowler's treatment also makes the carrying cost clear: toggles need categories, ownership, and removal discipline. Use a short-lived release toggle to shadow the new path, compare old and new aggregates, and roll back emission without changing delivery behavior. Do not create permanent combinations of telemetry modes across every worker.
Deploy in stages. First emit synthetic records. Next shadow real aggregate counts without alerting. Then enable dashboards, followed by alerts with a runbook that states the numerator, denominator, delay tolerance, and safe retry action. Finally, rehearse export and restore for any path whose exit depends on them. A backend that cannot produce evidence during that rehearsal has failed the test, regardless of its nominal price.
The cheapest defensible choice is the one that keeps attribution bounded and makes failure legible without creating a second unreliable service. For some teams that will be an existing cloud boundary; for others it will be hosted telemetry, event analytics, or a self-operated API. The workload packet and ownership model decide.
Sources
- Martin Fowler, "Feature Toggles": https://martinfowler.com/articles/feature-toggles.html
- IETF RFC 5424, "The Syslog Protocol": https://datatracker.ietf.org/doc/html/rfc5424
Top comments (0)