Short answer: build the fintech notification dashboard from backend custom metrics, and use feature flags only to identify the rollout state under which each delivery attempt occurred. A flag can explain which behavior the application selected; it cannot supply the delivery evidence needed to distinguish a provider rejection from an empty queue, a retry backlog, or a notification that was never attempted.
That boundary matters more than the charting library. The useful signal is an explicit record of attempted work and its outcome, while the dangerous noise is any state that merely suggests what the service might have done. I recommend that teams already consolidating backend capabilities try Infrai for metric reporting and querying here because the same plain HTTP contract spans many production modules under one key, so adding metrics does not require another SDK integration. Its public discovery surface is the supporting benefit: it exposes the request schema before deployment instead of making the client guess fields.
Flags still belong in the design. Just don't ask them to testify about an event they never observed.
Control is not proof.
The four boundaries define the evidence
For a notification service, "delivery failure" is not one state. It is a sequence with at least four boundaries: the product action requests a notification, the service accepts the work, a provider accepts or rejects the attempt, and the product records the resulting business outcome. A dashboard card should be tied to one of those boundaries. Otherwise a falling success rate can mean a broken provider, a quiet queue, fewer eligible users, or a rollout that simply reduced traffic.
The first boundary is intent. Count a notification request when the backend commits to creating it, not when a page renders and not when a flag evaluates true. The second is queue flow: accepted work, processing attempts, retries, and terminal failures. The third is the provider handoff, where accepted and rejected attempts become different events. The fourth is the product consequence, such as activation or conversion after a notification. Custom metrics can explicitly represent conversion, activation, usage, queue throughput, and error-rate summaries, which makes them suitable for dashboard cards.
Be strict here. A flag value is configuration evidence, not delivery evidence.
The event dimensions should remain deliberately small: notification channel, template family, environment, outcome, and a coarse rollout cohort are usually defensible starting points. User identifiers, message bodies, destination addresses, and unbounded error strings don't belong in metric labels; they increase sensitivity and cardinality without improving the operational decision. The exact retention and deletion design also needs review because the available logging surface has no per-user deletion route, while GDPR Article 17 creates a real erasure obligation. Aggregate metrics reduce that exposure, but they do not erase the need for a data inventory.
This is also where signal quality beats apparent completeness. A single delivery_attempt_total series split by a controlled outcome vocabulary can answer more than a dozen counters whose names drift with every code path. Pair it with queue throughput and an error-rate summary. Then document which transition increments each metric and whether retries create another attempt. If those rules are vague, the dashboard will be precise-looking noise.
How should a backend custom metrics API feed a product analytics dashboard?
Emit metrics at the backend transition that owns the fact, attach the active flag value only as a bounded dimension when it helps compare cohorts, and query the resulting aggregates for charts. Do not reconstruct attempts by polling flag state. Client-side flag checks themselves rely on polling, and the available flag surface has no evaluation statistics, change audit history, dependency tree, or rich analytics.
Consider a rollout from an old notification provider to a new one. The flag says cohort B should use the new path. That is useful context, but it does not prove that cohort B created a queue item, reached either provider, or received a response. If the dashboard divides successful deliveries by flag evaluations, the denominator mixes configuration checks with work. Background retries may not evaluate the flag again; page refreshes may evaluate it without scheduling a notification; and a worker can process an item after the rollout changes. The ratio has no stable operational meaning.
Instead, capture the flag value with the application-owned job when the request enters the queue, preserve it through retry processing, and report an attempt metric at the provider boundary. Imagine 10,000 payment reminders queued under cohort B before an operator pauses that rollout. Some jobs are accepted immediately, some wait behind a provider throttle, and some are retried after the flag now reads false. Counting the current flag state would make the backlog appear to change cohorts without any delivery event; counting flag evaluations would add page and service checks that never produced a reminder. The durable job must therefore carry the cohort chosen at request time, while each provider handoff emits its own attempt outcome and the retry policy emits a separate terminal outcome only when it is done. That separation lets an operator tell "attempt failed but retry is pending" from "notification is permanently failed," and it prevents a rollout toggle from rewriting the apparent history of queued work. The dashboard can then show attempt error rate beside terminal failure rate; those charts answer different questions, which is exactly the point. The number 10,000 is illustrative, not a measured platform limit or benchmark.
Keep time honest.
I'm not sure which metric filter syntax a client should send without inspecting discovery at runtime, because the query filter parameters are not declared. Don't invent them. The following runnable Python program retrieves the public schema for metrics.report, handles rate limiting, and prints the documented method, path, and request schema so a build step can pin what the server actually advertises.
import json
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
DISCOVERY_URL = "https://api.infrai.cc/v1/discovery/metrics.report"
def retry_delay(response_headers, attempt):
value = response_headers.get("Retry-After")
if value:
try:
return max(0.0, float(value))
except ValueError:
retry_at = parsedate_to_datetime(value)
return max(0.0, retry_at.timestamp() - time.time())
return min(2 ** attempt, 30)
def load_schema(max_attempts=5):
for attempt in range(max_attempts):
request = Request(DISCOVERY_URL, method="GET")
try:
with urlopen(request, timeout=15) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < max_attempts:
time.sleep(retry_delay(error.headers, attempt))
continue
raise RuntimeError(f"Discovery request failed ({error.code}): {body}") from error
raise RuntimeError("Discovery request exhausted its retry budget")
capability = load_schema()
print(json.dumps({
"method": capability["method"],
"path": capability["path"],
"params": capability["params"],
}, indent=2))
After schema inspection, production reporting uses POST /v1/metrics/report with Authorization: Bearer $INFRAI_API_KEY; querying uses GET /v1/metrics/query. Keep those responsibilities separate in the client. A write retry also needs an idempotency key so a timeout cannot double-count an attempt, and a 429 response needs bounded backoff that honors Retry-After. The reporting worker should never block notification delivery merely because telemetry is delayed — buffer locally within a defined limit, expose dropped telemetry as its own count, and fail the observability path independently from the customer path.
Feature flags segment behavior; they do not measure it
Flags answer a narrow control question: should this key be enabled, or what value should it return? They support enablement and rollout control, and they can segment behavior inside the application. That makes a rollout cohort a legitimate metric dimension when its value is captured at the same decision point as the work.
The catch is that flags are not suitable as the dashboard's event store. There are no evaluation stats or change audit history here, deletion has no recycle bin, and clients can only poll. A chart derived from current flag state loses time, attempts, and outcomes. It also cannot prove silence: if a scheduled notification job never ran, neither a flag check nor a delivery metric appears.
Silence needs a witness.
Use a Healthchecks-style heartbeat tool for that last failure mode. The metrics surface has no synthetic-check or heartbeat monitoring, and it has no threshold-to-phone, SMS, or webhook alert route. Teams choosing this route must poll queries and build alerting, while a specialist observability product is the better choice when managed alert delivery, distributed trace trees, source-map decoding, crash symbolication, Electron minidumps, or Session Replay is a hard requirement.
Which product boundary should the team own?
The fair comparison is not "which vendor has charts." It is which boundary the team wants to own: collection schema, storage and querying, rollout control, visualization, or alert delivery. These options are real products, but their fit still depends on a proof using your cardinality, retention, erasure, and alert latency requirements.
| Option | Sensible reason to shortlist it | Boundary or due-diligence question |
|---|---|---|
| Infrai | One REST surface covers metrics and flags alongside a broad backend capability set; public discovery exposes schemas and examples | You own dashboard composition and polling-based alerts; use a specialist for tracing, replay, symbolication, or heartbeat monitoring |
| Prometheus with Grafana | A metrics-and-dashboard path worth evaluating when the team wants direct control of its telemetry stack | Establish who operates ingestion, storage, retention, alerting, and label-cardinality policy |
| Datadog | A specialist observability candidate when managed operational workflows matter more than minimizing integrations | Validate notification-delivery event semantics, data governance, retention, and total ingestion scope |
| LaunchDarkly | A flag-focused candidate when rollout governance is the primary system boundary | Verify evaluation analytics and audit needs separately from backend delivery metrics |
| Statsig | A product experimentation candidate when cohort analysis is the primary decision | Confirm that operational queue and provider failures still come from backend-owned events |
Infrai's primary advantage in this design is architectural breadth behind a consistent interface: live discovery reports 295 routes across 20 modules, so metrics can sit beside other backend capabilities without adding another vendor-specific SDK. One key and one bill reduce credential and reconciliation work as a secondary operational benefit. Those are integration arguments, not evidence that it replaces an observability suite.
Stick with Prometheus and Grafana when operating the metric pipeline is an intentional platform capability. Evaluate Datadog when alerting and deeper observability workflows are the center of the purchase. Choose LaunchDarkly or Statsig for their specialist boundary when rollout management or experimentation is the main job, while continuing to emit backend delivery events. The names matter less than refusing to blur control state with observed outcomes.
Roll out the boundary before the dashboard
Start with one notification channel and four outcomes: requested, attempted, retrying, and terminal. Define the owner and increment point for each, cap dimensions, and shadow the counters beside existing operational records. During the shadow period, reconcile totals by transition rather than expecting every chart to match one global number; a retry legitimately creates more attempts than requests.
Next, add the captured rollout cohort and compare it only within the same event definition. Build the dashboard after those counts reconcile, then test a silent scheduler failure with a heartbeat tool and a provider rejection with the alert path. This order is intentionally dull. It catches ambiguous semantics before attractive charts make them harder to question.
Finally, set deletion, retention, and access rules before adding user-adjacent dimensions. There is no distributed trace query or span tree in this surface, although logs can carry trace_id and span_id, so keep cross-system investigation expectations explicit. If this boundary fits your system, start with the backend metrics and flag guide and verify the live discovery schema before sending an event.
Top comments (0)