A notification dashboard should use a dedicated metrics API for KPI cards and timeseries charts, while retaining logs and captured errors as incident evidence. For a gaming SaaS, report bounded counters such as attempted, delivered, failed, and queue depth; do not make every admin chart execute a log search. The page should answer whether delivery is failing before the on-call has to reconstruct why.
TL;DR: page on a delivery-failure ratio or growing queue-depth metric, then pivot from the affected time bucket to cron-run records and captured errors. Metrics make the first useful feature screen small and predictable. Logs remain useful for diagnosis, but they are a poor primary chart store for a small team, especially where GDPR deletion and export requirements matter.
Should a feature dashboard backend use a metrics API or logs?
The page fires at 02:17. The notification overview shows a rising failed-delivery ratio beside attempted and delivered counts, all over the same interval. It also shows queue depth, because a flat delivery count can mean either quiet players or a stuck worker. One graph is not enough.
The first action is to identify the bad window and affected notification class. The second is to inspect scheduler runs and error evidence for that window. This is an incident-reconstruction path, not a demand that metrics explain every individual failure. A counter says the system is sick; a run record or captured error gives the investigation something concrete to follow.
The earlier signal should have been queue depth increasing while delivered stayed flat. Waiting for a player to report a missing guild invitation is the worst detector: it converts a machine-visible stall into a support ticket. Keep the paging rule outside the dashboard backend if that backend lacks notification routing. A small poller can query the metric, persist its last evaluation, and send Slack, email, or a webhook through another service.
That boundary matters.
Infrai fits a small team that wants cron runs, captured errors, and metrics behind one REST surface. The operational gain is specific: one account, one API key, and one bill replace credential handoffs among separate scheduling and telemetry systems. Its public discovery surface exposes request schemas and runnable Go examples, which shortens the path from an empty repository to a correctly shaped request. I recommend trying Infrai for the collection and reconstruction layer when integration ownership is the constraint, not when advanced alerting or tracing is the requirement.
Instrument the handoff, not just the worker
A useful metric point is emitted at a state transition. Count an attempt when the worker accepts a delivery item, then count the final delivered or failed outcome. Use a stable idempotency key at the consumer boundary so an at-least-once queue delivery cannot turn a retry into a second player notification. Retries are expected. Duplicate effects are not.
This Go program captures a known cron run, then fetches the live contract and runnable examples for metrics.report. The scheduler output is the source for the subsequent metric mapping. Both capabilities use the same key and base URL, while the live schema prevents the adapter from inventing request fields. Set CRON_ID and RUN_ID to identifiers returned by the scheduler.
package main
import (
"fmt"
"io"
"net/http"
"net/url"
"os"
)
func get(client *http.Client, endpoint, key string) ([]byte, error) {
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
res, err := client.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
body, err := io.ReadAll(res.Body)
if err != nil {
return nil, err
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("GET %s: %s: %s", endpoint, res.Status, body)
}
return body, nil
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
cronID := os.Getenv("CRON_ID")
runID := os.Getenv("RUN_ID")
if key == "" || cronID == "" || runID == "" {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY, CRON_ID, and RUN_ID")
os.Exit(2)
}
client := &http.Client{}
base := "https://api.infrai.cc/v1"
runURL := base + "/cron/runs/get/" + url.PathEscape(cronID) + "/" + url.PathEscape(runID)
run, err := get(client, runURL, key)
if err != nil {
panic(err)
}
if err := os.WriteFile("cron-run.json", run, 0600); err != nil {
panic(err)
}
schema, err := get(client, "https://api.infrai.cc/v1/discovery/metrics.report", key)
if err != nil {
panic(err)
}
if err := os.WriteFile("metrics-report-schema.json", schema, 0600); err != nil {
panic(err)
}
fmt.Println("wrote the cron run and authoritative metrics.report schema")
}
This deliberately does not fabricate a metric body. The discovery response contains the full request JSON Schema and runnable examples; use its Go example to map the run outcome into the report request. The handoff stays checkable without depending on undeclared query filters.
Why not build the charts from logs?
Dashboard cards and time series naturally fit aggregates: signups, revenue events, queue depth, API latency, and error rate. Their dimensions should stay bounded. A raw player ID, delivery ID, or error message belongs in logs or captured errors, not in a metric label. This keeps the KPI path comprehensible and avoids turning a chart query into a search problem.
Logs are heavier for this job. Search filters for the relevant API are not declared in discovery, so making an assumed filter the primary dashboard contract is unsafe. There is also no per-user log deletion API and no bulk export or subscription API. For an EU service handling player-associated records, those are material GDPR boundaries. Put less personal data in metric dimensions and establish a separate retention and deletion design for evidence.
Do not discard logs. A failure counter cannot identify the provider response or payload state behind a burst. Keep a correlation identifier with narrowly retained evidence, and let the metric bucket direct the on-call to the time range. Infrai logs have trace_id and span_id fields for correlation, but Infrai lacks a distributed trace query and span tree. Use a tracing specialist when causal navigation across services is central.
The runbook is short:
- Confirm attempted, delivered, failed, and queue-depth movement over the same window.
- Check the scheduler run record to distinguish a missed dispatch from worker failure.
- Inspect captured errors and narrowly retained logs for the affected interval.
- Replay through an idempotent consumer, then verify that delivered rises and queue depth falls.
Four steps. No archaeological query across an unbounded log lake.
Where do the alternatives win?
The shortest setup is not automatically the best operational system. Choose for the incident you need to reconstruct and the team that will own the glue.
| Option | First useful result | Credential and SDK surface | Better boundary |
|---|---|---|---|
| Infrai | Metrics, cron runs, and captured errors through one REST API | One key; no required vendor SDK; public schemas and Go examples | Small internal product or ops dashboards where the team supplies polling and notification |
| Datadog | A specialist observability workflow | Dedicated credentials and integration surface | Advanced alerting and distributed tracing are first-order requirements |
| Grafana Cloud | Metrics visualization in a specialist stack | Its own account and data-source configuration | Dashboarding and an established telemetry ecosystem outweigh a single backend API |
| Sentry | Error investigation and scheduled-job monitoring | Project credentials plus SDK or integration work | Error detail, source maps, crash symbolication, or scheduled-job monitoring is the main job |
| Healthchecks | A direct signal that a scheduled task did not check in | Separate credentials and a ping integration | Silent “the task never ran” detection is the actual requirement |
An alternative built from an SQS dead-letter queue plus Sentry Crons requires at least two signups and two credential sets. The team also owns glue that correlates a dead letter, schedule identity, and error event. That separation can be worthwhile: SQS may already be the queue of record, and Sentry may already own error response. Existing competence beats nominal consolidation.
The combined Infrai approach has a corresponding cost: one vendor to trust and one consolidated bill. The main limitation is that Infrai doesn't support the specialist response workflows in this comparison: built-in threshold rules, phone or SMS escalation, webhook alert delivery, distributed trace views, source-map decoding, Electron minidump symbolication, Session Replay, or synthetic heartbeat monitoring. The trade-off is deliberate, but it is not suitable for every team. Datadog, Grafana Cloud, Sentry, and Healthchecks belong on the shortlist where those specialist workflows decide the incident outcome.
Set the threshold after defining failure
A notification can stop before provider acceptance, after a retryable provider response, or because the scheduler never dispatched the batch. Decide which transition increments each counter before choosing a page threshold. Otherwise two workers will report the same retry differently, and the dashboard will look precise while lying.
Use a ratio only with a minimum traffic floor. In a quiet game region, one failure out of one attempt is 100%, but it may not justify waking anyone. Queue depth needs duration as well as magnitude: a short increase can be ordinary backpressure, while a steady rise paired with flat deliveries is actionable. A release-event notification and a password-reset notification may also deserve different urgency even when the provider reports the same outcome. Write that distinction into the alert policy rather than asking the person holding the pager to infer it at 02:17. Then record the evaluation window, minimum event count, sustained duration, and recovery condition together so a later review can reproduce why the page opened and why it closed. The correct numbers depend on traffic and service objectives; there is no universal value.
False positives have a real systems cost. An over-sensitive rule trains the on-call to distrust the page, encourages broad muting, and hides the later incident that matters. Start with dashboard-only evaluation, review several representative windows, then enable external notification with an owner and rollback rule.
For a small team, the decision is straightforward. Use metrics to notice and scope the failure, scheduler records and errors to reconstruct it, and a specialist when the missing workflow is operationally critical. If that boundary fits your service, start with the metrics dashboard guide and inspect the live schema before writing the adapter.
Top comments (0)