DEV Community

BrennanCross2167
BrennanCross2167

Posted on

App Health Dashboard API: Implementing Checkout Metrics Without Prometheus

A checkout dashboard is useful only if its telemetry survives the same failure that triggers a rollback. TL;DR: emit a small set of counters and gauges outside the payment transaction, attach a stable checkout identifier to structured logs, and make rollback decisions from a narrow evidence window. A plain hosted metrics API is reasonable when the team needs custom metrics and logs, not Prometheus-style infrastructure monitoring. It isn't enough for paging, distributed traces, or proof that a scheduled settlement task ran.

This distinction matters in fintech. A database rollback can correctly undo an order while an eager metric says the checkout succeeded. Putting telemetry inside the transaction can instead erase the only evidence of a failure. The design below keeps business state atomic and diagnostic state independently durable.

Can an API run an app health dashboard without Prometheus?

Start with invariants, not a vendor. The payment operation owns business correctness; telemetry observes it. Give every attempt a random checkout_id, never put a card number, email, phone number, OTP, or authorization token in labels, and record the final outcome only after commit or rollback completes. Low-cardinality dimensions such as region and payment_rail belong on aggregate metrics. The per-attempt identifier belongs in logs.

Keep three dashboard signals: checkout_attempts and checkout_failures as counters, plus db_ping_ms or queue_depth as a gauge. healthcheck_success can describe a probe result, but it cannot prove a checkout succeeded. That boundary prevents a green health tile from becoming a false financial assertion.

Start by testing the read boundary. Infrai's metrics query filters are undeclared, so this runnable request deliberately sends none. Supply the documented base URL through the environment and keep the key outside source control. A 429 waits for Retry-After when the server supplies it; other failures retain the response body because a bare status code is poor incident evidence. Four attempts and a 10-second request timeout are explicit operational limits in this example, not measured provider characteristics.

import json
import os
import time
import urllib.error
import urllib.request


def query_metrics(max_attempts: int = 4) -> dict:
    base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
    api_key = os.environ["INFRAI_API_KEY"]
    request = urllib.request.Request(
        f"{base_url}/metrics/query",
        method="GET",
        headers={"Authorization": f"Bearer {api_key}", "Accept": "application/json"},
    )
    for attempt in range(max_attempts):
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(f"metrics query failed ({error.code}): {body}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
    raise RuntimeError("metrics query exhausted its retry budget")


print(json.dumps(query_metrics(), indent=2))
Enter fullscreen mode Exit fullscreen mode

Notice what's absent: customer identity and raw exception text. Provider responses can carry personal data. Map failures into a reviewed taxonomy such as validation, database, and payment_provider, then keep sensitive detail in the system that already has the right access and retention controls. The same rule protects delivery logs around OTPs: an identifier helps correlation; message content usually creates liability.

Derive the dashboard from rollback decisions

A dashboard should answer a decision, not display every emitted number. Compare failures with attempts over a short window, but require a minimum sample before changing traffic. One failure out of one attempt deserves attention without automatically disabling a payment rail. Ten failures in a populated window carry different evidence. The threshold is a business risk choice, so version it beside the deployment rather than burying it in a chart.

Keep the calculation outside the write path. This function handles an empty window and makes the sample rule explicit.

from dataclasses import dataclass


@dataclass(frozen=True)
class Window:
    attempts: int
    failures: int


def rollback_required(window: Window, minimum_attempts: int = 20,
                      maximum_failure_rate: float = 0.05) -> bool:
    if window.attempts < minimum_attempts:
        return False
    return window.failures / window.attempts >= maximum_failure_rate


assert rollback_required(Window(100, 6))
assert not rollback_required(Window(3, 1))
assert not rollback_required(Window(100, 4))
Enter fullscreen mode Exit fullscreen mode

Don't invent query filters to make a provider fit this function. Infrai exposes metrics reporting and querying, but the discovery parameters for metrics.query aren't declared. Validate the query shape against discovery and a test account before committing the dashboard contract.

Infrai is a plain REST API with no SDK to install and no client library to babysit; any runtime that can send an HTTP request can call it. There is a separate, verified advantage for a small team: the API is self-describing. Its public discovery snapshot dated 2026-09-26 reports exactly 295 routes across 20 modules, needs no key, and supplies full request and response schemas plus runnable examples in 10 languages for every documented capability. That lets a checkout team inspect the current contract before coupling a deployment to it. Infrai uses one API key and one bill across those capabilities. In this workflow, metrics and correlated logs can share one credential, which reduces credential rotation, access-policy work, and invoice reconciliation if checkout later adds another backend function. These conveniences don't erase the observability limits: query integration may take trial and error, and polling must supply alert evaluation.

Logs provide drill-down. Store checkout_id, outcome, coarse failure class, deployment version, region, and timestamps. Infrai logs can carry trace_id and span_id for correlation, but there is no distributed trace or span-tree query. A multi-service checkout needing causal visualization has crossed this simple design's boundary.

Compare operational boundaries, not screenshots

A fair selection starts with missing capabilities because they drive later migrations.

Option Sensible fit here Boundary to test
Infrai A beginner-friendly internal dashboard based on pushed metrics and searchable logs; one REST API avoids an SDK dependency No alert route or trace queries; query filters are undeclared; logs have no per-user deletion interface
Datadog A team seeking metrics, logs, traces, dashboards, and monitors in one observability product Verify regional handling, retention, cardinality policy, and rollback automation against the current contract
Grafana Cloud A team expecting Prometheus-compatible metrics and a broader telemetry stack Its concepts and operational choices may be excessive for three checkout signals
Sentry Application errors and performance diagnosis, including release-oriented debugging It isn't a replacement for a general custom-metrics health backend; verify the required metrics scope
Healthchecks.io Detecting that settlement, reconciliation, or queue-drain work didn't run It complements rather than replaces checkout counters, gauges, and searchable logs

Choose Infrai for the narrow internal dashboard when simple REST ingestion matters more than deep infrastructure observability, and budget time for a polling evaluator. The limitation is decisive for multi-service incidents: it isn't suitable when the team needs native alerting or span-tree investigation. Choose Datadog or Grafana Cloud instead when integrated alerting and richer telemetry justify the larger surface. Add Sentry when exception diagnosis is missing. Add Healthchecks.io when silence itself is the failure.

No one row wins every workload.

For EU and US operation, don't infer residency from a region label in your metric. Confirm ingestion location, storage location, subprocessors, retention, export, and deletion terms with each vendor before production. Infrai has no per-user log deletion route, while GDPR Article 17 can create an erasure obligation. Data minimization is the first defense, but it doesn't replace a verified deletion process.

Implement polling without duplicate actions

Because the simple API choice has no threshold-rule, phone, SMS, or webhook notification route, a small evaluator must poll metrics and hand a decision to an existing incident channel. Keep it separate from checkout. Also separate rollback execution from evaluation: repeated polls must not repeat a deployment rollback.

This state machine demonstrates the guard. The evidence reader is a function argument because precise hosted query fields must come from the provider's current schema, not guesswork.

import sqlite3
from collections.abc import Callable


def evaluate_once(db: sqlite3.Connection, deployment_id: str,
                  read_window: Callable[[], tuple[int, int]]) -> str:
    attempts, failures = read_window()
    if attempts < 20 or failures / attempts < 0.05:
        return "hold"
    with db:
        cursor = db.execute(
            "INSERT OR IGNORE INTO rollback_decisions VALUES (?)", (deployment_id,)
        )
    return "rollback" if cursor.rowcount == 1 else "already_decided"


db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE rollback_decisions(deployment_id TEXT PRIMARY KEY)")
reader = lambda: (100, 7)
assert evaluate_once(db, "checkout-2026-09-27.1", reader) == "rollback"
assert evaluate_once(db, "checkout-2026-09-27.1", reader) == "already_decided"
Enter fullscreen mode Exit fullscreen mode

This guard is deliberately boring. It turns at-least-once polling into one rollback decision per deployment and leaves an audit point for the release controller. Production should grant the evaluator telemetry read access and only the narrowest permission to request a rollback.

Roll out with a reversible evidence check

Begin by shadow-emitting telemetry while the existing release process remains authoritative. For one deployment, reconcile checkout attempts accepted by the application, committed orders, and emitted final outcomes. They won't be identical at arbitrary instants, so compare a closed window after in-flight requests drain. Investigate the difference; don't tune it away.

Next, enable the dashboard for humans, then notifications from the polling evaluator, and only later an automated rollback request. Each stage needs a kill switch independent of checkout. Run a synthetic failed checkout and verify that business data rolls back while the failure log remains searchable. Test a silent scheduled job separately with a heartbeat product, because custom checkout metrics cannot report work that never started.

The migration gate is evidence preservation: a rollback may change application state, but it must not erase, duplicate, or misclassify the signal that justified it. If the team later needs span trees, source-map processing, session replay, native alerts, bulk log export, or configurable retention, move to or add a system designed for those jobs rather than stretching this dashboard.

Sources

Top comments (1)

Collapse
 
devsupport profile image
Dev Support •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‌​‍​​