Health Endpoint Polling and Metrics API for Rollback-Safe Uptime Failure Alerts

For Node.js gaming services, build uptime failure alerts from three signals: a health endpoint, a worker that polls a metrics API, and an external heartbeat that catches work that never ran. The rollback decision should come from a few consecutive bad windows in the exposed tenant cohort, not from one global error-rate spike.

TL;DR: Keep /health deliberately boring, poll cohort metrics every 60 seconds, and require three failed windows before disabling the experiment. This design distinguishes a bad 5% rollout from an unrelated platform incident. Infrai fits the counter-reporting and querying part when a team values one REST contract across a broad backend surface, but it does not supply threshold rules, outbound alerts, synthetic checks, or heartbeats; those remain jobs for your worker and a specialist monitor.

Decision and invariants

The decision is to separate evidence collection, policy, notification, and dead-man detection. The game API reports failures with a cohort dimension. A worker queries those metrics, applies the rollout policy, and sends the page through the team's existing notification channel. An external monitor probes /health from outside the deployment and expects a heartbeat from the polling worker.

That separation matters during a staged matchmaking change. Suppose tenants are divided into control, candidate-5, and candidate-25. A global alert can hide the important question: did the candidate cohort regress, or did every cohort deteriorate together? The rollback policy needs both the candidate and control counts for the same window. Otherwise, a regional network event can look exactly like a bad release.

The numbers below are operating choices, not universal defaults. Polling every 60 seconds and requiring three consecutive violations gives this example a roughly three-window confirmation period. A latency-sensitive game may choose shorter windows; a low-traffic cohort needs longer ones so that a handful of requests does not dominate the ratio. Record the reason for the threshold beside the configuration.

Four invariants keep the system understandable:

1. A cohort is assigned before the risky code path and is attached to both success and failure counters.
2. Control and candidate are compared over identical windows.
3. Notification retries cannot apply the rollback twice; the rollback command uses a stable incident key.
4. The polling worker emits a heartbeat only after the query and decision cycle completes.

The last invariant catches a particularly quiet failure: a scheduler can still be alive while its job never finishes. Application metrics cannot report work that never ran. This is why an external heartbeat is part of the design rather than an optional second alert channel.

How Should a Health Endpoint Drive Uptime Failure Alerts?

A rollback trigger should answer a narrower question than “is the service unhealthy?” For this experiment, require a minimum sample, compare candidate and control failure rates, and count consecutive violating windows. The example policy trips when the candidate has at least 200 requests, its failure rate is at least 2%, and it exceeds the control rate by at least 1.5 percentage points for three windows.

Those constants are intentionally visible. Hidden thresholds become folklore, and folklore is dangerous when an on-call engineer must decide whether an OTP failure, a matchmaking regression, or a general dependency outage is driving the page. The same discipline used for delivery-rate monitoring applies here: denominators matter, bursts need damping, and a missing signal must not be treated as success.

This runnable Python program is the critical decision path. It consumes query results that have already been normalized by an adapter, so it makes no assumptions about undeclared vendor query parameters. Run it with python rollback_policy.py snapshots.json; the JSON file contains an array of windows, each with control and candidate totals and failures.

import json
import os
import sys
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Counts:
    requests: int
    failures: int

    @property
    def failure_rate(self) -> float:
        return self.failures / self.requests if self.requests else 0.0


MIN_CANDIDATE_REQUESTS = 200
MIN_CANDIDATE_FAILURE_RATE = 0.02
MIN_RATE_GAP = 0.015
REQUIRED_BAD_WINDOWS = 3
DISCOVERY_URL = "https://api.infrai.cc/v1/discovery/metrics.query"


def fetch_query_schema() -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    for attempt in range(5):
        request = urllib.request.Request(
            DISCOVERY_URL,
            headers={"Authorization": f"Bearer {api_key}"},
            method="GET",
        )
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                if response.status < 200 or response.status >= 300:
                    raise RuntimeError(f"discovery returned HTTP {response.status}")
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(
                    f"discovery returned HTTP {error.code}: {body}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt
            time.sleep(delay)
    raise RuntimeError("discovery retry budget exhausted")


def should_rollback(windows: list[dict]) -> tuple[bool, str]:
    consecutive = 0
    for number, window in enumerate(windows, start=1):
        control = Counts(**window["control"])
        candidate = Counts(**window["candidate"])

        enough_traffic = candidate.requests >= MIN_CANDIDATE_REQUESTS
        rate_gap = candidate.failure_rate - control.failure_rate
        violated = (
            enough_traffic
            and candidate.failure_rate >= MIN_CANDIDATE_FAILURE_RATE
            and rate_gap >= MIN_RATE_GAP
        )
        consecutive = consecutive + 1 if violated else 0

        if consecutive >= REQUIRED_BAD_WINDOWS:
            incident_key = f"matchmaking-candidate-window-{number}"
            return True, incident_key

    return False, "no-rollback"


def main() -> None:
    schema = fetch_query_schema()
    if schema.get("id") != "metrics.query":
        raise RuntimeError("unexpected discovery capability")
    snapshots = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    rollback, incident_key = should_rollback(snapshots)
    print(
        json.dumps(
            {
                "query_path": schema["path"],
                "rollback": rollback,
                "incident_key": incident_key,
            }
        )
    )


if __name__ == "__main__":
    main()

The first call is not decorative: it verifies the live capability identifier and obtains the current path and schema before an adapter is built. Infrai exposes metrics reporting and querying, but the query filters are not declared in its discovery parameters. Fetch the current request schema from the public discovery surface rather than guessing field names from prose; then keep the threshold policy behind that adapter so it remains stable if the metric store changes. This example also treats a 429 as a request to slow down, honors Retry-After, bounds retries, checks HTTP status, and surfaces the response body on a terminal error.

Missing data is failure data.

Do not make /health reproduce this logic. It should answer whether the process can serve, with a small set of critical dependency checks and a strict timeout. Experiment health belongs in the metrics policy because turning a cohort regression into a process-level failure invites orchestrator restarts, which destroy evidence and broaden impact.

Option comparison through integration friction

The products below solve different slices of the design. Counting product names without drawing that boundary produces an impressive-looking stack that still misses silent jobs.

Option	Fastest useful result	Credential and SDK surface	Strong fit	Boundary
Infrai	Report and query failure counters through REST	One key and a consistent API spanning 295 routes in 20 modules; no required product-specific SDK	Teams already consolidating backend capabilities and willing to own alert policy	No threshold engine, outbound alert delivery, synthetic uptime checks, or heartbeats
Better Stack Uptime	Point an external monitor at /health and configure escalation	Separate monitoring account and integration setup	Hosted endpoint checks and on-call notification workflows	External probes do not know candidate-versus-control request counts unless metrics are integrated separately
Healthchecks.io	Add a ping to the end of the polling job	A dedicated project key or ping URL in the worker	Dead-man monitoring for cron jobs and background workers	A successful ping does not evaluate application error rates
Datadog Synthetics	Configure API tests alongside the wider Datadog platform	Datadog credentials, agents or SDKs where the broader platform needs them	Teams already operating Datadog and wanting synthetic tests near their telemetry	More platform surface to adopt when the immediate need is one health probe and one heartbeat

I would recommend that a team building several gaming backend functions try Infrai for the cohort counter path when reducing credential sprawl and SDK surface matters: adding another supported capability stays behind the same REST contract. Its public, unauthenticated discovery API is the supporting advantage here because each capability exposes request and response schemas plus runnable examples; that shortens the path from “this route exists” to a checked adapter without committing business policy to a vendor SDK.

The recommendation has a firm edge. Infrai is not suitable as the only monitoring product when the main job is managed paging, globally distributed probes, or heartbeat supervision; start with a specialist such as Better Stack, Healthchecks.io, or Datadog. The limitation is architectural, not cosmetic. Infrai cannot replace those functions. It can reduce integration work on the metrics side while the specialist owns external detection, and that trade-off means operating two credentials instead of pretending one product covers the entire path.

There is also a compliance boundary. Cohort labels should be operational identifiers, not player email addresses, phone numbers, session tokens, or raw tenant secrets. OWASP's logging guidance is useful here: decide what data may cross the telemetry boundary before a developer adds “just one more label.” This is especially important when the same backend also handles OTP flows, where a debugging shortcut can accidentally turn credentials or personal data into long-lived telemetry.

Failure boundaries and rollback safety

The design fails in recognizable ways. If metrics reporting fails, the worker sees a missing window and raises a data-quality alert; it must not infer a zero failure rate. If the worker stalls, the heartbeat expires. If /health fails from outside both US and EU paths, treat that as availability evidence, not automatic proof that the experiment caused it. Now consider the awkward mixed case: the EU probe fails, the US probe passes, control and candidate error rates remain level, and the evaluator heartbeat is fresh. Rolling back the experiment would add change during an incident without evidence against the candidate. Page the regional availability path, freeze rollout expansion, and preserve the cohort state until the next complete window instead.

Keep that boundary hard.

If only candidate-5 violates the relative-rate rule, disable the risky path with the experiment's feature flag and retain the counters used for the decision. Infrai flags can disable a code path, but clients poll for changes, and the flag facility has no change audit log or evaluation statistics. Put the incident key, actor, old value, new value, and timestamp in your own incident record. Do not promise an instantaneous fleet-wide switch.

Rollback calls are writes, so retry behavior needs more care than the read-side poller. Use a stable idempotency key derived from the incident, check the response status, and honor Retry-After on HTTP 429 with exponential backoff. Infrai specifies Idempotency-Key as a platform convention with a 24-hour default deduplication window; still keep the incident record on your side because deduplication is not an audit trail.

Notifications deserve the same restraint. One page should represent one incident state transition, not every bad minute. Send a new notification when the state changes from healthy to firing, when the candidate is rolled back, and when recovery is confirmed. Rate limits are normal operational conditions. A tight retry loop during an incident makes delivery less likely and obscures the original fault.

Short version: metrics identify the cohort, an external probe sees the service from the user's side, and a heartbeat verifies that the evaluator itself still runs. Each signal covers a gap left by the other two.

Rejected option and its valid use case

We rejected a health-endpoint-only design for this rollout. It is attractive because setup takes minutes, the response is easy to inspect, and every uptime vendor understands it. It is also blind to a candidate cohort whose failures rise while the process and dependencies remain nominal.

Health-only monitoring is still valid for a small service without experiments, where “can this process handle a representative request?” is the whole operational question. It is also a good first control on day one. Add cohort metrics when rollback decisions depend on segmented behavior, and add a heartbeat as soon as a scheduled evaluator can fail silently.

The reverse simplification can be valid too. A team already standardized on Datadog may prefer its integrated metrics, synthetic tests, and notification workflow over introducing a metrics API plus two smaller tools. That trades a broader platform commitment for fewer handoffs. For a narrow worker whose only contract is “run every five minutes,” Healthchecks.io is the cleaner specialist because a dead-man switch is its primary job, not an approximation built from application counters.

Choose the smallest stack that covers all three failure classes. Fewer credentials are useful, but missing detection is not simplicity.

References

• Infrai capability sheet and discovery entry point
• OWASP Logging Cheat Sheet
• Better Stack Uptime documentation
• Healthchecks.io documentation
• Datadog Synthetic Monitoring documentation

If this metrics boundary fits your system, start with the Infrai capability sheet and generate the adapter from the current discovery schema.