Monitor the player-facing path from outside, monitor scheduled work with expiring heartbeats, and make rollback depend on per-cohort evidence rather than one global uptime number. For a game startup comparing an experiment across tenant cohorts, that separation matters more than finding the cheapest advertised monitor: a green aggregate can hide one damaged cohort, while a shared red status can expose an experiment that only a small tenant set received.
TL;DR: use three distinct signals: an external probe for reachability, a heartbeat deadline for cron jobs, and cohort-tagged application outcomes for the rollback decision. Keep the public status page coarse. Keep tenant identifiers out of URLs, labels, and public incident text. Evaluate a monitoring API by its data path, export path, retry semantics, and deletion controls before comparing its price.
What should a startup uptime monitoring API expect from a healthcheck endpoint?
A health endpoint answers a narrow question: can this process serve a cheap request and reach only the dependencies that define readiness? It cannot prove that a new matchmaking rule works for cohort treatment-eu-02, that rewards were granted once, or that the nightly leaderboard job finished before its deadline.
That gap is where experiment monitoring fails. Suppose the control and treatment endpoints both return success, yet the treatment cohort's completed-session ratio crosses the team's rollback boundary. The uptime check is correct. The release is still unsafe.
That is the trap.
So the decision rule needs layers. External reachability pages the service owner. A missed heartbeat pages the job owner. A cohort outcome breach pauses or rolls back the experiment. These events may share an incident timeline, but they should not collapse into one boolean called healthy.
This also keeps the status page honest. Public status should describe customer-visible components such as login, matchmaking, and purchases. An internal cohort key is diagnostic metadata, not a public component and not a useful subscriber filter.
The constraint that changed the design
The tempting design sends every probe, tenant, route, experiment arm, and build identifier as labels on one metric. It is flexible for about five minutes. Then each new label combination becomes another time series, and the monitoring bill is only one symptom; query cost and operational ambiguity grow too. Prometheus explicitly warns against overusing labels and recommends keeping cardinality low. For rollback safety, use bounded dimensions. control and treatment are useful. Raw tenant IDs are not. Store tenant-to-cohort membership in the experiment control plane, then emit only the stable cohort name to the metrics path. If tenant-level investigation is required, look it up in access-controlled logs or traces with a retention policy suited to that data. Privacy changes the API checklist as well. "EU region" is not a complete GDPR answer. The engineering review needs to identify what leaves the game service, where it is processed, who can access it, how long it remains, how deletion works, and which subprocessors receive it. Avoid query strings such as ?tenant_email=... on health and heartbeat calls. URLs tend to travel through proxies, logs, and incident tooling. The explicit trade-off is less ad hoc slicing in the metrics system in exchange for bounded cardinality, clearer ownership, and a smaller privacy surface. For an experiment gate, that is a good constraint: the gate needs stable comparison groups, while a tenant investigation needs controlled detail.
The cheapest plan can become the wrong unit of analysis. Count active probes, heartbeat jobs, status subscribers, retained events, API writes, and the cardinality created by experiment dimensions. Then test export and deletion. A low entry price does not repair an unsafe rollback signal or a data flow nobody can explain.
The smallest working implementation
Start with a deliberately boring contract. One external checker calls /health/ready. Each scheduled job calls an opaque heartbeat URL only after successful completion. The application exports aggregated cohort counters through a generic metrics adapter. No monitor gets a tenant identifier.
type Arm = "control" | "treatment";
type Outcome = "session_started" | "session_completed" | "server_error";
interface CounterSink {
increment(name: string, labels: Record<string, string>): void;
}
export function recordExperimentOutcome(
sink: CounterSink,
arm: Arm,
outcome: Outcome,
): void {
sink.increment("game_experiment_outcomes_total", {
experiment: "matchmaking_latency_guard",
arm,
outcome,
});
}
The label set is closed in the type definition.
That is intentional. Adding tenantId, playerId, or arbitrary error text should require a code review, not a convenient string assignment.
The readiness handler should do less, not more. It returns failure only when this instance must stop receiving traffic. Slow analytics, an experiment evaluation job, or a status-page API does not belong in that dependency chain.
type Check = () => Promise<boolean>;
export async function readiness(requiredChecks: Check[]): Promise<Response> {
const results = await Promise.allSettled(requiredChecks.map((check) => check()));
const ready = results.every(
(result) => result.status === "fulfilled" && result.value,
);
return Response.json(
{ status: ready ? "ready" : "not_ready" },
{
status: ready ? 200 : 503,
headers: { "cache-control": "no-store" },
},
);
}
Do not return dependency names, stack traces, deployment hashes, or secrets from the public endpoint. The external probe needs a status code and a tiny response. Operators can get detail from authenticated telemetry.
Heartbeat semantics need equal care. “Request received” is weaker than “job completed.” Send the heartbeat after the durable side effect and make the job idempotent, because network retries can happen around an ambiguous response. A deadline is also part of the contract: a daily reward job that reports success two hours late is not healthy merely because it eventually called home.
How should the rollback gate combine these signals?
Write the policy before launching the experiment. For example, require at least 1,000 completed sessions per arm, evaluate a 15-minute rolling window, and roll back treatment if its server-error rate exceeds control by 2 percentage points for two consecutive windows. Those numbers are an example policy, not a universal benchmark. Their value is that they are explicit, reviewable, and resistant to a single noisy sample.
Keep reachability as a veto. If the external probe fails across regions, stop interpreting cohort differences and handle the broader incident. If only the experiment outcome gate fails, revert the experiment without declaring the whole game offline. If the settlement heartbeat expires, disable the dependent feature or queue work for recovery according to the job's idempotency guarantees.
| Signal | Scope | Primary action | Public status effect |
|---|---|---|---|
| External readiness probe | Player-facing component | Remove traffic or page on-call | Update only if players are affected |
| Cron completion heartbeat | One scheduled workflow | Page owner and recover safely | Update only for visible impact |
| Cohort outcome gate | One experiment arm | Pause or roll back treatment | Usually none |
This table is also a useful API evaluation test. Can the system preserve these scopes, or does it turn every event into the same incident? Can automation read current state without scraping a dashboard? Are notifications deduplicated? Can maintenance suppress expected probe failures without muting cohort regressions? A ten-minute trial with one probe and one heartbeat reveals more DX friction than a feature grid.
Bad defaults show up fast. A monitoring client that needs a large configuration file for two calls adds glue to every service. A client that silently retries non-idempotent writes makes incident history suspect. A status API without a clean separation between private evidence and public messages invites accidental disclosure.
What I would change at scale
First, move rollback evaluation into a small control service that reads aggregated counters and writes an auditable decision event. Keep deployment, feature-flag, and monitoring systems loosely coupled through that event. A temporary outage in the status-page path should never block rollback.
Second, run synthetic checks from more than one failure domain and require corroboration before declaring a regional outage. The exact number of locations depends on the player footprint and false-positive budget; there is no honest universal count. Benchmark probe latency and alert delivery against the team's own recovery objective.
Third, export telemetry through a standard interface and test the escape hatch quarterly. The test is concrete: retrieve raw events, reconstruct one cohort window, delete a synthetic tenant record where applicable, and verify that the public status history contains no private cohort data.
This costs engineering time.
It also prevents monitoring from becoming the experiment control plane by accident. The trade is worthwhile when rollback safety is the primary decision axis.
The buying decision
Shortlist services only after the contract works with a generic adapter. Then score them on external probe coverage, heartbeat deadline behavior, API authentication, export fidelity, retention controls, deletion workflow, incident permissions, status-page separation, and the amount of configuration required for the first successful call.
Price belongs in the spreadsheet, but use the projected workload rather than a headline tier. Model the control and treatment probes, cron schedules, expected incident traffic, subscribers, and retention. Reject any option that cannot document the data path or preserve the three signal scopes, regardless of cost.
The result is less glamorous than a dashboard comparison. Good. A rollback gate should be dull, bounded, and testable. Green uptime then means reachability, a heartbeat means completed scheduled work, and cohort evidence decides the experiment. Each signal says exactly one thing.
Top comments (0)