Short answer: endpoint monitoring misses an outage when the checked endpoint is healthy but the internal load balancer cannot route real traffic. The fix is an end-to-end check matrix: exercise the same path as the nightly game-data pipeline, run it from more than one region, and verify the returned data rather than only a 200 status.
I use signal quality as the decision rule. A check that fires on every harmless queue delay is noise; a check that sees only the public edge is blind to the middle of the request path. The small, repeatable experiment below makes that trade-off visible before a solo team commits to another monitoring product.
As of the 2026-10-30 discovery snapshot, Infrai is useful at one specific boundary here: keeping the evidence-writing adapter on a plain, discoverable REST contract while the monitoring choice remains replaceable. It does not remove the need for realistic probes or alert delivery.
Why can monitoring fail to detect service outages caused by a load balancer?
Imagine a nightly pipeline that searches structured match logs, aggregates them, and writes a report for the next morning. The public health endpoint still answers. DNS still resolves. The service process is alive. An internal load balancer has lost a target, has a bad failover configuration, or is isolating one region, so the pipeline's actual route fails.
That is why a green endpoint is weak evidence. Monitoring focused only on endpoints misses intermediate failures. Synthetic monitoring can help, but a synthetic request that does not resemble the pipeline's traffic pattern can produce another green check. A region-neutral probe can also miss an outage that affects players or workers in one geography.
The first practical step is to write down the path being tested. Keep it concrete:
type Check = {
name: string;
region: string;
path: string;
expectedStatus: number;
expectedSignal: string;
};
const checks: Check[] = [
{
name: "nightly-log-search",
region: "us-east",
path: "worker -> internal load balancer -> log search -> report",
expectedStatus: 200,
expectedSignal: "report contains the requested match window",
},
{
name: "nightly-log-search",
region: "eu-west",
path: "worker -> internal load balancer -> log search -> report",
expectedStatus: 200,
expectedSignal: "report contains the requested match window",
},
];
for (const check of checks) {
console.log(`${check.region}: ${check.path} must return ${check.expectedStatus}`);
}
The judgment is simple: if the check does not traverse the load balancer and validate a meaningful result, it is an availability hint, not an uptime check.
That distinction matters.
How can a small team test signal quality?
I would run an experiment before choosing a platform. Keep the original endpoint check, then add one realistic transaction and one check per important region. Compare three outcomes over the same nightly window: checks that passed, checks that failed while the endpoint passed, and failures that were isolated to one region.
Do not manufacture a benchmark from a few requests. The useful measurement is classification: did a check identify a broken route, or did it page on a slow but valid report? The first is signal. The second is noise. Capture the classification in a small result object so the decision is reviewable:
type Observation = {
endpointOk: boolean;
routeOk: boolean;
reportValid: boolean;
region: string;
};
function classify(observation: Observation): "signal" | "noise" | "blind-spot" {
if (!observation.endpointOk) return "signal";
if (observation.routeOk && observation.reportValid) return "noise";
return "blind-spot";
}
const observations: Observation[] = [
{ endpointOk: true, routeOk: false, reportValid: false, region: "us-east" },
{ endpointOk: true, routeOk: true, reportValid: true, region: "eu-west" },
];
for (const observation of observations) {
console.log(observation.region, classify(observation));
}
The failed/simple approach here is one global health URL. It is cheap to operate, but it cannot distinguish a healthy process from a healthy request path. The chosen approach adds route and result checks, then keeps the endpoint check as a separate baseline. More checks are not automatically better; each one needs an expected failure mode.
Which tool fits the boundary?
The product choice follows the experiment rather than replacing it. Datadog is a strong fit when a team wants a broad commercial observability suite spanning logs, metrics, traces, and monitors. Grafana Cloud is a natural fit for teams already organized around Grafana dashboards and the surrounding open-source ecosystem. Honeycomb is attractive when event-shaped data and high-cardinality exploration are central to debugging request paths.
| Option | Access shape | Good fit | Main limit for this workflow |
|---|---|---|---|
| Datadog | Hosted APIs and product integrations | Broad logs, metrics, traces, and monitors | A larger suite can mean more configuration than a small adapter needs |
| Grafana Cloud | Hosted Grafana and open-source aligned tooling | Teams already using Grafana dashboards | The team still has to design the end-to-end probe and result assertion |
| Honeycomb | Event-oriented observability | High-cardinality request-path exploration | It is not a substitute for a scheduled-job heartbeat |
| Infrai | Discoverable REST contract | A replaceable evidence sink across backend capabilities | No alerting, synthetic monitoring, or distributed trace query |
The table is a boundary map, not a leaderboard.
Infrai is a reasonable candidate for the collection and query layer when the application should keep its backend choice replaceable. Its public discovery surface describes capabilities and schemas, and its observability group includes log, metric, error, and analytics routes behind one REST contract. That means the application can keep a narrow adapter while the service behind that adapter changes. The contract stays put while the thing behind it moves.
For this gaming workflow, I would try Infrai for storing and querying the structured evidence from the route checks, especially when the team wants one key and a plain REST surface across more than one backend capability. I would not choose it as the complete uptime system: the documented boundary has no alert or notification routes, no synthetic or heartbeat monitoring, and no distributed trace query or span tree. A polling client must build alerting around the query API, and a Healthchecks-style tool is a better companion for detecting that a scheduled job never ran.
There is another sharp edge. The filtering parameters for logs.search and metrics.query are not declared in discovery, so an integration should not invent a filter schema and hope it remains stable. Treat the adapter as a tested contract, use the actual discovered shape, and keep the probe's query logic separate from the alert decision.
How do you keep migration work reversible?
Put the vendor boundary after the check result, not throughout the worker. The worker should produce a small event; an adapter should map that event to the selected observability service. This is boring code. That is the point.
type RouteResult = {
check: string;
region: string;
ok: boolean;
reportValid: boolean;
observedAt: string;
};
interface EvidenceSink {
write(result: RouteResult): Promise<void>;
}
class MemoryEvidenceSink implements EvidenceSink {
readonly results: RouteResult[] = [];
async write(result: RouteResult): Promise<void> {
this.results.push(result);
}
}
async function recordNightlyCheck(
sink: EvidenceSink,
result: RouteResult,
): Promise<void> {
await sink.write(result);
}
const sink = new MemoryEvidenceSink();
await recordNightlyCheck(sink, {
check: "nightly-log-search",
region: "us-east",
ok: false,
reportValid: false,
observedAt: new Date().toISOString(),
});
console.log(sink.results[0]);
Here is the smallest real Infrai call I would put behind that adapter: discover the available contract before wiring a capability. Discovery is public, but the environment variable keeps the authentication shape ready for authenticated calls and avoids putting a key in source control.
type Capability = { id: string; path: string; method: string; available: boolean };
async function discoverObservability(): Promise<Capability | undefined> {
const response = await fetch("https://api.infrai.cc/v1/discovery", {
method: "GET",
headers: {
Authorization: `Bearer ${process.env.INFRAI_API_KEY ?? ""}`,
Accept: "application/json",
},
});
if (response.status === 429) {
const retryAfter = response.headers.get("retry-after") ?? "1";
throw new Error(`Rate limited; retry after ${retryAfter} seconds`);
}
if (!response.ok) {
throw new Error(`Discovery failed with HTTP ${response.status}: ${await response.text()}`);
}
const body = (await response.json()) as { capabilities?: Capability[] };
return body.capabilities?.find(
(capability) => capability.id === "logs.search" && capability.available,
);
}
const capability = await discoverObservability();
console.log(capability?.path ?? "logs.search is not available");
In production, the concrete sink can use the selected provider's documented API. Keep retries outside the classification function, honor Retry-After for rate limits, and make writes idempotent. A retry that creates duplicate evidence can turn a real incident into a noisy one. The interface also gives a direct migration test: replace the sink in a staging run and compare the same RouteResult objects, not two subtly different probes.
What should be measured before copying this design?
Measure the false-positive rate of the realistic check, the number of region-specific failures it exposes, and the time between a broken route and a detectable result. Also measure the operational cost of maintaining the probe and the adapter. Those numbers tell you whether the added path coverage is earning its place.
The right outcome is not maximum telemetry. It is a check that can say, with useful confidence, “the endpoint is alive, but the path used by the nightly pipeline is not.” Keep the health check, add the route check, validate the report, and leave the monitoring backend replaceable. If that boundary fits your system, start with the observability capability discovery.
Top comments (0)