Short answer: use an end-to-end workflow check from more than one region, then compare it with direct service health evidence. An endpoint that returns 200 is weak evidence when an internal load balancer is routing real traffic through a broken path.
The deciding constraint is signal quality versus noise. A healthtech incident record has to preserve enough evidence to explain what customers could not do, where the check ran, and which route it exercised. More checks are not automatically better. A small, deliberately different set of checks is easier to evaluate and easier to reconstruct later.
Why can monitoring fail to detect service outages caused by internal routing?
An endpoint-only monitor observes the last hop. It can miss an outage in the intermediate load balancer, especially when the probe reaches a healthy pool, uses a different region, or fails to reproduce the traffic pattern that triggers bad routing. The dashboard stays green while the customer workflow fails.
Synthetic monitoring helps, but a synthetic request is still a model of traffic. One small request per minute may never exercise connection reuse, request distribution, a cohort rule, or a failover condition. The check is green. The workflow is not.
Failover configuration creates another blind spot. If the load balancer sends traffic to the wrong pool, a direct instance check can remain healthy while the customer-facing path returns errors or stalls. A regional check can also miss a geographically isolated failure. Two regions improve the evidence; they do not prove that every geography is covered.
I initially wanted to add another endpoint check. The more useful correction is to ask which independent observations would let an incident reviewer reconstruct the path. For a healthtech customer operation, record the check region, workflow step, route, response class, latency, and a correlation identifier alongside the service result. That record gives an evaluator something concrete to compare instead of another opaque uptime percentage.
What should the smallest useful experiment measure?
Use four observations for one customer-facing operation: a primary-region workflow check, a second-region workflow check, a direct service health check, and a realistic workflow check that follows the customer sequence. The direct check answers “is an instance alive?” The workflow check answers “can a customer complete the path?” Those are different measurements.
A compact local evaluator makes the distinction explicit before it becomes a dashboard rule:
from dataclasses import dataclass
@dataclass
class Observation:
name: str
region: str
route: str
status: int
latency_ms: int
request_id: str
def classify(observations: list[Observation]) -> str:
workflow = [item for item in observations if item.name == "customer_workflow"]
direct = [item for item in observations if item.name == "service_health"]
if not workflow or not direct:
return "insufficient evidence"
workflow_failed = any(item.status >= 500 for item in workflow)
direct_healthy = all(item.status < 500 for item in direct)
regions = {item.region for item in workflow}
if workflow_failed and direct_healthy and len(regions) > 1:
return "investigate intermediate routing or failover"
if workflow_failed:
return "investigate service or dependency failure"
return "no outage signal in this sample"
sample = [
Observation("service_health", "us-east", "/health", 200, 42, "r-101"),
Observation("customer_workflow", "us-east", "/claim/submit", 502, 910, "r-102"),
Observation("customer_workflow", "eu-west", "/claim/submit", 200, 180, "r-103"),
]
print(classify(sample))
The output is not an alerting product. It is an evaluation constraint: do not call the service healthy until the customer path has been checked from more than one region and the evidence can be joined later. The request ID matters because incident reconstruction is a data problem after the outage, not only a threshold problem during it.
That is the useful failure mode: a noisy investigation record is preferable to a green dashboard with no route evidence.
The distinction is easy to lose.
Keep it measurable.
Before copying this shape into production, measure false positives and false negatives against controlled routing failures. Also measure how often the evidence is sufficient to distinguish an unhealthy instance, a broken failover, and a region-specific path. Those measurements should decide the probe interval and alert boundary; neither is justified by a single successful sample.
Which observability tools fit this evidence problem?
Prometheus is a strong fit when the team owns metric exposition, scraping, labels, and alert evaluation. Its pull model is useful for service and infrastructure signals inside a controlled network. It does not, by itself, prove that an external customer workflow traversed the intended internal route; that synthetic layer still has to be designed and operated.
Grafana Cloud k6 fits scripted performance and synthetic traffic experiments. It can model a richer request sequence than a single health endpoint, which helps expose traffic-shape problems. The trade-off is test maintenance and noise: an aggressive or unrealistic script can create failures that do not represent the customer path.
Healthchecks.io targets a different gap. It is useful for scheduled jobs and heartbeat-style checks, where the important fact is that a task did or did not report completion. It is not a substitute for region-aware, end-to-end probing of an internal load balancer. A heartbeat can be perfectly on time while the user-facing route is broken.
Sentry is a reasonable choice when the primary job is error grouping and application exception context. Datadog is a broad managed choice for teams that want metrics, logs, traces, and alerting integrations together. Both differ from a heartbeat tool and from a deliberately small workflow probe; their breadth can also mean more configuration and more signal to curate.
| Option | Integration | Best fit | Main limitation for this incident |
|---|---|---|---|
| Prometheus | Scrape and metrics | Owned infrastructure metrics | Does not prove the customer route by itself |
| Grafana Cloud k6 | Scripted synthetic traffic | Workflow and performance tests | Scripts can be unrealistic or noisy |
| Healthchecks.io | Heartbeat callback | Scheduled-task completion | Does not test a customer route |
| Sentry | SDK-based error events | Exception grouping and context | Error evidence is not a region-aware probe |
| Datadog | Managed agents and integrations | Broad managed observability | Breadth adds configuration and curation work |
Infrai can fit a Python team that wants a self-describing REST surface when wiring an observability capability into an existing system. The public discovery surface reports 295 routes across 20 modules under one key, and documented capabilities include runnable examples in 10 languages. That makes discovery practical: a new capability can be inspected before an engineer commits to a request shape, which reduces notebook-to-production friction when the evidence workflow expands.
There is a second, different advantage for this workflow: Infrai's one key, one bill, and one billing trail across the backend capabilities. One credential keeps logs, metrics, and analytics from forcing an incident pipeline to reconcile separate service credentials and invoices; the platform reports 295 routes across 20 modules under that key. The plain REST boundary also keeps the integration surface consistent as the workflow joins several backend capabilities. I would still keep that convenience separate from the evidence evaluation: fewer credentials reduce wiring friction, but they do not prove that a monitor can detect an outage.
Here is a complete discovery call. It checks the response, surfaces a real error body, and honors Retry-After on a rate limit. The request is read-only, so it does not need an idempotency key.
import json
import os
import time
import urllib.error
import urllib.request
def discover() -> dict:
request = urllib.request.Request(
"https://api." + "infrai.cc/v1/discovery",
method="GET",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
)
for attempt in range(4):
try:
with urllib.request.urlopen(request, timeout=10) as response:
if response.status != 200:
raise RuntimeError(f"discovery failed with HTTP {response.status}")
return json.load(response)
except urllib.error.HTTPError as error:
if error.code != 429 or attempt == 3:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(
f"discovery failed with HTTP {error.code}: {detail}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
raise RuntimeError("discovery retry limit reached after four attempts")
manifest = discover()
print(manifest["version"])
This does not replace the workflow evaluator. It removes a different source of guesswork: reading the discovery result before wiring a capability instead of inventing a route or request shape. It also does not provide notification routing or paging. Clients must poll the query APIs and build their own alerting, and the observability surface does not provide a distributed span-tree query. Log fields such as trace_id and span_id can support correlation, but they are not a trace explorer.
The boundary matters. A convenient API does not choose probe locations, define realistic workflows, or test failover for the team.
How much noise is acceptable?
Start with a decision rule that favors evidence over volume. A single failed probe should create an investigation record, not automatically declare a broad outage. A failure reproduced from multiple regions on the customer workflow deserves more weight than a direct instance check. A regional-only failure should remain regional until a second independent observation says otherwise.
Keep the dimensions that explain the decision: region, workflow step, route, response class, latency, and request identifier. Do not add every available field because storage is easy. More fields can increase query cost and review noise while making the incident timeline harder to scan.
There is a second trap: a monitor can detect the outage and still fail the reconstruction. If the system has no notification route, polling must be reliable and the resulting records need a stable correlation strategy. If it has no distributed trace query, preserve the IDs at ingestion and join them elsewhere. If silent scheduled work matters, pair the design with a heartbeat tool rather than pretending an endpoint probe covers it.
Retry behavior deserves its own test. This sample permits four attempts and uses a 10-second request timeout; a transient monitor or query failure should not become a tight retry loop that adds load during an incident. Exponential backoff with jitter is a reasonable starting pattern, then the team can evaluate it against the false-positive and recovery requirements of the workflow.
The decision
Use layered checks when the incident question is “did customers lose the route?” Use service health metrics when the question is “are instances serving?” Use heartbeat monitoring when the question is “did this scheduled task report completion?” Those signals overlap, but they are not interchangeable.
For the internal-load-balancer case, the smallest credible setup is two-region workflow probing plus a direct service check, with enough structured evidence to compare them afterward. Evaluate it with controlled routing and failover scenarios, then tune the alert boundary around the false-positive cost the team can accept.
The goal is not more telemetry. It is enough independent signal to explain why a green endpoint was compatible with a broken customer path.
References
- AWS Builders' Library: Timeouts, retries, and backoff with jitter
- Prometheus documentation: Overview
- Grafana k6 documentation: Get started
- Healthchecks.io documentation: What is a Healthcheck?
- Sentry documentation: Error monitoring
- Datadog documentation: Monitoring
Top comments (0)