Short answer: monitoring misses an outage behind an internal load balancer when it checks the path to the balancer, but not the delivery of the business result. A healthy listener, a successful TCP connection, or a 200 response from a shallow health endpoint can all coexist with a nightly property-data pipeline that never reaches its worker.
For cost attribution, that distinction matters. The useful question is not only "is the service up?" It is "which stage accepted the work, which stage delivered it, and where did the failure spend time and compute?" Treat the route as a chain of independently observed handoffs. Give each handoff a correlation ID, a bounded deadline, and a business completion signal. Then an outage becomes a missing delivery edge instead of an ambiguous green dashboard.
The field guide at a glance
| Option | Detects | Pick this when | Cost attribution signal |
|---|---|---|---|
| External synthetic check | Public or operator-visible availability | A customer must be able to start or finish a workflow | Record the workflow step and response class, not just HTTP status |
| Internal dependency probe | Reachability across the private route | The load balancer can be healthy while a target pool or route is not serving | Split route, target, and application timings |
| End-to-end completion check | Delivered business outcome | A nightly job can accept requests yet fail before producing its artifact | Attribute work by run ID, stage, and retry attempt |
| Logs plus metrics | Diagnosis and trend | Operators need to explain a failed check without guessing | Join counters, durations, bytes, and structured events |
There is no single best check. A synthetic request tells you what a caller experiences. A dependency probe tells you which private edge is unreachable. A completion check tells you whether the property export actually arrived. Use their disagreement as evidence.
For this scenario, the completion check is the deciding signal. The pipeline is not successful because an internal listener answered. It is successful when the expected run produces a validated artifact and records that fact.
Why can monitoring fail to detect service outages behind an internal load balancer?
A health endpoint usually answers a narrower question than the one the pipeline asks. It may prove that a process is alive, that a listener can accept a connection, or that a lightweight handler can return a response. It does not automatically prove that the selected target can fetch a property partition, write its output, or survive the same timeout budget as the real job.
Picture the path as words: scheduler to internal load balancer, load balancer to target, target to data source, target to artifact store, artifact store to completion marker. A shallow check samples the first arrow. The outage may sit on the third or fifth. The dashboard stays green because the sampled arrow is still intact.
The route can be green. The job can still be down.
The artifact is the proof.
The reverse failure is common too. A deep probe can create noise if it performs production work, competes with the nightly job, or retries faster than the dependency can recover. Monitoring is part of the system. Give it a small, explicit workload and a clear deadline.
Timeouts and retries need a budget, not just a number copied into every client. A caller with a 30-second deadline cannot sensibly give three downstream attempts 30 seconds each. Retries can amplify load while a dependency is already impaired. Exponential backoff with jitter reduces synchronized retry bursts, but it cannot turn an unreachable route into a successful delivery. The AWS Builders' Library describes this interaction in detail.
That is the first trap: availability is a property of a path and an outcome, not a property of one green component.
Which check should own the alert?
Use layered ownership. Let the shallow check page for a broad reachability failure. Let the private probe create a route-specific signal. Let the end-to-end check own the business incident when a run misses its completion deadline. The signals should share a run ID or probe ID so an operator can compare them without reconstructing the event from timestamps.
A practical alert rule has two parts: a failed condition and a useful explanation. "Five probes failed" is a condition. "The listener responded, target selection succeeded, and no completion marker arrived within the run deadline" is an explanation. The latter points the next investigation toward delivery and downstream dependencies.
Do not make one failed synthetic request an outage by itself. Network paths have transient failures. Consecutive failures, a ratio over a defined window, or a missed completion deadline is more meaningful. The exact threshold belongs to the pipeline's schedule and recovery objective; it should be written down and tested, not inferred from a default dashboard.
Keep the alert payload small but decisive:
- run ID and probe ID
- route or target-pool label
- start time, deadline, and attempt count
- status class and failure phase
- input partition and expected artifact
This is enough to separate "the request never reached a target" from "the target ran but did not publish".\n+
A cost-attribution event that survives routing failures
The event schema is the hinge. Put the cost dimensions on every stage event, including failures, and keep them stable across retries. A log line that says only "request failed" cannot answer which property portfolio, partition, or attempt consumed resources.
Here is a small TypeScript shape for the event emitted by each handoff. It is deliberately generic: the collector, storage engine, and dashboard can change without changing the accounting contract.
type PipelinePhase =
| "scheduled"
| "route"
| "target"
| "fetch"
| "publish"
| "complete"
| "failed";
type PipelineEvent = {
eventName: "property_pipeline";
runId: string;
probeId?: string;
phase: PipelinePhase;
portfolioId: string;
partition: string;
attempt: number;
startedAt: string;
durationMs: number;
outcome: "ok" | "error";
errorClass?: "timeout" | "route_unreachable" | "target_unavailable" | "publish_rejected";
bytesRead?: number;
recordsWritten?: number;
};
The runId identifies one scheduled execution. The attempt identifies work that may be repeated. The portfolio and partition fields support attribution without putting customer data into the event. The phase tells you where the path stopped.
A route failure should still emit an event. That event has no recordsWritten, but it does have a duration, attempt, partition, and error class. Otherwise the failed work disappears from the cost view and successful retries look more expensive than they really are.
For a query, group by runId, then order by startedAt. A missing target event after a route error is a different operational story from a complete fetch followed by a rejected publish. Both may produce the same top-level alert. They should not produce the same remediation.
How do you test the invisible edge without creating a second outage?
Start with a dry-run probe that exercises routing and authentication but does not write a production artifact. Give it a separate identity and a narrow target. Its contract should say what it proves and what it intentionally does not prove. A route probe can prove private reachability; it cannot prove that the full partition is processable.
Then test the completion path with a canary partition. The canary must be recognizable in logs and excluded from customer output, or written to an isolated destination. Run it on the same schedule and through the same private route as the real pipeline. This catches configuration drift that an external uptime check cannot see.
Failure injection belongs here, too, but keep the blast radius explicit. A test can make a target reject the probe, delay a response until the deadline, or refuse a canary publish. The expected result is a phase-specific event and an alert with the right run ID. Do not judge the test only by whether an alert fired; check whether the alert explains the missing handoff.
I initially treated the health endpoint as the cheapest useful monitor. The correction was simple: cheap to execute is not the same as cheap to interpret. A shallow check saved probe work but shifted investigation into log archaeology. A small canary and stable event schema made the failure more expensive to generate than to explain, which is the trade-off a nightly pipeline needs.
Use separate budgets for probe traffic and application retries. Keep retry counts visible. If a dependency returns a failure quickly, backoff should prevent a synchronized wave. If the dependency remains unavailable, the caller should stop before the pipeline's completion deadline and record the abandoned work.
Limits and decision rules
Layered monitoring still has blind spots. A probe can pass while only one production partition is broken. A canary can pass while a large export exhausts a downstream quota. A completion marker can arrive after partial output unless publication is atomic or the output is validated before the marker is written. Observability cannot repair an ambiguous success contract.
The decision rule is therefore plain: choose the deepest check that proves the outcome you are promising, then add shallower checks to localize failures. Keep route reachability, target availability, processing, and publication as separate phases. Attach the same run identity and cost dimensions to each one.
That design catches the outage a green internal listener hides, while preserving enough evidence to say what failed and what work was repeated. For a property-management pipeline, that is the difference between a vague uptime incident and an attributable delivery gap.
Sources
- AWS Builders' Library, “Timeouts, retries, and backoff with jitter”: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- OpenTelemetry Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
- OpenTelemetry Metrics Data Model: https://opentelemetry.io/docs/specs/otel/metrics/data-model/
- W3C Trace Context Recommendation: https://www.w3.org/TR/trace-context/
Top comments (0)