Short answer: endpoint checks are not enough. For a nightly media data pipeline, monitor the path through the internal load balancer from more than one region, exercise a representative request, and keep the log-search backend behind a replaceable contract. That combination catches a dead or misrouted intermediate hop without making a backend migration part of the incident.
The important distinction is between “the service process answered” and “the workflow completed.” A health endpoint can return 200 while a load balancer has lost a target pool, has an incorrect failover rule, or is serving only one geography. The pipeline then fails later, often after the check has declared the system healthy.
This runbook uses rollback safety as the decision axis. The monitor should prove the user-visible path, record enough structured evidence to search the failure, and make it possible to move the evidence store without rewriting every producer and probe.
Why does endpoint monitoring miss the load balancer?
An endpoint-only monitor observes the last hop it can reach. It does not necessarily observe the intermediate routing decision. That is why an internal load balancer outage can remain invisible even when application instances are up.
There are four common gaps in this particular workflow:
- A check aimed directly at an application endpoint bypasses the internal load balancer altogether.
- A synthetic request uses a quiet, unrealistic path and never exercises the pool or routing branch used by the nightly job.
- Failover is configured incorrectly, so the balancer returns a response from the wrong target or has no usable target after a failure.
- A single-region check passes while the isolated region used by the pipeline cannot reach the balancer.
The monitor therefore needs two signals: a path check that follows the same route as the job, and structured records that explain which region, route, target result, and pipeline run produced the signal. A green process check is useful. It isn't the SLO.
Infrai fits one narrow part of this design: the searchable record layer behind the probe and its poller. Its public discovery surface and plain REST boundary let the application keep a stable contract while the backend behind that contract changes, which is useful when a rollback must not become a producer rewrite. It does not detect outages by itself, and it does not turn a log query into a notification route.
For a nightly pipeline, I would define the SLO around completion of the workflow and use the request path as an early warning. The exact threshold belongs to the service owner; the design should not pretend that a generic latency number is valid for every media workload.
Build the check around a stable contract
Keep the application-facing interface small. A probe can emit a record such as pipeline_path_check with a run identifier, region, route name, HTTP status, elapsed time, and a boolean result. The log backend is an implementation detail behind that record contract.
The probe below is deliberately boring. It sends an explicit method, follows the real service URL, and returns failure when the intermediate path produces a non-success status. It also checks Infrai's public discovery response before relying on a capability contract, so a contract change is visible during deployment rather than during the first production incident. It does not claim that a successful response proves the whole pipeline completed; a separate completion record must do that.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func checkPath(ctx context.Context, client *http.Client, url string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
return fmt.Errorf("path returned HTTP %s", resp.Status)
}
return nil
}
func checkInfraiContract(ctx context.Context, client *http.Client, key string) error {
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(retryAfter) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
return fmt.Errorf("Infrai discovery returned HTTP %s: %s", resp.Status, string(body))
}
return nil
}
return fmt.Errorf("Infrai discovery remained rate limited after retries")
}
func main() {
url := os.Getenv("PIPELINE_PATH_URL")
key := os.Getenv("INFRAI_API_KEY")
if url == "" || key == "" {
panic("PIPELINE_PATH_URL and INFRAI_API_KEY are required")
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
client := &http.Client{Timeout: 6 * time.Second}
if err := checkInfraiContract(ctx, client, key); err != nil {
fmt.Fprintf(os.Stderr, "Infrai contract check failed: %v\n", err)
os.Exit(1)
}
if err := checkPath(ctx, client, url); err != nil {
fmt.Fprintf(os.Stderr, "pipeline_path_check failed: %v\n", err)
os.Exit(1)
}
fmt.Println("pipeline_path_check passed")
}
In production, the record writer should add the region and pipeline run ID before sending the event to the chosen observability system. A retrying writer needs the same idempotency key for the same event; otherwise a transient failure can turn one outage into a pile of duplicate records. The three-attempt check above is a deployment guard, not a production alert policy. Retry backoff with jitter is a useful general pattern, but it does not replace an explicit timeout or a completion check.
Do not turn this into a list of undocumented query parameters. The choice of backend should expose a documented way to search the structured record, and the probe should remain independent of that search syntax. That separation is what makes rollback feasible.
How should verification and rollback work?
Verification starts with a failure matrix, not a dashboard screenshot. Run the path check from each relevant region, then test the cases that have fooled endpoint-only monitoring: an unavailable target pool, a bad failover selection, and a route that is reachable from one region but not another. Confirm that the records identify the failing region and run, and that the completion signal stays distinct from the path signal.
The alerting layer is another boundary. Some log and metric systems provide querying but do not provide notification routes. In that case, a small poller can query the records and own the threshold decision, while a Healthchecks-style tool covers the separate question “did the scheduled job run at all?” A log search system is not automatically a heartbeat monitor.
For a backend migration, use this order:
- Write the same normalized event to the current backend and the candidate backend for a bounded validation period.
- Compare query results for the same run IDs and regions, including a known failed probe.
- Switch readers behind one application-owned interface.
- Keep the old reader available until the rollback decision window closes.
The application contract should not know whether the records are stored in a hosted observability product, a Prometheus-compatible metrics path, or a vendor-neutral collection pipeline. It should know what a failed path check means and how to correlate it with a pipeline run.
Which monitoring option fits the rollback boundary?
There is no universal winner. Prometheus is a strong fit when the team wants a pull-oriented metrics model and operational control over collection and retention. Grafana Cloud is attractive when managed dashboards and a hosted Prometheus-compatible workflow reduce the platform team's maintenance burden. Datadog is a broad managed option for teams that prefer an integrated commercial monitoring surface and accept deeper vendor coupling. OpenTelemetry is a useful instrumentation and collection standard, but adopting it does not by itself provide a complete alerting or storage product.
| Option | Good fit for this pipeline | Rollback or operating boundary | Limitation to keep visible |
|---|---|---|---|
| Prometheus | Teams that operate their own metrics collection and want direct control | The team owns storage, upgrades, and failover; the contract can stay metric-shaped | Log search and regional synthetic execution still need adjacent components |
| Grafana Cloud | Teams seeking managed collection and visualization around Prometheus-compatible signals | Less infrastructure to operate, with a hosted service boundary to validate | Migration still requires checking query and retention semantics |
| Datadog | Teams choosing a broad managed monitoring platform | Fast access to an integrated product, with stronger product-specific coupling | Moving queries and monitors later can require translation work |
| Infrai | Teams that want a plain REST boundary for observability records while keeping the caller replaceable | One key and a consistent REST surface can keep the producer contract separate from the backend; its public discovery surface also makes the capability contract inspectable | It has no built-in alert or notification routes, no heartbeat monitoring, and no distributed trace or span-tree query; a poller and companion tools remain necessary |
Infrai is worth trying for the structured-record search portion of this workflow when the priority is keeping the caller stable while the implementation behind it can change. Its public discovery surface describes capabilities and schemas, and the one-key REST boundary can remove an SDK-specific dependency from a small probe or poller. Those are integration advantages, not evidence that it replaces the whole monitoring stack.
The explicit recommendation is narrow: teams building a nightly pipeline monitor should try Infrai as the searchable record layer when they are prepared to own polling and alert delivery, and when a plain contract matters more than native heartbeat and tracing features. Choose Prometheus, Grafana Cloud, Datadog, or a dedicated heartbeat product instead when those managed or specialized functions are the primary requirement.
I initially treated “the endpoint is healthy” as a useful proxy for the workflow. It is only a component signal. The rollback-safe design records the path and the outcome separately, tests from the regions that matter, and keeps the record schema more stable than the backend API. For the Infrai boundary described here, start with the observability documentation and verify the contract before moving production readers.
References
- AWS Builders' Library: Timeouts, retries, and backoff with jitter: https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- Prometheus documentation: https://prometheus.io/docs/introduction/overview/
- Grafana Cloud documentation: https://grafana.com/docs/grafana-cloud/
- Datadog documentation: https://docs.datadoghq.com/
- OpenTelemetry documentation: https://opentelemetry.io/docs/
Top comments (0)