How to Reconstruct Checkout Outages with Go and Internal Load Balancer Signals

#observability #sre #go #monitoring

A checkout outage can exist while the public health check stays green. That happens when the probe reaches a healthy edge or a different backend path, while the internal load balancer is failing to deliver requests to the checkout service. The useful answer is a reconstruction path: pair an external transaction check with backend delivery signals, then correlate them by request ID, zone, and time window.

Short answer: monitor the customer-visible transaction and the internal handoff separately. A TCP connection, a 200 from a shallow health endpoint, and normal process metrics do not prove that a checkout request crossed the internal load balancer and completed. Alert on the gap between those observations, and retain enough context to explain the first failed payment attempt without guessing.

How can monitoring fail to detect service outages caused by routing issues?

The page usually arrives late and looks misleading. Customer support reports that checkout spins or returns an error. The public uptime probe still gets a 200. CPU is ordinary. The checkout pods report no crash loop.

The missing observation is the delivery boundary. An internal load balancer can accept a connection while its target selection, health state, route, or zone path prevents the request from reaching the application. From the probe's point of view, the front door answered. From the customer's point of view, the checkout did not finish. Those are different facts. This is why monitoring can fail to detect outages caused by internal routing issues even when every individual check looks plausible.

For a customer-support workflow, the first useful artifact is not a red dashboard. It is a timeline: checkout attempt received, internal handoff attempted, backend handler entered, payment authorization called, response returned. A request ID carried through those events lets the responder distinguish “the handler was never reached” from “the handler failed after reaching the payment dependency.”

I start with the page and work backward. If support says failures began at 14:07 UTC, I compare the external transaction result with internal handoff failures in a narrow window around 14:07, split by zone and target group. A broad five-minute average can hide a short routing failure.

That comparison is the detector. The dashboard is only the evidence shelf.

Why is a green health check weak evidence?

A health endpoint answers a narrow question: can this path produce an HTTP response now? It may bypass authentication, session state, checkout dependencies, and the internal route used by the real transaction. A load balancer can therefore be healthy enough to answer /healthz while the checkout path is unavailable.

The remedy is layered evidence, not a more complicated single endpoint. Keep a cheap liveness check for process failure, add readiness checks for dependencies that must be available, and run a synthetic checkout with a test cart that cannot create a real charge. The synthetic must traverse the same routing boundary as a customer request. It should record a transaction result, not just latency.

The distinction matters for alert policy. A liveness failure is a strong signal for a dead process. A synthetic failure with normal liveness is a routing or dependency investigation. Treating both as “service down” loses the clue that makes incident reconstruction fast.

Instrument the handoff before tuning the threshold

The instrumentation change is small: emit one event when the checkout request enters the service and one when the internal handoff fails or times out. Include a stable request ID, route name, zone, target identity when available, outcome, and elapsed milliseconds. Do not put payment credentials, tokens, or full customer payloads in the event.

Here is a minimal Go shape for an event emitter. It uses an interface so the same fields can go to a log, an OpenTelemetry span, or a metrics adapter without tying the runbook to one vendor.

package checkout

import (
    "context"
    "time"
)

type HandoffEvent struct {
    RequestID string
    Route     string
    Zone      string
    Outcome   string
    Elapsed   time.Duration
}

type EventSink interface {
    Record(context.Context, HandoffEvent)
}

func recordHandoff(ctx context.Context, sink EventSink, requestID, route, zone, outcome string, started time.Time) {
    sink.Record(ctx, HandoffEvent{
        RequestID: requestID,
        Route:     route,
        Zone:      zone,
        Outcome:   outcome,
        Elapsed:   time.Since(started),
    })
}

The decision rule is more important than the field list. If the synthetic transaction fails and backend entry events fall at the same time, investigate routing, target health, or zone reachability. If backend entry continues but completion falls, investigate the handler or its downstream dependency. If only one zone diverges, remove that zone from service according to the existing traffic procedure and preserve the evidence before changing more variables. A 30-second alert window may catch a brief outage sooner, but it also gives an independent probe less time to prove that the failure is real; choose that boundary from customer impact and probe reliability, then verify it against failure records.

How should retries affect the incident picture?

Retries can turn one delivery problem into several misleading signals. A client timeout may cause a second attempt, while the first request is still running. The customer sees one failed checkout, but the service sees duplicate work. A retry policy also changes the rate at which a damaged internal path is exercised.

Use bounded timeouts and exponential backoff with jitter, and make the checkout operation idempotent with a request or order key. Record the attempt number and the idempotency key alongside the handoff event. The retry policy should be visible in the trace; otherwise a responder may mistake retries for independent customers or conclude that the load balancer created duplicates.

The relevant trade-off is detection speed versus false positives. A threshold that pages on one failed synthetic attempt catches a short outage but can page on a probe network blip. A threshold that waits for ten consecutive failures gives cleaner pages and delays recognition. Pick the threshold from the customer impact and probe independence, then test it against recorded failure windows. Do not tune it from a quiet dashboard.

AWS describes timeouts, retries, backoff, and jitter as interacting controls rather than isolated settings. That is the right operational frame here: the alert must show both the original attempt and the retry behavior.

The following example shows the judgment a runbook needs. It does not decide whether to page from a single metric; it compares transaction failures with missing backend entries in the same window.

package main

import "fmt"

type Attempt struct {
    RequestID string
    Outcome   string
    BackendIn bool
    Zone      string
}

func classify(a Attempt) string {
    if a.Outcome == "failed" && !a.BackendIn {
        return "delivery boundary: inspect internal load balancer path in " + a.Zone
    }
    if a.Outcome == "failed" && a.BackendIn {
        return "application or downstream failure: inspect checkout trace " + a.RequestID
    }
    return "no checkout outage classification"
}

func main() {
    a := Attempt{RequestID: "req-4821", Outcome: "failed", BackendIn: false, Zone: "zone-a"}
    fmt.Println(classify(a))
}

This is deliberately boring. A useful incident tool should make the next query obvious, preserve the original request ID, and avoid pretending that correlation proves causation. The responder still checks target health, route configuration, recent deployments, and zone-specific reachability.

What should the runbook preserve after recovery?

When traffic recovers, keep the failure window, probe results, handoff counts, target-health transitions, and deployment timeline. Capture the false-positive cost too: how many pages came from an isolated probe, and how many customer transactions were affected before the page? That number guides the next threshold review better than an abstract availability target.

The final test is a replayable explanation. Someone who was not on call should be able to follow one failed checkout from the external transaction to the internal boundary, see where the evidence stops, and understand why the alert fired. If the record only says “uptime green” and “pods healthy,” monitoring did not observe the customer journey. It observed two healthy fragments.

The goal is not to make every layer page independently. It is to make the layers explain one another.

DEV Community