upordown.online

Posted on Nov 8 Edited on Nov 8

Quick tip: your load balancer hides a dead server. Check each backend with curl --resolve

#devops #monitoring #curl #sre

Here's a failure mode that drives people up the wall: users report that your site "sometimes" shows an error. You check. It works. You check again. Works. A colleague checks: error. You refresh: works.

Very often the explanation is boring. There are several servers behind a load balancer, and one of them is broken. Every request is a roll of the dice, and your uptime monitor, going through the load balancer like everyone else, rolls the same dice. It might catch the bad server occasionally, or it might not, depending on how the balancer spreads traffic and whether health checks pull the server out.

The quick tip: test every backend directly, using the real hostname, with curl --resolve.

Why not just curl the server's IP?

You could try curl https://10.0.1.12/, but it usually gives you the wrong answer:

TLS breaks. The certificate is for www.example.com, not for an IP address, so curl complains (and -k hides real certificate problems).
Virtual hosts break. The server picks which site to serve based on the hostname. Request it by IP and you may get a default page, not your app.
SNI is missing. Without the hostname in the TLS handshake, the server may present a different certificate entirely.

You want a request that's identical to a real user's, except it goes to a server you choose.

The flag

--resolve tells curl: "for this host and port, use this IP instead of asking DNS".

curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' \
  --resolve www.example.com:443:10.0.1.12 \
  https://www.example.com/health

Curl connects to 10.0.1.12 but sends the hostname in SNI and the Host header, validates the certificate for www.example.com, and behaves exactly as a browser would, if DNS had pointed at that server.

Loop over every backend

#!/usr/bin/env bash
set -euo pipefail
HOST=www.example.com
PATH_TO_CHECK=/health
BACKENDS=(10.0.1.11 10.0.1.12 10.0.1.13)

for ip in "${BACKENDS[@]}"; do
  out=$(curl -sS -o /dev/null --max-time 10 \
        -w '%{http_code} %{time_total}' \
        --resolve "$HOST:443:$ip" "https://$HOST$PATH_TO_CHECK" 2>&1) || out="ERR $out"
  printf '%-12s %s\n' "$ip" "$out"
done

Output looks like:

10.0.1.11    200 0.084
10.0.1.12    502 0.031
10.0.1.13    200 0.091

That second line is the "sometimes broken" bug, found in one second.

Where do the backend IPs come from?

Depends on your setup:

Cloud load balancers: the target group or backend pool lists instance IPs; most cloud CLIs can print them, so the script can fetch the list instead of hard-coding it.
Kubernetes: kubectl get endpoints <service> -o wide lists pod IPs. For pods you can't reach from outside the cluster, run the loop from a debug pod inside it.
DNS round robin: dig +short www.example.com returns them all. Run it a few times; some DNS servers rotate or limit the answers.
CDN in front: check the origins, not the CDN edges. Use the origin IPs with --resolve, and be aware the origin might only accept traffic from the CDN, in which case run this from an allowed network.

Variations worth knowing

Check a specific port or protocol. --resolve takes host:port:address. For a service on 8443, use www.example.com:8443:10.0.1.12.

Compare responses, not just status codes. A backend serving an old release returns 200 too. Fetch a version endpoint from each:

curl -s --resolve "$HOST:443:$ip" "https://$HOST/version"

If one server reports a different build, you've found a failed deploy that health checks happily accept.

IPv6 backends. Wrap the address in brackets: --resolve 'www.example.com:443:[2001:db8::12]'.

Test a new server before switching DNS. The same trick lets you check a migration target with the real hostname and certificate before a single user is sent there.

Turning it into an alert

Once the loop exists, turning it into a monitor is small. Run it from cron or a scheduled CI job every minute or two, and fail if any backend returns something other than 200, takes longer than a threshold, or reports a different version from the others. Send the failing line, with the IP, straight into the alert message, so whoever gets paged knows which server to look at without repeating the investigation.

Be careful with alerting on a single failed request from one backend; a brief blip during a deploy is normal. Requiring two consecutive failures from the same server is usually enough to filter out noise while still catching a server that is genuinely broken.

Why this belongs in monitoring, not just debugging

Running the loop by hand during an incident is useful. Running it every minute is better. Checking through the load balancer tells you whether users are generally fine. Checking every backend tells you whether every server is fine, which is what you need to know before the balancer runs out of healthy ones.

A good setup has both: an external check through the public hostname, the way users see it, and a per-backend check from inside your network. The first is what we focus on at upordown.online. The second is a ten-line script you can write today.

One caution

Health-check endpoints sometimes lie: they return 200 as long as the process is alive, even if the database connection is gone. When you check each backend directly, hit a path that exercises real dependencies, or a lightweight page that renders actual data, so a "200" means the server can really do its job.

The takeaway

When something is "intermittently broken", stop refreshing and start asking each server individually. curl --resolve is the single most useful flag we know for that question, and once you have the loop, you'll use it in every incident.

DEV Community