DEV Community: Ionut-Robert Sandu

Your Health Check Is Green and Your Service Is Down

Ionut-Robert Sandu — Fri, 25 Sep 2026 17:11:37 +0000

Monitoring has two ways to be wrong, and we treat them as if they were
symmetrical. They are not, and the asymmetry should drive how you build checks.

A false alarm costs you an interruption. Someone looks, finds nothing, mutters,
goes back to what they were doing. Annoying, self-correcting, and — this is the
important part — it tells you the monitoring is alive.

A false all-clear costs you the entire outage. Nobody looks, because nothing
asked them to. The first signal is a customer, and by then you have lost both the
time and the claim that you knew before they did.

We spend most of our tuning effort on the first failure mode. Almost all of the
damage comes from the second.

Three ways a check reports green on a broken service

1. The check is narrower than the thing it claims to cover

GET /health returns 200. The endpoint returns a literal {"status":"ok"} from a
handler that touches nothing. It confirms that the process is running and the HTTP
listener is bound. It says nothing about the database connection pool, the expired
credential to the payment provider, or the disk that filled up twenty minutes ago.

This is not laziness; it is drift. The check was written when the service had two
dependencies and was honest about both. Four years and eleven dependencies later,
the check still passes because it still does exactly what it did in 2022.

The useful question is not "is the check passing" but "what would have to break
for this check to fail?" If the answer is "the process would have to be dead",
you have a liveness probe wearing a health check's name tag. Both are legitimate,
but only one of them is allowed on the dashboard the on-call engineer looks at.

2. The check runs somewhere that cannot fail the way users do

Checks that run inside the cluster share fate with the thing they measure. They use
internal DNS, skip the load balancer, bypass the CDN, sit inside the network policy,
and often talk to the pod directly rather than through the service. Every one of
those is a component that can break for users while leaving the check perfectly
happy.

Certificate monitoring makes this concrete. Probe public HTTPS endpoints from inside
an environment whose egress
terminates and re-signs TLS
and every certificate that comes back is issued by the gateway, not by the site you
think you are watching. Build a certificate monitor there, point it at production,
and it reports healthy forever — because it is measuring the appliance one hop away
rather than the service on the other side of the internet. The check is fine. The
vantage point makes it meaningless.

The rule that falls out: at least one check has to traverse the same path a user
does, including the parts you do not own. Internal checks tell you which component
broke. Only an external check tells you whether anyone is actually affected.

3. The check stopped running

This is the one that produces the longest outages, and it is structurally
invisible. A cron job that no longer fires, a worker that died at 03:00, a pipeline
whose credential silently expired — none of these emit anything. And the absence of
an alert is indistinguishable from the absence of a problem.

Think about what your dashboard actually renders when a checker is dead. Most of
them show the last known state, which was green, with a timestamp nobody reads. The
system is not lying to you. You just asked it a question it has no way to answer.

The heartbeat inverts the logic

The fix for the third case is to stop asking "did something report a failure" and
start asking "did something report at all".

A dead man's switch is an external service expecting a ping on a schedule. Your
checker pings it after every successful run. If the ping stops, the external
service alerts. The alert now fires on silence, which means the failure of your
monitoring is itself a monitored event.

#!/bin/sh
set -e

run_all_checks            # exits non-zero if anything is wrong

# only reached when the checks ran AND passed
curl -fsS -m 10 --retry 3 "https://hc-ping.example/$CHECK_UUID" > /dev/null

Three things about that snippet are load-bearing.

set -e and the ordering. The ping is last. If the checks fail, or crash, or
the box runs out of memory halfway through, the ping never happens, and you get an
alert from the outside. Ping first and you have built a system that reports success
before knowing whether there is any.

The pinged service must be external. A heartbeat monitored by the same
infrastructure that runs the checker shares fate with it. Hosted dead man's switch
services exist and are cheap; the point is the independence, not the vendor.

The expected period should be roughly twice the run interval. One missed run on
a five-minute job is usually a blip. Two is a pattern. Alert on the pattern, or you
will train yourself to ignore the alert — which returns you to a false all-clear by
a slower route.

Designing checks that fail loudly

A few habits that follow from all of this.

Make "unknown" a distinct state from "healthy". If your checker cannot reach a
target, that is not a pass and it is not necessarily a fail — it is stale data, and
it should render differently. Most dashboards have two colours where they need
three. A check that has not reported in an hour should look visibly wrong even if
its last result was green.

Test the failure path, not just the success path. A check nobody has ever seen
fail is a check nobody has verified. Break the dependency deliberately in a staging
environment and confirm the alert actually arrives, at the actual destination, in
the actual channel. Alert routing rots quietly: people leave, channels get archived,
webhooks expire. The routing is part of the system and needs the same treatment as
the code.

Write down what each check does not cover. One line next to the definition.
"Does not verify database connectivity." It costs nothing to write and it is the
sentence that saves you during the post-mortem, because it converts an unknown
unknown into a known gap that somebody can choose to close or accept.

Prefer a check that occasionally cries wolf over one that never speaks. Given
the asymmetry at the top of this post, a slightly noisy check is a trade you should
be willing to make on purpose rather than one you drift into.

The uncomfortable summary

Green means one of two things: the system is healthy, or the system is not being
measured. Most monitoring setups cannot distinguish between the two, and the
dashboard renders both identically.

The work is not making the checks more sensitive. It is making sure that when the
measurement itself fails, something notices — because that is the failure mode that
costs you the whole outage, and it is the only one that gets quieter the worse it
gets.

Your Certificate Monitoring Only Checks the Leaf. That Is Not the Same Thing as Your Chain Being Valid.

Ionut-Robert Sandu — Fri, 18 Sep 2026 08:10:41 +0000

Here is a monitoring result. It is from a server built in a lab a few minutes earlier,
and every number in it is real.

$ echo | openssl s_client -connect app.lab.example:443 -servername app.lab.example \
  2>/dev/null | openssl x509 -noout -enddate -checkend 2592000

notAfter=Dec 11 14:12:02 2026 GMT
Certificate will not expire

Ninety days of runway, comfortably past the thirty-day threshold. Green.

The site stops working in twenty days.

What the check missed

openssl s_client hands back the leaf certificate by default. That is the
certificate for the hostname, the one with your domain in it, the one everybody
means when they say "the certificate". It is also only the first entry in a list.
A browser does not trust a leaf because the leaf says a date. It trusts it
because it can build a path from that leaf to a root it already trusts, and
every certificate on that path has to be valid at the same moment.

Here is the same server, with every certificate it sends checked instead of only
the first one:

CN = app.lab.example               Dec 11 14:12:02 2026 GMT   ok
CN = Lab Intermediate CA (short)   Oct  2 14:12:02 2026 GMT   EXPIRES WITHIN 30d

The leaf outlives the intermediate that signed it. On October 2nd, the chain
stops validating, and the leaf's own December expiry becomes irrelevant. Clients
will report an expired certificate. Your dashboard will report ninety days
remaining, because the thing your dashboard is looking at does, in fact, have
ninety days remaining.

Nothing here is exotic. A CA can perfectly well issue you a certificate that
outlives its own issuing intermediate, and the certificate is not malformed when
it does. The constraint lives in path validation at the client, not in the
issuance.

Build it yourself in two minutes

Do not take my word for the shape of this. The whole thing is four openssl
invocations.

# Root CA, long lived
openssl req -x509 -newkey rsa:2048 -nodes -keyout root.key -out root.crt \
  -days 3650 -subj "/CN=Lab Root CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

# Intermediate CA that expires in 20 days
openssl req -newkey rsa:2048 -nodes -keyout inter.key -out inter.csr \
  -subj "/CN=Lab Intermediate CA (short)"
printf 'basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign\n' > inter.ext
openssl x509 -req -in inter.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -out inter.crt -days 20 -extfile inter.ext

# Leaf valid for 90 days, signed by that intermediate
openssl req -newkey rsa:2048 -nodes -keyout leaf.key -out leaf.csr \
  -subj "/CN=app.lab.example"
printf 'basicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectAltName=DNS:app.lab.example\n' > leaf.ext
openssl x509 -req -in leaf.csr -CA inter.crt -CAkey inter.key -CAcreateserial \
  -out leaf.crt -days 90 -extfile leaf.ext

cat leaf.crt inter.crt > fullchain.crt

Serve fullchain.crt from anything, point your existing certificate monitor at
it, and watch it tell you everything is fine.

Checking the whole chain

The fix is not complicated, which is part of why the gap is annoying. Pull every
certificate the server actually sends, and run the expiry check against each one.

#!/bin/sh
# Usage: tls-chain-check.sh host:port [sni] [days]
HOST="$1"; SNI="${2:-${1%%:*}}"; DAYS="${3:-30}"
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT

echo | openssl s_client -connect "$HOST" -servername "$SNI" -showcerts 2>/dev/null \
| awk -v d="$TMP" '
    /-----BEGIN CERTIFICATE-----/ { n++; p=1 }
    p                             { print >> (d "/c" n ".pem") }
    /-----END CERTIFICATE-----/   { p=0 }'

[ -f "$TMP/c1.pem" ] || { echo "no certificates received from $HOST"; exit 2; }

rc=0
for f in "$TMP"/c*.pem; do
    subj=$(openssl x509 -in "$f" -noout -subject | sed 's/^subject=[ ]*//')
    end=$(openssl x509 -in "$f" -noout -enddate | sed 's/^notAfter=//')
    if openssl x509 -in "$f" -noout -checkend $((DAYS * 86400)) >/dev/null 2>&1; then
        state="ok"
    else
        state="EXPIRES WITHIN ${DAYS}d"; rc=1
    fi
    printf '%-34s %-26s %s\n' "$subj" "$end" "$state"
done
exit $rc

Non-zero exit when anything on the path is inside the window, which is what you
want for a cron job or a CI gate.

Two details in there are deliberate and worth stealing.

-showcerts before anything else. Without it you get the leaf and nothing
else, which is how you ended up here.

-servername separate from the connect address. On a host serving multiple
sites, the certificate you get depends on the SNI you send, not on the IP you
connected to. If you monitor by IP and omit SNI, you are checking whatever the
default virtual host happens to present, which may be a completely different
certificate from the one your users receive. This is a good way to monitor
something real and useless for eighteen months.

Three more places the same mistake hides

The chain the server sends is not always the chain the client builds.
A client with a cached intermediate, or one that follows the AIA extension to
fetch a missing issuer, can successfully validate a chain your server sent
incompletely. Your desktop browser says fine; a stripped-down container with no
cached intermediates and no outbound access to the CA's AIA endpoint says
handshake failure. Checking from a machine with a rich trust state hides the
problem from you specifically.

Your checker's trust store is not your users' trust store. If the checker
runs openssl verify against the OS bundle on a long-lived VM, you are
validating against whatever roots that VM had at its last update. Mobile clients,
older Java runtimes, and embedded devices all carry different sets. "Valid" is
not a property of a certificate. It is a relationship between a certificate and
a particular trust store at a particular time.

The interception layer has its own copy. If traffic crosses a proxy that
terminates and re-signs TLS,
there are now two certificates on the path to your user, issued by two different
authorities, renewed by two different processes.
Probe a public site from inside an environment with intercepted egress and every
certificate comes back signed by the gateway's own CA — the leaf you would be
"monitoring" was manufactured about a second earlier. If you run monitoring from
inside a corporate network and alert on the certificate you receive, you may be
monitoring the health of your own proxy.

What to actually alert on

Alert on the minimum remaining lifetime across the full path, not on the
leaf. It is one number, it is the one that determines when things break, and it
is strictly more conservative than what most tools give you today.

Then make the thresholds reflect who fixes what. A leaf inside thirty days is
your problem and your automation should have handled it. An intermediate inside
sixty days is usually your CA's problem, and the remediation is to re-issue and
redeploy — a different task, on a different timeline, often owned by a different
team. Folding both into a single "certificate expiring" alert guarantees that at
least one of them gets handled by someone who cannot fix it.

This matters more every year, not less. Public certificate lifetimes are already
down from 398 days to 200, drop to 100 in March 2027, and reach 47 in March 2029.
Renewal stops being a calendar event and becomes a pipeline. Pipelines fail
quietly, which means the monitoring has to be right about what it is looking at.

Checking the leaf was a reasonable approximation when certificates lasted a year.
It is not one anymore.

Your App Works Everywhere Except the Corporate Network

Ionut-Robert Sandu — Sat, 12 Sep 2026 13:57:36 +0000

You ship something. It works on your machine, in CI, in staging, and for every user who tries it. Then one customer opens a ticket: it doesn't work for them. Same version, same config, same everything. It just hangs, or throws a certificate error, or fails in some way your error handling never anticipated.

They're on a corporate network. Somewhere between their machine and your server, a device is opening every TLS connection, reading it, and building a new one.

This is normal. Large organisations are often legally required to inspect traffic leaving their network, and they've been doing it for twenty years. The problem isn't that it happens. The problem is that from where you're standing, it's nearly invisible — and the failures it causes look like bugs in your code.

Here's what's actually happening, why five different things break in five different ways, and how to work out which one you're looking at without access to the customer's network team.

What interception actually does

A normal TLS connection is between your client and your server. The server presents a certificate, the client checks it chains to a certificate authority it trusts, and the two negotiate keys that nobody in the middle can derive. That's the entire point.

An inspection proxy breaks this into two connections. It terminates the client's TLS session itself, then opens a separate one to the real server. In the middle, it holds plaintext.

For the client to accept this, the proxy has to present a certificate for the site the client asked for. It generates one on the fly, signed by a CA whose root certificate the organisation has installed on every managed machine. On a managed laptop, this works transparently. The browser sees a valid chain to a root it trusts, and shows a padlock.

One consequence is worth stating plainly, because a lot of developers get it backwards: this isn't a vulnerability being exploited. It works because an administrator deliberately installed a root certificate. Browsers make a specific exception for locally-installed roots — Certificate Transparency requirements and most key-pinning checks are enforced for publicly-trusted CAs but relaxed for locally-added ones. If they weren't, corporate inspection would break the web for a large fraction of enterprise users, so the exception is deliberate.

That exception is also the reason the failures are so confusing. Interception works fine for the browser, which is why the customer insists their internet is "working" — and breaks for everything else.

Five mechanisms, five different failure signatures

1. Your language runtime doesn't use the system trust store

This is the most common one and the least known.

The corporate root CA gets installed into the operating system trust store. The browser picks it up. Your application might not, because several runtimes ship their own bundled list of certificate authorities and ignore the OS entirely:

Node.js bundles its own root store. It does not read the system store by default. The escape hatch is NODE_EXTRA_CA_CERTS, pointing at a PEM file.
Python, when using requests or anything else built on certifi, uses the certifi bundle rather than the system store. REQUESTS_CA_BUNDLE and SSL_CERT_FILE override it.
Java uses its own cacerts keystore, managed with keytool, entirely separate from the OS.
Go does read system roots on Linux and macOS, but any code that sets a custom tls.Config with its own RootCAs pool has opted out.

Signature: the site loads in the browser on the same machine, but your application throws a certificate verification error. That specific combination — browser fine, code broken — almost always means a trust store mismatch rather than anything wrong with the network.

Firefox is worth a special mention. It maintains its own trust store independently of the OS, so an environment where Chrome works and Firefox doesn't is the same problem wearing a different hat.

2. Certificate pinning

If your application pins a specific certificate or public key — common in mobile apps and in anything handling payments — you've explicitly said you will only accept one identity, regardless of what the trust store says. An inspection proxy cannot satisfy that. It doesn't have the private key.

Signature: fails on every corporate network, works everywhere else, and no amount of installing certificates fixes it. Unlike the trust store problem, this one is unfixable from the client side. That's the point of pinning.

If you pin, you need a documented way for administrators to allow-list your domain from inspection, and your error message should say so. Most apps that pin fail with a generic network error, which turns a five-minute fix into a week of support tickets.

3. Mutual TLS

If your server requires a client certificate, the proxy has to present one on the client's behalf. It doesn't have the client's private key, so it can't. Some proxies detect this and pass the connection through untouched; others don't, and the handshake dies.

Signature: the handshake fails after the server requests a certificate, often with an unhelpful message about a missing or bad certificate. The tell is that it fails at a specific handshake stage, not at verification.

4. Protocol upgrades and long-lived connections

WebSockets, server-sent events, gRPC streams and anything else that holds a connection open or upgrades it mid-flight depend on the middlebox handling that correctly. Many do. Some strip the Upgrade header, some apply an idle timeout far shorter than your keepalive interval, and some buffer responses that you intended to stream.

Signature: the initial request succeeds and then the connection dies, or the upgrade silently doesn't happen and you fall back to polling. Reconnect loops with no error are typical.

This one is nasty because it is intermittent and timing-dependent, which makes it look like a bug in your reconnection logic. If your connections die at a suspiciously round interval — sixty seconds, five minutes — that's an idle timeout somewhere in the path, not your code.

5. QUIC and HTTP/3

QUIC encrypts most of its transport metadata, which defeats the inspection techniques built for TLS over TCP. The usual enterprise response is to block UDP/443 outright and force clients back to TCP.

Browsers handle this gracefully — they try QUIC, fail, and fall back. Custom clients often don't.

Signature: a delay of several seconds before every connection, or a hang, on one network only. If your HTTP client prefers HTTP/3, test it with QUIC disabled before blaming anything else.

Triage from the client side

You usually can't get access to the customer's network. You can get them to run two commands.

The first question is always: is the connection being intercepted at all?

openssl s_client -connect example.com:443 -servername example.com

Look at the issuer line. On an unintercepted connection you'll see a public CA. On an intercepted one you'll see something internal — the organisation's name, or a firewall vendor's. That single line answers the question.

To see the whole chain, including what the proxy is signing with:

openssl s_client -connect example.com:443 -servername example.com -showcerts

Then compare what your application sees against what the system sees:

curl -v https://example.com

If curl succeeds and your application fails on the same machine, you're in mechanism 1 — a trust store mismatch. If curl fails too, the problem is at the network layer and affects everything.

To confirm a certificate issue specifically rather than a network one, try without verification:

curl -vk https://example.com

If it works with -k and fails without, it's trust. If it fails both ways, it isn't. Never leave -k in anything but a diagnostic.

For a suspected HTTP/3 problem, there is a trap worth knowing about. curl does not negotiate HTTP/3 by default — that needs a build with QUIC support and an explicit flag. So check what you actually got before drawing conclusions:

curl -sS -o /dev/null -w "%{http_version}\n" https://example.com

If that prints 2, curl never attempted QUIC, and comparing it against --http1.1 proves nothing at all. Force the attempt instead:

curl -v --http3 https://example.com

If the explicit HTTP/3 attempt hangs or fails while the ordinary request succeeds, UDP/443 is being blocked somewhere in the path. Browsers conceal this, because they try QUIC, fail, and fall back silently — which is precisely why testing in a browser will tell you everything is fine.

What to build so this is less painful

You will not stop corporate networks from inspecting traffic. What you can do is stop your application from being the hardest part of the diagnosis.

Say what actually failed. "Network error" is worthless. "Certificate verification failed: issuer not trusted" points the user at their administrator immediately. Surface the issuer name if you have it.

Document the domains you need. Every serious enterprise vendor publishes a list of hostnames to allow-list, and states plainly if a domain must be exempt from inspection. If you pin certificates, this documentation is not optional.

Make the trust store configurable. Respect NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, or whatever your runtime's equivalent is, and say so in your docs. Administrators know what to do with that.

Never disable verification as a workaround. The number of production systems running with verification switched off because someone hit this once and needed it working by Friday is genuinely alarming. It converts a support ticket into a permanent vulnerability, and nobody ever turns it back on.

Test against an interception proxy. mitmproxy in front of your integration tests, with its CA installed, reproduces most of this in an afternoon. It's a better use of time than debugging it live against a customer who can't share their network configuration.

The short version

If a customer reports that your application fails only on their network, the question isn't whether your code is wrong. The question is which of five things a middlebox is doing to your connection — and the first openssl s_client tells you most of it.