DEV Community

lastmileops
lastmileops

Posted on

Your certificate renewed. Does your service know?

#bash #sysadmin #security #devops

Tested with OpenSSL 3.6 and GNU date.

Automatic certificate renewal was invented to make expiry a non-event. Most of the time it is. But when something does break,
it breaks quietly and with imagination:

  • the DNS provider's API token expired, and domain validation no longer passes;
  • a new firewall politely drops the HTTP challenge, and nobody knows;
  • the deploy hook restarts a service that was renamed six months ago;
  • renewal went perfectly, and the service keeps serving the old certificate from memory.

And if you were counting on reminder emails from your CA: Let's Encrypt stopped sending them back in 2025. Nobody is going to
remind you now except you.

One simple idea covers that whole list: a check that doesn't trust the renewal job at all. It connects to the service the way
an ordinary client does and looks at the certificate that actually comes back. Not the file on disk, not the certbot log, but what
your users' browsers see.

#!/usr/bin/env bash
# check-cert.sh HOST[:PORT] [MIN_DAYS] — check the certificate a service actually presents
set -uo pipefail
target="$1"; min="${2:-14}"
host="${target%%:*}"; [[ "$target" == *:* ]] || target="$target:443"
pem=$(echo | timeout 10 openssl s_client -servername "$host" -connect "$target" 2>/dev/null \
      | openssl x509 2>/dev/null) || true
if [[ -z "$pem" ]]; then echo "UNREACHABLE: $target presented no certificate"; exit 2; fi
end=$(openssl x509 -noout -enddate <<<"$pem" | cut -d= -f2)
days=$(( ( $(date -u -d "$end" +%s) - $(date -u +%s) ) / 86400 ))
if ! openssl x509 -noout -checkhost "$host" <<<"$pem" | grep -q "does match"; then
  echo "MISMATCH: $target presents a certificate not valid for $host (expires in $days days)"; exit 2
fi
if (( days < 0 )); then echo "EXPIRED: $target expired on $end"; exit 1; fi
if (( days < min )); then echo "EXPIRING: $target expires in $days days ($end)"; exit 1; fi
echo "OK: $target valid for $days more days"
Enter fullscreen mode Exit fullscreen mode

What it says:

OK: intranet.example.com:443 valid for 54 more days
EXPIRING: wiki.example.com:443 expires in 9 days (Nov  2 18:13:52 2026 GMT)
EXPIRED: old-vpn.example.com:443 expired on Feb  1 00:00:00 2025 GMT
MISMATCH: files.example.com:443 presents a certificate not valid for files.example.com (expires in 399 days)
UNREACHABLE: mail.example.com:465 presented no certificate
Enter fullscreen mode Exit fullscreen mode

Pick a threshold below the point where renewal should already have happened. From February 10, 2027, Let's Encrypt will issue
64-day certificates by default, and it already recommends renewing at about two-thirds of the lifetime, which leaves roughly
21 days. An alert at 14 days stays quiet on normal renewals and fires when one is a week late. A 21-day threshold would page
someone on every routine renewal, and within a month everyone learns to ignore it. Which is exactly what we were trying to avoid.

MISMATCH is the sneakiest one. A virtual host lost its own certificate and quietly serves the server's default one. Technically
there is a certificate, and it's valid. Just not for this name. Your users' browsers will disagree with you.

Run it for every name and port that matters, and from a different machine than the one doing the renewals. Otherwise the
check and the renewal break together and stay silent together.

#!/usr/bin/env bash
# check-all.sh — run check-cert.sh for every host:port in certs.txt, print only problems
while read -r t _; do
  [[ -z "$t" || "$t" == \#* ]] && continue
  /opt/tools/check-cert.sh "$t" 14 || true
done < /opt/tools/certs.txt | grep -v '^OK'
Enter fullscreen mode Exit fullscreen mode 
0 7 * * * out=$(/opt/tools/check-all.sh); [ -z "$out" ] || echo "$out" | notify-oncall
Enter fullscreen mode Exit fullscreen mode

Send the result to a role or a shared queue, not to whoever set all this up once. The address that gets the alert should outlive
the person who typed it in. Otherwise you end up like one thread on Spiceworks: renewal failed, and the notifications were going
to the mailbox of a former contractor who was long gone.

What it doesn't do. It doesn't validate the chain or revocation: if you need that, add -verify_return_error and your own CA
bundle to s_client. Services that upgrade with STARTTLS (SMTP on 25 and 587, IMAP on 143) need -starttls smtp or
-starttls imap. And it tells you that something broke, not why. For the why, keep a record of every renewal run: which
version, when, and how it ended.

Whenever I get the chance to make my geeky thoughts more pleasant for a wider audience, I don't shy away from what modern AI can do.

Top comments (0)