Let's Encrypt made certificates free and short-lived. It did not make them renew themselves. Renewal fails silently when a DNS record changes, a firewall rule blocks port 80, a cron job stops, or someone moves the site to a new server and forgets the hook. The certificate keeps working right up to the second it does not.
Check expiry from the shell
echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
| openssl x509 -noout -dates -issuer -subject
Output includes notAfter=. Anything under 14 days on a Let's Encrypt certificate (90-day lifetime, renewed by certbot at 30 days remaining, twice a day) means dozens of renewal attempts have already failed. The -servername flag matters: it sets SNI, and without it a server hosting several sites returns its default certificate, which is a common source of false alarms in home-grown checks.
Check the chain, not just the leaf
Browsers accept a certificate only when they can build a chain to a trusted root. A server that sends the leaf without the intermediate works in Chrome (which caches intermediates) and fails in curl, in mobile apps and in every monitoring script:
curl -sSI https://example.com -o /dev/null && echo chain ok
If this fails with unable to get local issuer certificate while the browser is happy, the intermediate is missing. Chrome and Firefox fetch missing intermediates through the Authority Information Access extension or use a cache; OpenSSL-based clients do not. Fix it in the web server's ssl_certificate (nginx wants fullchain.pem, not cert.pem); in Caddy nothing is needed, because Caddy manages it. To see what the server actually sends, openssl s_client -showcerts prints every certificate in the handshake, and a chain of one is the tell.
Check the hostname
A certificate for www.example.com does not cover example.com unless it is in the Subject Alternative Names; since 2017 browsers ignore the Common Name entirely, so a certificate with only a CN fails everywhere modern. A wildcard *.example.com matches one label only: it covers api.example.com but not example.com and not v2.api.example.com. Redirects hide these gaps: the bare domain redirects to www before the browser complains, but a monitoring probe, an API client or a webhook hitting the bare host gets a hostname mismatch.
echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
| openssl x509 -noout -ext subjectAltName
Why "we have auto-renewal" is not a monitoring strategy
Auto-renewal is a process. Monitoring watches the outcome. The outcome is "the certificate the public sees expires on date X", and only an external check sees exactly what a visitor sees, including the wrong certificate served by a load balancer that still holds last year's file.
Getting a warning 14 days out
Three options, from most to least effort:
- A cron job around the
opensslcommand above that comparesnotAfterwithdateand posts to Slack. Works, breaks when the box that runs it is decommissioned. - A Prometheus
blackbox_exporterprobe with theprobe_ssl_earliest_cert_expirymetric and an alert rule. Solid if you already run Prometheus. - An uptime monitor that checks the certificate as part of its HTTPS check. In Vigil every https monitor runs a dedicated TLS handshake every 12 hours, records the expiry, and alerts once per certificate when the days left cross the threshold you set per monitor (14 by default). Because the alert is keyed to the certificate's expiry date, a renewal resets it automatically and a certificate that keeps not renewing does not spam you. The chain and hostname problems above show up as a failed check with the exact
opensslerror attached, so you are not guessing.
You can run all three checks on any hostname right now without an account at /tools/ssl-check.
A short checklist for the next incident review
- Is the expiry alert sent to a channel someone reads, not only to the ops mailbox nobody opens?
- Does the alert fire at 14 days, when there is still time to debug renewal, rather than at 24 hours?
- Are the bare domain,
www, the API host and the status page all monitored separately? They can have four different certificates. - Does the monitoring probe run from outside your network? A probe on the same box sees a different picture from your customers.
- Is OCSP stapling on, and is the stapled response fresh? A stale staple with
Must-Stapleset fails the handshake in Firefox even when the certificate itself is fine.openssl s_client -statusshows the staple.
Originally published on the PuzariCloud engineering blog. Drafted with AI assistance and reviewed, edited and tested by the author, who builds Vigil by PuzariCloud, the monitoring service the examples use.
Top comments (0)