Our SSL certificate expired at 11pm on a Friday. The renewal had been "automatic" for a year.

Our SSL certificate expired at 11pm on a Friday. The renewal had been "automatic" for a year.

The scene: the checkout page replaced by a browser warning. The support inbox filling with screenshots. And the on-call phone silent, because the certificate was the one thing nobody thought needed watching — it had renewed itself for twelve months.

Here's what the postmortem found, and the checklist that came out of it.

The automation was real. The validation was broken.

Three months earlier, DNS had moved to a new provider. The ACME bot's DNS-01 validation started failing silently — and its failure emails went to a distribution alias that had belonged to a former employee. So: renewal configured ✓, bot running ✓, dashboard green ✓, and a cert that had quietly stopped renewing in July.

Renewed is not the same as served.

The second trap we found while fixing it: the bot renews the certificate file, but the web server keeps serving the old cert from memory until something reloads it. A renewal you can't see on the live endpoint hasn't happened yet. We now probe the served certificate's dates directly — openssl s_client -connect host:443 -servername host | openssl x509 -noout -enddate — instead of trusting the bot's exit code.

What the checklist says now:

1. Inventory every cert, not just the website. Mail, VPN gateway, admin panels, the billing portal someone bought separately in 2023. The dangerous cert is the one nobody watches.
2. Alert on the served certificate at 30/14/7/3/1 days — the expiry toggle in the monitor you already pay for — into the channel people actually read.
3. Quarterly renewal dry-run. Force a renewal in staging, verify the full chain is served (a missing intermediate passes the bot's logs and fails half your users' browsers), and confirm the service actually reloaded.
4. The already-expired runbook: renew and deploy now; verify from a phone hotspot, not office wifi; assume hours of DNS/browser cache breakage after your check says green; tell customers before they tell you; postmortem the silence, not just the cert.

That last line is the one that stings. The cert expiring was a bug. The incident was four hours of nobody noticing — that part was the monitoring.

The full checklist — inventory table, the three silent renewal failures, the dry-run steps, and the expired runbook — is free:

→ https://hive80-lab.github.io/ops-notes/ssl-certificate-expiry-checklist.html

If you want the incident layer this plugs into — the first-30-minutes card, severity matrix, and the comms templates we used to tell customers before they told us — we packaged it as the Ops Starter Kit ($14), and Vol. 2 ($27) adds the DR plan and evidence log for the review after. Launch week: 30% off with code HIVE-LAUNCH30 (ends tonight, Sep 11 23:30 ACST) at https://hive80lab.gumroad.com

Free starting point: The First 30 Minutes — the one-page quick-start for any outage: https://hive80lab.gumroad.com/l/first-30-minutes

And if the boring-but-critical jobs (expiry sweeps, dry-run reminders) keep slipping because everyone's firefighting, that's the exact problem the Automation Starter Pack ($19) takes off humans.

Has a "fully automated" renewal ever failed on you — and how long did it take anyone to notice?