Today's Let's Encrypt outage failed renewals for an hour. Here's the 30-minute playbook for when that's you.

At 01:40 UTC today (Sep 11), Let's Encrypt started failing domain-control validation — a DNS resolver problem on their secondary validation path. For roughly an hour, issuance and renewal just failed. They identified it, reverted the change, and resolved by 02:39 UTC. Status page did its job. That's the good news.

The bad news: for a lot of small teams, the outage wasn't the incident. The incident is what happens thirty days from now, when a cert whose renewal fell inside that window silently expires on a Tuesday morning.

Here's the playbook we run — it fits on one page, and it's written for teams without a full-time SRE, without PagerDuty, and without a budget.

1. Re-run failed renewals — but pace them

ACME rate-limits failed validation attempts: 5 failures per hour, per account, per hostname. If your automation retried every minute during the outage window, you may be temporarily throttled and not know it.

Don't debug the client. Just let your normal renewal loop (certbot renew, whatever you use) pick up the un-renewed certs on its next run — it will. What matters is that the next run happens, not that you force it now.

2. Monitor expiry, not uptime

A failed renewal is invisible until the cert expires. Uptime checks go green because the current cert is still valid. The failure already happened — you just can't see it yet.

Cheap fix, ten minutes:

• blackbox_exporter TLS probe, alert at < 21 days remaining, or
• a cron line: echo | openssl s_client -connect host:443 2>/dev/null | openssl x509 -noout -enddate

That second one is genuinely the whole monitoring stack for cert expiry. Alert on the date, not the outage.

3. Use the staging endpoint to answer "is it me or them"

acme-staging-v02.api.letsencrypt.org has much higher rate limits and uses fake trust roots. When renewals fail and you don't know why, reproduce against staging first. It tells you in one attempt whether your client config is broken or the CA is on fire — without burning your production rate limit.

4. If something actually expired: reissue, then interrogate the gap

Reissue now, obviously. Then ask the real question: why did nothing retry between the outage window and expiry? If your renewal runs once a day at 03:00 and the outage window was 01:40–02:39, a single-day outage shouldn't have killed you — the next day's run should have caught it. If it didn't, your renewal frequency is the bug, not the CA.

That silent gap — the thing that "should have retried and didn't" — is the actual root cause of most cert incidents, and it survives long after the CA's status page goes green.

The bigger pattern

Every step above is one instance of the same 30-minute flow we use for any dependency outage:

1. Verify blast radius — what actually failed, what silently degraded
2. Stabilize — restore service without making rate limits worse
3. Communicate — who needs to know, right now
4. Timeline — write it down while it's fresh; it's your postmortem

Dependency outages (CAs, DNS, payment providers, clouds) are where small teams lose the most time, because there's no dashboard that says "your renewals failed three hours ago." The flow is the same whether the dependency is a certificate authority or a third-party API.

We publish a free first-response checklist for exactly this — the "first 30 minutes" of an incident, built for teams of 2–50 people who don't have an SRE on call: The First 30 Minutes — free checklist. If you want the full version (postmortem templates, escalation ladder, comms templates), there's the Ops Starter Kit.

No signup wall on the free one. Take it, adapt it, ship it.

---

Hive80Lab publishes ops playbooks and runbook templates for small teams — written by an ops crew that watches outages so you don't have to learn them the hard way.