DEV Community

Cover image for ACME Automation Failures: Renewal Outages, Rate Limits & DNS-01 Pitfalls
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

ACME Automation Failures: Renewal Outages, Rate Limits & DNS-01 Pitfalls

#security #webdev #privacy #devops

ACME made certificates free and automatic, which is exactly why expiry outages still happen: nobody watches a cron job that has worked for two years. Then a firewall rule changes, a DNS API token rotates, or a deploy script wipes the client's config directory, and the renewal fails silently for 30 days until browsers start throwing NET::ERR_CERT_DATE_INVALID. With certificate lifetimes shrinking, the gap between "renewal broke" and "site down" keeps getting shorter.

I wrote this with help from AI tools and checked every command and claim myself against current Let's Encrypt documentation. I build PixelValut, a set of free browser-based developer and network tools, and this is Part 5 of Web PKI & Certificate Forensics in Practice. Part 3 covered revocation and short-lived certificates. Part 4 covered Certificate Transparency monitoring. Here we debug the automation that issues all those certificates.

📚 Series Roadmap — Web PKI & Certificate Forensics in Practice

Related: Subdomain Takeover & Dangling DNS Reconnaissance · Zero-Trust Reverse Proxy Hardening in Nginx

Why do ACME certificate renewals fail in production?

Direct answer: Most ACME renewal outages come from three things: validation paths that changed (HTTP-01 blocked by redirects, firewalls or CDNs; DNS-01 broken by expired API tokens or slow propagation), clients that lose state and burn rate limits by re-issuing, and nobody alerting on certificate age. Fix it with ARI-aware clients, staging dry runs, and expiry monitoring independent of the client.

Brass padlock on a dark network appliance in moody side light, AI-generated

The failure taxonomy

Failure Typical symptom Root cause Fix
HTTP-01 blocked Invalid response from http://.../.well-known/acme-challenge/...: 404 or timeout Forced redirect to an app route, WAF/geo-block, CDN cache, port 80 closed Serve the challenge path before redirects; allow port 80 from anywhere
DNS-01 stale Incorrect TXT record or NXDOMAIN for _acme-challenge Propagation delay, wrong zone, split-horizon DNS Wait on authoritative NS, use CNAME delegation
DNS API token expired Client log shows 401/403 from the DNS provider Token rotated or scoped too narrowly Scoped long-lived token + alert on hook failures
CAA mismatch CAA record for example.com prevents issuance CAA lists another CA or wrong accounturi Update CAA before switching CA or account
Rate limited too many certificates (5) already issued for this exact set of identifiers Lost client state, rebuild loops, testing in production Persist client state; use staging for testing
Silent cron failure Cert expires with no alert Timer disabled, container restarted, hook errors swallowed External expiry monitoring
Reload missing New cert on disk, old cert served No deploy hook to reload Nginx/HAProxy --deploy-hook + verify served cert

The last row is the sneakiest: Certbot renews, the file on disk is fresh, and Nginx keeps serving the old certificate from memory.

Let's Encrypt rate limits, stated correctly

Rate limits attract the most folklore. These are the current production limits from Let's Encrypt's documentation (token buckets that refill gradually):

Limit Value Refill Override?
New certificates per registered domain 50 per 7 days 1 every 202 minutes Yes, by request
New certificates per exact set of identifiers 5 per 7 days 1 every 34 hours No
New orders per account 300 per 3 hours 1 every 36 seconds Yes, by request
Authorization failures per identifier per account 5 per hour 1 every 12 minutes No
Consecutive authorization failures per identifier per account 1,152 1 per day; resets on success No (self-service unpause)
New registrations per IP address 10 per 3 hours 1 every 18 minutes No

Three details matter during an incident:

  • Renewals via ACME Renewal Info (ARI) are exempt from all rate limits. Non-ARI renewals (same exact identifier set) skip the per-domain and per-account order limits but still count against the exact-set and authorization-failure limits.
  • Revoking certificates does not reset rate limits. Revoke-and-retry makes things worse.
  • Staging has much higher limits. Point every test, rebuild and CI job at https://acme-staging-v02.api.letsencrypt.org/directory.

The classic self-inflicted outage: an entrypoint runs certbot certonly on every start with no persistent /etc/letsencrypt. A few crash-loop restarts exhaust "5 per exact set", then you wait ~34 hours per token.

HTTP-01 pitfalls and a correct Nginx config

Let's Encrypt follows redirects for HTTP-01, but app-level redirects, auth middleware and CDN rules often break the path. Serve the challenge directory explicitly before anything else:

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;

    # Must come before the catch-all redirect
    location ^~ /.well-known/acme-challenge/ {
        root /var/www/acme;
        default_type "text/plain";
        try_files $uri =404;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}
Enter fullscreen mode Exit fullscreen mode

Validation runs from multiple network perspectives, so geo-blocking or IP allowlists on port 80 can fail it even when your own curl works. Check the path from outside: request a test file and confirm the status code with the PixelValut HTTP status checker, or:

echo ok | sudo tee /var/www/acme/.well-known/acme-challenge/probe
curl -sI http://example.com/.well-known/acme-challenge/probe
# Expect: HTTP/1.1 200 OK, not 301 to an app route or 403 from a WAF
Enter fullscreen mode Exit fullscreen mode

DNS-01 pitfalls: propagation, delegation and wildcards

Wildcards require DNS-01, and it is the fallback when port 80 cannot be exposed. Its failures look different:

  1. Propagation. Your provider's API returns success before all authoritative nameservers serve the TXT record. Query the authoritative servers directly, not your resolver cache.
  2. Wrong zone. _acme-challenge.app.example.com must exist in whichever zone is authoritative for that name. Delegated subzones catch people out.
  3. Split-horizon DNS. Internal views hide what the public sees.
  4. Over-privileged tokens. A zone-wide DNS token on every web server is a takeover risk. 
# Check what the CA will actually see: query each authoritative NS
for ns in $(dig +short NS example.com); do
  echo "== $ns"; dig +short TXT _acme-challenge.example.com @"$ns"
done

# CNAME delegation: point the challenge name at a dedicated validation zone
# _acme-challenge.example.com.  CNAME  _acme-challenge.example.com.acme-validation.example.net.
dig +short CNAME _acme-challenge.example.com
Enter fullscreen mode Exit fullscreen mode

With CNAME delegation, the client only needs write access to a small validation zone. Verify the CNAME, TXT and CAA records from a public vantage point with the PixelValut DNS lookup tool.

CAA records that help instead of hurt

CAA lets you restrict which CAs may issue for your domain. RFC 8657 adds accounturi and validationmethods, which pin issuance to your ACME account and allowed challenge types:

example.com.  CAA 0 issue "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/123456789; validationmethods=dns-01"
example.com.  CAA 0 issuewild "letsencrypt.org; validationmethods=dns-01"
example.com.  CAA 0 iodef "mailto:security@example.com"
Enter fullscreen mode Exit fullscreen mode

This pairs with Part 4: CT tells you after the fact, CAA prevents other accounts from issuing at all. The trade-off: a new ACME account fails issuance until you update the record, so put it in the runbook.

Monitoring that does not trust the client

Never trust the client to report failure. Monitor the served certificate:

#!/usr/bin/env bash
# cert-age-check.sh: alert if the served cert expires within N days
host="$1"; days="${2:-14}"
cert=$(echo | openssl s_client -connect "$host:443" -servername "$host" 2>/dev/null)
end=$(openssl x509 -noout -enddate <<<"$cert" | cut -d= -f2)
if ! openssl x509 -noout -checkend $((days*86400)) <<<"$cert" >/dev/null; then
  echo "ALERT $host expires $end (< ${days}d)"; exit 1
fi
echo "OK $host expires $end"
Enter fullscreen mode Exit fullscreen mode 
# Certbot: test renewal against staging and reload on success
sudo certbot renew --dry-run
sudo certbot renew --deploy-hook "systemctl reload nginx"
systemctl list-timers | grep -i certbot
Enter fullscreen mode Exit fullscreen mode

Set the threshold relative to lifetime. The default classic profile still issues 90-day certificates, but Let's Encrypt's published plan moves it to 64 days on February 10, 2027 and 45 days on February 16, 2028; the opt-in tlsserver profile is already 45 days and shortlived is about six days. Hardcoded "renew at 60 days" schedules will break, so prefer ARI windows or renewing around two-thirds of lifetime.

Diagnostic checklist

  1. Does an external monitor check the served certificate's expiry on every hostname?
  2. Is the ACME client state (/etc/letsencrypt, acme.sh home, cert-manager secrets) on persistent storage?
  3. Do all test and CI runs use the staging directory?
  4. Does your client support ARI so renewals are rate-limit exempt?
  5. Is /.well-known/acme-challenge/ served before redirects, auth and WAF rules?
  6. Is port 80 reachable from all regions, not just your office?
  7. For DNS-01, do you verify TXT records on every authoritative nameserver?
  8. Is the DNS API token scoped (or CNAME-delegated) and its expiry tracked?
  9. Do CAA records match your current CA and ACME account?
  10. Does a deploy hook reload the server, and do you verify the new serial is served?

FAQ

Does revoking a broken certificate let me issue again sooner?

No. Let's Encrypt states revocation does not reset rate limits. Fix the cause, then wait for the bucket to refill.

I hit "5 per exact set of identifiers". What now?

Wait for refill (1 every 34 hours), or add another hostname to change the set. That new set is not a renewal and counts against the per-domain and per-account limits.

Is DNS-01 more secure than HTTP-01?

Neither is inherently safer. DNS-01 avoids exposing port 80 but puts DNS write credentials somewhere. CNAME delegation and scoped tokens reduce that risk.

Why does validation fail when curl works from my laptop?

Validation runs from multiple vantage points; geo-blocking, allowlists or regional DNS can fail some.

Should Kubernetes clusters use cert-manager or a cron'd client?

cert-manager, usually: state lives in Secrets and renewals reconcile. Still monitor externally.

What do shorter certificate lifetimes change?

Less tolerance for silent failures. Automate fully, enable ARI, and alert on overdue renewals rather than days remaining.

Next in this series, Part 6 covers mutual TLS for internal APIs: running a private CA, rotating client certificates without downtime, and when pinning helps or hurts.

Follow & Connect

Top comments (0)