DEV Community

TokenTimer
TokenTimer

Posted on Originally published at tokentimer.ch

Why Certificate Automation Fails Quietly Before an Outage

Why Certificate Automation Fails Quietly Before an Outage

A certificate can renew successfully for months and then fail after a DNS migration, hosting change, validation change, or new network path. The current certificate remains valid, so the service looks healthy while the expiry date keeps moving closer.

That pattern appeared in four public incidents involving Tailscale, jsDelivr, ServiceNow, and IPinfo.

The failure happened before the outage

Tailscale's March 2024 outage lasted about 90 minutes. A hosting migration months earlier had produced an unusual IPv4 and IPv6 setup. The expiry probe checked the IPv6 proxy, which still served a valid certificate. Users reaching the IPv4 path received the certificate that eventually expired.

In May 2024, some jsDelivr users received an expired certificate for more than five hours. A managed certificate had moved from DigiCert to Google Trust Services, and the domain-validation method changed. In a multi-CDN design, HTTP validation could reach a different provider and fail.

ServiceNow showed that this problem is not limited to public websites. In September 2024, an expired TLS cross-chain certificate disrupted integrations, discovery, orchestration, upgrades, and other workflows for 616 customers.

IPinfo's September 2025 incident followed a DNS move to Cloudflare. DNS-01 records were still being created in Google Cloud DNS, which was no longer authoritative. cert-manager logged the errors, but nobody received an expiry warning before the API and website went down for 2 hours and 36 minutes.

The common sequence was simple:

  1. An infrastructure or provider change broke renewal indirectly.
  2. The current certificate stayed valid, hiding the problem.
  3. An error existed in logs, provider state, or an unmonitored path.
  4. The team discovered the failure at or after expiry.

A green health check is not a lifecycle check

An HTTP probe asks whether an endpoint responds now. Certificate lifecycle monitoring asks additional questions:

  • Which certificate was served?
  • Which IPv4, IPv6, CDN, region, SNI, or internal trust path did we observe?
  • How many days remain?
  • Who owns the certificate?
  • Did renewal run?
  • Is the renewed certificate actually deployed?

A green uptime check can coexist with a broken renewal process for most of a 90-day certificate's lifetime.

Controls that catch the problem earlier

  • Keep a certificate inventory independent of the system that issues certificates.
  • Alert at several thresholds. A practical baseline is 45 or 30 days, followed by 14, 7, 3, and 1 day.
  • Retest renewal after DNS, hosting, CDN, load-balancer, ACME-account, or CA changes.
  • Probe every path that matters to users and internal dependencies.
  • Verify the live certificate after issuance and deployment.
  • Assign an owner and escalation path to roots, intermediates, cross-chains, and leaf certificates.

ACME remains the issuance protocol, and your CA remains the issuer. Independent lifecycle monitoring provides the separate signal that tells you when automation has drifted.

TokenTimer approaches this by keeping inventory, owners, expiry alerts, renewal work, and deployment evidence together. CertOps execution stays in the customer's environment, so private keys and DNS credentials do not move into the TokenTimer control plane.

The complete incident analysis and source links are in the canonical TokenTimer article.

What do you test after a DNS or load-balancer change to prove certificate renewal still works end to end?

Top comments (0)