DEV Community

Gaberial Sofie
Gaberial Sofie

Posted on

When Your Resilience Vendor Is Your Single Point of Failure: A Multi-CDN Failover Threat Model After the November 2025 Cloudflare Outage

#architecture #infrastructure #security #sre

The exposure

At 11:20 UTC on November 18, 2025, Cloudflare fell over, and our stack fell over with it. Every dashboard we owned threw HTTP 500, 522, and 530. The detail that reframed the whole morning: when we tried to log into Cloudflare to disable a WAF rule and route around the problem, the Cloudflare dashboard itself was down. We were locked out of the controls that could have saved us.

It is worth stating the exposure in security terms, not just operational ones, because "the vendor had a bad day" is the wrong lesson. Availability is a security property — it sits alongside confidentiality and integrity — and a multi-CDN failover posture is the control that protects it against a single-provider failure. What the outage actually revealed was that our "resilient" architecture had exactly one point of failure, and it was the vendor whose entire job was resilience. The event hit an estimated 20% of all HTTP requests, roughly 7 million websites, and 30% of the top 1,000 sites; per Cloudflare's own post-mortem, a database permissions change caused a Bot Management feature file to double in size and blow past a hardcoded limit, crashing the traffic-routing software across the network and dragging the control plane down with it. I kept that post-mortem in one tab and a thorough third-party writeup of how one team scrambled and rebuilt afterward → in another while reworking our own single points of failure.

Threat model

Blaming the vendor is comfortable and useless. The outage exposed our own decisions, and naming them as threats is what made them fixable.

  • Single-provider control-plane failure. Cloudflare was authoritative DNS for every zone and proxied every hostname. When its control plane died we had zero ability to re-point anything — the failure of one provider was a total loss of both traffic routing and our ability to intervene.
  • Cascading failure with the control plane as victim. The overload did not stay contained; it rippled outward until the very system needed to recover was itself down — the textbook pattern Google's SRE book describes in its chapter on cascading failures. A recovery mechanism that shares a fate with the thing it recovers is not a recovery mechanism.
  • Undersized origins behind a cache. Origins sized for cache-hit traffic collapse in minutes when the CDN stops absorbing load — turning a CDN outage into an origin outage, a second-order failure we had built in.
  • Responder lockout via SSO through the failing provider. Engineers authenticated to internal tooling through Cloudflare Access. With Access degraded, the people who needed to respond could not reach the tools to do it. Concentrating the response path behind the same provider as the incident is an access-control failure mode, not just an availability one.

Controls we added

We gave ourselves a six-week sprint, in priority order.

Control 1 — secondary DNS, so no one control plane can strand us

DNS is the first and cheapest line of defense. We moved to a dual-provider authoritative setup — Cloudflare DNS primary, AWS Route 53 secondary — with both providers' nameservers registered at the registrar and both zones synchronized from a single Terraform source of truth so they cannot drift. Health checks follow the active-passive pattern AWS documents for Route 53 DNS failover. The point is not that Route 53 is "better" — it is that no single provider's control plane can strand us anymore.

Control 2 — a warm CDN fallback, not a slide-deck one

Cloudflare stays primary; AWS CloudFront is a pre-configured, continuously smoke-tested secondary, so "activate fallback" is a DNS weight change rather than a two-day build under fire. A fallback you have never exercised is a hypothesis, not a control.

Control 3 — a direct-origin escape hatch

We created a permanent, un-proxied path to origin: origin.ourdomain.com with proxying disabled, a 24-hour TTL so the record is cached far and wide before we ever need it, and documented copy-paste commands in the runbook. The security-critical detail is the certificate: the escape hatch carries a real, publicly-trusted TLS cert, not a Cloudflare Origin CA cert. Cloudflare's own SSL/TLS docs note that visitors hit untrusted-certificate errors the moment you disable proxying — which is exactly useless when Cloudflare is the thing that is down. An Origin CA cert only chains behind Cloudflare, so a failover that relies on it fails on TLS at the worst possible moment.

Control 4 — monitoring that does not depend on the thing it watches

Monitoring services through the CDN is like testing a smoke detector by asking the fire. We moved to direct-IP health checks (HTTP and TCP) against origin, origin-domain monitoring separate from the public hostname, two independent providers (UptimeRobot and Pingdom) so a single monitoring vendor blip does not blind us, and synthetic checks on the real login and checkout paths. Lost visibility during an incident is lost detection, so the monitoring path is deliberately decoupled from the serving path.

Control 5 — break-glass authentication outside the failing provider

This is the access-control control, and the one most directly in my lane. We kept Cloudflare Access for day-to-day SSO but added backup OIDC providers (Google Workspace and GitHub) wired up ahead of time, break-glass accounts that live entirely outside Cloudflare Access, and a one-page emergency-access document every on-call engineer holds offline. The principle is that the response path must never share a single point of failure with the incident — if the way in to fix things depends on the thing that broke, you have no way in.

Verifying the controls, not just shipping them

Infrastructure changes are worthless if nobody knows the sequence at 11:20 on a Tuesday, so the controls are backed by a timed runbook we drill quarterly against a deliberately broken staging environment: minute 0-5 detect and verify origin health independently; minute 5-15 flip DNS weight, activate the fallback CDN, enable origin-direct for critical paths; minute 15-30 communicate over a channel that is not behind the failing CDN; minute 30+ stabilize and start the post-mortem timeline. The drills are the verification — our first one took 40 minutes because nobody remembered where the break-glass doc lived, which is precisely the kind of finding you want in a drill and not in an incident. A simulated CDN loss now recovers to redirected traffic in under four minutes instead of three-plus hours hard down, but that number is only true as long as the next drill confirms it.

Residual risk / what we're still watching

The rebuild removed the single-provider single point of failure, but resilience work relocates risk rather than eliminating it, and the honest edges matter.

  • The escape-hatch certificate and TTL cut both ways. A publicly-trusted cert on origin. must be renewed and monitored like any production cert — an expired escape-hatch cert is a silent failure that only surfaces during the incident it was meant to survive. The 24-hour TTL that makes the record widely cached also means a mistake on that record lingers, so it changes only through reviewed IaC.
  • Origin autoscaling lag. When the CDN stops shielding origin, direct traffic is spikier than models predict. Pre-warming capacity is in the runbook, but the residual is that a large enough spike still outruns the scaler; we are tuning headroom.
  • Drills decay. A runbook is only as good as the last rehearsal, and quarterly is a compromise — long enough that details rot between drills. We watch drill times as the leading indicator of whether the control still works.
  • Hidden third-party dependencies. The scariest dependencies are the ones you did not know you had until they fail. We are mapping every third-party dependency in the stack and moving toward light chaos engineering — deliberately black-holing the primary CDN in staging on a schedule — to surface them before an outage does.
  • Adding providers adds surface. A second DNS provider, a second CDN, and backup OIDC providers each expand the set of vendors whose compromise or misconfiguration could affect us. More redundancy is more attack surface, so each addition is hardened and its credentials scoped rather than assumed benign.

The mindset shift the outage forced is the durable part: design assuming every provider will fail, then make the failure boring. What we keep watching is the gap between "we rehearsed this" and "the fire came at a time we did not pick" — because you only ever control the second half of that.

Sources & further reading

Top comments (0)