Failover beyond DNS

So you've built a service and it's running in the cloud. It sits in a Kubernetes cluster, it's designed for high availability, and every component has a replica. Then the data centre loses its connection to the internet, and none of that helps.

It happens more often than we'd like to admit. Data centres with multiple independent points of presence still have outages. On 20 October 2025, a DNS fault inside AWS's US-EAST-1 region cascaded into hours of disruption for a long list of services that had nothing to do with each other, apart from where they were hosted. Cloudflare has had more than one bad day too, including an outage of its 1.1.1.1 DNS resolver in July 2025 and a network-wide failure that November. When a large share of the web sits on a handful of providers, one provider's bad afternoon becomes everyone's.

The uncomfortable lesson is that your hardware can be perfectly healthy and you can still be offline. Spread yourself across data centres with failover between them, and a DNS problem can still undo the whole arrangement.

Why this mattered to us

Web-Engine Sentry (Sentry for short) is our site-protection plugin. It cuts spam by making a site hard work for bots while keeping it easy for humans. It's built for WordPress first, and it ships with an API and examples so it can protect any site. Sentry gives sites several ways to check that a visitor is real: captcha, email verification and mobile verification. As soon as a site puts one of those checks in front of its contact form or login page, our uptime becomes their uptime. If we're unreachable, their visitors can't get in touch or sign in.

So "it's in the cloud" wasn't a good enough answer. We needed a plan that keeps the service running wherever possible, whatever fails underneath it.

Option one: throw hardware at it

You can solve most availability problems with hardware, if you have the budget. Large cloud providers do exactly that, and they price it in. For most teams, though, a purely hardware-based answer is expensive, and you're still bound by the physical limits of someone else's estate.

The conventional approach would be to host in two geographic regions and use DNS failover to switch between them. It works, but it isn't instant. Once the primary goes down, health checks have to notice, records have to change, and resolvers around the world have to let go of cached answers. Short TTLs help, but not every resolver honours them. That leaves a window, often a few minutes, where requests fail. For a verification step sitting in front of someone's checkout, a few minutes is a long time.

Option two: make the software smarter

The alternative is to let the client decide. Give the consumer of the service an ordered list of endpoints, add some basic error handling, and when a request fails, move to the next one straight away.

The real advantage is that those endpoints can be completely independent: different hosting, different networks, even different DNS providers. Failover no longer waits for a record to propagate. It happens on the next request, in milliseconds. Our weak point becomes our own code and the service itself, rather than the rigid physical constraints of our suppliers, and our own code is something we can test, improve and ship.

DNS failover vs client-side fallback

The idea in its simplest form:

// Simplified illustration, not the Sentry SDK itself
const ENDPOINTS = [
  'https://a.example-primary.net',
  'https://b.example-secondary.org',
  'https://c.example-tertiary.io',
];

async function callService(path, options = {}) {
  for (const base of ENDPOINTS) {
    try {
      const res = await fetch(base + path, {
        ...options,
        signal: AbortSignal.timeout(3000),
      });
      if (res.ok) return res;
    } catch (e) {
      // network error or timeout: try the next endpoint
    }
  }
  throw new Error('All endpoints unavailable');
}

Decentralise by design

Real resilience comes from breaking the application into its key parts and giving each one its own availability strategy. Once you treat each component as a separate service, you can tune how each one fails over. Here's how that played out for Sentry.

DNS

We spread our endpoints across several domains, each managed by a different DNS provider, with multiple endpoints behind each domain. If one provider has a bad day, the others are unaffected. Relatively short TTLs mean we can repoint routes quickly when we need to.

Frontend

The browser script loads from whichever CDN endpoint is available. The frontend knows about the alternatives, so if a script fails to load it moves on to the next one. That gives us geographically separate fallbacks managed entirely on the visitor's side. Once it finds an endpoint that works, it sticks with it for the rest of the session, so visitors don't pay the cost of a failed attempt twice.

Backend

On the site's own server, the Sentry SDK acts as a gateway to the available service endpoints. That pushed us to treat each backend as a separate node from the start and to design a proper synchronisation strategy between them. We had to decide early what the service provides and how to deliver it in a decentralised way, rather than bolting resilience on later.

Data

Each node also runs its own separate database, holding a specific set of tables duplicated across several sites. Our APIs keep these in sync with the master. Some of that data is read-only reference data; the rest lets a node keep tracking activity on its own. That's enough for the service to keep running even when the primary site is unavailable. When the master comes back, each node syncs what it recorded back to it.

Behind everything sits a highly available database cluster, and the nodes treat it as the ultimate source of truth. On paper, that cluster should never go down. We deliberately don't assume it won't. Making no promises about its availability gives us another layer of fault tolerance, and forced us to plan up front how data is reconciled once everything is back online.

The result

None of this is exotic. It's a fallback list, some careful error handling, and the discipline to assume every layer can fail. Together those give a site running Sentry verification that keeps working through regional outages, DNS failures and the occasional very bad day in a data centre.

Try it, and tell us what you think

You can install Sentry from the WordPress plugin directory, or read more about Web-Engine Sentry and the Sentry API to use it on any site.

How do you handle failover in your own services? Have you been caught out by DNS? Join the discussion and let us know.