DEV Community

OneLifeVibe
OneLifeVibe

Posted on

Split-Brain DNS Is What Actually Breaks Multi-Cloud — And How to Design Around It

Split-Brain DNS Is What Actually Breaks Multi-Cloud — And How to Design Around It

Most multi-cloud designs start with the network diagram: VPCs, transit hubs, private interconnects, routing tables. DNS gets one small box in the corner, labelled "DNS", and that is usually where the trouble starts — not during the design review, but three months later, at 2 a.m., when the same hostname returns three different answers depending on which cloud you ask from.

1. The symptom: one name, several truths

The classic failure looks like this. api.internal.example.com resolves to a private address in AWS because Route 53 has a private hosted zone attached to that VPC. In Azure, the same name falls through to public DNS and returns the public load balancer — or NXDOMAIN, because Private DNS zones are linked per virtual network and one link was never added after a new subscription was created. In Google Cloud, Cloud DNS peering forwards some queries and not others, depending on which network the VM sits in.

Nothing is "down". Every resolver is working exactly as configured. The estate simply has no single answer to the question "what does this name mean?", and applications experience that as intermittent connectivity: retries that sometimes land on the private path and sometimes on the public one, TLS errors when a name resolves to an endpoint presenting a different certificate, and health checks that pass in one cloud and fail in another for the same service.

2. Why it keeps happening

Three structural reasons show up again and again:

Private zones are local by default. Every cloud scopes its private DNS to the networks you explicitly attach. Add a new VPC, subscription, or project and, until someone remembers the DNS step, that network lives in a different naming world from the rest of the estate.

Forwarding loops are easy to build by accident. Cloud A forwards corp.example.com to Cloud B's resolver, Cloud B forwards unknown queries back to Cloud A, and a typo'd zone turns into a query loop that only shows up under load. SERVFAIL responses get cached downstream, so a two-minute forwarding mistake can look like a thirty-minute outage.

Teams reuse the same root name for different things. The public zone and two different private zones all claim example.com. Split-horizon DNS is a legitimate pattern, but only when the split is deliberate and documented. Accidental split-horizon — where the private zone shadows public records nobody knew existed — breaks things like SaaS endpoints and certificate validation in ways that are painful to trace.

3. A design that holds up

A pattern that survives growth across AWS, Azure, Google Cloud, and Oracle estates:

  • Pick one authoritative home per zone. Each private zone lives in exactly one place — often a central networking account or subscription — and other clouds forward to it. Forward, do not copy: replicated zones drift, forwarded zones do not.
  • Use a dedicated internal namespace. Something like internal.example.com (or a domain you own but never publish publicly) for private records, kept separate from the public zone. This removes most shadowing accidents by construction.
  • Keep service discovery out of DNS failover. DNS TTLs, negative caching, and resolver behaviour you do not control make DNS a blunt instrument for traffic shifting. Use it for stable name-to-location mapping; let load balancers, health-checked endpoints, and your service layer handle failover decisions.
  • Automate the attachment step. New network creation — VPC, VNet, or VPC network in any cloud — should attach or peer the DNS configuration in the same pipeline that creates the network. If attaching DNS is a manual ticket, it will eventually be a forgotten ticket.
  • Set TTLs deliberately. Short TTLs on records that genuinely change during failover windows, longer ones on stable internal services. And know your negative TTLs: an NXDOMAIN cached for 300 seconds will outlive the fix you just deployed.

4. Test resolution like you test routing

DNS in a multi-cloud estate deserves the same verification discipline as the network itself:

  • Maintain a small query matrix: a fixed set of critical hostnames, queried from a host in every cloud and every major network segment, with expected answers recorded. Run it after every network change and on a schedule.
  • Log resolver queries centrally for a while after any DNS change. Loops, unexpected NXDOMAIN spikes, and queries leaking to public resolvers are all visible there before users report them.
  • Rehearse the failure: detach a zone link or disable a forwarder in a staging estate and watch what applications actually do. The gap between "DNS is redundant" on a diagram and observed behaviour is usually instructive.

None of this is exotic. It is mostly the discipline of treating name resolution as shared infrastructure with one owner, rather than a per-cloud side effect of creating networks.

5. Where this fits in a wider design

DNS is only one layer of a multi-cloud connectivity design — routing, address planning, identity between services, and evidence for controls all interact with it. Teams working through the full picture sometimes use a managed connectivity layer to keep those pieces consistent across clouds; that is the problem space CloudVerg works in, at cloudverg.com, with private multi-cloud networking and zero-trust design across AWS, Azure, Google Cloud, and Oracle estates.

Get the naming layer boring first. Multi-cloud systems fail in interesting ways often enough without DNS helping them.

Top comments (0)