DEV Community

speed engineer
speed engineer

Posted on

We Failed Over to a Healthy Region. The JVM Never Noticed — It Had Cached the DNS Answer Forever.

#java #dns #networking #reliability

The problem

3:40 AM. One region's load balancers started throwing 5xx at roughly 8% of traffic — enough to page, not enough to look catastrophic. We did the standard move: flipped the Route 53 weighted record to send 100% of traffic to the healthy region and watched the dashboard.

Error rate didn't move. Not "improved slowly" — didn't move at all, for 45 minutes, on a subset of hosts that kept hammering the dead region like nothing had happened.

dig from a bastion host showed the DNS answer had updated within seconds, exactly as expected. The record was correct. The resolvers were correct. And a chunk of our fleet was still connecting to a region that no longer existed as far as DNS was concerned.

The affected hosts had one thing in common: they were long-running JVM processes that made outbound HTTP calls through Java's built-in HttpURLConnection / InetAddress resolution path, not through a client that did its own re-resolution.

Why it happens

The JVM does not use your OS resolver's TTL. It has its own DNS cache, controlled by two properties most teams never set: networkaddress.cache.ttl and networkaddress.cache.negative.ttl.

The default behavior, baked in since the early 2000s for a reason that made sense at the time (mitigating DNS-rebinding attacks against applets running under a SecurityManager), is this: if a SecurityManager is installed, successful lookups are cached forever — TTL of -1, meaning "never expire, never re-resolve." If no SecurityManager is installed, the JDK falls back to a default of 30 seconds, which is more reasonable but still isn't reading the actual DNS record's TTL — it's a hardcoded JVM constant that has nothing to do with what your DNS provider configured.

Our long-running services had a SecurityManager set (leftover from an old compliance requirement, unrelated to this code path) and had never touched networkaddress.cache.ttl in java.security. So the first successful resolution of the load balancer's hostname, made whenever that JVM process last happened to open a connection to it, was cached in-process for the lifetime of that JVM. Some of those processes had been running for eleven days. They were never going to re-resolve on their own, no matter what Route 53 said, no matter how many times dig came back clean.

This is the part that makes it a nasty bug rather than a simple misconfiguration: it's invisible under normal operation. Everything works fine for months because your load balancer's IP rarely changes. The cache only becomes a liability at the exact moment you need DNS-based failover to work — during an actual regional failure — which is the worst possible time to discover it.

What to do about it

Set the TTL explicitly and don't rely on the JDK default either way:

# In $JAVA_HOME/lib/security/java.security, or as a JVM property:
networkaddress.cache.ttl=30
networkaddress.cache.negative.ttl=10
Enter fullscreen mode Exit fullscreen mode

Or per-process, without touching the shared security file:

-Dsun.net.inetaddr.ttl=30
-Dsun.net.inetaddr.negative.ttl=10
Enter fullscreen mode Exit fullscreen mode

A few things worth knowing beyond just setting the number:

sun.net.inetaddr.ttl only takes effect when no SecurityManager is present — if you do run one, you have to set networkaddress.cache.ttl in the security policy itself, not the system property. We'd set the wrong knob on our first attempt and spent twenty minutes confused about why nothing changed.

Don't rely on DNS TTL as your only failover mechanism for anything that matters. Pair it with an active health check at the client layer — a connection pool that evicts dead backends, or a client-side load balancer (Envoy, a service mesh sidecar, or even a simple periodic re-resolve-and-swap in application code) that doesn't depend on any single cache expiring correctly. DNS-based failover is a blunt instrument; treat a 30-second cache as the floor of your recovery time, not the whole plan.

Test failover on a live, long-running process, not a freshly started one. A JVM that's been up for ten minutes and one that's been up for ten days can behave completely differently here, and most staging environments get restarted far more often than production ever does.

Key takeaways

  • The JVM caches successful DNS lookups independently of the OS and independently of the record's real TTL — forever, by default, if a SecurityManager is present.
  • This is invisible until the one moment it matters: an actual failover event.
  • Set networkaddress.cache.ttl explicitly; know that -Dsun.net.inetaddr.ttl is a no-op under a SecurityManager.
  • DNS TTL is not a failover mechanism on its own — pair it with active health checking at the client.
  • Test failover against long-lived processes, not fresh ones. That's where caches like this one hide.

Top comments (0)