You deploy an urgent microservice update, push DNS records, and everything seems fine locally. Ten minutes later, PagerDuty lights up: workers cannot resolve the API hostname, inbound webhooks fail, and support tickets pour in.
DNS is often treated as simple static infrastructure. But under production workloads, subtle protocol behaviors cause silent failures that waste hours of engineering time.
Here are five DNS resolution traps that consistently catch engineering teams off guard, and how to fix them.
1. The CNAME at Zone Apex Trap (RFC 1912)
A common mistake is pointing an apex domain (example.com) to a CDN using a CNAME record.
Under RFC 1912 §2.4, if a CNAME record exists for a node, no other record types may coexist for that same node. Because your apex must contain SOA and NS records to function, creating a CNAME at the apex violates the DNS specification:
;; Invalid RFC 1912 configuration:
example.com. 300 IN CNAME d1234.cloudfront.net.
example.com. 300 IN SOA ns-1.awsdns.com. ...
example.com. 300 IN NS ns-1.awsdns.com.
Resolvers behave unpredictably—many drop MX records entirely, silently killing company email.
Fix: Use DNS providers supporting CNAME flattening or ALIAS records. These synthesize direct A and AAAA answers at query time while preserving apex SOA and MX records.
2. Negative Caching and the SOA Minimum TTL (RFC 2308)
During deployment, engineers often test whether a new subdomain resolves before records finish provisioning. You run curl api-staging.example.com, get NXDOMAIN, and wait for the record to be added.
Once created, your application still fails with ENOTFOUND for hours.
Resolvers do not just cache successful answers; they also cache negative results (NXDOMAIN). Under RFC 2308, negative caching duration is governed by the minimum TTL in the zone’s SOA record:
example.com. 86400 IN SOA ns1.example.com. hostmaster.example.com. (
2026090401 ; Serial
7200 ; Refresh (2h)
3600 ; Retry (1h)
1209600 ; Expire (14d)
86400 ; Negative Cache TTL (24h)
)
If your SOA negative TTL is 86,400 seconds (24 hours), querying an uncreated record poisons recursive resolvers for a full day.
When chasing an incident, you rarely have terminal access to clean external networks. Verifying record sets through an in-browser utility like Nutilz DNS Lookup (which queries Cloudflare DNS over HTTPS directly) lets you inspect A, AAAA, MX, CNAME, and SOA responses from an independent resolver without poisoning local OS cache.
3. UDP Truncation and Missing TCP Fallback (RFC 7766)
DNS over UDP is historically bounded by 512 bytes, expanded by EDNS0 to 1,232 bytes to avoid packet fragmentation.
However, DNSSEC signatures (RRSIG) and verbose TXT verification strings (SPF, DKIM) routinely exceed 1,232 bytes. When a response exceeds buffer size, the nameserver sets the TC (truncated) bit:
$ dig +bufsize=512 TXT large-zone.example.com
;; Truncated, retrying in TCP mode.
Under RFC 7766, resolvers MUST fall back to TCP port 53. Yet many firewalls, cloud security groups, and Docker bridges only permit outbound UDP 53. When outbound TCP 53 is blocked, large queries fail silently with connection timeouts or SERVFAIL.
Fix: Ensure egress security rules permit both TCP and UDP on port 53.
4. The Kubernetes ndots:5 Latency Multiplier
In Kubernetes, pod /etc/resolv.conf files default to options ndots:5:
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
This tells the resolver that any domain with fewer than 5 dots must be queried against each search suffix before attempting the domain as an absolute FQDN.
When your microservice queries api.stripe.com (2 dots), the resolver sequentially queries three internal search domains before trying the external name. This introduces three wasted round-trips, quadruples CoreDNS load, and exhausts kernel conntrack tables.
Fix: Append a trailing dot to external hostnames in configs (https://api.stripe.com./) or set ndots: 2 in pod dnsConfig.
5. Happy Eyeballs with Broken IPv6 Routing
When teams add AAAA records, dual-stack clients query A and AAAA in parallel.
Modern OS resolvers implement Happy Eyeballs (RFC 8305), giving IPv6 a 250ms head start. If your load balancer advertises IPv6, but internal firewall rules drop IPv6 packets rather than sending an immediate TCP RST, clients hang waiting for SYN timeouts before falling back to IPv4. Users experience 1–3 second delays on every new connection.
Summary
DNS reliability requires understanding how resolvers, caches, and transport layers interact. Avoid apex CNAMEs, keep your SOA minimum TTL low (e.g., 300 seconds), allow egress TCP 53, and tune ndots in containers. Keeping testing tools like dig +trace and Nutilz DNS Lookup in your toolkit ensures you can diagnose resolution bottlenecks before they impact users.
Top comments (0)