Every hybrid environment I've ever touched has the same ghost story: an application in AWS that sometimes can't resolve an on-premises hostname. Not always — sometimes. It works from one VPC and not another. It works for the sysadmin testing with nslookup and fails for the Java app two minutes later. Somebody eventually says "it's never DNS", everyone laughs nervously, and it's DNS.
I run the AWS side of a hybrid environment for a court system — 22 accounts, production EKS, and a datacenter full of systems that aren't going anywhere. Our cloud workloads need to resolve internal names that live on on-prem DNS servers, and on-prem systems need to resolve names that only exist in Route 53 private hosted zones. Getting that to work reliably in both directions is one of those problems that looks simple on the whiteboard and then eats a week of your life.
This is the end-to-end setup that works, plus the six mistakes I see over and over — including one that creates a genuine infinite loop between your datacenter and AWS.
First, understand who answers DNS inside a VPC
Before touching Resolver endpoints, you need to be clear about one thing: every VPC already has a DNS server. It lives at the VPC CIDR base plus two — for a 10.20.0.0/16 VPC, that's 10.20.0.2. People call it "VPC+2", the docs call it the Amazon Route 53 Resolver, your instances get it via DHCP as AmazonProvidedDNS.
That resolver is what makes AWS DNS magic work: it answers for private hosted zones associated with the VPC, resolves AWS service endpoints (including PrivateLink and VPC endpoint names), and recurses to the public internet for everything else. Your EC2 instances, your pods (via CoreDNS, which forwards to it), your Lambda functions in a VPC — everyone asks VPC+2.
Here's the thing VPC+2 can't do by itself: it has no idea your datacenter exists. Ask it for intranet.corp.internal and it will go look on the public internet, come back with NXDOMAIN, and your app breaks. And the reverse is also true — your on-prem DNS servers have no idea what db.prd.internal.example.com in a private hosted zone means.
Route 53 Resolver endpoints are the bridge, and there are exactly two kinds because there are exactly two directions.
The two directions
Outbound endpoint: AWS asks on-prem. An outbound endpoint is a set of ENIs in your VPC that can forward queries out. You pair it with forwarding rules: "any query for corp.internal → send it to 192.168.10.10 and 192.168.10.11". The rule is the brain, the endpoint is the pipe.
Inbound endpoint: on-prem asks AWS. An inbound endpoint is a set of ENIs with fixed private IPs that your on-prem DNS servers can forward to. On the datacenter side, you create conditional forwarders on your AD DNS or BIND: "anything under internal.example.com → ask 10.20.5.10 and 10.20.6.10". Those IPs answer exactly like VPC+2 would — private hosted zones, PrivateLink names, all of it.
Most real environments need both. Ours certainly does.
Building it: the order that avoids rework
Step 1 — plan the subnets and security groups. Each endpoint needs IPs in at least two AZs (that's your HA — don't cheat on this). The security group needs both TCP and UDP on port 53. Write that down, because it's mistake #3 below and it produces the most maddening symptom in this entire article.
Step 2 — the outbound endpoint and rules. In Terraform:
resource "aws_route53_resolver_endpoint" "outbound" {
name = "to-onprem"
direction = "OUTBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address { subnet_id = aws_subnet.private_a.id }
ip_address { subnet_id = aws_subnet.private_b.id }
}
resource "aws_route53_resolver_rule" "corp" {
name = "corp-internal"
domain_name = "corp.internal"
rule_type = "FORWARD"
resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id
target_ip { ip = "192.168.10.10" }
target_ip { ip = "192.168.10.11" }
}
resource "aws_route53_resolver_rule_association" "corp_vpc_main" {
resolver_rule_id = aws_route53_resolver_rule.corp.id
vpc_id = aws_vpc.main.id
}
A rule only takes effect in VPCs it's associated with. That word is doing a lot of work — remember it for mistake #5.
Step 3 — share the rules across the Organization. If you have more than a handful of accounts, do not recreate rules everywhere. Create them once in a networking account and share via AWS RAM:
resource "aws_ram_resource_share" "resolver_rules" {
name = "resolver-rules"
allow_external_principals = false
}
resource "aws_ram_resource_association" "corp" {
resource_arn = aws_route53_resolver_rule.corp.arn
resource_share_arn = aws_ram_resource_share.resolver_rules.arn
}
resource "aws_ram_principal_association" "org" {
principal = data.aws_organizations_organization.this.arn
resource_share_arn = aws_ram_resource_share.resolver_rules.arn
}
Spoke accounts then just associate the shared rule with their VPCs. In our 22-account setup this is the difference between "DNS is managed" and "DNS is 22 slightly different snowflakes". One outbound endpoint in the hub serves the whole Organization — the rule points at the hub's endpoint regardless of which VPC the query starts in, as long as your Transit Gateway or peering can carry the traffic back.
Step 4 — the inbound endpoint and the on-prem side.
resource "aws_route53_resolver_endpoint" "inbound" {
name = "from-onprem"
direction = "INBOUND"
security_group_ids = [aws_security_group.resolver.id]
ip_address { subnet_id = aws_subnet.private_a.id, ip = "10.20.5.10" }
ip_address { subnet_id = aws_subnet.private_b.id, ip = "10.20.6.10" }
}
Pin the IPs explicitly — your on-prem DNS team will configure conditional forwarders pointing at them, and you do not want those IPs changing on a rebuild. On Windows DNS: New Conditional Forwarder → internal.example.com → both inbound IPs. On BIND, a forwarders block in a zone stanza. Same idea anywhere: for this domain, ask AWS.
Step 5 — test both directions before declaring victory. From an EC2 instance: dig intranet.corp.internal (should come back with an on-prem answer). From an on-prem box: dig db.prd.internal.example.com @10.20.5.10, then again without the @ once the forwarder is configured. Test from a spoke VPC too, not just the hub. Then test something that only exists in a private hosted zone, and something behind PrivateLink. Five minutes of testing here saves the 2am call later.
The six classic mistakes
1. The circular forwarder (the infinite DNS loop). This is the one that earns war-story status. On-prem forwards example.com to the inbound endpoint. Someone later creates an outbound rule forwarding example.com back to on-prem — maybe because one subdomain genuinely lives there. Now a query for a name that doesn't exist anywhere bounces: on-prem → inbound → (no answer in AWS, rule matches) → outbound → on-prem → inbound → ... until something times out. Symptoms: slow SERVFAILs, resolver ENIs busy doing nothing useful, query volume that makes no sense.
The rule that prevents it: never forward a domain to a resolver that forwards the same domain back to you. Split the namespace instead — aws.example.com lives in Route 53 and on-prem forwards it to inbound; corp.example.com lives on-prem and AWS forwards it out. Clean boundaries, no cycles. If a domain genuinely must be split at a deeper level, forward the specific subdomains, never the parent from both sides.
2. Pointing instances at on-prem DNS via DHCP option sets. It feels like the obvious shortcut: change the VPC's DHCP options to hand out 192.168.10.10 and skip the whole endpoint conversation. Then private hosted zones stop resolving, VPC endpoint and PrivateLink names stop resolving, EKS gets weird, and every DNS packet from every instance crosses your Direct Connect. Leave AmazonProvidedDNS alone and let VPC+2 do the routing via rules. The DHCP shortcut is how you trade one week of proper setup for a year of mystery incidents.
3. Security group without TCP/53. DNS uses UDP until the answer doesn't fit or DNSSEC gets involved — then it retries over TCP. If your resolver security group only allows UDP, small answers work and large ones fail. The symptom is beautiful in its cruelty: dig works, the application intermittently doesn't, and the failure correlates with which name you look up. Allow TCP and UDP 53, both directions that apply, and move on with your life.
4. Forgetting rule associations in spoke VPCs. The rule exists, the endpoint works, the hub VPC resolves fine — and the new application VPC gets NXDOMAIN. Rules don't apply Organization-wide by magic; each VPC needs the association (RAM sharing gets the rule visible to the account; associating it to the VPC is still a step). Bake the association into your VPC Terraform module so no one has to remember. Same story for private hosted zones — a PHZ answers only in VPCs it's associated with, and cross-account PHZ association is a two-step CLI dance (create-vpc-association-authorization on the zone owner, associate-vpc-with-hosted-zone on the VPC owner) that nobody discovers until it fails.
5. Under-provisioned endpoints and mystery latency. Each endpoint IP handles up to about 10,000 queries per second, and — the part people miss — VPC+2 itself caps at 1,024 packets per second per ENI. A busy EKS node can hit that limit all by itself, thanks to ndots:5 turning every lookup of api.example.com into five queries. The fix on the cluster side is NodeLocal DNSCache (and sensible ndots in pod specs); on the endpoint side, add IPs before you need them and watch the InboundQueryVolume/OutboundQueryVolume metrics. DNS latency looks exactly like application latency to everyone upstream of you.
6. Nobody can see what's being asked. Turn on Resolver query logging from day one — to S3, cheap, same pattern as flow logs (I wrote a whole piece on querying that kind of data with Athena). When something misbehaves, the query log answers in one search what "can you try nslookup again?" answers in an afternoon: what was asked, by which ENI, and what came back. The pair of flow logs + query logs is the closest thing AWS gives you to network omniscience, and together they cost less than one incident bridge call.
The mental model that makes it all stick
Resolution order inside a VPC, simplified but honest: the resolver first matches your forwarding rules (most specific domain wins), then private hosted zones and autodefined AWS names, then public recursion. When two rules could match, the longer domain name takes precedence — db.corp.internal beats corp.internal. That's the entire debugging framework: for any failing name, ask "which of the three paths should this take, and which one is it actually taking?" dig from inside the VPC plus the query logs answer the second half.
Cost, for the FinOps-minded: endpoints are billed per ENI-hour plus per-query, and the minimum HA setup (2 IPs inbound + 2 outbound) runs a few hundred dollars a year — check current pricing for your region. Compare that to a shared services architecture where one hub endpoint pair serves 22 accounts, and the per-account cost becomes a rounding error. Centralize; this is one of the clearest hub-and-spoke wins in AWS networking.
Hybrid DNS is unforgiving of improvisation and very kind to boring, explicit design: one clean namespace boundary, rules shared from a hub, TCP and UDP open, associations automated in the VPC module, query logging on. Do those things and DNS quietly stops being on your incident bingo card — which, in this line of work, is the highest compliment an infrastructure component can earn.
I write about AWS networking from the packet up — VPC, hybrid connectivity, EKS networking. Previous article: querying VPC Flow Logs with Athena. Follow along here or on LinkedIn.
Top comments (0)