Nitesh More

Posted on Aug 17

Automating DNS with ExternalDNS on EKS and Istio: Lessons From Real-World Gotchas

#devops #sre #dns #istio

When I first set up Istio on Amazon EKS, I thought DNS automation would be the easy part. Just install ExternalDNS, connect it to Route 53, and boom — api.yourdomain.com and grafana.yourdomain.com should resolve to my cluster.

Most of it did work smoothly. The EKS add-on installed fine, IAM permissions were clear from the docs, and Route 53 was wired up. But there was one thing missing in the documentation that cost me a lot of time: how to make ExternalDNS notice Istio hostnames.

In this post, I’ll explain:

How ExternalDNS, cert-manager, and Istio fit together
The exact setup I used
The one gotcha that slowed me down (annotations!)
A debugging checklist you can use when DNS records don’t appear

🧩 The Basics

🔹 ExternalDNS

A Kubernetes controller that creates and updates DNS records automatically in providers like Route 53. Annotate a Service or Ingress with a hostname, and ExternalDNS will manage the record for you.

🔹 cert-manager

cert-manager is a Kubernetes add-on that automates the management of TLS certificates. It integrates with Let’s Encrypt, Vault, or private CAs to issue and renew certificates automatically.

At its core, cert-manager uses two main components:

Issuer/ClusterIssuer → defines where certificates are requested from (e.g., Let’s Encrypt).

Certificate → defines what domain you want the cert for (e.g., *.yourdomain.com).

In my setup, I used it with a wildcard certificate (*.yourdomain.com) from Let’s Encrypt so all subdomains were automatically covered.

🔹 Istio

A service mesh that adds traffic routing and security. The Istio Gateway terminates TLS and forwards traffic based on the requested host (api.yourdomain.com, grafana.yourdomain.com, etc.).

📐 Architecture

EKS cluster with Istio installed
Istio ingressgateway (type LoadBalancer) → backed by an AWS NLB
ExternalDNS EKS add-on → manages Route 53 records
Route 53 hosted zone for yourdomain.com
cert-manager → wildcard TLS cert
VirtualServices → route subdomains to the right service

Flow:

⚙️ Setup

1. Install ExternalDNS Add-On

With EKS it’s easier to use the managed add-on. Example config:

provider: aws
policy: upsert-only
txtOwnerId: archops
domainFilters:
  - yourdomain.com
sources:
  - service
  - istio-gateway

2. Annotate the Istio IngressGateway

This was the missing piece. ExternalDNS does not see Istio VirtualServices directly. You need to annotate the Istio ingressgateway Service with the subdomains you want managed:

kubectl -n istio-system annotate svc istio-ingressgateway \
  external-dns.alpha.kubernetes.io/hostname="api.yourdomain.com,grafana.yourdomain.com" \
  --overwrite

Once I did this, ExternalDNS created the records in Route 53 instantly.

3. Istio Gateway + VirtualServices

After ExternalDNS creates the DNS records, Istio takes over to handle TLS termination and routing based on host headers.

First, you need a Gateway that uses the wildcard TLS certificate from cert-manager. Then you create separate VirtualService objects for each subdomain.

Gateway with Wildcard Certificate

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway    # use istio’s default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: wildcard-yourdomain-com   # cert-manager Secret
    hosts:
    - api.yourdomain.com
    - grafana.yourdomain.com

VirtualService for API

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-vs
  namespace: api
spec:
  hosts:
  - api.yourdomain.com
  gateways:
  - istio-system/my-gateway
  http:
  - route:
    - destination:
        host: api-svc.api.svc.cluster.local
        port:
          number: 8080

…and similar VirtualService for Grafana just change host, service name and port.

🚧 The Gotcha: Annotations

Everything else was well-documented, but this part wasn’t obvious:

ExternalDNS doesn’t read Istio VirtualServices
Without annotations, no DNS records get created
Adding external-dns.alpha.kubernetes.io/hostname to the ingressgateway Service solved it

That was the one real stumbling block for me.

🔍 Debugging Checklist

If your DNS records don’t show up, here’s what to check:

Logs
check logs using kubectl logs deploy/external-dns -n kube-system
Domain filter
Make sure domainFilter matches your hosted zone exactly (yourdomain.com).
IAM permissions
ExternalDNS service account needs ChangeResourceRecordSets and ListHostedZones.
Stale TXT records
Sometimes old TXT entries block updates. Clean them in Route 53.
Annotations
ExternalDNS discovers hostnames in different ways:

If you use a standard Kubernetes Ingress, the spec.rules.host field is enough.
If you expose a Service of type LoadBalancer (like Istio’s ingressgateway), add external-dns.alpha.kubernetes.io/hostname with the hostnames.

❌ Why I Didn’t Use AWS Load Balancer Controller

Quick note: I skipped AWS Load Balancer Controller because:

Istio already handled L7 routing
We needed TCP/gRPC in addition to HTTP
TLS was managed inside Istio
One NLB was simpler and cheaper

✅ Final Thoughts

ExternalDNS, cert-manager, and Istio work beautifully together — once you know the trick with annotations.

Now I can add new subdomains like api.yourdomain.com or grafana.yourdomain.com without touching Route 53. ExternalDNS manages the records, cert-manager handles TLS, and Istio routes traffic where it belongs.

Hopefully, this walkthrough saves you the debugging time I went through!

💡 Have you run into similar DNS headaches with Istio on EKS? Share your story — I’d love to compare notes.

📬 Connect with me on LinkedIn

DEV Community