DEV Community

Cover image for Kubernetes Networking [Level-4: Services Discovery/DNS]
ADITYA RAJ
ADITYA RAJ

Posted on

Kubernetes Networking [Level-4: Services Discovery/DNS]

#docker #linux #container #kubernetes

This is Level 4 of our Kubernetes networking series. In Level 3, we introduced the Service — a stable virtual IP that solves the problem of ever-changing Pod IPs. But that left one question unanswered:

How does an application find a Kubernetes Service without hardcoding its IP address?

Instead of applications relying on this:

Frontend → 10.96.120.50:80
Enter fullscreen mode Exit fullscreen mode

we want something far more human-friendly:

Frontend → backend:80
Enter fullscreen mode Exit fullscreen mode

That's exactly what Service Discovery solves, and in Kubernetes, it's built on top of DNS.

Table of Contents

  1. The Problem: Hardcoded IPs Are Fragile
  2. What Is DNS?
  3. Who Provides DNS in Kubernetes? CoreDNS
  4. Where CoreDNS Actually Runs
  5. How a Pod Knows Where DNS Is
  6. The Complete DNS Flow
  7. Why You Can Just Write "backend"
  8. The Kubernetes Service DNS Name Format
  9. Why Namespaces Matter for DNS
  10. Cross-Namespace Service Access
  11. A Full DNS Name Example
  12. DNS Doesn't Directly Find the Pod
  13. DNS and Service Are Different Jobs
  14. Hands-On: Testing DNS Yourself
  15. Using nslookup for Debugging
  16. A Layered Debugging Sequence
  17. When DNS Works But the App Still Fails
  18. Headless Services
  19. Normal Service vs Headless Service
  20. What About Pod DNS?
  21. Understanding Search Domains
  22. The Complete Request, Step by Step
  23. The Mental Model to Memorize
  24. Interview Questions
  25. Level 4 Checkpoint
  26. What's Next: Ingress and Gateway API

The Problem: Hardcoded IPs Are Fragile

Recall our Service from Level 3:

Service: backend
ClusterIP: 10.96.120.50
Enter fullscreen mode Exit fullscreen mode

We could technically configure the frontend to call http://10.96.120.50:80 directly — but that's brittle. What happens if the Service gets recreated and its ClusterIP changes? The application shouldn't have to care.

What we really want is to write:

http://backend:80
Enter fullscreen mode Exit fullscreen mode

Kubernetes makes this possible using DNS.

What Is DNS?

At its core, DNS does one simple thing: it converts names into IP addresses. On the public internet, this looks like:

google.com → IP address
Enter fullscreen mode Exit fullscreen mode

Kubernetes applies the exact same idea inside the cluster:

backend → 10.96.120.50
Enter fullscreen mode Exit fullscreen mode

So the basic flow is:

Application
    | "backend"
    ↓
   DNS
    | "10.96.120.50"
    ↓
 Service
Enter fullscreen mode Exit fullscreen mode

Who Provides DNS in Kubernetes? CoreDNS

Kubernetes clusters typically run CoreDNS — think of it as the cluster's built-in DNS server.

                 Kubernetes Cluster

Frontend Pod
     |
     | DNS query: "backend"
     ↓
 CoreDNS
     |
     | answer: 10.96.120.50
     ↓
 Service
Enter fullscreen mode Exit fullscreen mode

You don't need to understand CoreDNS internals yet. For now, just remember: CoreDNS = the Kubernetes cluster's DNS server.

Where CoreDNS Actually Runs

CoreDNS itself runs as regular Pods inside your cluster. Check it out:

kubectl get pods -n kube-system
Enter fullscreen mode Exit fullscreen mode

You'll typically see something like:

coredns-xxxxx
coredns-yyyyy
Enter fullscreen mode Exit fullscreen mode

Interestingly, CoreDNS is itself exposed through a Kubernetes Service:

kubectl get svc -n kube-system
Enter fullscreen mode Exit fullscreen mode

You'll likely see a Service named kube-dns — this name sticks around for historical reasons even when CoreDNS (not the older kube-dns implementation) is what's actually running behind it.

How a Pod Knows Where DNS Is

When Kubernetes creates a Pod, it automatically configures DNS settings inside it. You can inspect this yourself:

kubectl exec -it <pod-name> -- cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode

You'll typically see something like:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
Enter fullscreen mode Exit fullscreen mode

The exact IP depends on your cluster, but the key line is:

nameserver <cluster-DNS-IP>
Enter fullscreen mode Exit fullscreen mode

This tells the Pod: "Whenever you need to resolve a name, ask this DNS server."

The Complete DNS Flow

Suppose we have a Frontend Pod and a Service named backend with ClusterIP 10.96.120.50. The frontend runs:

curl http://backend
Enter fullscreen mode Exit fullscreen mode

The application needs to answer: what does backend actually point to? Here's the full chain:

Application
     | DNS lookup: backend
     ↓
/etc/resolv.conf
     ↓
CoreDNS
     ↓
10.96.120.50
     ↓
Service
     ↓
Backend Pod
Enter fullscreen mode Exit fullscreen mode

That entire sequence is Kubernetes Service Discovery in action.

Why You Can Just Write "backend"

This works thanks to Kubernetes DNS search domains. If your Pod lives in the default namespace and your Service is named backend, a query for just backend gets expanded automatically through the configured search domains:

backend
backend.default
backend.default.svc
backend.default.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Eventually, one of these resolves to the Service's full DNS name — which is why you rarely need to type the whole thing out.

The Kubernetes Service DNS Name Format

Every Kubernetes Service gets a predictable, standard DNS name:

<service>.<namespace>.svc.<cluster-domain>
Enter fullscreen mode Exit fullscreen mode

The cluster domain is usually cluster.local. So a Service named backend in the default namespace becomes:

backend.default.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Breaking it down:

  • backend → the Service name
  • default → the namespace
  • svc → indicates this is a Service
  • cluster.local → the cluster's DNS domain

Put together, backend.default.svc.cluster.local unambiguously means "the Service backend in namespace default."

Why Namespaces Matter for DNS

Imagine two namespaces — default and production — both containing a Service named backend:

default:    backend
production: backend
Enter fullscreen mode Exit fullscreen mode

So which one does a plain backend refer to? By default, a Pod resolves short names relative to its own namespace.

  • A Pod in default resolving backend effectively means backend.default.svc.cluster.local.
  • A Pod in production resolving backend effectively means backend.production.svc.cluster.local.

This is exactly why namespaces matter for Service discovery — they prevent naming collisions across teams and environments.

Cross-Namespace Service Access

Suppose a Frontend Pod living in the frontend namespace needs to talk to a Backend Service living in the backend namespace. You'd use:

backend.backend.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

Notice the two different meanings of "backend" here — the first backend is the Service name, and the second backend is the namespace. Together they resolve to "the Service named backend, in the namespace named backend."

A Full DNS Name Example

Let's use a clearer example. Suppose:

Service:   payments
Namespace: production
Enter fullscreen mode Exit fullscreen mode

Its full DNS name is:

payments.production.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

A frontend could call it in a few different ways:

curl http://payments.production.svc.cluster.local
curl http://payments.production
curl http://payments   # only if the caller is already in the "production" namespace
Enter fullscreen mode Exit fullscreen mode

The fully qualified name is always the safest and clearest option, especially across namespaces.

DNS Doesn't Directly Find the Pod

Here's a subtle but crucial point. When you resolve backend, DNS gives you the Service's ClusterIP — not a Pod IP:

backend → 10.96.120.50
Enter fullscreen mode Exit fullscreen mode

From there, a completely separate mechanism takes over:

10.96.120.50
      ↓
Service dataplane
      ↓
Backend Pod
Enter fullscreen mode Exit fullscreen mode

So the full picture is:

DNS → Service IP → Service routing → Pod
Enter fullscreen mode Exit fullscreen mode

Don't confuse DNS resolution with Service traffic forwarding — they're two distinct steps handled by two distinct systems.

DNS and Service Are Different Jobs

This distinction is worth memorizing permanently:

DNS answers: "What IP belongs to this name?"

backend → 10.96.120.50
Enter fullscreen mode Exit fullscreen mode

Service answers: "Where should traffic sent to this IP actually go?"

10.96.120.50 → Pod A / Pod B / Pod C
Enter fullscreen mode Exit fullscreen mode

Chained together:

Name → DNS → Service IP → Service dataplane → Pod
Enter fullscreen mode Exit fullscreen mode

Hands-On: Testing DNS Yourself

Let's verify all of this with a real Pod. Create a simple test Pod:

kubectl run test --image=busybox:1.36 --restart=Never -- sleep 3600
Enter fullscreen mode Exit fullscreen mode

Check it's running:

kubectl get pod test
Enter fullscreen mode Exit fullscreen mode

Shell into it:

kubectl exec -it test -- sh
Enter fullscreen mode Exit fullscreen mode

Inside the Pod, inspect its DNS configuration:

cat /etc/resolv.conf
Enter fullscreen mode Exit fullscreen mode

Then try resolving the built-in Kubernetes API Service:

nslookup kubernetes
Enter fullscreen mode Exit fullscreen mode

You should get back a Service IP. If you created the backend Service from Level 3, try that too:

nslookup backend
Enter fullscreen mode Exit fullscreen mode

Using nslookup for Debugging

nslookup is one of the most useful tools for troubleshooting DNS issues:

nslookup backend
Enter fullscreen mode Exit fullscreen mode

Conceptually, this asks: "What is the IP of backend?" — and gets back an answer like 10.96.120.50. Once you have that answer, curl http://backend is really just using this same resolution behind the scenes.

A Layered Debugging Sequence

When curl http://backend fails, don't immediately blame "the network." Break the problem down layer by layer instead:

Step 1 — Does the Service exist?

kubectl get svc
Enter fullscreen mode Exit fullscreen mode

Step 2 — Does the Service have live endpoints?

kubectl get endpointslices
Enter fullscreen mode Exit fullscreen mode

Step 3 — Does DNS resolve the name?

nslookup backend
Enter fullscreen mode Exit fullscreen mode

Step 4 — Does the resolved Service IP respond? Try connecting directly to the IP returned above.

Step 5 — Can the backend Pod itself be reached? Try connecting directly using the Pod's own IP.

This structured approach cleanly separates a DNS problem from a Service problem from a Pod networking problem — an extremely useful skill for real production debugging.

When DNS Works But the App Still Fails

Suppose nslookup backend correctly returns 10.96.120.50 — DNS is clearly working. But curl http://backend still fails. Where's the problem?

DNS       ✅
  ↓
Service   ❓
  ↓
Endpoint  ❓
  ↓
Pod       ❓
Enter fullscreen mode Exit fullscreen mode

The key insight: DNS success does not mean application connectivity is working. It only confirms that name resolution succeeded — the actual request still has to travel through the Service dataplane and reach a healthy Pod.

Headless Services

Now for an important special case. Normally, a Service gets a ClusterIP:

backend → 10.96.120.50
Enter fullscreen mode Exit fullscreen mode

But sometimes you don't want a virtual ClusterIP at all. For that, Kubernetes offers headless Services, created using clusterIP: None:

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  clusterIP: None
  selector:
    app: backend
  ports:
    - port: 80
Enter fullscreen mode Exit fullscreen mode

With a headless Service, DNS behaves differently: instead of returning a single virtual IP, it returns the individual IPs of every matching Pod:

backend
   ↓
DNS
   ↓
10.244.1.5
10.244.2.7
10.244.2.9
Enter fullscreen mode Exit fullscreen mode

This is especially useful for applications that need to know about individual backend instances directly — common examples include databases, distributed systems, and other stateful applications. We'll revisit this when we cover StatefulSets in a later part of the series.

Normal Service vs Headless Service

Normal Service Headless Service
ClusterIP Yes No
clusterIP field Virtual IP None
DNS resolves to Service IP Individual Pod endpoints
Typical use Normal application access Direct endpoint discovery

Mental picture — normal Service:

DNS → 10.96.x.x → Service → Pods
Enter fullscreen mode Exit fullscreen mode

Headless Service:

DNS → Pod A, Pod B, Pod C (directly)
Enter fullscreen mode Exit fullscreen mode

What About Pod DNS?

Kubernetes can also provide DNS names for individual Pods under certain configurations. However, for standard application-to-application communication:

Prefer using a Service over relying on individual Pod DNS names or IPs.

Why? Because Pods are inherently temporary:

Pod A (10.244.1.5) dies → Pod B (10.244.4.9) takes over
Enter fullscreen mode Exit fullscreen mode

A Service hides all of that churn behind a stable name and IP — exactly the abstraction you want your applications depending on.

Understanding Search Domains

Revisit /etc/resolv.conf once more:

search default.svc.cluster.local svc.cluster.local cluster.local
Enter fullscreen mode Exit fullscreen mode

This search line is what allows short names to "just work." A query for backend gets automatically expanded through each of these search domains until one of them resolves successfully. This is precisely why Kubernetes applications can conveniently write:

http://backend
Enter fullscreen mode Exit fullscreen mode

instead of always spelling out the full:

http://backend.default.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

The Complete Request, Step by Step

Let's trace one full request end-to-end. The frontend runs:

curl http://backend
Enter fullscreen mode Exit fullscreen mode

Step 1 — Application asks DNS: "What is backend?"

Step 2 — Pod's DNS configuration: /etc/resolv.conf points the query at the cluster's DNS server.

Step 3 — CoreDNS resolves it: CoreDNS knows backend.default.svc.cluster.local maps to 10.96.120.50.

Step 4 — Application receives the IP: backend → 10.96.120.50.

Step 5 — Application sends traffic: to 10.96.120.50:80.

Step 6 — Service dataplane takes over: traffic gets forwarded to an appropriate backend Pod.

10.96.120.50 → Pod 1 / Pod 2 / Pod 3
Enter fullscreen mode Exit fullscreen mode

Putting the whole picture together:

┌──────────────┐
│ Frontend Pod │
└──────┬───────┘
       │ "backend"
       ↓
┌──────────────┐
│   CoreDNS    │
└──────┬───────┘
       │ 10.96.120.50
       ↓
┌──────────────┐
│   Service    │
└──────┬───────┘
       │
   ┌───┼────┐
   ↓   ↓    ↓
 Pod1 Pod2 Pod3
Enter fullscreen mode Exit fullscreen mode

The Mental Model to Memorize

This entire article boils down to one chain:

Application
    | "backend"
    ↓
   DNS
    | "10.96.120.50"
    ↓
 Service
    | selects backend
    ↓
 EndpointSlice
    ↓
  Pod
Enter fullscreen mode Exit fullscreen mode

Or even more compactly:

NAME → DNS → SERVICE IP → POD
Enter fullscreen mode Exit fullscreen mode

Interview Questions

Q1. What is CoreDNS?
The DNS server commonly used inside Kubernetes clusters.

Q2. Why do we need DNS in Kubernetes?
So applications can use stable, human-readable names instead of hardcoding IP addresses that can change.

Q3. What is the standard DNS format for a Service?
<service>.<namespace>.svc.<cluster-domain> — for example, backend.default.svc.cluster.local.

Q4. What does a Service name usually resolve to?
For a normal ClusterIP Service, its virtual ClusterIP.

Q5. Does DNS send traffic to the Pod?
No — DNS only resolves the name. The Service dataplane handles the actual traffic forwarding afterward.

Q6. What is a headless Service?
A Service configured with clusterIP: None. It skips the normal virtual ClusterIP and instead allows DNS-based discovery of individual Pod endpoints.

Q7. Where can you check a Pod's DNS configuration?
kubectl exec <pod> -- cat /etc/resolv.conf

Q8. How do you test DNS resolution from inside a Pod?
kubectl exec -it <pod> -- nslookup backend

Level 4 Checkpoint

By now, you should be comfortable explaining this entire chain:

backend
   ↓
CoreDNS
   ↓
10.96.120.50
   ↓
Service
   ↓
EndpointSlice
   ↓
Backend Pod
Enter fullscreen mode Exit fullscreen mode

And you should be able to clearly state the difference:

DNS     = "name → IP"
Service = "stable IP → backend Pods"
Enter fullscreen mode Exit fullscreen mode

What's Next: Ingress and Gateway API

In Level 5, we'll tackle the next natural question:

I have a Service inside Kubernetes. How does a user on the public internet reach my application using a domain like app.example.com?

We'll cover Ingress, Ingress Controllers, the Gateway API, host/path-based routing, TLS termination, and the full traffic flow from Internet → Load Balancer → Ingress/Gateway → Service → Pod — while still saving the low-level dataplane internals for Level 8.

Conclusion

Service Discovery is what makes Kubernetes applications feel effortless to configure: instead of chasing down ever-changing IP addresses, you just use a name like backend, and CoreDNS quietly resolves it to a stable Service IP behind the scenes. Understanding that DNS and Service routing are two separate, sequential steps — name resolution first, traffic forwarding second — is the key to debugging connectivity issues quickly and confidently.

The next time something doesn't connect, resist the urge to guess. Walk the layers: check the Service, check the EndpointSlice, check DNS with nslookup, then check the Pod itself. That habit alone will save you hours of confused troubleshooting.

Enjoyed this article? Follow along for Level 5, where we finally expose these Services to the outside world using Ingress and the Gateway API. Drop your questions in the comments, and share this with a teammate who's still typing raw ClusterIPs into their code. Let's keep building this Kubernetes networking roadmap together. 🚀

Top comments (0)