DEV Community

Metronom
Metronom

Posted on

Kubernetes Service 'Endpoints: <none>': The One Mismatched Label That Drops All Traffic

The service was deployed. The Pod was Running. The Service existed. Zero traffic reached it. When a Kubernetes Service shows Endpoints: <none>, it is not broken — it's matching nothing, because its selector doesn't match the pod labels. The fix was one hyphen in one YAML file. Understanding why changed how I read every manifest since.

Symptom

Three manifests, the ones everyone starts with: a Namespace, a Deployment, a Service. Applied clean. Then a curl through the Service:

$ kubectl apply -f ./k8s/
namespace/myapp created
deployment.apps/myapp created
service/myapp created

$ curl http://myapp.myapp.svc.cluster.local
curl: (7) Failed to connect: Connection refused
Enter fullscreen mode Exit fullscreen mode

No error on apply. Pod healthy. Logs clean. Nothing to grab onto. I'd copied fragments from three tutorials and somewhere in the stitching a label drifted. There's a solid breakdown of the minimal skeleton and how labels glue it together at this manifests walkthrough.

Root cause

The thing nobody had made click: nothing connects these objects with hard references. No "Service, here are the pod IDs." It's all labels — arbitrary key/value tags — and label selectors matching against them, per the Kubernetes labels-and-selectors concept. Three separate things reference the same app: myapp label:

  1. The Deployment stamps app: myapp onto every pod it creates via spec.template.metadata.labels.
  2. The same Deployment uses spec.selector.matchLabels: {app: myapp} to recognize those pods as its own.
  3. The Service uses its own spec.selector: {app: myapp} to find pods, collect their addresses into an EndpointSlice, and balance traffic — the Service docs describe the controller continuously scanning for matching pods.

Two selectors, same pod labels, two problems, and no schema anywhere enforcing that they agree — the string just has to match, character for character. If the Service's selector points at a label no pod has, the Service binds to nothing. It exists, it has an IP, traffic goes nowhere.

The fix

Diagnosis was instant:

$ kubectl describe svc myapp -n myapp
...
Endpoints:   <none>
Enter fullscreen mode Exit fullscreen mode

The Deployment stamped app: myapp; I'd typed app: my-app in the Service selector. One hyphen. Endpoints: <none> means the pod label and the Service selector didn't match — full stop. Fixed the selector, re-applied, endpoints populated immediately:

Endpoints:   10.42.0.14:8080,10.42.0.15:8080
Enter fullscreen mode Exit fullscreen mode

A related rule that would've caught me later, and the Deployment docs make it hard: spec.selector.matchLabels must match spec.template.metadata.labels, or Kubernetes rejects the manifest at apply time. Different failure (loud, not silent), same root cause — labels that don't line up.

The runbook I use now for "Service, but no traffic"

Whenever a Service exists and nothing reaches it, I run this in order. It localizes the break in under a minute, every time:

# 1. Does the Service have any endpoints at all?
kubectl get endpointslices -n myapp -l kubernetes.io/service-name=myapp
#    empty -> selector matches nothing. Stop here, fix labels.

# 2. What selector is the Service actually using?
kubectl get svc myapp -n myapp -o jsonpath='{.spec.selector}'; echo

# 3. What labels do the pods actually carry?
kubectl get pods -n myapp --show-labels

# 4. Do they intersect? Ask the API directly with the Service's selector.
kubectl get pods -n myapp -l app=myapp
#    no pods listed -> confirmed mismatch between (2) and (3)
Enter fullscreen mode Exit fullscreen mode

Steps 2 and 3 side by side make the mismatch obvious — my-app next to myapp jumps out once you're looking at both. Before I had this list I'd start by tcpdumping the pod network, which is exactly the wrong end of the problem.

The other trap: ports that don't mean what you think

While in there, I finally understood the port fields I'd been cargo-culting:

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: myapp
spec:
  selector:
    app: myapp
  ports:
    - protocol: TCP
      port: 80          # the port the Service listens on
      targetPort: 8080  # the port on the pod it forwards to
Enter fullscreen mode Exit fullscreen mode

port is where cluster clients connect (80). targetPort is the pod's actual port (8080). Callers hit myapp on 80; the app only ever listens on 8080; the Service translates. Also why kubectl port-forward svc/myapp 8080:80 and port-forward pod/... 8080:8080 use different numbers — one targets the Service port, the other the container port.

Before / after

Before After
Symptom Service exists, no traffic, no error traffic flows
Root cause selector my-app vs label myapp labels aligned
Diagnosis hours of guessing describe svcEndpoints: <none>
Mental model "objects are wired together" "labels are the only glue"
Manifests pasted from 3 tutorials one reviewed k8s/ folder in git

The namespace traps hiding underneath

Once labels made sense, a second layer did too — all about namespaces. Objects live in a dedicated myapp namespace, not defaultnamespaces make it easier to delete everything at once, attach limits, and avoid colliding with other teams. But namespaces shape DNS. Services get a name of the form <service>.<namespace>.svc.cluster.local, per the DNS-for-Services spec. From the same namespace the short name myapp works; from a different namespace you need the FQDN myapp.myapp.svc.cluster.local. A job in another namespace "couldn't reach" the service purely because it used the short name — looks like a networking failure, is really a DNS-scope misunderstanding. The tell is that the error is DNS resolution, not connection refused: could not translate host name "myapp" means the name never resolved, so the packet never left. nslookup myapp.myapp.svc.cluster.local from inside a debug pod confirms which form the cluster actually answers.

Guardrail

  • Endpoints: <none> = label/selector mismatch. First thing to check when a Service "works" but nothing reaches it.
  • Deployment selector must equal template labels, or apply is rejected outright.
  • Forgetting -n myapp makes kubectl get look in default and report nothing — the "where did everything go?" panic. Pin it once: kubectl config set-context --current --namespace=myapp.
  • port vs targetPort are not interchangeable. Backwards and traffic dies at the pod.
  • Two selector syntaxes, same meaning. The Service writes flat selector: {app: myapp} (equality-based); the Deployment uses selector.matchLabels (which can also do matchExpressions). Different syntax generations, both compare labels.
  • kubectl diff -f deployment.yaml before every apply. apply is declarative and idempotent, easy to run blind; diff catches an accidental label or replica edit before the cluster does.
  • A green apply proves nothing about routing. The three objects can each be valid and still not connect. Endpoints are the only proof that the wiring worked.

What I'd do differently

The deeper problem wasn't the hyphen — it was why I had one. My manifests were fragments on a laptop, applied from the desktop, no history, no review. Now they live in a flat k8s/ folder at the repo root next to the code, applied as a directory with kubectl apply -f ./k8s/, and every label change goes through a PR where a mismatch like mine is a one-line diff a teammate spots. apply (declarative) beats create (imperative) for anything in version control — create fails if the resource exists and can't update, useless for repeatable manifests — and the declarative-config guide covers the reasoning end to end. I'd have started there instead of pasting from three tutorials.

Bottom line: when a Service drops all traffic with no error, run kubectl describe svc and read the Endpoints line — <none> is a label/selector mismatch every time, not a networking failure.

Sources

Top comments (0)