DEV Community

Sagar
Sagar

Posted on

Pods Running, Service returns 503: the selector bug that fools everyone

#kubernetes #debugging #devops #sre

Every request to the app fails with a 503. The body says no available server. And yet kubectl get pods looks perfect:

$ curl -i http://go-api.snowops.localhost/health
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 12 Oct 2026 14:25:45 GMT
Content-Length: 20

no available server

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-5d8497557b-7pfhz   1/1     Running   0          4m42s
Enter fullscreen mode Exit fullscreen mode

One of one ready, zero restarts, nothing deployed. I broke a small Go service on purpose to walk through this one, because it fools people precisely by looking healthy everywhere you'd normally look. The commands below work on any cluster. The cause in my drill is one renamed label, and at the end I'll go through the other things that leave a Service with nowhere to send traffic.

The piece most people never check

A request from outside reaches a pod in three hops. The ingress controller receives it and looks up the Service the route points at. The Service doesn't forward anything to "the app" in general. It forwards to a list of pod addresses, its endpoints. Then the request lands on one of those pods.

That list isn't something you write by hand. A Service has a selector, a set of labels, and Kubernetes keeps the list up to date with every pod that carries all of those labels and is ready to serve. Today the list lives in EndpointSlice objects (Kubernetes docs: EndpointSlices). The older Endpoints object still exists, and kubectl get endpoints still works, but newer clusters print a warning when you use it.

Here is the healthy state, before I broke anything:

$ kubectl get endpoints go-api -n go-api
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS          AGE
go-api   10.42.0.136:8080   67s

$ kubectl get endpointslices -n go-api
NAME           ADDRESSTYPE   PORTS   ENDPOINTS     AGE
go-api-kt9b6   IPv4          8080    10.42.0.136   82s
Enter fullscreen mode Exit fullscreen mode

One address, the pod's IP, on port 8080. Keep that picture in mind. If the list is empty, the ingress has a route and a Service and nowhere to send the request. That is exactly what no available server means. It comes from Traefik, the ingress controller in this lab, not from the app.

What the outage looks like from the inside

Before going to the terminal, I looked at the dashboard. A load generator (k6) had been sending the app steady traffic from inside the cluster, so there's a before and an after.

This was the part that surprised me most:

Grafana during the outage: the app's own 5xx error rate stays at zero while k6's failed request rate climbs to 100%

Grafana during the outage: the app's own 5xx error rate stays at zero while k6's failed request rate climbs to 100%

On the left, the app's own error rate. Flat at zero for the whole outage. On the right, what the client saw: every request failing. Both panels are telling the truth. No request reached the pod, so the pod had nothing to count, good or bad. Its metrics didn't go red, they went quiet.

The panel that shows it best puts both sides on one chart:

Offered vs served: k6 keeps offering about 50 requests a second (green) while the requests the app handles (yellow) fall away

Offered vs served: k6 keeps offering about 50 requests a second (green) while the requests the app handles (yellow) fall away

Green is what k6 offered, yellow is what the app handled. Normally the two lines sit on top of each other. The gap between them is the outage.

If your alerting only looks at what a service says about itself, this outage looks like a quiet afternoon. You need at least one signal from the client side: a synthetic check, a probe from outside the cluster, or the ingress controller's own metrics. An alert that watches pods won't help either, because the pods really are fine. What does work is an alert on the thing that broke. The lab ships one, and it was firing:

Prometheus: the LabFaultServiceEndpointsEmpty rule firing, absent() over the Service's ready EndpointSlice endpoints for 1 minute

Prometheus: the LabFaultServiceEndpointsEmpty rule firing, absent() over the Service's ready EndpointSlice endpoints for 1 minute

It uses absent() rather than count(...) == 0, and the lab's rule file explains why: kube-state-metrics exports one series per endpoint address, so when there are no addresses there is no series at all, and a count would never fire. That's an easy rule to get wrong, and worth stealing.

Find the tell: an empty endpoint list

Back in the terminal, the order of checks follows the request. The pods are healthy, so check the link in the middle:

$ kubectl get endpoints go-api -n go-api
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS   AGE
go-api   <none>      7m27s

$ kubectl get endpointslices -n go-api
NAME           ADDRESSTYPE   PORTS     ENDPOINTS   AGE
go-api-kt9b6   IPv4          <unset>   <unset>     7m41s
Enter fullscreen mode Exit fullscreen mode

<none>. A few minutes ago that column held the pod's address. The EndpointSlice is still there, but it's empty. That's the tell: the Service exists, the pod exists, and the Service isn't selecting it.

Nothing else would have told me. Nothing crashed, nothing restarted, and Kubernetes emits no event when a selector stops matching, because from its point of view nothing failed. The Service is doing exactly what its spec says.

Compare the selector with the labels

-o wide prints a Service's selector in the last column:

$ kubectl get svc go-api -n go-api -o wide
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
go-api   ClusterIP   10.43.223.247   <none>        8080/TCP   8m    app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api-v2,app=go-api
Enter fullscreen mode Exit fullscreen mode

(My terminal wrapped that last line, I joined it back.) Three labels. Now the labels the pod actually carries:

$ kubectl get pods -n go-api --show-labels
NAME                      READY   STATUS    RESTARTS   AGE     LABELS
go-api-5d8497557b-7pfhz   1/1     Running   0          8m16s   app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api,app=go-api,pod-template-hash=5d8497557b
Enter fullscreen mode Exit fullscreen mode

Line them up and the difference is one value:

Service wants:  app.kubernetes.io/name=go-api-v2
Pod carries:    app.kubernetes.io/name=go-api
Enter fullscreen mode Exit fullscreen mode

Two out of three labels match, and that counts for nothing. A selector is an AND: a pod has to carry every label in it (Kubernetes docs: labels and selectors). One key off, and not one pod matches.

A small tip if you're doing this on your own cluster: I used get -o wide rather than kubectl describe svc here. describe shows the selector too, but on this lab it also prints an annotation the drill leaves behind with the original selector in it, which gives the answer away. On your cluster describe is fine.

Which side is wrong?

A mismatch has two sides, and you could make them agree from either one. The Deployment settles it. It selects its pods by labels too, and it stamps those labels onto every pod it creates:

$ kubectl get deploy go-api -n go-api -o wide
NAME     READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES          SELECTOR
go-api   1/1     1            1           8m33s   go-api       go-api:v1.2.0   app=go-api,app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api
Enter fullscreen mode Exit fullscreen mode

name=go-api. The pods are right, and the Service is the odd one out. There's a structural reason to fix it there as well: a Deployment's selector can't be changed after it's created (in apps/v1 it's immutable), while a Service's selector can be changed at any time, to anything, and Kubernetes accepts it without a warning. That's how this kind of drift gets in.

The tempting fix

The Service wants a pod labelled go-api-v2. So why not give the pod that label? It's one command:

$ kubectl label pod -n go-api go-api-5d8497557b-7pfhz app.kubernetes.io/name=go-api-v2 --overwrite
pod/go-api-5d8497557b-7pfhz labeled
Enter fullscreen mode Exit fullscreen mode

And it works, in the sense that requests flow again. The endpoint list fills back in and the health check returns 200. If you stopped here, you'd close the incident. But list the pods:

$ kubectl get pods -n go-api --show-labels
NAME                      READY   STATUS    RESTARTS   AGE     LABELS
go-api-5d8497557b-2qwx4   1/1     Running   0          18s     app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api,app=go-api,pod-template-hash=5d8497557b
go-api-5d8497557b-7pfhz   1/1     Running   0          9m16s   app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api-v2,app=go-api,pod-template-hash=5d8497557b
Enter fullscreen mode Exit fullscreen mode

There are two now. A ReplicaSet also finds its pods by their labels. The moment I relabelled the pod, it stopped matching the ReplicaSet's selector, so as far as the ReplicaSet was concerned it had lost a pod, and it made a replacement with the original labels. The docs describe this as isolating a pod from its ReplicaSet (Kubernetes docs: ReplicaSet), and it's normally used on purpose, to pull a misbehaving pod out of service for debugging.

So the Service is now held up by one orphan pod that nothing owns. If it's evicted, or its node goes away, nothing replaces it and the 503s come back. And the next rollout creates pods from the template, which still says go-api. The person doing that rollout won't connect it to today's incident.

The lab's check wasn't fooled:

!The SnowOps Labs check refuses the relabel: Not resolved yet, the Service wants "go-api-v2", pods are labelled "go-api"

The SnowOps Labs check refuses the relabel: Not resolved yet, the Service wants "go-api-v2", pods are labelled "go-api"

It compares the Service's selector with the Deployment's pod template rather than with whatever labels happen to be on a pod right now, for exactly this reason.

The real fix: change the Service

First, undo the shortcut. Nothing owns the relabelled pod, so deleting it is permanent:

$ kubectl delete pod -n go-api go-api-5d8497557b-7pfhz
pod "go-api-5d8497557b-7pfhz" deleted from go-api namespace
Enter fullscreen mode Exit fullscreen mode

Then patch the Service's selector back to what the Deployment uses:

$ kubectl -n go-api patch svc go-api -p '{"spec":{"selector":{"app.kubernetes.io/name":"go-api"}}}'
service/go-api patched
Enter fullscreen mode Exit fullscreen mode

kubectl patch sends only the fields you give it, and for a Service the selector map is merged, so the other two labels stay as they were. Only name changes.

Prove it, in the order you found it

I check in the same order I diagnosed, from the inside out. First the Service:

$ kubectl get endpoints go-api -n go-api
NAME     ENDPOINTS          AGE
go-api   10.42.2.168:8080   10m
Enter fullscreen mode Exit fullscreen mode

(I dropped the deprecation warning from here on.) An address again, and a different one from the start. It belongs to the pod the ReplicaSet created, which is the one that should be serving. Then the same question from outside:

$ curl -i http://go-api.snowops.localhost/health
HTTP/1.1 200 OK
Content-Length: 16
Content-Type: application/json
Date: Mon, 12 Oct 2026 14:32:28 GMT

{"status":"ok"}
Enter fullscreen mode Exit fullscreen mode

Same request as at the start, different answer. On the dashboard the app's request rate came back up to meet k6's, and the lab's check passed.

Try it on your own cluster

You don't need the lab for this one. Pick a Service in a test namespace, note its selector, and change one label to a value no pod carries:

kubectl get svc <your-service> -o wide
kubectl patch svc <your-service> -p '{"spec":{"selector":{"<key>":"<value no pod has>"}}}'
kubectl get endpoints <your-service>
Enter fullscreen mode Exit fullscreen mode

The endpoints go to <none> straight away. Your pods stay Running and Ready, nothing restarts, and no event warns you. Patch the label back to undo it. Only do this on a test Service: a selector change moves traffic immediately.

If you'd rather practise the whole incident, with traffic, the dashboards, the alert and a check that won't accept the relabel shortcut, it's one of the drills in SnowOps Labs, a free, open source Kubernetes lab that runs on a laptop. Disclosure: I build it. Every screenshot above is from it, and the code is on GitHub.

Other ways to end up with no endpoints

A renamed label is one cause. When kubectl get endpoints comes back empty and your pods look fine, these are the others I'd check, roughly in this order. The Kubernetes guide to debugging Services walks through the same ground.

  • The pods aren't Ready. Running isn't Ready. A pod failing its readiness probe is left out of the Service's ready endpoints, so a broken readiness probe empties the list just as well. kubectl get pods shows 0/1 in the READY column. Check that first, which is why it comes first in the recap below.
  • A typo in the label key, not the value. app.kubernetes.io/name against app.kubernetes.io/Name, or app against application. Same result, harder to spot by eye. Compare the two outputs character by character.
  • The wrong namespace. A Service only selects pods in its own namespace. A Service applied to default while the pods live in go-api has a perfect selector and no endpoints.
  • A Service with no selector at all. Then Kubernetes doesn't manage its endpoints, and they stay empty unless something creates EndpointSlices for it. That's intended for pointing a Service at something outside the cluster, and a surprise when it's a mistake.
  • Endpoints exist, but requests still fail. Then it isn't this problem. Look at the Service's targetPort against the port the container really listens on, and at what the ingress route points to.

At work, the label usually changes somewhere other than the cluster: a Helm chart refactor that renames the selector labels, or a Service copied from another app with its app name left in. Whatever produced the change, fix it there as well (the chart, the Kustomize overlay, the GitOps repo), or the next sync puts the bad selector straight back.

The short version

When users get 503s and the pods look healthy:

  1. curl -i the URL, or open it, and confirm what users see.
  2. kubectl get pods: pods that aren't Ready also empty the endpoint list, so rule that out first.
  3. kubectl get endpoints <service> (or get endpointslices): an empty list means the Service selects nothing.
  4. Compare kubectl get svc <service> -o wide with kubectl get pods --show-labels. One mismatched label is enough.
  5. Fix the Service (or whatever generates it), not the pods, and check that the endpoints come back before you check the URL.

This incident is resolved using snowops labs, which is the open source project I am building for simulating real world platform engineering scenarios. Checkout this project: snowopslabs

Top comments (1)

Collapse
 
nathan_fixes profile image
Nathan •

Good drill. One gap in the alert though: absent() catches an empty endpoint list, not a wrong one.

Ship a second deployment with one extra label (canary naming, a chart bump, a team label) and have the Service selector match it, and the EndpointSlice is non-empty, the rule stays green, and traffic goes to the wrong pods. You get partial 503s, or one pod carrying all of it, which reads as a load problem instead of a selector problem.

The assertion that catches both: count the ready pods matching the Service selector and compare it to the number of addresses in the EndpointSlice. Mismatch either way is the bug. Cheap as a probe, and it fires before the first 503 instead of after.

Why this class keeps happening, from my own version of it: the check shares the thing it's checking. Mine was a write path that derived a storage key one way and a read path that derived it another. Every test was green because the tests used the write path. A check that asks whether my config agrees with itself can't see a config that's wrong in both places. It has to come from outside what it measures, which is the same reason your offered-vs-served panel works.

Does the rule cover the partial case at all, or only zero endpoints?