Troubleshooting Kubernetes Application Failures: A Real Debugging Session

Here's a scenario every Kubernetes learner eventually runs into: your Pods are all Running, your Deployments say 1/1 or 3/3 AVAILABLE, everything looks healthy in kubectl get... and yet when you try to actually hit your app, you get nothing but silence or a flat-out connection refusal.

That's exactly the situation I found myself in on my two-node cluster (one control-plane node, one worker) while running the classic Example Voting App — vote, redis, worker, db, and result. This post walks through the actual debugging session, the dead ends I hit, and the three separate root causes I eventually tracked down. If you're new to Kubernetes networking, this is the kind of thing you'll hit constantly, so it's worth understanding the pattern rather than just memorizing the fix.

The Setup

A simple 5-component app running across 2 nodes:

$ k get nodes
NAME          STATUS   ROLES           AGE   VERSION
k8s-worker1   Ready    <none>          23h   v1.36.3
osboxes       Ready    control-plane   24h   v1.36.3

$ k get deploy
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
db       1/1     1            1           113m
redis    1/1     1            1           113m
result   1/1     1            1           113m
vote     1/1     1            1           113m
worker   3/3     3            3           113m

Everything's READY. Every Pod is Running. By every surface-level signal, this cluster is healthy. So when curl to the vote service just hung and then refused the connection, it was a good reminder that "Running" only tells you the container started — it says nothing about whether traffic can actually reach it.

The Architecture

Before diving into the debugging, it helps to see the full request path this app takes. Five components, two different NodePorts for user-facing traffic, and three internal ClusterIP hops in between:

Walking through it end to end:

1. The user hits the vote frontend (Python) from outside the cluster, through a NodePort service on 31000.
2. vote writes each vote into Redis, reached internally via a ClusterIP service — Redis never needs to be exposed outside the cluster.
3. The worker (.NET) consumes votes off Redis as they come in.
4. worker writes the processed result into Postgres (the db component), again over ClusterIP.
5. The result frontend (Node.js) pulls the current tally from Postgres.
6. The user checks the live results through a second NodePort, 31001.

This is a useful diagram to keep next to your terminal while debugging, because it tells you exactly which Service type to expect at each hop: vote and result are the only two components that should ever be reachable via NodePort — everything else (redis, worker, db) should only ever be reachable ClusterIP-to-ClusterIP, inside the cluster. That distinction is what made the three bugs below easy to isolate: I knew from this diagram alone that if vote (step 1) wasn't reachable, the problem had to be in the vote Service or Pod itself — and if redis couldn't be reached by worker (step 3), the problem had to be either a Service selector or a NetworkPolicy sitting between them.

Where To Even Start

When an app "isn't working" in Kubernetes, the traffic path you're debugging usually looks like this:

Client → NodePort/Service → Endpoints → Pod IP:containerPort → Your app

A failure can happen at any link in that chain, so the debugging flow is basically peeling back one layer at a time:

k get pods -o wide          # Are the pods actually running, and where?
k get svc                   # Do the services exist, with the right type/ports?
k describe svc <name>       # What's the selector, targetPort, and endpoint?
k get ep                    # Does the service actually have any endpoints?
k get po --show-labels      # Do the pod labels match what the service expects?

That last one ended up being the key to almost everything below. Let's go through what actually broke.

Issue #1: NodePort Connects... to Nothing

First symptom — trying to hit the vote app from outside the cluster just failed outright:

$ curl http://10.0.2.4:31000
curl: (7) Failed to connect to 10.0.2.4:31000 after 4 ms: Could not connect to server

The service existed, and it had an endpoint, so my first instinct (checking for missing endpoints) was a dead end:

$ k describe svc vote
Name:                     vote
Selector:                 app=vote
Type:                     NodePort
Port:                     vote-service  5000/TCP
TargetPort:               8080/TCP
NodePort:                 vote-service  31000/TCP
Endpoints:                192.168.194.74:8080

That all looks fine at a glance — there is an endpoint. So I went a layer deeper and checked what port the container inside the Pod was actually bound to:

$ kubectl get pod vote-6c95bd6c8-hjz4s -o jsonpath='{.spec.containers[*].ports}'
[{"containerPort":80,"name":"vote","protocol":"TCP"}]

$ kubectl logs vote-6c95bd6c8-hjz4s
[2026-08-18 12:42:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)

And there's the mismatch: the container was listening on port 80, but the Service's targetPort was set to 8080. The Service was faithfully forwarding traffic to a port nothing was listening on — which from the outside just looks like a dead connection.

The fix was a one-line edit to the Service spec:

$ k edit svc vote
# change targetPort from 8080 to 80
service/vote edited

$ kubectl describe svc vote
TargetPort:               80/TCP
Endpoints:                192.168.194.74:80

$ curl http://10.0.2.4:31000
<!DOCTYPE html>
<html>
  ...
  <h3>Cats vs Dogs!</h3>
  ...

Lesson: targetPort on a Service has to match the actual containerPort your app is listening on inside the Pod — not the port you think it should be, and not the port field on the Service itself (those are two different numbers doing two different jobs). Always check kubectl logs or the container's declared port when a Service "has an endpoint" but still won't respond.

Issue #2: A Service With Zero Endpoints At All

Next up, the result service. This one was worse — it didn't even have a broken endpoint, it had no endpoint whatsoever:

$ k describe svc result
Name:                     result
Selector:                 app=results
Type:                     NodePort
Port:                     result-service  5001/TCP
TargetPort:               80/TCP
NodePort:                 result-service  31001/TCP
Endpoints:                

An empty Endpoints: field is one of the most common Kubernetes networking bugs, and it always means the same thing: the Service's selector doesn't match any Pod's labels. Kubernetes isn't "broken" here — it's doing exactly what it's told, which is "find me Pods with these exact labels." If nothing matches, it just quietly gives you nothing.

So I checked the actual Pod labels:

$ k get po --show-labels
NAME                     LABELS
result-58c64976b-7qqh4   app=result,pod-template-hash=58c64976b

Found it. The Service selector was looking for app=results (plural), but the Pod was labeled app=result (singular). One character, zero traffic. This is an incredibly easy typo to make and an incredibly hard one to see just by glancing at YAML — describe output puts it right in front of you.

Fix: edit the Service selector to match the Pod's real label.

$ k edit svc result
service/result edited

$ k get ep
NAME     ENDPOINTS
result   192.168.194.83:80

Endpoint populated immediately — no Pod restart needed, since the Service just watches for matching labels live.

Issue #3: NetworkPolicies Fail Silently Too

At this point vote and result were both reachable, but vote still couldn't talk to redis internally — a NetworkPolicy was involved:

$ k describe netpol access-redis
Name:         access-redis
Spec:
  PodSelector:     app=redis
  Allowing ingress traffic:
    From:
      PodSelector: app=frontend
  Policy Types: Ingress

This policy says: "Only allow ingress to redis Pods from Pods labeled app=frontend." Sounds reasonable — except nothing in this cluster was labeled app=frontend. The actual frontend-facing app was labeled app=vote:

$ k get po --show-labels | grep vote
vote-6c95bd6c8-hjz4s   app=vote,pod-template-hash=6c95bd6c8

Same root cause as Issue #2 — a selector referencing a label that doesn't exist anywhere in the cluster — just applied to a NetworkPolicy instead of a Service. That's worth internalizing: NetworkPolicies are also just label selectors under the hood, and they fail the exact same silent way. There's no error, no event, no warning. Traffic is simply dropped, and unless you specifically go look at k get netpol you might never think to check it.

$ k edit netpol access-redis
# change podSelector under "From" to app=vote
networkpolicy.networking.k8s.io/access-redis edited

A Quick Debugging Checklist

If your app is deployed, Pods are Running, but traffic isn't flowing, walk this list roughly top to bottom:

k get pods -o wide                              # Pods running? Which node/IP?
k get svc                                        # Service exists, right type/ports?
k describe svc <svc-name>                        # selector, targetPort, endpoints
k get ep                                         # any endpoints at all?
k get po --show-labels                           # do labels actually match the selector?
kubectl logs <pod>                               # is the app listening where you expect?
kubectl get pod <pod> -o jsonpath='{.spec.containers[*].ports}'  # real containerPort
k get netpol                                     # any policies restricting traffic?
k describe netpol <name>                         # does its selector match real Pods?

A couple of things I ran into along the way that are worth a quick mention: some minimal app images don't ship with ss, netstat, or even much of a shell (exec: "ss": executable file not found in $PATH was a fun one), so don't assume you can always debug from inside the container — sometimes kubectl logs and the Pod's declared ports are more reliable than trying to exec in. Also, kubectl get endpoints is deprecated in favor of discovery.k8s.io/v1 EndpointSlice from v1.33 onward, so kubectl get endpointslice -l kubernetes.io/service-name=<svc> is the more future-proof command going forward.

None of these were exotic bugs — a wrong port number, a pluralized label, and a stale selector in a policy. But that's honestly the nature of most Kubernetes networking issues: small, easy-to-miss mismatches between YAML files that Kubernetes will never flag as an "error," because as far as the control plane is concerned, you asked for exactly what you got. Learning to methodically walk the Service → Endpoints → Pod chain is what turns "my app doesn't work and I have no idea why" into a two-minute fix.