Pod is Healthy, Service is Green, Users See 502: The Anatomy of a Kubernetes Ghost Outage

At 2:07 AM, the page goes off.

Users are reporting that the application is returning 502 Bad Gateway. You open Grafana expecting the usual disaster: crashed Pods, failing readiness probes, CPU at 100%, or a node running out of memory.

Nothing.

Every Pod says Running. Readiness and liveness probes have been green for hours. The Service has healthy endpoints. CPU is normal. Memory is normal. Even the ingress controller looks healthy.

You refresh the application. 502. Refresh again. 200 OK. Again. 502. This is the kind of incident that makes Kubernetes look broken.

Usually, Kubernetes isn't broken. The problem is that your dashboards are looking at the wrong layer.

Kubernetes can tell you that a Pod exists and that an HTTP health endpoint responds. It cannot, by itself, tell you that a particular TCP connection sitting inside an ingress controller's connection pool is still usable, that the application's listen() queue isn't overflowing, or that a terminating process just closed a socket while traffic was still being routed toward it.

To understand these outages, you have to go below Pods and Services.

You have to look at processes, file descriptors, sockets, TCP state, kernel queues, and syscalls.

The Nightmare Scenario: All Pods Are Green, Probes Pass, but Ingress Returns 502

Consider a simple request path:

Client
   |
   v
Load Balancer
   |
   v
Ingress / NGINX / Envoy
   |
   v
Kubernetes Service
   |
   v
Pod
   |
   v
Application process
   |
   v
Linux TCP socket

A request does not magically travel from an Ingress to a Pod.

Eventually, somebody has to perform operations equivalent to:

socket()
connect()
send()
recv()
close()

On the backend, the application eventually does something equivalent to:

socket()
bind()
listen()
accept()
recv()
send()
close()

Between those system calls sits the Linux networking stack, TCP state machines, queues, connection buffers, and the application runtime.

That is where many "healthy Pod, mysterious 502" incidents actually live. A Pod can be perfectly healthy according to Kubernetes while one particular connection between the proxy and application is completely broken.

What an HTTP 502 Really Means at the Network Layer

A 502 Bad Gateway generally means the component acting as a gateway or reverse proxy could not obtain a valid response from its upstream backend. The important word is proxy.

Suppose NGINX receives:

GET /api/users HTTP/1.1
Host: example.com

NGINX selects a backend:

Service
   |
   +---- Pod A
   +---- Pod B
   +---- Pod C

It then needs to communicate with one of those Pods.

A simplified successful exchange looks like this:

NGINX Backend
  | |
  | -------- TCP SYN ----------> |
  | <------- SYN + ACK --------- |
  | -------- ACK --------------> |
  | |
  | -------- HTTP request -----> |
  | |
  | <------- HTTP response ----- |
  | |

If the backend closes the connection unexpectedly, resets it, fails to accept the connection, or sends something the proxy cannot interpret as a valid HTTP response, the proxy may return:

HTTP/1.1 502 Bad Gateway

The critical detail is that the 502 can be generated by the proxy. The application itself may never have generated a 502.

For instance:

Client
  |
  | HTTP request
  v
NGINX
  |
  | connection reset
  X
Backend

NGINX can only tell the client:

"I couldn't successfully talk to my upstream."

So the first question during a 502 incident should not be:

"Why is my Pod unhealthy?"

It should be:

"What happened between the proxy and the upstream socket?"

That change in thinking saves a lot of debugging time.

Root Cause Breakdown

1. Race Condition: The HTTP Keep-Alive Timeout Mismatch

This is one of the nastiest causes because everything can look perfectly healthy. Imagine your ingress controller maintains a pool of persistent TCP connections to your application.

Instead of doing this for every request:

Request 1 -> TCP connection -> close
Request 2 -> TCP connection -> close
Request 3 -> TCP connection -> close

it does this:

TCP connection
      |
      +--> Request 1
      +--> Request 2
      +--> Request 3
      +--> Request 4

That's HTTP keep-alive. It reduces TCP connection setup overhead and is normal in production. But now imagine the proxy and application disagree about how long an idle connection should remain alive.

For instance:

NGINX upstream keep-alive: 60 seconds
Node.js server timeout: 30 seconds

The application is allowed to close an idle connection after 30 seconds. NGINX believes the connection can remain available for 60 seconds. Now the race looks like this:

Time →

0s Request completes
         |
         | connection becomes idle
         |
30s Application closes socket
         |
         X
         |
60s NGINX still thinks connection is reusable
         |
         | next request
         v
      stale socket
         |
         X ECONNRESET / closed connection
         |
         v
       502

This can produce an extremely confusing pattern:

200
200
200
502
200
200
502

The Pod is still running. The process is still running. The readiness probe still succeeds. But a connection in the proxy's connection pool is stale.

Why ECONNRESET matters

At the application or proxy level, you may see something similar to:

ECONNRESET
upstream prematurely closed connection
connection reset by peer
upstream timed out

ECONNRESET means the TCP connection was reset rather than completing normally. It doesn't automatically prove a keep-alive mismatch, but it is an important clue.

The fix

The basic rule is:

Don't let the proxy assume an idle upstream connection will survive longer than the backend actually keeps it alive.

For example, if the backend closes idle connections after 30 seconds, configure the proxy's reusable connection lifetime appropriately below that boundary. The exact settings depend on the proxy and application runtime.

For an NGINX-style configuration, you might encounter settings such as:

upstream backend {
    server app:8080;

    keepalive 32;
    keepalive_timeout 25s;
}

On the application side, the corresponding timeout might be configured around:

30s

The exact numbers are not universal. The important thing is that you understand which component owns each timeout. Do not blindly set everything to five minutes because "more keep-alive is better."

Longer idle connections mean more persistent sockets and more resources.

2. Kernel Queues: Sockets, SOMAXCONN, and TCP Backlog Drops

Now let's move lower. Suppose traffic suddenly increases. Your application normally receives: 100 connections/sec. Then an event sends: 5,000 connections/sec. Your Pod does not necessarily fail immediately.

Instead, connections begin accumulating in queues. When a server calls:

listen(fd, backlog);

it tells the kernel that the socket is a listening socket and provides a requested backlog.

For instance:

listen(server_fd, 128);

The actual behavior depends on the operating system and runtime, and Linux also has system-wide limits such as:

cat /proc/sys/net/core/somaxconn

You might see: 4096

SOMAXCONN places an upper bound relevant to the listen backlog requested by applications. The important mental model is:

Incoming connections
        |
        v
+----------------------+
| Linux TCP structures |
| and connection queues |
+----------------------+
        |
        v
     accept()
        |
        v
Application

If connections arrive faster than the application can accept them, queues can fill.

What happens when the application cannot keep up?

Imagine:

Traffic:
5000 connections/sec

Application accepts:
1000 connections/sec

The difference has to go somewhere. As queues fill, connection attempts can experience delays, retransmissions, or failures depending on the exact TCP state, kernel settings, workload, and application behavior.

From the proxy's perspective, that can eventually become:

connect() failed
connection timed out
connection refused
upstream unavailable

And the client may see: 502 Bad Gateway.

Again, Kubernetes might still report:

Pod: Running
Ready: True

because the process hasn't crashed. The process is simply struggling at the socket boundary.

Inspecting the socket state

Start with:

ss -lntp

For more detail:

ss -lnt

You might see:

State Recv-Q Send-Q Local Address:Port
LISTEN 128 0 0.0.0.0:8080

For a listening socket, the queue-related values can give you clues about whether connections are accumulating. You can also inspect established connections:

ss -ant

Or filter by port:

ss -ant '( sport = :8080 or dport = :8080 )'

During an incident, look for unusual growth in:

SYN-SENT
SYN-RECV
ESTAB
CLOSE-WAIT
TIME-WAIT

Each state tells a different story. For instance, a large number of:

CLOSE-WAIT

can indicate that the remote side has closed the connection but the application has not closed its local socket. That is very different from a TCP backlog problem. This is why simply looking at "number of connections" isn't enough. You want to know what state those connections are actually in.

Kernel-level tuning

You may encounter:

sysctl net.core.somaxconn

and:

sysctl net.ipv4.tcp_max_syn_backlog

These control different parts of the connection-handling path. A common mistake is to increase them blindly:

sysctl -w net.core.somaxconn=65535

and declare victory. That doesn't magically make an application capable of accepting 65,535 connections.

If the application has a slow event loop, blocked workers, exhausted file descriptors, or insufficient CPU, a larger queue can simply allow more work to accumulate.

Tune the kernel alongside the application. Also remember that Kubernetes does not make node-level kernel settings disappear. The network stack is still Linux underneath.

3. The Termination Gap: SIGTERM, Endpoint Propagation, and Inflight Traffic Drops

This is another classic source of intermittent 502s. Suppose Kubernetes wants to terminate a Pod. The lifecycle looks roughly like:

Pod termination begins
        |
        v
SIGTERM sent to process
        |
        +----------------------+
        | |
        v v
Endpoint removal Application shutdown
        | |
        v v
Traffic stops Socket closes
eventually

The problem is that these operations are not one instantaneous atomic event. There is propagation time. Imagine a request arrives just as a Pod starts terminating.

A proxy or load-balancing path may still have traffic associated with that Pod while Kubernetes networking components and endpoint consumers are converging on the new state.

Meanwhile, your application receives:

SIGTERM

and immediately does:

close(listening_socket);
exit(0);

Now imagine the timing:

T0 Pod is serving traffic

T1 Kubernetes starts termination

T2 Application receives SIGTERM

T3 Application closes listening socket

T4 Some traffic still reaches old Pod path

T5 Proxy attempts connection

T6 Connection fails/reset

T7 Proxy returns 502

The exact packet path depends on the Kubernetes networking implementation, kube-proxy mode, CNI, service topology, proxy behavior, and timing. But the underlying problem remains:

traffic draining and process shutdown must be coordinated.

Don't treat SIGTERM as "exit immediately"

A production application should interpret SIGTERM as:

"Stop accepting new work and gracefully finish what you're already doing."

Not:

"Die now."

A typical graceful shutdown sequence is:

SIGTERM
   |
   v
Stop accepting new work
   |
   v
Allow existing requests to finish
   |
   v
Close keep-alive connections
   |
   v
Close listening socket
   |
   v
Exit

Kubernetes gives you mechanisms to help with this.

For instancee:

spec:
  terminationGracePeriodSeconds: 30

You can also use a preStop lifecycle hook.

A simple example:

lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - "sleep 10"

The purpose isn't to randomly sleep because Kubernetes likes sleeping Pods. The purpose is to give the system time to drain traffic and propagate endpoint changes before the process disappears.

The correct duration depends on your environment. Ten seconds isn't a magic number. Thirty seconds isn't a magic number either. Measure the actual propagation and request-drain behavior in your cluster.

Beyond Kubernetes Abstractions: Are Syscalls and Sockets Enough?

If you are debugging this class of outage, yes, you need to get comfortable with syscalls and sockets. But raw syscalls alone aren't enough.

You need to correlate four layers:

Layer 1: Kubernetes
    Pod / Service / EndpointSlice
              |
Layer 2: Proxy
    NGINX / Envoy / HAProxy
              |
Layer 3: Linux
    sockets / TCP / queues
              |
Layer 4: Application
    Node.js / Go / Python / Java

Kubernetes tells you where traffic should go. The proxy tells you what it experienced. Linux tells you what happened to the connection. The application tells you why it behaved that way. You need all four when the failure is subtle.

Start with ss

First inspect listening sockets:

ss -lntp

Then established connections:

ss -antp

For a specific port:

ss -antp | grep ':8080'

Look for patterns rather than individual connections.

For instance:

many CLOSE-WAIT
many SYN-RECV
rapid connection churn
unexpected connection resets

These patterns can tell you whether you are dealing with application cleanup, connection pressure, or connection-establishment problems.

Use strace to See What the Process Actually Does

When the application is behaving strangely, strace can expose the system calls underneath the runtime.

For instance:

strace -f -e trace=network -p <PID>

You may see operations resembling:

socket(...)
bind(...)
listen(...)
accept4(...)
recvfrom(...)
sendto(...)
close(...)

Or something like:

accept4(...) = -1 EMFILE

That is a very different incident. EMFILE means the process has exhausted its file descriptors. The Pod can still be running but the application cannot accept new sockets.

Now the Kubernetes health dashboard suddenly makes a lot more sense: the Pod isn't dead; the application is resource-starved.

You can inspect limits with:

ulimit -n

and:

cat /proc/<PID>/limits

This is one of the reasons SRE debugging eventually moves beyond container-level metrics.

Use tcpdump When You Need the Truth

When logs disagree with reality, capture packets. Inside a suitable network namespace or node:

tcpdump -ni any host <POD_IP> and port 8080

Or:

tcpdump -ni any tcp port 8080

Now you can observe things such as:

SYN
SYN-ACK
ACK
PSH
ACK
FIN
RST

A TCP reset (RST) is especially interesting.

For instance:

Proxy Pod
  | |
  | -------- SYN ------------> |
  | <------- SYN/ACK --------- |
  | -------- ACK ------------> |
  | |
  | -------- HTTP ------------>|
  | |
  | <--------- RST ------------|
  | |
  v
502

Now you have something much more useful than:

Pod: Healthy

You have evidence that a TCP connection was reset. The next question becomes:

Who sent the reset, and why?

That's a question you can investigate.

How to Fix It Permanently

The goal isn't to make the dashboard green. The goal is to make the connection lifecycle deterministic.

Step 1: Align Proxy and Application Keep-Alive Behavior

Inventory the relevant timeouts.

For instance:

Client idle timeout
        |
Load balancer timeout
        |
Ingress keep-alive timeout
        |
Proxy upstream keep-alive
        |
Application keep-alive
        |
Application request timeout

Write the actual values down. Don't assume them.

For instance:

Proxy idle connection: 60s
Application idle timeout: 30s

That's a potential mismatch. Adjust the configuration so the proxy does not retain reusable upstream connections beyond the backend's effective lifetime. Then test it under real keep-alive traffic.

A useful test is to deliberately leave connections idle and reuse them later. You want to prove that:

idle connection
      |
      v
backend timeout
      |
      v
proxy reuse

doesn't result in a stale connection being handed to a request.

Step 2: Inspect and Tune Socket Queues

Check:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog

Inspect the application's listener:

ss -lntp

Then check the application's file descriptor limits:

ulimit -n

and:

cat /proc/<PID>/limits

If the application allows configuring its listen backlog, make sure it is appropriate for the workload.

For instance, a Go application may explicitly configure a listener:

listener, err := net.Listen("tcp", ":8080")
if err != nil {
    log.Fatal(err)
}

The runtime and operating system then determine the actual socket behavior. For other runtimes, the configuration may expose backlog directly.

The important part is to avoid treating:

SOMAXCONN

as a magic performance knob. If the application cannot process connections fast enough, increasing the queue only changes where the pressure accumulates.

Also consider:

CPU saturation
file descriptor exhaustion
worker/thread exhaustion
event-loop blocking
connection pool exhaustion

These often appear alongside socket pressure.

Step 3: Build a Real Graceful Shutdown Path

A better Kubernetes deployment might look like:

spec:
  terminationGracePeriodSeconds: 30

  containers:
    - name: app
      image: example/app:1.0

      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - "sleep 10"

But don't stop there. The application itself should handle SIGTERM.

For instance:

SIGTERM
   |
   v
Set "shutting down" state
   |
   v
Fail readiness
   |
   v
Stop accepting new requests
   |
   v
Drain existing requests
   |
   v
Close listeners
   |
   v
Exit

This is much stronger than simply adding:

preStop:
  exec:
    command: ["sleep", "10"]

and hoping everything works. The sleep is a drain window, not a replacement for graceful shutdown.

Step 4: Verify Endpoint Propagation

During a controlled rollout, watch:

kubectl get pods -w

and:

kubectl get endpointslices -A -w

You want to understand the sequence:

Pod begins termination
        |
        v
Readiness changes
        |
        v
EndpointSlice changes
        |
        v
Traffic drains
        |
        v
Application exits

The exact ordering and timing depend on your Kubernetes version and networking stack, so measure it rather than assuming there is a fixed delay.

If you routinely see requests reaching terminating Pods, investigate:

• readiness behavior
• endpoint propagation
• ingress connection reuse
• load balancer draining
• application shutdown handling
• preStop
• terminationGracePeriodSeconds

Step 5: Make the Failure Observable

A generic metric like:

Pod availability = 100%

is not enough. Track the failure boundary. For the ingress layer, useful signals include:

HTTP 502 count
upstream connection errors
upstream resets
upstream timeouts
active connections
connection reuse

At the application layer:

request latency
active connections
open file descriptors
connection errors
graceful shutdown duration
request rejection

At Linux level:

TCP retransmissions
socket states
listen queue pressure
file descriptor usage
network drops

Now you can distinguish:

Pod is unhealthy

from:

Pod is healthy, but proxy-to-Pod TCP connections are failing

Those are completely different incidents.

A Practical Debugging Sequence

When users report intermittent 502s while Kubernetes says everything is healthy, don't randomly restart Pods.

Walk down the stack.

1. Confirm where the 502 originates

Check ingress/proxy logs.

Look for:

upstream reset
upstream prematurely closed connection
connect() failed
connection refused
timeout
ECONNRESET

2. Identify the backend Pod

Determine which Pod the proxy attempted to reach.

kubectl get pods -o wide

3. Check the application directly

From inside the cluster:

curl -v http://<POD_IP>:8080/health

Then test the actual application endpoint:

curl -v http://<POD_IP>:8080/api/users

A healthy /health endpoint doesn't prove that the application can successfully process every real request.

4. Inspect sockets

ss -antp
ss -lntp

5. Check file descriptors

ulimit -n
cat /proc/<PID>/limits

6. Trace system calls when necessary

strace -f -e trace=network -p <PID>

7. Capture packets

tcpdump -ni any tcp port 8080

8. Compare timeouts

Write down the values for:

Ingress
Load Balancer
Application
TCP keep-alive
Request timeout
Idle timeout

Don't rely on memory.

9. Reproduce under load

Use a controlled load test. A failure that occurs once every 20,000 requests is difficult to diagnose manually. A reproducible failure under controlled traffic is much easier to understand.

---

Why Kubernetes Health Checks Miss This

A readiness probe might execute:

GET /health

and receive:

200 OK

That proves something useful. It proves that at that moment, the kubelet could successfully perform that health check according to the configured probe mechanism.

It does not prove:

Every existing TCP connection is healthy.
Every proxy connection is fresh.
The listen queue isn't saturated.
The process isn't close to its FD limit.
No connection will reset.
The application won't terminate during an inflight request.
The proxy's connection pool contains no stale sockets.

This is the gap between health and correctness under traffic.

A Pod can be alive. A Pod can be ready. A Service can have endpoints. And a specific TCP connection can still be dead. That's not contradictory. They're different layers observing different things.

---

The Mental Model to Keep

When you see:

502 Bad Gateway

think:

Client
  |
  | request
  v
Proxy
  |
  | "I couldn't successfully talk to upstream"
  |
  X
Backend connection
  |
  +-- TCP failure?
  +-- stale keep-alive?
  +-- backlog pressure?
  +-- application closed socket?
  +-- graceful shutdown race?
  +-- FD exhaustion?
  +-- timeout?

Then move downward:

Ingress logs
      ↓
Service / EndpointSlice
      ↓
Pod IP
      ↓
Application process
      ↓
Linux socket
      ↓
TCP state
      ↓
Packets

That's how you turn a "ghost outage" into a concrete failure.

---

3 Key Takeaways

• A green Pod does not mean every network connection to that Pod is healthy. A 502 usually means the proxy failed to get a valid upstream response, and the failure may exist entirely below the Kubernetes health-check layer.

• Keep-alive mismatches, socket queue pressure, and termination races are connection-lifecycle problems. Fix them by aligning proxy/application timeouts, understanding listen() backlogs and kernel limits, and giving terminating Pods enough time to drain traffic.

• When Kubernetes abstractions stop explaining the outage, inspect Linux. ss shows socket state, strace shows what the process is asking the kernel to do, and tcpdump shows what actually crossed the network. Together, they can expose the failure that a green dashboard cannot.