DEV Community: Sagar

Kubernetes CPU throttling: p99 in seconds, no restarts, and a limit that's too low

Sagar — Fri, 30 Oct 2026 08:15:33 +0000

Your service is slow and nothing looks wrong. The p99 latency is in seconds, the median is fine, every pod is Running and Ready with 0 restarts, and the nodes are nearly idle. That pattern usually means a CPU limit is throttling the container: it isn't killed, it's paused. Compare kubectl top pod with the CPU limit in kubectl describe pod. If the two numbers match, the pod is capped.

I broke a small Go service on purpose to show this on a real cluster: a "cost review" lowered its CPU limit while a load generator sent it steady traffic. Every output below is copied from that recording (I lined up the columns).

What throttling looks like from outside

A throttled pod passes every health check you'd normally look at. The latency alert fired because the p99 stayed above half a second, but the error rate was flat and the pod looked healthy:

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-649df775bd-md4lq   1/1     Running   0          4m15s

The nodes weren't busy either, so this wasn't a neighbour stealing CPU:

$ kubectl top nodes
NAME                    CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
k3d-snowops-agent-0-0   151m         3%       860Mi           10%
k3d-snowops-agent-1-0   154m         3%       444Mi           5%
k3d-snowops-server-0    399m         9%       3085Mi          38%

kubectl rollout history showed a second revision, so the pod template had changed. That's the "what changed?" clue, but the history doesn't say what.

Compare top with the limit

kubectl top pod shows what a container uses, and kubectl describe pod shows what it's allowed. Put them side by side:

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-649df775bd-md4lq   11m          29Mi

$ kubectl describe pod go-api-649df775bd-md4lq -n go-api
...
    Limits:
      cpu:     11m
      memory:  128Mi
    Requests:
      cpu:     11m
      memory:  32Mi
...

(I cut the rest of the describe output.) 11m of CPU looks like an idle pod. It isn't. It's exactly the limit, and a pod that sits on its limit in every sample is using everything it's allowed. The lab picks this limit from a measurement of your machine, so your number will differ. The match is what matters.

Grafana's Pod Resources dashboard made it plain. The CPU throttled (share of periods) panel had go-api's pod throttled in nearly every period:

Grafana panel "CPU throttled (share of periods)": the go-api pod sits near 100% for the whole five minutes

That panel is the ratio of two cAdvisor counters, which you can graph on any cluster with Prometheus:

sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[2m]))
  / sum by (namespace, pod) (rate(container_cpu_cfs_periods_total{container!=""}[2m]))

Why a CPU limit slows a pod instead of killing it

A CPU limit is enforced as a quota of CPU time per scheduling period. The kernel gives the container its limit's share of each period (100 ms by default), and once the container has spent it, the container waits for the next period (kubernetes.io, how pods with resource limits are run). A memory limit is different: go over it and the container is killed with OOMKilled. A CPU limit only holds the container back (kubernetes.io, assign CPU resources).

So most requests land early in a period and finish fast. The unlucky ones arrive after the quota is spent, queue behind the work in progress, and wait one period or several. That's how the median stays flat while the p99 climbs to seconds, with no error anywhere.

Why does kubectl top show low CPU while the pod is throttled?

Because a capped pod can never use more than its cap. kubectl top reports what the container actually used, averaged over a short window (kubernetes.io, resource metrics pipeline). It can't show what the pod wanted. A pod pinned at its limit reads as a small, flat, round number. And a multi-threaded app can be throttled even when its average is below the limit, because several threads can burn a period's quota in a few milliseconds. Watch the throttled ratio, not only usage.

Adding replicas didn't fix it

Scaling out is the usual reflex for a slow service, so I tried it:

$ kubectl scale deploy/go-api --replicas=3 -n go-api
deployment.apps/go-api scaled

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-649df775bd-gx4p8   8m           7Mi
go-api-649df775bd-md4lq   9m           25Mi
go-api-649df775bd-n2q6c   9m           7Mi

Each pod got a third of the traffic and still sat just under the same 11m ceiling. The throttled panel showed all three throttled in roughly half or more of their periods:

!The same panel after scaling to three replicas: the three go-api pods are still throttled in about 45% to 100% of periods

The same panel after scaling to three replicas: the three go-api pods are still throttled in about 45% to 100% of periods

The lab's check stayed red: FAIL: go-api/go-api is still not sized for the load it serves. Replicas spread the load, but every pod keeps a ceiling below what it needs, and the cost review that set the limit now pays for three pods instead of one.

Measure first, then set the limit

You can't size a limit while the pod is capped, because kubectl top reads the cap. So the fix has two steps. I scaled back to one replica and lifted the limit far above anything it had used, as a measuring step:

$ kubectl scale deploy/go-api --replicas=1 -n go-api
$ kubectl set resources deploy/go-api -n go-api --limits=cpu=500m
deployment.apps/go-api resource requirements updated

A minute into steady load, kubectl top showed the real demand:

$ kubectl top pod -n go-api
NAME                      CPU(cores)   MEMORY(bytes)
go-api-6d5ddc4d8b-db7gc   24m          7Mi

24m, and 23m a minute later. More than twice the old limit. Then I set it properly: a request near the steady use, so the scheduler is told the truth, and a limit with plenty of headroom for bursts:

$ kubectl set resources deploy/go-api -n go-api --requests=cpu=50m --limits=cpu=200m
deployment.apps/go-api resource requirements updated

$ kubectl rollout status deploy/go-api -n go-api
Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination...
deployment "go-api" successfully rolled out

(I left out the scale command's output and one repeated Waiting line.) The request is the pod's guaranteed share of a busy node. The limit is the ceiling.

Proving the fix

The fix shows on the user signal, not only on the resource signal. The server-side p99 fell from about 8 seconds back down to the median:

Grafana "Latency percentiles (server-side)": p99 drops from about 8 s to near zero after the limit is lifted, while p50 stays flat at zero the whole time

On the throttled panel, the old pods' lines ended and the new pod drew no line at all. That panel only draws pods throttled in more than 5% of periods, so the new pod is under that. The lab's check then passed and closed the incident.

Questions people ask next

Should I remove CPU limits altogether?

It's a common choice for latency-sensitive services, and a reasonable one. Without a limit the pod can use idle CPU on the node, and its request still guarantees it a share when the node is busy. Two catches: a namespace LimitRange may add a default limit nobody chose (check kubectl describe limitrange -n <ns>), and a pod only gets the Guaranteed QoS class if its limits equal its requests. Keep a limit where you need either of those. And this is about the CPU limit only: memory can't be throttled, so keep the memory limit.

How much throttling is a problem?

There's no universal threshold, so judge it against latency. In this lab, a limit of 1.5 times the measured demand still left the app throttled in about a sixth of its periods with p99 near 150 ms. At twice the demand, throttling was near zero and p99 went back to a few milliseconds. Graph the throttled ratio next to your p99 and look for the two moving together.

Other causes of slow pods with no restarts

Throttling is the cause where the pod's own CPU reads flat at a round number that equals its limit. If it doesn't, look elsewhere:

A busy node: kubectl top nodes near 100%, and a neighbour taking the CPU. The fix is placement or requests, not this pod's limit.
Too few replicas for the traffic: CPU per pod is high but below the limit. That's a job for scaling, or a HorizontalPodAutoscaler.
A slow dependency: the app waits on a database or another service. Traces or the dependency's own latency show it.
Memory pressure: a tight memory limit causes garbage-collection pauses before it causes OOMKilled.

Try it yourself

On any test cluster with steady traffic on a Deployment, set a CPU limit far below what it uses:

kubectl set resources deploy/<app> -n <ns> --limits=cpu=10m

Then work through it: p99 against the median, pods, nodes, top against the limit, the throttled ratio, then measure and set.

I ran this drill in SnowOps Labs, an open-source Kubernetes lab where you inject faults like this one into a real app and fix them (GitHub). Disclosure: I build it. It measures your machine before it sets the limit, runs the load for you, and checks the limit you set, not just the moment's latency, so scaling out or stopping the traffic won't pass.

The ladder to remember

Compare p99 with the median. Slow but not broken means something is waiting.
kubectl get pods and kubectl top nodes: no restarts, no busy node.
kubectl top pod against the limit in kubectl describe pod. Equal means capped.
Check the throttled ratio (container_cpu_cfs_throttled_periods_total).
Lift the limit, measure the real use under load.
Set a request near that use and a limit with headroom, or remove the CPU limit.

*This is part of a series I'm writing on debugging Kubernetes in production. I am building Snowopslabs which I have used to simulate this incident and resolve it. Check out this tool if you want to play around with Kubernetes incidents and scenarios and learn along the way.

Pods Running, Service returns 503: the selector bug that fools everyone

Sagar — Sun, 25 Oct 2026 06:58:32 +0000

Every request to the app fails with a 503. The body says no available server. And yet kubectl get pods looks perfect:

$ curl -i http://go-api.snowops.localhost/health
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 12 Oct 2026 14:25:45 GMT
Content-Length: 20

no available server

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-5d8497557b-7pfhz   1/1     Running   0          4m42s

One of one ready, zero restarts, nothing deployed. I broke a small Go service on purpose to walk through this one, because it fools people precisely by looking healthy everywhere you'd normally look. The commands below work on any cluster. The cause in my drill is one renamed label, and at the end I'll go through the other things that leave a Service with nowhere to send traffic.

The piece most people never check

A request from outside reaches a pod in three hops. The ingress controller receives it and looks up the Service the route points at. The Service doesn't forward anything to "the app" in general. It forwards to a list of pod addresses, its endpoints. Then the request lands on one of those pods.

That list isn't something you write by hand. A Service has a selector, a set of labels, and Kubernetes keeps the list up to date with every pod that carries all of those labels and is ready to serve. Today the list lives in EndpointSlice objects (Kubernetes docs: EndpointSlices). The older Endpoints object still exists, and kubectl get endpoints still works, but newer clusters print a warning when you use it.

Here is the healthy state, before I broke anything:

$ kubectl get endpoints go-api -n go-api
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS          AGE
go-api   10.42.0.136:8080   67s

$ kubectl get endpointslices -n go-api
NAME           ADDRESSTYPE   PORTS   ENDPOINTS     AGE
go-api-kt9b6   IPv4          8080    10.42.0.136   82s

One address, the pod's IP, on port 8080. Keep that picture in mind. If the list is empty, the ingress has a route and a Service and nowhere to send the request. That is exactly what no available server means. It comes from Traefik, the ingress controller in this lab, not from the app.

What the outage looks like from the inside

Before going to the terminal, I looked at the dashboard. A load generator (k6) had been sending the app steady traffic from inside the cluster, so there's a before and an after.

This was the part that surprised me most:

Grafana during the outage: the app's own 5xx error rate stays at zero while k6's failed request rate climbs to 100%

On the left, the app's own error rate. Flat at zero for the whole outage. On the right, what the client saw: every request failing. Both panels are telling the truth. No request reached the pod, so the pod had nothing to count, good or bad. Its metrics didn't go red, they went quiet.

The panel that shows it best puts both sides on one chart:

Offered vs served: k6 keeps offering about 50 requests a second (green) while the requests the app handles (yellow) fall away

Green is what k6 offered, yellow is what the app handled. Normally the two lines sit on top of each other. The gap between them is the outage.

If your alerting only looks at what a service says about itself, this outage looks like a quiet afternoon. You need at least one signal from the client side: a synthetic check, a probe from outside the cluster, or the ingress controller's own metrics. An alert that watches pods won't help either, because the pods really are fine. What does work is an alert on the thing that broke. The lab ships one, and it was firing:

Prometheus: the LabFaultServiceEndpointsEmpty rule firing, absent() over the Service's ready EndpointSlice endpoints for 1 minute

It uses absent() rather than count(...) == 0, and the lab's rule file explains why: kube-state-metrics exports one series per endpoint address, so when there are no addresses there is no series at all, and a count would never fire. That's an easy rule to get wrong, and worth stealing.

Find the tell: an empty endpoint list

Back in the terminal, the order of checks follows the request. The pods are healthy, so check the link in the middle:

$ kubectl get endpoints go-api -n go-api
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME     ENDPOINTS   AGE
go-api   <none>      7m27s

$ kubectl get endpointslices -n go-api
NAME           ADDRESSTYPE   PORTS     ENDPOINTS   AGE
go-api-kt9b6   IPv4          <unset>   <unset>     7m41s

<none>. A few minutes ago that column held the pod's address. The EndpointSlice is still there, but it's empty. That's the tell: the Service exists, the pod exists, and the Service isn't selecting it.

Nothing else would have told me. Nothing crashed, nothing restarted, and Kubernetes emits no event when a selector stops matching, because from its point of view nothing failed. The Service is doing exactly what its spec says.

Compare the selector with the labels

-o wide prints a Service's selector in the last column:

$ kubectl get svc go-api -n go-api -o wide
NAME     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE   SELECTOR
go-api   ClusterIP   10.43.223.247   <none>        8080/TCP   8m    app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api-v2,app=go-api

(My terminal wrapped that last line, I joined it back.) Three labels. Now the labels the pod actually carries:

$ kubectl get pods -n go-api --show-labels
NAME                      READY   STATUS    RESTARTS   AGE     LABELS
go-api-5d8497557b-7pfhz   1/1     Running   0          8m16s   app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api,app=go-api,pod-template-hash=5d8497557b

Line them up and the difference is one value:

Service wants:  app.kubernetes.io/name=go-api-v2
Pod carries:    app.kubernetes.io/name=go-api

Two out of three labels match, and that counts for nothing. A selector is an AND: a pod has to carry every label in it (Kubernetes docs: labels and selectors). One key off, and not one pod matches.

A small tip if you're doing this on your own cluster: I used get -o wide rather than kubectl describe svc here. describe shows the selector too, but on this lab it also prints an annotation the drill leaves behind with the original selector in it, which gives the answer away. On your cluster describe is fine.

Which side is wrong?

A mismatch has two sides, and you could make them agree from either one. The Deployment settles it. It selects its pods by labels too, and it stamps those labels onto every pod it creates:

$ kubectl get deploy go-api -n go-api -o wide
NAME     READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES          SELECTOR
go-api   1/1     1            1           8m33s   go-api       go-api:v1.2.0   app=go-api,app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api

name=go-api. The pods are right, and the Service is the odd one out. There's a structural reason to fix it there as well: a Deployment's selector can't be changed after it's created (in apps/v1 it's immutable), while a Service's selector can be changed at any time, to anything, and Kubernetes accepts it without a warning. That's how this kind of drift gets in.

The tempting fix

The Service wants a pod labelled go-api-v2. So why not give the pod that label? It's one command:

$ kubectl label pod -n go-api go-api-5d8497557b-7pfhz app.kubernetes.io/name=go-api-v2 --overwrite
pod/go-api-5d8497557b-7pfhz labeled

And it works, in the sense that requests flow again. The endpoint list fills back in and the health check returns 200. If you stopped here, you'd close the incident. But list the pods:

$ kubectl get pods -n go-api --show-labels
NAME                      READY   STATUS    RESTARTS   AGE     LABELS
go-api-5d8497557b-2qwx4   1/1     Running   0          18s     app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api,app=go-api,pod-template-hash=5d8497557b
go-api-5d8497557b-7pfhz   1/1     Running   0          9m16s   app.kubernetes.io/instance=go-api,app.kubernetes.io/name=go-api-v2,app=go-api,pod-template-hash=5d8497557b

There are two now. A ReplicaSet also finds its pods by their labels. The moment I relabelled the pod, it stopped matching the ReplicaSet's selector, so as far as the ReplicaSet was concerned it had lost a pod, and it made a replacement with the original labels. The docs describe this as isolating a pod from its ReplicaSet (Kubernetes docs: ReplicaSet), and it's normally used on purpose, to pull a misbehaving pod out of service for debugging.

So the Service is now held up by one orphan pod that nothing owns. If it's evicted, or its node goes away, nothing replaces it and the 503s come back. And the next rollout creates pods from the template, which still says go-api. The person doing that rollout won't connect it to today's incident.

The lab's check wasn't fooled:

!The SnowOps Labs check refuses the relabel: Not resolved yet, the Service wants "go-api-v2", pods are labelled "go-api"

The SnowOps Labs check refuses the relabel: Not resolved yet, the Service wants "go-api-v2", pods are labelled "go-api"

It compares the Service's selector with the Deployment's pod template rather than with whatever labels happen to be on a pod right now, for exactly this reason.

The real fix: change the Service

First, undo the shortcut. Nothing owns the relabelled pod, so deleting it is permanent:

$ kubectl delete pod -n go-api go-api-5d8497557b-7pfhz
pod "go-api-5d8497557b-7pfhz" deleted from go-api namespace

Then patch the Service's selector back to what the Deployment uses:

$ kubectl -n go-api patch svc go-api -p '{"spec":{"selector":{"app.kubernetes.io/name":"go-api"}}}'
service/go-api patched

kubectl patch sends only the fields you give it, and for a Service the selector map is merged, so the other two labels stay as they were. Only name changes.

Prove it, in the order you found it

I check in the same order I diagnosed, from the inside out. First the Service:

$ kubectl get endpoints go-api -n go-api
NAME     ENDPOINTS          AGE
go-api   10.42.2.168:8080   10m

(I dropped the deprecation warning from here on.) An address again, and a different one from the start. It belongs to the pod the ReplicaSet created, which is the one that should be serving. Then the same question from outside:

$ curl -i http://go-api.snowops.localhost/health
HTTP/1.1 200 OK
Content-Length: 16
Content-Type: application/json
Date: Mon, 12 Oct 2026 14:32:28 GMT

{"status":"ok"}

Same request as at the start, different answer. On the dashboard the app's request rate came back up to meet k6's, and the lab's check passed.

Try it on your own cluster

You don't need the lab for this one. Pick a Service in a test namespace, note its selector, and change one label to a value no pod carries:

kubectl get svc <your-service> -o wide
kubectl patch svc <your-service> -p '{"spec":{"selector":{"<key>":"<value no pod has>"}}}'
kubectl get endpoints <your-service>

The endpoints go to <none> straight away. Your pods stay Running and Ready, nothing restarts, and no event warns you. Patch the label back to undo it. Only do this on a test Service: a selector change moves traffic immediately.

If you'd rather practise the whole incident, with traffic, the dashboards, the alert and a check that won't accept the relabel shortcut, it's one of the drills in SnowOps Labs, a free, open source Kubernetes lab that runs on a laptop. Disclosure: I build it. Every screenshot above is from it, and the code is on GitHub.

Other ways to end up with no endpoints

A renamed label is one cause. When kubectl get endpoints comes back empty and your pods look fine, these are the others I'd check, roughly in this order. The Kubernetes guide to debugging Services walks through the same ground.

The pods aren't Ready. Running isn't Ready. A pod failing its readiness probe is left out of the Service's ready endpoints, so a broken readiness probe empties the list just as well. kubectl get pods shows 0/1 in the READY column. Check that first, which is why it comes first in the recap below.
A typo in the label key, not the value. app.kubernetes.io/name against app.kubernetes.io/Name, or app against application. Same result, harder to spot by eye. Compare the two outputs character by character.
The wrong namespace. A Service only selects pods in its own namespace. A Service applied to default while the pods live in go-api has a perfect selector and no endpoints.
A Service with no selector at all. Then Kubernetes doesn't manage its endpoints, and they stay empty unless something creates EndpointSlices for it. That's intended for pointing a Service at something outside the cluster, and a surprise when it's a mistake.
Endpoints exist, but requests still fail. Then it isn't this problem. Look at the Service's targetPort against the port the container really listens on, and at what the ingress route points to.

At work, the label usually changes somewhere other than the cluster: a Helm chart refactor that renames the selector labels, or a Service copied from another app with its app name left in. Whatever produced the change, fix it there as well (the chart, the Kustomize overlay, the GitOps repo), or the next sync puts the bad selector straight back.

The short version

When users get 503s and the pods look healthy:

curl -i the URL, or open it, and confirm what users see.
kubectl get pods: pods that aren't Ready also empty the endpoint list, so rule that out first.
kubectl get endpoints <service> (or get endpointslices): an empty list means the Service selects nothing.
Compare kubectl get svc <service> -o wide with kubectl get pods --show-labels. One mismatched label is enough.
Fix the Service (or whatever generates it), not the pods, and check that the endpoints come back before you check the URL.

This incident is resolved using snowops labs, which is the open source project I am building for simulating real world platform engineering scenarios. Checkout this project: snowopslabs

CrashLoopBackOff: how to find the real cause (and why deleting the pod won't help)

Sagar — Wed, 21 Oct 2026 11:48:23 +0000

A config change goes out. A couple of minutes later every request to the app fails, and kubectl get pods shows the status nobody wants to see:

NAME                      READY   STATUS             RESTARTS      AGE
go-api-7b45497dfd-qxqx2   0/1     CrashLoopBackOff   4 (15s ago)   115s

I broke a small Go service on purpose to walk through this one properly, from the first symptom to a fix you can prove. The commands work on any cluster. The specific bug is just one example, and I'll come back at the end to the other things that cause the same status.

What CrashLoopBackOff actually tells you

It is easy to read CrashLoopBackOff as an error. It isn't one. It is Kubernetes describing what it is doing about an error.

The container starts and then exits. The kubelet on the node restarts it, because the pod's restartPolicy says so. If it exits again, the kubelet waits before the next try, and the wait grows each time: 10 seconds, 20, 40, and so on, capped at five minutes (Kubernetes docs: container restarts). While it is waiting, the pod shows CrashLoopBackOff.

So the status answers "what is happening?" (the container keeps dying) and says nothing about why. That second question is the whole job, and the pod has already written the answer down somewhere. You just have to know where to read it.

First, check whether anyone is still being served

Before digging into the crash, it's worth thirty seconds to find out how bad things are. A crash loop during a rollout is often harmless: with a rolling update and a readiness probe, the old pods keep serving until a new one becomes Ready. If the new pod never gets there, the rollout just stalls.

A Deployment keeps one ReplicaSet per version of its pod template, so listing them tells you whether an older version is still up:

$ kubectl get rs -n go-api
NAME                DESIRED   CURRENT   READY   AGE
go-api-5d8497557b   0         0         0       5m46s
go-api-7b45497dfd   1         1         0       2m9s

Not good. The old ReplicaSet is at zero, and the new one wants a pod and has none ready. Nothing is serving. In my drill I set it up that way on purpose, but you get the same picture in real life with a Recreate strategy, or when the crash only starts after the pod has already passed its readiness check. Either way, this isn't a stuck rollout any more. It's an outage.

The browser agreed:

That message comes from the ingress. It had a route for the app and no ready pod to send the request to.

A detour worth taking: the dashboard that says everything is fine

This was the part of the drill that surprised me most. Here is the Grafana dashboard during the outage. On the left is the app's own error rate (HTTP 5xx). On the right is the failure rate seen by k6, the load generator sending traffic from outside.

The app reports zero errors. The client sees 100% of requests fail. Both are telling the truth: a process that never finishes starting can't count the requests it never received, so its own metrics go quiet rather than red. If your alerting only looks at what the service says about itself, a total outage can look like a calm afternoon. Watch at least one signal from the client side too, whether that's an ingress or load balancer metric, a synthetic check, or a probe from outside the cluster.

Read the crash's last words: `kubectl logs --previous`

Plain kubectl logs shows the current container. In a crash loop the current container has often just started, or hasn't started yet, so you get nothing useful. The container that just died is still on the node, and --previous (or -p) asks for its logs:

$ kubectl logs -n go-api go-api-7b45497dfd-qxqx2 --previous
{"level":"INFO","msg":"server starting","port":"80800","service":"go-api"}
{"level":"ERROR","msg":"server error","error":"listen tcp: address 80800: invalid port"}

(I trimmed the timestamps.) Two lines, and the second one is the answer. The server tried to listen on port 80800. Port numbers stop at 65535, so the bind fails and the process exits.

Most of the time the logs get you this far. When they don't, the next command usually does.

Confirm it from Kubernetes' side: `kubectl describe pod`

describe is long, so I pipe it into less and search for Last State:

$ kubectl describe pod -n go-api go-api-7b45497dfd-qxqx2 | less
...
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Ready:          False
    Restart Count:  4
    ...
    Environment:
      PORT:                 80800
      READINESS_FAILURE:    false
      SHUTDOWN_TIMEOUT:     10s

Three things to take from this block.

The exit code. 1 means the app chose to exit with an error. Nothing killed it. If you see 137 instead, the process got a SIGKILL (128 + 9), and a Reason of OOMKilled tells you it ran out of memory. Those two numbers send you down completely different paths, so it's worth reading this line before anything else.

The environment. The same 80800 again, now in the spec. So the bad value isn't something the app made up at runtime. Kubernetes handed it over.

The events at the bottom. In this case they only confirm the loop: Back-off restarting failed container. For other causes they carry the real clue, such as Liveness probe failed, a missing ConfigMap or Secret, or an image that can't be pulled.

The fix everyone tries first

With the pod clearly broken, deleting it is tempting. I tried:

$ kubectl delete pod -n go-api go-api-7b45497dfd-qxqx2
pod "go-api-7b45497dfd-qxqx2" deleted

$ kubectl get pods -n go-api
NAME                      READY   STATUS             RESTARTS      AGE
go-api-7b45497dfd-4c995   0/1     CrashLoopBackOff   1 (21s ago)   22s

A new name, the same crash, within seconds. The pod was never the problem. A pod owned by a Deployment is a copy of the Deployment's pod template, and the ReplicaSet will keep making fresh copies of the same mistake for as long as you delete them. The only thing deleting achieved was resetting the restart counter, which made the situation look less serious than it was.

Fix it where it lives: the pod template

So the question becomes where the template gets PORT=80800 from. describe deploy shows the template:

$ kubectl describe deploy go-api -n go-api | less
...
  Containers:
   go-api:
    Image:      go-api:v1.2.0
    Port:       8080/TCP (http)
    Liveness:   http-get http://:http/health delay=15s timeout=5s period=10s
    Readiness:  http-get http://:http/ready delay=10s timeout=3s period=5s
    Environment:
      PORT:                 80800

The container declares port 8080, both probes use that named port, and the environment variable says 80800. One zero too many. The fix goes on the Deployment:

$ kubectl -n go-api set env deploy/go-api PORT=8080
deployment.apps/go-api env updated

Changing the template starts a new rollout. In a real team you'd make the same change in whatever produced the manifest (the Helm values, the Kustomize overlay, the GitOps repo) or the next sync will put the typo straight back. kubectl set env is fine for stopping the bleeding, as long as the real fix follows it.

kubectl rollout undo deploy/go-api -n go-api would have worked too. It returns the template to the previous revision, environment included. When you're paged and you know the last change caused it, rolling back first and investigating second is often the right call.

Prove the fix instead of assuming it

Don't eyeball get pods a few seconds after the change. rollout status waits until the new pods are actually Ready:

$ kubectl -n go-api rollout status deploy/go-api
Waiting for deployment "go-api" rollout to finish: 1 old replicas are pending termination...
deployment "go-api" successfully rolled out

$ kubectl get pods -n go-api
NAME                      READY   STATUS    RESTARTS   AGE
go-api-5d8497557b-r47rn   1/1     Running   0          26s

$ kubectl get rs -n go-api
NAME                DESIRED   CURRENT   READY   AGE
go-api-5d8497557b   1         1         1       9m8s
go-api-7b45497dfd   0         0         0       5m31s

Look at the ReplicaSet names. Kubernetes didn't create a third one. The fixed template is identical to the one from before the bad change, so the Deployment scaled the old ReplicaSet back up. It's a small detail, but it's a nice confirmation that the template now matches the last version that worked.

Then check from outside, the way users see it. The health URL answered again, and the request rate on the dashboard came back.

Try it on your own cluster

You don't need anything special to reproduce this. On a test cluster, take any Deployment whose app reads its port from an environment variable and give it an impossible one:

kubectl set env deploy/<your-test-app> PORT=80800
kubectl get pods -w

With the default rolling update you'll likely see the stalled-rollout version: the old pod stays Running while the new one crash-loops. That's a useful thing to see for yourself, because it's exactly why the ReplicaSet check at the start matters. Put it back with:

kubectl rollout undo deploy/<your-test-app>

If you'd rather practise the full outage version, with traffic, dashboards, alerts and a check that tells you when you've actually fixed it, this drill is one of the incidents in SnowOps Labs, a free, open-source Kubernetes lab that runs on a laptop. Disclosure: I build it. Every screenshot above is from it, and the code is on GitHub.

Other causes of the same status

The port typo is one cause among many. The Kubernetes docs group the usual suspects into application errors, configuration errors, resource limits, and failing health checks. In practice that looks like:

A bug or a missing dependency in the app. Logs with --previous usually say so, often as a stack trace or a "connection refused" to a database that isn't there.
Missing or wrong configuration. A typo like this one, a variable renamed in one place and not the other, a config file mounted at the wrong path, or a value that's fine in staging and wrong in production.
Not enough memory. Exit code 137 with Reason: OOMKilled in describe. Raise the limit or fix the leak, but find out which one first.
A liveness probe that kills a healthy app. The app starts, the probe fails (too strict, wrong path, too short a delay for a slow start), and the kubelet restarts the container. The events say Liveness probe failed.
The container never really starts. A wrong command or args, an image built for another CPU architecture, a missing binary. Here the logs are often empty, because nothing ran long enough to write any. describe and its events carry the message instead.

One extra tip from the docs on finding why a pod failed: set terminationMessagePolicy: FallbackToLogsOnError on your containers. When a container fails without writing a termination message, Kubernetes then copies the last lines of its log into the pod status, so describe shows the error even after the logs are gone.

The short version

When a pod is in CrashLoopBackOff, this order gets you to the cause quickly, whatever the cause turns out to be:

kubectl get pods shows the symptom and the restart count.
kubectl get rs tells you whether anything is still serving, which decides how urgent this is.
kubectl logs <pod> --previous gives you the dead container's last words.
kubectl describe pod <pod> gives the exit code, the reason and the events.

Then fix the thing the pod was copied from, not the pod, and let rollout status tell you when it's really done.

DEV Community: Sagar

Kubernetes CPU throttling: p99 in seconds, no restarts, and a limit that's too low

What throttling looks like from outside

Compare top with the limit

Why a CPU limit slows a pod instead of killing it

Why does kubectl top show low CPU while the pod is throttled?

Adding replicas didn't fix it

Measure first, then set the limit

Proving the fix

Questions people ask next

Should I remove CPU limits altogether?

How much throttling is a problem?

Other causes of slow pods with no restarts

Try it yourself

The ladder to remember

Pods Running, Service returns 503: the selector bug that fools everyone

The piece most people never check

What the outage looks like from the inside

Find the tell: an empty endpoint list

Compare the selector with the labels

Which side is wrong?

The tempting fix

The real fix: change the Service

Prove it, in the order you found it

Try it on your own cluster

Other ways to end up with no endpoints

The short version

CrashLoopBackOff: how to find the real cause (and why deleting the pod won't help)

What CrashLoopBackOff actually tells you

First, check whether anyone is still being served

A detour worth taking: the dashboard that says everything is fine

Read the crash's last words: kubectl logs --previous

Confirm it from Kubernetes' side: kubectl describe pod

The fix everyone tries first

Fix it where it lives: the pod template

Prove the fix instead of assuming it

Try it on your own cluster

Other causes of the same status

The short version

Read the crash's last words: `kubectl logs --previous`

Confirm it from Kubernetes' side: `kubectl describe pod`