TL;DR:
set requests.cpu, requests.memory, and limits.memory, but remove limits.cpu.
"Always set memory limit == request. Never set CPU limit"
~ Tim Hockin, Kubernetes co-founder (source)Google's GKE docs say the same thing: for the CPU request, "specify the minimum CPU needed... according to your own SLOs," then "set an unbounded CPU limit" (source).
Disclaimer: this post is a short version of the full analysis I posted on GitHub here. Some of the images below should be animated (but SVGs are blocked on DEV.to), so I highly recommend reading it there as well.
Most teams still set a CPU limit anyway, then spend their time fighting the throttling that follows. This analysis explains why the advice is right, and why so many people still get it wrong. Most charts set two CPU numbers, a request and a limit, and people treat them like the same setting with a bit of headroom.
- The request is how much CPU is reserved for your pod if it needs it.
- The limit is a hard cap: hit it and the kernel throttles the pod, even when the node still has spare CPU.
There are exceptional use cases for using CPU limits (they exist for a reason), but your situation is unlikely one of them, meaning you probably should NOT set CPU limits (see conclusion).
Why CPU requests are enough
Linux already shares a node's CPU fairly among pods. That sharing is called CFS (Completely Fair Scheduler). A request is a weight in the cgroup, not a pinned core. If the node is busy, CFS splits time in proportion to those weights: a 16 CPU request gets about sixteen times the CPU of a 1 CPU request. If the other pods are idle, a pod can use the leftover; when they need CPU again, those cores go back. None of this requires a CPU limit.
A limit is a separate cap, also enforced by CFS. Every 100 ms the kernel gives your pod a budget of CPU time, and every thread in the pod shares that budget. When it is gone, the pod is throttled until the next 100 ms. The node can be idle and the pod still waits. A 1 CPU limit on a 32-core node can still run on all 32 cores for a few milliseconds, then sit out the rest of the window. Four threads working at once burn a 500m budget in about 12 milliseconds. It is an average, not a reserved core.
Grafana shows the CPU is fine! No, your metrics are lying.
The CPU graph on your Kubernetes dashboard usually averages a minute. CPU throttling lasts a tenth of a second. So the graph can look fine while the app is being throttled all the time. Watch container_cpu_cfs_throttled_periods_total, not average CPU.
Two views of the same 30 seconds: an average CPU graph that looks flat and fine, next to a CPU throttling graph on the same window showing the pod repeatedly hitting its limit
People put a limit on because they are afraid some other app will starve their pod. That is what the request is for. The app you care about is protected by its own request. A runaway next door can use leftover CPU, but it cannot take the share you reserved. If leftover on the node is huge, the requests are too small. If teams can deploy with no request at all, give them a default request instead of putting a CPU limit on whoever looks greedy.
The same blindness poisons right-sizing. Usage recorded under a CPU limit can never go above the limit: the cap clips every burst, so the history shows what the kernel allowed, not what the app wanted. Size a request from that history and you copy the cap's distortion into the request. Drop the limit first, let the app run for a while, then measure and set requests from numbers that were free to move.
Proof: the same app with and without a CPU limit
In a controlled burst test, adding a CPU limit took typical latency from 23 ms to roughly 87 ms (about 4x slower), with the limited pod throttled in half of all CFS windows, while the average CPU graph looked fine the whole time.
How CPU limits cause memory issues and OOMKills
A service with any kind of in-memory queue: a consumer buffering messages from Kafka or RabbitMQ faster than it can process them, an unbounded channel between two components, a server holding request payloads while handlers run behind.
Any kind of application that has a garbage collector (GC) is a potential victim. In my experiment: every request builds a reference-dense graph of 10,000 small objects (~1.5 MiB) and holds it only while doing real work; once the request finishes, the graph is garbage. Nothing is retained anywhere, on purpose. What runs short here is the garbage collector's own CPU budget: on the capped pod (limits.cpu: 100m, the same value as the production incident this test is based on), in-flight requests pile up, the live object count grows with them, each GC cycle gets more expensive, and the collector fights the workload for the same shrinking quota.
Even a simple HTTP API server is not spared. As http requests come in, and each one is holding its memory. Nothing stops the pile-up: web frameworks (ASP.NET Core, Node, Go, and friends) do not cap how many requests may run at once, and the await hands the thread back but keeps the buffer, so the thread pool does not cap it either.
All three shapes end the same way on a graph: memory climbing into the limit. It looks like a leak, the usual fix is a bigger memory limit, and the actual cause is the CPU limit. Check container_cpu_cfs_throttled_periods_total before you blindly raise limits.memory.
Conclusion and FAQs
Dropping limits.cpu moves a pod from Guaranteed to Burstable QoS. In practice this rarely matters: the kubelet evicts for memory pressure, not CPU, and it ranks pods by how far usage exceeds the request, not by QoS class alone. Keep requests.memory equal to limits.memory and eviction exposure barely moves. Full answer, with the eviction docs link, in the FAQ.
FAQ
Should I set CPU limits in Kubernetes? For most services, no. Keep requests.cpu so CFS reserves your share, keep limits.memory, and drop limits.cpu. The exceptions (benchmarks, pinned cores, per-customer billing) are listed in the conclusion (read more).
Doesn't a limit stop a bad pod from eating the node? Monopolizing a node is a myth. Spare CPU is borrowed, not taken: the moment another pod wants CPU, CFS pulls those cores back within milliseconds and splits time by request weights again. A limit on the busy pod only stops it using CPU that would otherwise sit idle. It does not give CPU to anyone else: the neighbor is protected by its request. If several pods burst at once, the leftover is not first come first served, CFS divides it in proportion to their requests. If leftover on the node is huge, the requests on that node are too small.
After I drop limits, can a burster hurt the node itself? Not the other pods, but kubelet, containerd, the CNI, and log shippers often run with tiny or no CPU reservation. A pod bursting into all spare CPU can delay exec probes and flap readiness. The fix is system-reserved and kube-reserved in the kubelet config, which carves out CPU for the node's own daemons. A per-pod CPU limit is the wrong tool for this.
Top comments (0)