DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

Your Kubernetes Workloads Are Mis-Sized. The Eviction and Throttle Signals Prove It.

Your workloads are already telling you their size. The signals are in the metrics.

Size by evidence, not by vibes.

Open any Kubernetes manifest and you will find a resources block. Half the time it was copied from another service. The other half it was a guess someone made on a Tuesday and never revisited. Those blocks fossilize: requests too generous waste capacity and money, limits too tight silently strangle performance, and nobody notices because the signals that reveal the mismatch are scattered across metrics most dashboards never graph.

This post is a field guide to reading those signals. CPU throttling, memory working set, OOMKills, ephemeral-storage evictions — each one is a workload telling you its size. The method is a loop: observe the signal, compare it against a recommendation, adjust one knob, re-observe. No guesswork, no cargo-culted values, no “we set it to 2 CPU because that felt right.”

Requests are a promise; limits are a ceiling

Before the signals make sense, the two knobs have to be straight. They control completely different things, and confusing them is the root of most bad sizing.

Requests are a scheduling promise. They tell the scheduler how much CPU and memory to reserve on a node before the pod is allowed to land. If the sum of requests on a node exceeds capacity, the pod stays pending. Requests decide where your pod runs and how densely nodes pack. They are never enforced at runtime — a container can happily use more CPU than its request if the node has spare cycles.

Limits are a runtime ceiling enforced by the kernel. CPU limits become a CFS quota: the container gets a fixed slice of CPU time per 100ms window, and when it burns through it, the kernel freezes the container until the next window. Memory limits are harsher still — exceed them and the kernel’s OOM killer terminates the container with prejudice. Limits decide how hard your pod is allowed to run.

Requests are a promise to the scheduler. Limits are a ceiling enforced by the kernel. Size them from different evidence.

A typical block looks like this:

resources:
  requests:
    cpu: 500m # scheduling promise: reserve half a core
    memory: 512Mi # scheduling promise: reserve 512 MiB
    ephemeral-storage: 1Gi # scheduling promise: reserve 1 GiB of disk
  limits:
    cpu: "1" # hard ceiling: CFS quota of 1 core
    memory: 1Gi # hard ceiling: exceed it and the OOM killer fires
    ephemeral-storage: 2Gi # hard ceiling: exceed it and the pod is evicted
Enter fullscreen mode Exit fullscreen mode

The sizing question is therefore two questions: what should the scheduler reserve (requests), and where should the kernel draw the line (limits)? Each signal below answers one of them.

The CPU throttle signal: stop watching utilization

The most common sizing mistake is reading average CPU utilization and calling it a day. Utilization lies. A container can show 20% average CPU while the kernel is freezing it for 90% of every 100ms window — because CFS throttling happens in microscopic slices that dashboards average away. The number that tells the truth is the throttle ratio : the fraction of CFS periods in which the container was throttled.

cAdvisor exposes the raw counters. The robust query — the one that works on both cgroup v1 and v2 — is:

# Throttle ratio: fraction of CFS periods where the container was frozen
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_throttled_periods_total{container!="", container!="POD"}[5m])
)
/
sum by (namespace, pod, container) (
  rate(container_cpu_cfs_periods_total{container!="", container!="POD"}[5m])
)
Enter fullscreen mode Exit fullscreen mode

One caveat worth knowing: the companion metric container_cpu_cfs_throttled_seconds_total is not exported on cgroup v2 clusters, so dashboards built on it go silently blank after a node upgrade. The periods-based ratio above works on both. If you take one query from this post, take this one.

How to read it: a ratio hovering near zero means the limit is fine. A ratio consistently above roughly 0.1 on a latency-sensitive service means the container is spending a meaningful share of its life frozen — raise the CPU limit. The flat-topped usage graph is the visual twin of this number: usage pressed flat against the limit line is a workload asking for more CPU.

A throttled container's CPU usage flattened against its 500m limit versus a right-sized one at 1200m with natural peaks.

A flat top on the usage graph is the throttle ratio made visible. The dashed line is CPU the workload wanted but never got.

Stop looking at utilization. Look at throttling. Utilization tells you what the container got; the throttle ratio tells you what it was denied.

Memory: working set vs RSS, and the OOMKill verdict

Memory has its own trap: the number most people watch — RSS, or container_memory_usage_bytes — includes page cache that the kernel can reclaim under pressure. The number the kernel actually enforces against is the working set (container_memory_working_set_bytes): the memory that cannot be reclaimed. Size against the working set, not RSS, or you will set limits from a number that overstates real need — or worse, miss a leak hiding inside the cache.

# Working set as a fraction of the limit — the number that actually matters
container_memory_working_set_bytes{container!="", container!="POD"}
/
container_spec_memory_limit_bytes
Enter fullscreen mode Exit fullscreen mode

Unlike CPU, memory has no graceful degradation. There is no “memory throttling” — there is only the OOM killer. An OOMKilled container is the bluntest sizing signal Kubernetes produces: you set the memory limit too low , full stop. It is not ambiguous, it is not a gray area, and it should be treated as a sizing defect rather than an application bug until the working-set data says otherwise.

# Pods killed by the OOM killer — each one is a memory limit set too low
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} > 0
Enter fullscreen mode Exit fullscreen mode

A practical pattern (illustrative): a service whose working set sits at 85% of its limit with occasional OOMKills under load spikes is not “flaky” — it is under-provisioned by a predictable margin. Raise the limit to cover the p99 working set plus headroom, or better, find out what the spike allocates. The fix for repeated OOMKills is almost never “restart it and hope”; it is a bigger limit or a smaller appetite.

The forgotten eviction: ephemeral storage

Ephemeral storage is the sizing signal teams forget exists — until the kubelet starts evicting pods for it. Every container writes somewhere: logs, temp files, emptyDir volumes, image layers pulled onto the node. When a pod’s ephemeral-storage usage crosses its limit, there is no throttle and no warning. The kubelet evicts the pod outright, and the event reads like a node problem when it is really a sizing problem.

The defenses are the same shape as memory: set the request so the scheduler accounts for the disk, set the limit so runaway writes get caught, and watch actual usage. The field name in the manifest is ephemeral-storage, and it belongs in both blocks:

resources:
  requests:
    ephemeral-storage: 1Gi # reserve disk at scheduling time
  limits:
    ephemeral-storage: 2Gi # exceed this and the kubelet evicts the pod
Enter fullscreen mode Exit fullscreen mode

When a pod is evicted, check the eviction reason before blaming the node. If the message names ephemeral storage, the sizing loop applies: find what is writing (unbounded logs and debug dumps are the usual suspects), then raise the request/limit to match reality — or stop the writing. An eviction for disk is the storage twin of an OOMKill: blunt, unambiguous, and telling you the number was wrong.

The sizing loop: observe, recommend, adjust, re-observe

Signals tell you something is wrong; they don’t hand you the right number. That is what the Vertical Pod Autoscaler’s recommender is for. Run it in recommendation mode — updateMode: "Off" — and it watches usage history and publishes target requests and limits without touching a single running pod:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
  namespace: payments
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off" # recommendations only — never touches pods
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: "4"
          memory: 4Gi
Enter fullscreen mode Exit fullscreen mode

Read what it learned with kubectl get vpa api-vpa -o jsonpath='{.status.recommendation}'. Then the loop: compare the recommendation against the throttle ratio, the working set, and the eviction history; adjust one knob in the manifest; roll it out; re-observe. The update modes are explicit for a reason — Off for recommendations, Initial to apply at pod creation, Recreate to evict and resize, and the newer InPlaceOrRecreate where the cluster supports it. (Auto is deprecated upstream and behaves like Recreate.) Start in Off; promote only when the recommendations have earned your trust.

Telemetry flows from kubelet and cAdvisor through Prometheus to the VPA recommender, which feeds sizing decisions back into manifests.

Every arrow is a checkable step. Telemetry in, recommendations out, humans in the loop.

The decision order matters — check the signals in sequence, because each one points at exactly one knob:

Decision flowchart: throttling leads to raising the CPU limit, OOMKills to raising the memory limit, evictions to raising ephemeral storage, idle usage to lowering requests.

One signal, one knob. Work the chart top to bottom and the sizing stops being guesswork.

A quick-start checklist for the next workload you size:

  • [] Graph the throttle ratio for every container — anything sustained above ~0.1 gets a bigger CPU limit

  • [] Graph working set (not RSS) against the memory limit — headroom below the p99 spike, or expect OOMKills

  • [] Set ephemeral-storage requests and limits on anything that writes to disk — logs, temp files, emptyDir

  • [] Deploy a VPA in Off mode and let it observe for a full business cycle before trusting it

  • [] Change one knob at a time, roll it out, and re-observe — the loop is the method

  • [] Revisit quarterly: traffic patterns drift, and fossilized numbers rot

And the signal-to-action summary to pin next to your dashboards:

  • Throttle ratio > ~0.1 — Kernel is freezing the container. Knob to turn: Raise CPU limit.

  • Working set near limit — Real memory need exceeds the ceiling. Knob to turn: Raise memory limit.

  • OOMKilled — Memory limit was definitively too low. Knob to turn: Raise memory limit.

  • Evicted (ephemeral storage) — Disk usage exceeded the limit. Knob to turn: Raise ephemeral-storage; find the writer.

  • Usage far below requests — Capacity reserved but never used. Knob to turn: Lower requests.

The takeaway

Right-sizing is not a one-time exercise and it is not a guess. The signals are already there — throttle counters, working-set bytes, OOMKill events, eviction reasons — and the loop is simple: observe, compare against a recommendation, adjust one knob, re-observe. Workloads tell you their size continuously. The teams with the cheapest, fastest clusters are not the ones with the cleverest initial guesses; they are the ones that kept listening.

Pick one service this week. Graph its throttle ratio and its working set against its limits. If either number surprises you, the resources block was a fossil — and now you know exactly which knob to turn.


© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

I write about the unsexy work that keeps systems alive at 3am — monitoring, SRE, Terraform, and platform engineering.

New posts regularly. Follow along — and send this to the teammate whose pods keep getting OOMKilled.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.

  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

  • Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.

  • Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.

Top comments (0)