Every VPA article you've read covers the same three things: install it, set updateMode, watch it resize your pods. A few of the better ones now cover in-place resize and its fallback-to-eviction gotcha. Almost none of them explain what the recommender is doing internally to arrive at a number, which is a problem, because that's exactly the part that makes VPA feel wrong half the time. You bump a pod's memory during an incident, VPA quietly walks it back down a week later, and it looks like a bug. It isn't. It's a specific statistical model doing exactly what it was built to do, and once you see the model, the behavior stops being mysterious.
VPA doesn't track usage. It tracks a decaying histogram of usage.
The recommender doesn't average your pod's CPU and memory over some window and call it a day. For each container, it builds two histograms, one for CPU, one for memory, and every sample it takes gets added as a weighted bucket. The weight isn't equal across time. Older samples get discounted through exponential decay, with a half-life on the order of a day for CPU and closer to a week for memory. That means a spike from three weeks ago hasn't vanished from the histogram. It's still in there, just contributing a fraction of what it did on day one.
This is why a memory recommendation can take what feels like forever to come back down after an incident, and why it never quite drops to what you'd expect from "current" usage alone. You're not looking at current usage. You're looking at a weighted blend of current and historical usage, and the blend only clears out gradually.
The recommendation is a percentile, not a peak or an average
VPA doesn't recommend "the highest memory usage we ever saw" or "the average." It targets a percentile of the histogram, and it computes three separate numbers off the same distribution:
- Target: usually the 90th percentile of usage
- Lower bound: a conservative floor, roughly the 50th percentile with a safety margin
- Upper bound: a ceiling near the 95th-99th percentile, used to decide whether an in-place resize is even worth attempting
The gap between lower and upper bound isn't cosmetic. It's a confidence interval that shrinks as more samples accumulate. A pod that's been running for six hours has a wide gap because the recommender doesn't trust its own histogram yet. A pod running for two weeks has a tight gap because the sample count is large enough to be confident in the shape of the distribution. This is also why a freshly deployed workload gets what looks like an overly generous recommendation at first: low sample count means wide bounds, and VPA rounds up when it isn't sure.
The OOM bump: the one input that skips the histogram entirely
There's exactly one signal that doesn't go through the normal decay-weighted percentile math: an OOMKill. When the recommender sees a container got OOMKilled, it doesn't wait for the next scheduled recommendation cycle to react gracefully. It immediately bumps the memory recommendation for that container, typically by adding a fixed increment on top of whatever the container was using when it died, specifically so the next restart doesn't die the same way.
This matters because it means your memory recommendations are not purely a statistical function of usage. They're a statistical function of usage, plus a ratchet that only moves in one direction after an OOM event. If you're wondering why a container's memory floor jumped by exactly 250Mi (a common default bump) right after an incident and didn't drift back down smoothly, this is why. It's not part of the percentile curve. It's a separate correction layered on top.
Three datasets, worked by hand
The math above is easy to state and hard to picture, so here are three small datasets run through it by hand. Real VPA samples every few seconds, but the mechanics are identical with one sample a day, and one-a-day is what a human can actually check with a calculator.
Weight formula used in all three: weight = 0.5^(age_in_days / half_life). CPU below uses a 1-day half-life, matching VPA's default order of magnitude for CPU.
Dataset A: a steady service with one unrelated spike
Ten days of CPU samples in millicores, oldest to newest: 110, 105, 120, 115, 108, 112, 300, 118, 122, 119. The 300 on day 7 was a one-off, a deploy that briefly pegged CPU during a cache warm.
| Day (age) | Value | Weight (0.5^age) | Weighted value |
|---|---|---|---|
| 10 (age 0) | 119 | 1.000 | 119.0 |
| 9 (age 1) | 122 | 0.500 | 61.0 |
| 8 (age 2) | 118 | 0.250 | 29.5 |
| 7 (age 3) | 300 | 0.125 | 37.5 |
| 6 (age 4) | 112 | 0.0625 | 7.0 |
| 5 (age 5) | 108 | 0.03125 | 3.4 |
| 4 (age 6) | 115 | 0.015625 | 1.8 |
| 3 (age 7) | 120 | 0.0078 | 0.9 |
| 2 (age 8) | 105 | 0.0039 | 0.4 |
| 1 (age 9) | 110 | 0.00195 | 0.2 |
Total weight ≈ 1.998. Sort by value and walk cumulative weight until you cross 90% of the total (1.998 × 0.9 = 1.798): 105, 108, 110, 112, 115, 118, 119 get you to a cumulative weight of 1.365; adding 120 gets you to 1.373; adding 122 gets you to 1.873, which crosses the 1.798 threshold. The 90th percentile lands at 122m, not anywhere near 300. The spike is still in the histogram, contributing its 0.125 of weight, but four days of decay knocked it down enough that it barely nudges the target. This is the mechanism, in numbers, behind "VPA doesn't overreact to a single bad day."
Dataset B: the same shape, but the spike keeps recurring
Same ten days, same 1-day half-life, but now the "spike" is a nightly batch job that runs every day: 95, 600, 92, 600, 88, 600, 91, 600, 90, 600, alternating baseline and batch run. Every recent day still carries a near-full weight, and half of those recent days are 600. Walking the same cumulative-weight process, the 600 values alone account for roughly half the total weight and nearly all of the weight in the most recent, least-decayed days. The 90th percentile comes out at 600m, essentially the batch peak itself. Nothing about the histogram math changed between Dataset A and B. What changed is that a recurring pattern keeps refreshing its own weight before it can decay, while a one-off spike only ever gets to decay. That's the entire difference between "VPA ignores this" and "VPA sizes for this."
Dataset C: an OOMKill overrides the histogram entirely
A container has a clean, boring memory history: ten days of samples between 200Mi and 230Mi, decay-weighted 90th percentile comes out to 230Mi using the same method as above. On day 11, the container gets OOMKilled while sitting at its 256Mi limit. The recommender doesn't wait for day 12's data point to update the percentile gracefully. It applies the bump directly: new recommendation = max(histogram target, usage at OOM + fixed bump), and with a typical 250Mi bump that's max(230Mi, 256Mi + 250Mi) = 506Mi. The next recommendation is 506Mi, more than double the pre-incident percentile, and it did not come from the histogram at all. If you're watching the recommendation feed and see a number that doesn't fit the trend line of the last ten days, this is almost always why: an OOM event took a shortcut around the whole percentile calculation.
The resource hierarchy VPA has to fit inside
None of the math above happens in a vacuum. A VPA recommendation gets applied to a pod, but the pod sits inside a deployment, the deployment sits inside a namespace with its own quota and default ranges, and the namespace sits inside a cluster with finite node capacity. VPA can compute a mathematically correct number and still have it rejected, deferred, or silently capped at any of those layers.
Cluster level. Twelve nodes at 16 vCPU / 64Gi each gives roughly 192 vCPU and 768Gi of raw capacity, and after kubelet and system reservations, maybe 172 vCPU and 700Gi actually schedulable. This is the outer ceiling. Nothing below can exceed it in aggregate, no matter what any recommender computes.
Namespace level. A ResourceQuota on the checkout namespace might cap requests.cpu at 32 and limits.cpu at 64. A LimitRange in the same namespace might set a default request of 250m, a default limit of 500m, and a hard per-container max of 2 vCPU. These aren't advisory. They're enforced at admission time.
Deployment level. Say checkout-api runs 8 replicas, each currently requesting 620m CPU and 780Mi memory (numbers that could plausibly come straight out of Dataset A or B's math). Total requested across the deployment: 8 × 620m = 4.96 vCPU, 8 × 780Mi ≈ 6.1Gi. Comfortably inside the namespace quota of 32 vCPU and, say, 64Gi.
Pod level, and where it breaks. Now traffic grows and HPA scales the same deployment from 8 replicas to 30. Total requested CPU becomes 30 × 620m = 18.6 vCPU, still under the 32 vCPU quota, so this scales fine. But suppose the VPA recommender, watching the same growth in per-pod load, also raises its per-container target from 620m to 1.2 vCPU. Now 30 × 1.2 = 36 vCPU, which exceeds the 32 vCPU namespace quota. The admission controller won't let all 30 pods hold that request simultaneously. What actually happens in practice: some pods get resized in-place successfully before the quota is exhausted, and the rest stay at their old, smaller request, deferred, retried every reconciliation loop, and rejected every time until either quota goes up or replica count comes down. You end up with a deployment where pods of the same spec are running with two different actual CPU requests, which looks like a bug in kubectl output and is actually quota exhaustion happening one pod at a time.
There's a second, quieter version of this same failure at the LimitRange layer. If that namespace's LimitRange caps per-container CPU at 2 vCPU (2000m) and the recommender's own upper bound, computed from actual usage, comes out to 2.4 vCPU, the patch VPA tries to apply gets rejected by the LimitRange admission check, not by quota. Unless the VPA object's own resourcePolicy.containerPolicies[].maxAllowed is explicitly set at or below 2000m, VPA will keep proposing 2.4 vCPU, keep getting rejected, and keep showing a pending resize with no obvious error in kubectl describe pod. The fix is boring and easy to miss: maxAllowed on the VPA object should always be set at or below whatever the namespace's LimitRange maximum is, otherwise the two systems disagree forever and nobody gets an alert about it.
Why VPA and HPA fight when they're pointed at the same resource
This part gets mentioned in passing in a lot of docs but rarely explained mechanically. HPA scales replica count based on a metric, commonly CPU utilization as a percentage of the requested value. VPA changes the request itself. If VPA lowers a container's CPU request from 500m to 300m while HPA is watching CPU utilization percentage, the percentage calculation's denominator just moved. The same absolute CPU usage now reads as a higher utilization percentage, which can trigger HPA to scale out even though nothing about actual load changed. Two controllers, each doing exactly what they're supposed to do, produce a scaling decision neither of them would make in isolation.
The fix isn't "don't use both." It's scoping them to different resources or different signals: let VPA own memory (where HPA rarely operates well anyway) and let HPA own CPU-based horizontal scaling, or move HPA to a custom metric that doesn't shift when VPA changes requests. Running both against CPU utilization on the same object is the one combination that reliably produces the "why is this pod count oscillating for no reason" tickets.
Forcing the recommender to forget faster
If you know a spike was a one-time event (a load test, a migration, a Black Friday spike that isn't representative of steady state), you have two real options, and neither of them is "wait it out."
- Delete and let the VPA object rebuild its histogram. The recommender's state lives in the VPA controller's memory and checkpoints, tied to the object. Deleting and recreating the VerticalPodAutoscaler resource clears the accumulated histogram for that target, at the cost of losing the confidence built up from normal traffic too. Blunt, but sometimes exactly what you want.
-
Shrink the decay half-life via recommender flags (
--memory-histogram-decay-half-life,--cpu-histogram-decay-half-lifeon the recommender deployment). Lowering these makes the whole system more reactive to recent behavior and less resistant to noise, which is a real trade-off: you'll adapt to genuine trend changes faster, and you'll also chase noise more aggressively during a bad afternoon.
There's no setting that says "forget just that one Tuesday." The histogram doesn't have a concept of an anomalous day versus a representative one. It only has decayed weight.
What to actually do with this
- Don't be surprised that a memory recommendation lags behind current usage for days after an incident. That's the decay half-life working as designed, not the recommender being slow to update.
- Watch for OOMKill-driven jumps that don't match your regular percentile trend. Those come from the OOM bump logic, not from the histogram, and they won't smoothly re-derive back down without another cycle of clean data pushing the histogram in a different direction.
- If HPA and VPA are both watching CPU on the same workload, expect oscillation. Split them across resources or metrics before you spend a week debugging replica counts that seem to move on their own.
- If you need the recommender to forget a known one-off event, decide up front whether to reset the object (losing all history) or tune the decay half-life globally (accepting more noise sensitivity), because there's no per-incident undo button.
VPA isn't guessing. It's running a specific, fairly conservative statistical model with decay, percentiles, and confidence bounds, layered with one hard-coded escape hatch for OOM events. Once you know that's what's under the hood, the recommendations stop looking arbitrary. They start looking like exactly what a bounded, memory-limited, slightly stubborn statistical process would produce.
Top comments (0)