Kubernetes Cost Optimization: How to Right-Size Workloads with Goldilocks and VPA

Kubernetes resource requests and limits are one of the most consequential configuration decisions in cloud-native infrastructure, and one of the least data-driven. Most teams set them once during initial deployment, based on estimates or copied from a Stack Overflow answer, and never revisit them. The result is predictable, workloads are systematically over-provisioned to avoid OOM kills, node utilization stays at 20–40%, and cloud bills grow faster than the business demands it.

The irony is that right-sizing is not guesswork, it's a measurement problem. Every Kubernetes cluster already collects the utilization data needed to set accurate resource requests. The gap is tooling that surfaces those measurements as actionable recommendations without requiring engineers to manually analyze Prometheus queries for every workload.

Goldilocks and the Vertical Pod Autoscaler (VPA) close that gap. VPA observes actual resource consumption and computes right-sized recommendations. Goldilocks surfaces those recommendations in a readable dashboard, organized by namespace, with diff-ready output that shows exactly what to change in your Helm values or Kubernetes manifests.

This guide covers how to deploy and configure both tools, interpret their recommendations, implement right-sizing changes safely, and integrate continuous cost optimization into your engineering workflow.

The Resource Request Problem

To understand why right-sizing matters, start with how Kubernetes uses resource requests:

Resource requests are what the Kubernetes scheduler uses to decide which node to place a pod on. A pod requesting 500m CPU and 512Mi memory will only be scheduled on a node with that much unallocated capacity.

Resource limits cap what a pod can actually consume. A pod that exceeds its CPU limit is throttled. A pod that exceeds its memory limit is OOM killed.

The consequence of over-provisioning requests is that capacity is reserved on nodes that the workload never uses. A pod requesting 2000m CPU that actually uses 200m is holding a slot for 1800m CPU that no other pod can use, you're paying for 10x the compute the workload needs.

The consequence of under-provisioning limits is that pods get OOM killed, generating false incidents. CPU throttling degrades latency without generating visible errors, a silent performance problem that many teams misdiagnose.

Accurate resource requests and limits require knowing what workloads actually consume under real load, which requires measurement, not estimation.

Architecture: How VPA and Goldilocks Work Together

Vertical Pod Autoscaler (VPA) is a Kubernetes controller that observes pod resource utilization over time and computes recommendation ranges:

• Lower Bound - the minimum request that prevents resource starvation
• Target - the recommended request for typical workload behavior
• Upper Bound - the maximum consumed under peak conditions

VPA has three modes: Auto (applies recommendations automatically, with pod restarts), Recreate (applies on pod recreation only), and Off (computes recommendations but applies nothing). For most production environments, Off mode (recommendations without automatic application) is the right starting point.

Goldilocks wraps VPA's Off mode with a namespace-level dashboard that aggregates VPA recommendations across all deployments in a namespace. It creates a VerticalPodAutoscaler object in Off mode for every deployment it monitors, collects the resulting recommendations, and presents them in a UI showing current requests vs. recommended requests with quality-of-service (QoS) class implications.

The workflow is:

Workloads run → Metrics Server collects utilization →
VPA Recommender analyzes history →
Goldilocks reads VPA recommendations →
Dashboard shows what to change →
Engineers apply changes to Helm values / manifests →
Next cycle validates the new settings

Step 1 - Install Metrics Server

VPA depends on the Kubernetes Metrics Server for resource utilization data. Verify it is running:

kubectl top nodes
kubectl top pods -A

If kubectl top returns an error, install Metrics Server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify
kubectl get deployment metrics-server -n kube-system

Step 2 - Install the Vertical Pod Autoscaler

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA components
./hack/vpa-up.sh

# Verify all three VPA components are running
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxx   1/1   Running
# vpa-recommender-xxx            1/1   Running
# vpa-updater-xxx                1/1   Running

Alternatively, install via Helm:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace kube-system \
  --set recommender.enabled=true \
  --set updater.enabled=false \    # disable auto-updates - use Off mode
  --set admissionController.enabled=false

The VPA recommender is the only component required for Goldilocks, it computes recommendations. The updater and admission controller are needed only for automatic recommendation application, which we'll address separately.

Step 3 - Install Goldilocks

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace \
  --set dashboard.enabled=true \
  --set dashboard.service.type=ClusterIP

Verify Goldilocks is running:

kubectl get pods -n goldilocks
# goldilocks-controller-xxx    1/1   Running
# goldilocks-dashboard-xxx     1/1   Running

Step 4 - Enable Goldilocks Per Namespace

Goldilocks only monitors namespaces explicitly labeled for analysis. Label the namespaces you want to optimize:

# Enable Goldilocks for your application namespaces
kubectl label namespace orders-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace users-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace payments-service goldilocks.fairwinds.com/enabled=true

Once labeled, Goldilocks automatically creates a VerticalPodAutoscaler object in Off mode for every Deployment in the namespace. VPA begins observing resource utilization immediately.

Important: Allow at least 24–72 hours of observation before acting on recommendations. VPA's accuracy improves with more history, recommendations based on 15 minutes of data reflect startup behavior, not steady-state production utilization.

Step 5 - Accessing the Goldilocks Dashboard

# Port-forward the dashboard
kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80

# Open http://localhost:8080

The dashboard displays, per namespace and per deployment:

• Current requests and limits - what your manifests currently specify
• VPA recommendations - Lower Bound, Target, and Upper Bound for CPU and memory
• QoS class impact - whether your current and recommended settings result in Guaranteed, Burstable, or BestEffort quality-of-service
• Helm values diff - the exact resources block to paste into your Helm values file

A typical recommendation output looks like:

# Current (what you have)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "1Gi"

# Goldilocks recommended (VPA Target)
resources:
  requests:
    cpu: "87m"
    memory: "179Mi"
  limits:
    cpu: "262m"
    memory: "358Mi"

This workload is requesting 5.7x more CPU than it actually uses. Applying the recommendation would reduce its node footprint by ~83%, allowing the cluster to either schedule significantly more workloads on existing nodes or rightsize to fewer nodes.

Step 6 - Interpreting Recommendations Safely

Goldilocks recommendations are starting points, not gospel. Apply these interpretation rules before implementing changes:

Use the Upper Bound for Memory Limits

VPA's memory Target represents typical consumption. Set your memory limit to the Upper Bound, the peak observed usage, to avoid OOM kills during traffic spikes. Memory limits below observed peak are the primary cause of OOM kills.

resources:
  requests:
    memory: "179Mi"      # VPA Target → use as request
  limits:
    memory: "358Mi"      # VPA Upper Bound → use as limit (not Target)

Add a Safety Buffer for Memory Requests

For stateful or memory-sensitive workloads, add a 20–30% buffer above the VPA Target for memory requests:

# VPA Target: 179Mi → apply with 25% buffer
requests:
  memory: "224Mi"    # 179 × 1.25 = 223.75 → round up

Be Conservative with CPU Limits

Unlike memory, CPU throttling does not kill pods, it degrades latency. Setting CPU limits close to observed peak reduces burst headroom. Consider setting CPU limits to the Upper Bound and leaving headroom for traffic spikes:

requests:
  cpu: "87m"         # VPA Target - accurate for scheduling
limits:
  cpu: "500m"        # Upper Bound + buffer - prevents pathological consumption

Ignore Recommendations for Stateful Workloads

VPA recommendations for StatefulSets (databases, message brokers, caches) should be applied with extra caution, memory consumption patterns for stateful workloads are often highly variable and spike during compaction, reindexing, or failover operations. Treat VPA recommendations for stateful workloads as lower bounds, not targets.

Step 7 - Implementing Changes Safely

Never apply all VPA recommendations at once. A phased approach minimizes risk:

Phase 1 - Non-production environments first. Apply recommendations to staging and development. Observe for one week. Verify no OOM kills, no CPU throttling-related latency degradation, no unexpected behavior.

Phase 2 - Lowest-traffic production workloads. Apply to internal tools, admin services, and background workers, workloads with the lowest blast radius if a recommendation is wrong.

Phase 3 - Core application services. Apply to API services and user-facing workloads after observing Phase 2 outcomes. Monitor error rates, latency p95/p99, and OOM kill rates for 48 hours after each change.

For Helm-managed deployments, apply recommendations via values override:

# values/production/orders-service.yaml — after Goldilocks review
resources:
  requests:
    cpu: "100m"       # up from VPA's 87m — minor safety buffer
    memory: "224Mi"   # VPA Target (179Mi) + 25% buffer
  limits:
    cpu: "500m"       # VPA Upper Bound + margin
    memory: "360Mi"   # VPA Upper Bound

helm upgrade orders-service ./charts/orders-service \
  -f values/production/orders-service.yaml \
  --namespace orders-service

Step 8 - VPA Auto Mode for Non-Critical Workloads

For non-critical workloads (batch jobs, internal tools, development namespaces) where pod restarts are acceptable, VPA's Auto mode applies recommendations without manual intervention:

# vpa/orders-worker-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-worker-vpa
  namespace: orders-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  updatePolicy:
    updateMode: "Auto"   # applies recommendations automatically
  resourcePolicy:
    containerPolicies:
      - containerName: orders-worker
        minAllowed:
          cpu: "50m"
          memory: "64Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]

The minAllowed and maxAllowed bounds prevent VPA from recommending values outside your operational comfort zone, a safeguard against runaway recommendations if a workload behaves unusually during the observation window.

Step 9 - CI/CD Integration for Continuous Right-Sizing

Make right-sizing a recurring engineering practice rather than a one-time event by integrating Goldilocks into your workflow:

# .github/workflows/cost-review.yml
name: Monthly Cost Optimization Review

on:
  schedule:
    - cron: '0 9 1 * *'   # first day of each month at 9am

jobs:
  generate-recommendations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Export Goldilocks recommendations
        run: |
          kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80 &
          sleep 5

          # Export recommendations via Goldilocks API
          curl -s http://localhost:8080/api/v1/namespaces \
            | jq '.[] | {namespace: .name, recommendations: .recommendations}' \
            > recommendations.json

      - name: Create GitHub Issue with recommendations
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const recs = JSON.parse(fs.readFileSync('recommendations.json'));

            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Monthly Cost Optimization Review — ${new Date().toISOString().slice(0, 7)}`,
              body: `## Goldilocks Recommendations\n\n\`\`\`json\n${JSON.stringify(recs, null, 2)}\n\`\`\`\n\nReview and apply relevant recommendations following the right-sizing runbook.`,
              labels: ['cost-optimization', 'infrastructure'],
              assignees: ['platform-team'],
            });

A monthly GitHub issue with current recommendations creates a lightweight, trackable cost optimization cadence, without requiring engineers to proactively access the dashboard.

Measuring the Impact

Track right-sizing outcomes with these Grafana/Prometheus metrics:

# Node CPU utilization — target 60-70% for cost efficiency
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (node) * 100

# Pod CPU request vs actual usage (request efficiency)
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# OOM kill rate — should be zero after right-sizing
sum(increase(kube_pod_container_status_restarts_total[24h])) by (pod, namespace)

# Memory request efficiency
sum(container_memory_working_set_bytes) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="memory"}) by (pod, namespace)

A request efficiency ratio below 0.3 (pods using less than 30% of requested resources) is a strong signal of over-provisioning. A ratio above 0.85 sustained means requests are too tight, burst traffic risks throttling or OOM.

Common Pitfalls to Avoid

Acting on fresh recommendations. VPA recommendations based on less than 24 hours of data reflect startup patterns, not steady-state behavior. Wait at least 72 hours before applying recommendations to production workloads.

Setting memory limits equal to requests. This creates Guaranteed QoS class pods, Kubernetes evicts Burstable pods before Guaranteed ones, which sounds good but means any memory spike above the limit results in an OOM kill with no burst headroom. Set limits meaningfully above requests.

Applying VPA Auto mode to HPA-managed deployments. VPA and HPA conflict when both target the same metric. If you use HPA for CPU-based scaling, configure VPA to manage only memory: controlledResources: ["memory"].

Ignoring the QoS class implications. Guaranteed QoS (requests = limits) gets the highest scheduling priority but no burst headroom. Burstable QoS (requests < limits) allows bursting but can be evicted under node pressure. Understand which class your workloads land in after right-sizing.

Right-sizing without load testing. VPA recommendations are based on historical traffic. If your staging environment doesn't reflect production load patterns, its recommendations don't reflect production requirements. Right-size production workloads using production observation data.

Conclusion

Kubernetes cost optimization is not a one-time project, it's an ongoing engineering discipline driven by measurement rather than estimation. VPA provides the utilization data. Goldilocks makes that data actionable. The right-sizing process, observe, recommend, review, apply, measure, is the feedback loop that keeps cluster efficiency high as workloads evolve.

The typical outcome of a systematic right-sizing initiative is a 30–50% reduction in resource requests, translating directly to fewer nodes needed to run the same workloads, and a proportional reduction in cloud spend. More importantly, accurate resource requests improve scheduling density, reduce node pressure evictions, and eliminate the silent CPU throttling that degrades latency without generating visible errors.

Start with one namespace. Label it for Goldilocks, wait 72 hours, review the dashboard, and apply the most conservative recommendations first. The data will tell you what to do, you just have to make time to look at it.

Using Karpenter for node provisioning alongside VPA for pod right-sizing? The combination is particularly effective, Karpenter provisions exactly the node types needed for right-sized pod requests, eliminating both over-provisioned pods and over-provisioned nodes simultaneously.

Kubernetes #CostOptimization #Goldilocks #VPA #CloudCost