DEV Community

Deep Fix
Deep Fix

Posted on

Automating Kubernetes Pod Scaling Issues: Best Practices & Scripts

#kubernetes #automation #scaling #devops

Automating Kubernetes Pod Scaling Issues: Best Practices & Scripts

Scaling Kubernetes pods automatically is a cornerstone of cloud‑native reliability, but hidden pitfalls can cause erratic behavior, wasted resources, or even outages. In this guide we walk through the most common scaling hiccups, provide step‑by‑step remediation, and ship a ready‑to‑run script that keeps your Horizontal Pod Autoscaler (HPA) humming.

Why Scaling Can Fail

Symptom Typical Root Cause
HPA never fires Metrics‑Server not installed or API aggregation blocked
Pods over‑scale Incorrect targetCPUUtilizationPercentage or missing request/limit definitions
Pods under‑scale Custom metrics unavailable, or scaleDownDelay too aggressive
Flapping (scale up/down repeatedly) Low stabilizationWindowSeconds or noisy metric spikes

Understanding the why makes troubleshooting deterministic.

Prerequisites 

# Kubernetes 1.24+ (any managed service works)
# kubectl configured with cluster admin rights
# Helm 3 (optional but recommended)
Enter fullscreen mode Exit fullscreen mode

Step 1 – Install a Reliable Metrics Server 

# Using the official Helm chart
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
helm repo update
helm upgrade --install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --set args={--kubelet-insecure-tls,--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS}
Enter fullscreen mode Exit fullscreen mode

Verify:

kubectl get deployment metrics-server -n kube-system
kubectl top nodes
Enter fullscreen mode Exit fullscreen mode

Step 2 – Configure HPA with Proper Requests & Limits 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web
        image: nginx:stable-alpine
        resources:
          requests:
            cpu: "250m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
Enter fullscreen mode Exit fullscreen mode 
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 60
Enter fullscreen mode Exit fullscreen mode

Apply both manifests and watch:

kubectl apply -f deployment.yaml -f hpa.yaml
kubectl get hpa web-app-hpa --watch
Enter fullscreen mode Exit fullscreen mode

Step 3 – Automate HPA Validation with a Bash Helper

The script below checks the health of the Metrics Server, validates that every pod in the target deployment has CPU requests/limits, and prints a concise report. It can be scheduled via a CronJob or CI pipeline.

#!/usr/bin/env bash
set -euo pipefail

DEPLOYMENT=${1:-web-app}
NAMESPACE=${2:-default}

# 1️⃣ Verify Metrics Server
if ! kubectl get apiservices v1beta1.metrics.k8s.io -o jsonpath='{.status.conditions[?(@.type=="Available")].status}' | grep -q True; then
  echo "❗ Metrics Server unavailable. Install or troubleshoot first."
  exit 1
fi

echo "✅ Metrics Server is healthy"

# 2️⃣ Check resource requests/limits
MISSING=$(kubectl get deployment "$DEPLOYMENT" -n "$NAMESPACE" -o json |
  jq -r '.spec.template.spec.containers[] | select(.resources.requests.cpu==null or .resources.limits.cpu==null) | .name')

if [ -n "$MISSING" ]; then
  echo "❗ Containers missing CPU requests/limits: $MISSING"
  exit 1
fi

echo "✅ All containers define CPU requests & limits"

# 3️⃣ Summarize current HPA status
kubectl get hpa -n "$NAMESPACE" "$DEPLOYMENT" -o wide

# Optional: auto‑patch missing resources (use with caution)
# if [ "$AUTO_FIX" = "true" ]; then
#   kubectl set resources deployment "$DEPLOYMENT" -n "$NAMESPACE" --limits=cpu=500m,memory=256Mi --requests=cpu=250m,memory=128Mi
#   echo "🔧 Applied default resources"
# fi

echo "🎉 HPA validation completed"
Enter fullscreen mode Exit fullscreen mode

Save this as hpa-check.sh, make it executable (chmod +x hpa-check.sh), and run:

./hpa-check.sh web-app default
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Checklist

  1. Metrics Server – kubectl top pods returns data?
  2. API Aggregation – kubectl get apiservices shows v1beta1.metrics.k8s.io as Available.
  3. Resource Requests – Every container defines cpu request & limit.
  4. HPA Spec – Verify targetCPUUtilizationPercentage (or averageUtilization) aligns with actual load.
  5. Behavior Settings – Adjust stabilizationWindowSeconds to prevent flapping.
  6. Custom Metrics – If using Prometheus Adapter, ensure the adapter CRDs are installed and the metric name matches.

If you hit a wall, the script above will surface the most common mis‑configurations.

Ready‑to‑Deploy? Grab the Full Toolkit

With these resources you’ll spend less time firefighting scaling glitches and more time delivering value.

Happy autoscaling!

Top comments (0)