DEV Community

Deep Fix
Deep Fix

Posted on

Automating Kubernetes Pod Scaling: A Complete Guide to Fix Common HPA Issues

#kubernetes #scaling #automation #devops

Introduction

Scaling Kubernetes pods automatically is essential for modern cloud‑native applications, but the Horizontal Pod Autoscaler (HPA) can misbehave in subtle ways. This post walks you through the most common scaling pitfalls and shows how to automate detection and remediation.

Typical Scaling Issues

Symptom Likely Cause
HPA never scales up Metrics server down, wrong metric name, missing resource requests
Pods oscillate Aggressive behavior settings or low cpuUtilizationTarget
Scaling delays > 5 min --horizontal-pod-autoscaler-sync-period too high

Step‑by‑Step Automated Troubleshooting

1. Verify the Metrics Server 

kubectl get pods -n kube-system | grep metrics-server
kubectl logs -n kube-system deployment/metrics-server
Enter fullscreen mode Exit fullscreen mode

If the server is not running, redeploy it:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Enter fullscreen mode Exit fullscreen mode

2. Inspect the HPA Definition 

kubectl get hpa -o yaml > hpa-backup.yaml
kubectl describe hpa my‑app‑hpa
Enter fullscreen mode Exit fullscreen mode

Key fields to check:

  • spec.minReplicas / spec.maxReplicas
  • metrics (e.g., resource.name: cpu)
  • behavior (stabilization window)

3. Ensure Pods Declare Requests & Limits

A pod without resource requests will never generate metrics for the HPA.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: app
        image: myrepo/app:latest
        resources:
          requests:
            cpu: "250m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
Enter fullscreen mode Exit fullscreen mode

4. Simulate Load to Verify Scaling 

kubectl run load-generator --image=busybox --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://my-app; done"
Enter fullscreen mode Exit fullscreen mode

Watch the HPA react:

watch kubectl get hpa my-app-hpa
Enter fullscreen mode Exit fullscreen mode

5. Automate the Fixes with a Script

The following Bash script checks the three checkpoints above and applies corrective actions automatically.

#!/usr/bin/env bash
set -euo pipefail

# 1️⃣ Metrics server health
if ! kubectl get pods -n kube-system -l k8s-app=metrics-server -o jsonpath='{.items[0].status.phase}' | grep -q Running; then
  echo "Metrics server not running – reinstalling…"
  kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
fi

# 2️⃣ HPA sanity
HPA_NAME="my-app-hpa"
if ! kubectl get hpa "$HPA_NAME" >/dev/null 2>&1; then
  echo "HPA $HPA_NAME missing – creating a default one"
  cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: $HPA_NAME
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
EOF
fi

# 3️⃣ Resource requests check
if ! kubectl get deployment my-app -o jsonpath='{.spec.template.spec.containers[*].resources.requests.cpu}' | grep -q .; then
  echo "Adding default CPU/memory requests to my-app deployment…"
  kubectl patch deployment my-app --type='json' -p='[{"op":"add","path":"/spec/template/spec/containers/0/resources","value":{"requests":{"cpu":"250m","memory":"128Mi"},"limits":{"cpu":"500m","memory":"256Mi"}}}]'
fi

echo "All checks passed. Your HPA should now scale correctly."
Enter fullscreen mode Exit fullscreen mode

Pro tip: Schedule this script with a CronJob or GitOps pipeline to keep your cluster self‑healing.

Natural Outbound Links

Conclusion

By systematically validating the metrics pipeline, HPA spec, and pod resource declarations, you can eliminate the majority of scaling failures. Automating these checks turns a reactive debugging session into a proactive reliability layer.

Top comments (0)