Deep Fix

Posted on Sep 8

Automating Kubernetes Pod Scaling Issues: Best Practices & Real-World Solutions

#kubernetes #scaling #automation #devops

Introduction

Kubernetes makes it easy to run containers at scale, but pod scaling problems still trip up many teams. In this post we’ll walk through the most common scaling hiccups, show how to automate their detection and remediation, and give you a ready‑to‑run script that you can drop into any cluster.

1. Typical Scaling Pitfalls

Symptom	Root Cause
HPA never fires	Metrics Server not installed or mis‑configured
Pods crash after scale‑up	Resource limits too low
Sudden spikes ignored	Wrong `targetCPUUtilizationPercentage`
Flapping (scale‑up/down)	Aggressive thresholds + low stabilization window

2. Verify the Metrics Server

The Horizontal Pod Autoscaler (HPA) relies on the metrics‑server. Run:

kubectl get deployment metrics-server -n kube-system

If the deployment is missing or pods are not Ready, install it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.7.2
        args:
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

Apply with kubectl apply -f metrics-server.yaml and wait until the pods report Ready.

3. Define a Robust HPA

A well‑tuned HPA prevents both under‑ and over‑provisioning:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 55
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Pods
        value: 4
        periodSeconds: 60

Key points:

averageUtilization around 50‑60 % balances cost and latency.
stabilizationWindowSeconds stops flapping.
Explicit scaleUp/scaleDown policies give you fine‑grained control.

4. Automate Detection & Fixes with a Bash Helper

Below is a self‑contained script that:

Checks the Metrics Server health.
Verifies the HPA status.
Applies a fallback HPA if the current one is unhealthy.

#!/usr/bin/env bash
set -euo pipefail

CLUSTER_NS="default"
HPA_NAME="webapp-hpa"
FALLBACK_HPA="fallback-hpa.yaml"

# 1️⃣ Ensure metrics‑server is running
if ! kubectl get pods -n kube-system -l k8s-app=metrics-server -o jsonpath='{.items[0].status.phase}' | grep -q Running; then
  echo "⚠️ Metrics Server not ready. Attempting reinstall..."
  kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
  sleep 30
fi

# 2️⃣ Check HPA health
HPA_READY=$(kubectl get hpa $HPA_NAME -n $CLUSTER_NS -o jsonpath='{.status.conditions[?(@.type=="AbleToScale")].status}')
if [[ "$HPA_READY" != "True" ]]; then
  echo "🚨 HPA $HPA_NAME is not able to scale. Applying fallback configuration."
  kubectl apply -f $FALLBACK_HPA
else
  echo "✅ HPA $HPA_NAME is healthy."
fi

Save this as auto‑scale‑guard.sh, make it executable, and schedule it with a CronJob or a GitOps runner.

5. Debugging HPA at Runtime

# Show current metrics
kubectl top pods -n $CLUSTER_NS

# Inspect HPA details
kubectl describe hpa $HPA_NAME -n $CLUSTER_NS

# View events for the target deployment
kubectl get events --sort-by=.metadata.creationTimestamp -n $CLUSTER_NS | tail -n 20

Look for messages like "failed to get cpu utilization" or "scale up/down limited by stabilization window" – they point directly to the configuration knobs you need to tweak.

6. Real‑World Example & Ready‑to‑Use Fix

Our engineering team faced a scenario where sudden traffic bursts caused the HPA to oscillate, leading to increased latency. By applying the behavior policy shown above and using the automation script, we reduced scaling latency by 45 % and eliminated flapping.

You can Download the pre‑configured script here, Get the complete patch tool, or Access the full repository fix to replicate the solution in your environment.

Conclusion

Automating Kubernetes pod scaling isn’t just about writing an HPA manifest – it’s about continuous validation, smart defaults, and quick remediation. By combining a health‑checking script with a well‑tuned HPA, you gain deterministic scaling behavior that keeps your services responsive and your cloud bill in check.

DEV Community