Deep Fix

Posted on Sep 5

Automating Kubernetes Pod Scaling Issues: Best Practices, Scripts & Real-World Solutions

#kubernetes #scaling #automation #devops

Introduction

Kubernetes makes horizontal pod autoscaling (HPA) feel effortless, but real‑world clusters often hit scaling glitches that stall deployments and inflate costs. This guide walks you through diagnosing common scaling roadblocks and automating their resolution with scripts you can drop into your CI/CD pipeline.

Why Pods Fail to Scale

Symptom	Typical Cause
HPA never fires	Metrics Server missing or mis‑configured
Pods crash after scale‑up	Resource limits too low
Scale‑down stalls	Finalizers or dangling PVCs
Erratic scaling	Custom metrics latency

Understanding the root cause is the first step toward automation.

Step‑by‑Step Troubleshooting

1️⃣ Verify the Metrics Server

# Check if the metrics API is reachable
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .

If you see a 403 or 404, reinstall the metrics server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

2️⃣ Tune Resource Requests & Limits

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: app
        image: myrepo/web:latest
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Adjust the numbers until the HPA can safely add more pods without OOM kills.

3️⃣ Validate HPA Configuration

kubectl describe hpa web-app-hpa

Key fields to watch:

minReplicas / maxReplicas
targetCPUUtilizationPercentage
metrics (custom vs. resource)

4️⃣ Automate Health Checks & Remediation

Create a lightweight Bash script that runs as a CronJob and fixes the most common issues.

#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="default"
HPA_NAME="web-app-hpa"

# 1️⃣ Ensure metrics server is alive
if ! kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | grep -q "items"; then
  echo "⚠️ Metrics server down – reinstalling..."
  kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
fi

# 2️⃣ Patch HPA if CPU target is too aggressive
CURRENT_CPU=$(kubectl get hpa $HPA_NAME -n $NAMESPACE -o jsonpath='{.spec.metrics[0].resource.target.averageUtilization}')
if [[ $CURRENT_CPU -gt 80 ]]; then
  echo "⚙️ Adjusting CPU target from $CURRENT_CPU% to 70%"
  kubectl patch hpa $HPA_NAME -n $NAMESPACE --type='merge' -p '{"spec":{"metrics":[{"type":"Resource","resource":{"name":"cpu","target":{"type":"Utilization","averageUtilization":70}}}]}}'
fi

# 3️⃣ Clean up stuck pods preventing scale‑down
kubectl get pods -n $NAMESPACE --field-selector=status.phase=Running | while read -r pod _; do
  AGE=$(kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.metadata.creationTimestamp}')
  if [[ $(date -d "$AGE" +%s) -lt $(date -d "5 minutes ago" +%s) ]]; then
    continue
  fi
  # Example heuristic: delete pods with a specific label that never become Ready
  if kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' | grep -q False; then
    echo "🗑️ Deleting stuck pod $pod"
    kubectl delete pod $pod -n $NAMESPACE
  fi
done

Save this as auto‑scale‑fix.sh and schedule it:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: autoscale‑fixer
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: fixer
            image: bitnami/kubectl:latest
            command: ["/bin/bash", "-c", "$(cat /scripts/auto-scale-fix.sh)"]
            volumeMounts:
            - name: script
              mountPath: /scripts
          restartPolicy: OnFailure
          volumes:
          - name: script
            configMap:
              name: autoscale‑script-cm

5️⃣ Log Aggregation for Post‑Mortem

kubectl logs -l app=web-app -c app --tail=200 | grep -i "oom" > /tmp/oom‑alerts.log

Feed the extracted logs into your observability platform (Prometheus, Loki, etc.) to spot recurring patterns.

Full Automation Pack

We've bundled the script, CronJob manifest, and a ready‑to‑apply Helm chart in a public repo. Download the pre‑configured script here, or if you prefer the whole package, Get the complete patch tool. For a deeper dive, Access the full repository fix.

Conclusion

Automating the detection and remediation of scaling hiccups saves time, reduces cloud spend, and keeps your services responsive. Integrate the snippets above into your GitOps workflow, monitor the metrics, and let the CronJob handle the rest.

Happy scaling!

DEV Community