Hands-On Kubernetes Autoscaling: Building a Local HPA Lab with KinD and Python

Have you ever wanted to experiment with Kubernetes Horizontal Pod Autoscaler (HPA) without spinning up costly cloud clusters on AWS, GCP, or Azure?

Testing autoscaling locally in your homelab or dev environment is one of the most effective ways to master Kubernetes workload mechanics. But when you try setting up HPA locally using KinD (Kubernetes in Docker), you quickly discover that it is not just about applying an HPA YAML file. You have to wire up metrics collection, handle container-to-host networking, configure CPU requests accurately, and navigate self-signed certificates.

In this walkthrough, I will take you through building a complete, local autoscaling lab from scratch—from writing a lightweight Python web service to triggering real-time pod scaling under simulated traffic surges.

🏛️ Architecture at a Glance

Here is how the components interact inside our local KinD cluster:

flowchart TD
    subgraph Traffic["Workload Ingress"]
        Generator["Load Generator Pod (BusyBox)"]
    end

    subgraph Cluster["KinD Cluster (Docker)"]
        SVC["Service: autoscaling-service (ClusterIP: 80 -> 8000)"]

        subgraph Workload["Deployment: autoscaling-app"]
            P1["Pod 1 (Base)"]
            P2["Pod 2 (Scaled)"]
            PN["Pod ... (Up to 5)"]
        end

        subgraph MetricsControl["Metrics & Autoscaling Pipeline"]
            Kubelet["Kubelet Summary API"]
            MS["Metrics Server\n(--kubelet-insecure-tls)"]
            HPA["HPA Controller\n(Target: 50% CPU)"]
        end
    end

    Generator -->|HTTP Flood| SVC
    SVC --> P1
    SVC -.-> P2
    SVC -.-> PN

    Kubelet -->|Scrapes CPU/Mem| MS
    MS -->|Custom Metrics API| HPA
    HPA -->|Adjust Replicas: 1 ➔ 5| Workload

The Workflow:

1. A Python HTTP app runs in a Kubernetes deployment with defined CPU requests (100m) and limits (500m).
2. An HPA controller targets 50% average CPU utilization across pods.
3. Metrics Server collects pod resource consumption from the Kubelet.
4. When we simulate traffic using a load generator pod, the CPU threshold is breached, and Kubernetes automatically provisions up to 5 replicas. Once load drops, the cluster safely scales back down to 1 replica.

---

🛠️ Prerequisites & Local Tooling

To follow along, make sure you have:

• Docker Desktop (or Docker Engine)
• KinD (brew install kind or via mise)
• kubectl (brew install kubectl or via mise)

(Optional) If you use mise or VS Code Dev Containers, you can manage these CLI tools declaratively with a mise.toml configuration:

[tools]
docker-cli = "latest"
kind = "latest"
kubectl = "latest"
python = "latest"

---

Step 1: Writing the Python Microservice

We keep the application clean and lightweight using Python's built-in http.server. It exposes a root route for service requests and a /health endpoint for Kubernetes health probes.

app.py:

from http.server import BaseHTTPRequestHandler, HTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/health":
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"OK")
            return

        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"Kubernetes Autoscaling App")

    def log_message(self, format, *args):
        return


server = HTTPServer(("0.0.0.0", 8000), Handler)
server.serve_forever()

Next, wrap it in a slim Docker image using python:3.12-slim:

Dockerfile:

FROM python:3.12-slim

WORKDIR /app
COPY app.py .
EXPOSE 8000

CMD ["python", "app.py"]

Build the image locally:

docker build -t kubernetes-autoscaling:1.0 .

---

Step 2: Creating the KinD Cluster (with DevContainer / Host Networking Support)

If you work inside Dev Containers or DevPod, running kubectl from inside a container to reach a KinD cluster running on the host requires special attention: 127.0.0.1 inside a container refers to the container itself, not the host machine!

To fix this, we configure certSANs in kind-config.yaml to include host.docker.internal:

kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      certSANs:
        - host.docker.internal

nodes:
  - role: control-plane

Create the cluster:

kind create cluster --name autoscaling-demo --config kind-config.yaml

Now load the locally built Docker image into the KinD node:

kind load docker-image kubernetes-autoscaling:1.0 --name autoscaling-demo

💡 Tip: In local KinD setups, Kubernetes cannot pull images from your host's local Docker daemon unless you explicitly load them using kind load docker-image. Also set imagePullPolicy: IfNotPresent in your deployment manifest.

---

Step 3: Deploying and Patching Metrics Server

By default, Kubernetes HPA cannot autoscale without Metrics Server.

In production, Kubelets have valid certificates. In KinD, Kubelet serving certificates are self-signed, causing Metrics Server to throw certificate verification errors.

1. Deploy the official Metrics Server:

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

1. Patch Metrics Server with --kubelet-insecure-tls:

   kubectl patch -n kube-system deployment metrics-server --type='json' -p='[
     {"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}
   ]'

1. Verify that metrics are flowing:

   kubectl rollout status -n kube-system deployment metrics-server
   kubectl top nodes

Once kubectl top nodes outputs CPU and memory metrics without errors, your metrics pipeline is operational.

---

Step 4: Crafting the Kubernetes Manifests

1. The Deployment (k8s/deployment.yaml)

Notice the resources.requests.cpu: "100m" specification. This is critical. HPA calculates CPU utilization percentage as:

$$\text{Utilization} = \frac{\text{Current CPU Usage}}{\text{Requested CPU}} \times 100$$

Without a defined CPU request, HPA cannot compute percentage-based targets and will report <unknown>/50%.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: autoscaling-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: autoscaling-app
  template:
    metadata:
      labels:
        app: autoscaling-app
    spec:
      containers:
        - name: autoscaling-app
          image: kubernetes-autoscaling:1.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "500m"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 2
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10

2. The Service (k8s/service.yaml)

A ClusterIP service provides a steady, stable DNS entry point (autoscaling-service) regardless of how many pods scale up or down:

apiVersion: v1
kind: Service
metadata:
  name: autoscaling-service
spec:
  selector:
    app: autoscaling-app
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP

3. The HorizontalPodAutoscaler (k8s/hpa.yaml)

We configure scaling boundaries between 1 and 5 replicas, targeting 50% average CPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: autoscaling-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: autoscaling-app
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

Deploy everything to your cluster:

kubectl apply -f k8s/

Check the initial state:

kubectl get hpa autoscaling-app

You should see:

NAME              REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
autoscaling-app   Deployment/autoscaling-app   0%/50%    1         5         1          1m

---

Step 5: Putting the Autoscaler to the Test

Now comes the fun part: simulating high traffic load and watching Kubernetes respond in real time.

1. Watch HPA and Pods

Open two separate terminal windows or split panes:

Terminal 1 (Watch HPA):

kubectl get hpa autoscaling-app -w

Terminal 2 (Watch Pods):

kubectl get pods -l app=autoscaling-app -w

2. Trigger the Load Generator

In a third terminal, run an ephemeral BusyBox pod that hammers the service with continuous HTTP requests:

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://autoscaling-service; done"

3. The Scale-Up Reaction

Within 1–2 minutes, watch the CPU utilization spike well above the 50% target:

NAME              REFERENCE                    TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
autoscaling-app   Deployment/autoscaling-app   0%/50%     1         5         1          2m
autoscaling-app   Deployment/autoscaling-app   78%/50%    1         5         2          3m
autoscaling-app   Deployment/autoscaling-app   134%/50%   1         5         4          4m
autoscaling-app   Deployment/autoscaling-app   55%/50%    1         5         5          5m

Kubernetes automatically scales the deployment to maximum capacity (5 pods) to absorb the incoming flood.

4. The Scale-Down Cooldown

Terminate the load generator by pressing Ctrl + C.

Kubernetes uses a default stabilization window (typically 5 minutes) before scaling down to prevent pod "flapping" (rapid oscillation between scale-up and scale-down). Once the cooldown elapses and CPU utilization drops to 0%, HPA smoothly brings replica count back to 1.

---

🧠 Key Takeaways & Troubleshooting Wisdom

Setting up this lab highlights several foundational concepts every Kubernetes practitioner should master:

1. Host Networking vs Container Contexts: When running inside Docker containers or DevContainers, 127.0.0.1 refers to the container itself. Use host.docker.internal along with certSANs in your control plane config to enable seamless cross-environment communication.
2. CPU Requests are Not Optional for HPA: If you don't define resources.requests.cpu, HPA cannot calculate percentage-based utilization and will stall in an <unknown> status.
3. Kubelet TLS in Local Clusters: KinD nodes rely on self-signed certs. Metrics Server requires --kubelet-insecure-tls in local testbeds to successfully pull node metrics.
4. Readiness Probes Prevent Traffic Blackholes: During sudden scale-up, newly provisioned pods must pass their /health readiness check before receiving ingress traffic from the Service.
5. The 7-Layer Kubernetes Debugging Workflow: Whenever autoscaling behaves unexpectedly, check in order:
   • Pod Status (kubectl get pods)
   • Deployment Status (kubectl get deployment)
   • Service & Endpoints (kubectl get endpoints)
   • Metrics Server Health (kubectl top pods, nodes)
   • HPA Status & Events (kubectl describe hpa)
   • Cluster Events (kubectl get events --sort-by='.metadata.creationTimestamp')
   • Pod Logs (kubectl logs)

---

🏁 Conclusion

Building an autoscaling environment on your local machine gives you deep visibility into how Kubernetes orchestrates scaling events, monitors workloads, and manages service discovery—all without cloud provider costs or latency.

Try tweaking the HPA algorithm with custom metrics, memory thresholds, or scaling behaviors next!

Have questions or run into issues setting up your own KinD HPA lab? Drop a comment below and let's discuss!