DEV Community: Duncan

Misadventures in Kubernetes: Autoscaling Workers

Duncan — Sun, 10 May 2026 01:31:11 +0000

So we’ve already done a few things for setting up our own custom cluster. We’ve manually configured a Kubernetes Control Plane and joined worker nodes by hand. While that was a great way for us to learn the components, let’s be honest: setting up every server by hand is just not scalable for a real production environment. Or for creating a more resilient cluster as well! Really the main issue boils down to how we set up the initial pool of worker nodes.

If what I’m talking about us doing isn’t familiar you should first read our prior post!

The Problem with Manual Nodes

Right now, our cluster is static, and honestly, it’s a bit of a liability. If worker-1 decides to take an unscheduled vacation and crashes, it’s just gone. Our capacity takes a hit, and we are stuck in the dark until someone manually notices and provisions a replacement. In a modern setup, having to SSH into every new VM just to run a join command isn’t just tedious, it’s a bottleneck that keeps us from scaling when it actually matters. Not only that, but it also makes it hard to meet the expectations of what we thought Kubernetes would be able to help as well, no!?

There are a few things that we need our cluster to be able to do:

Automatic Joining: New nodes should just join the cluster the moment they boot up, no human needed.
Self-Healing: If a node dies, the system should recognize the loss and automatically spin up a healthy replacement.
Smart Scaling: The cluster needs to get bigger when the load increases and shrink back down when things quieten down to save money.

Startup Scripts

The key to our automation is ensuring a new VM joins the cluster automatically when it boots. We can’t be there to SSH in and run kubeadm join every time.

To achieve this, we will use a GCP Startup Script to run the join command for us.

Step 1: Generate a Permanent Token

Standard kubeadm tokens expire after 24 hours. For our autoscaling group that might last months, we need a permanent token.

On your Control Plane, run:

kubeadm token create --print-join-command --ttl 0

Copy the output command, as you’ll need it for the next step.

Step 2: Create an Instance Template

An Instance Template tells GCP how to build a VM by specifying the image, machine type, and scripts to run. Note that the k8s-node-family referenced here is a custom image we built in the previous part of this series.

We use the — metadata startup-script flag to inject our join command.

gcloud compute instance-templates create k8s-worker-template \
--image-family=k8s-node-family \
--machine-type=e2-standard-2 \
--tags=k8s-worker \
--metadata startup-script='#! /bin/bash &lt;PASTE_YOUR_JOIN_COMMAND_HERE&gt;'

Step 3: Create the Managed Instance Group (MIG)

We will create a Regional MIG. This means GCP will spread our nodes across multiple zones, such as us-central1-a, b, and c, for high availability.

gcloud compute instance-groups managed create k8s-worker-mig \
--template=k8s-worker-template \
--size=1 \
--region=us-central1

GCP will immediately spin up 1 node. It will boot, run the startup script, and join your cluster automatically.

Step 4: Enable Autoscaling

Now we tell GCP to watch the CPU usage of these nodes. If the average CPU usage exceeds 60%, it will add more nodes for us (up to 5).

gcloud compute instance-groups managed set-autoscaling k8s-worker-mig \
--max-num-replicas=5 \
--min-num-replicas=1 \
--target-cpu-utilization=0.60 \
--region=us-central1

Stress Testing the Autoscaler

Let’s prove it works. We’ll use a local kubectl connection to create artificial load.

Create a Load Generator We’ll deploy a simple busybox container that does nothing but an infinite loop to burn CPU.

kubectl create deployment load-generator --image=busybox -- /bin/sh -c "while true; do :; done"

2. Request CPU This step is absolutely critical. We must tell Kubernetes. exactly how much CPU this pod requires; without these resource requests, the cluster autoscaler will not recognize that the node is at capacity and will fail to trigger the scale-up event.

kubectl set resources deployment load-generator --requests=cpu=200m

3. Scale the Load Scale it to 20 replicas.

kubectl scale deployment load-generator --replicas=20

4. Watch the Magic Open two terminal windows. In one, watch your nodes:

kubectl get nodes -w

In the other, watch GCP instances:

gcloud compute instance-groups managed list-instances k8s-worker-mig --region=us-central1

You will see the single node fill up, pods go into Pending state, and the GCP Autoscaler provision new VMs to handle the load.

The Easy Way (GKE)

It’s always worth pausing to appreciate just how much effort we put into this. If you were using Google Kubernetes Engine (GKE), this entire process could have been replaced by a single “easy button” command:

gcloud container clusters create k8s-easy-cluster \
--zone us-central1-a \
--num-nodes 3 \
--machine-type e2-medium

While GKE is powerful, understanding the “hard way” makes us better operators because we know exactly which components to investigate when things go wrong.

Ready for the next challenge? Follow along to the next part of the series: upgrading your control plane without downtime!

Are We Done? (Or Just Getting Started?)

At this stage, we have a functional Kubernetes cluster on Compute Engine that includes self-healing and auto-scaling capabilities. This setup provides control over the operating system, kernel, and networking without GKE management fees, demonstrating that automation can be integrated into a manual build. However, this configuration is merely the foundation for a much broader architectural exploration. What are some things that you might want this cluster to do that it doesn’t already?

Go Deeper

The Original Blueprint: This entire series is inspired by Kelsey Hightower’s foundational guide, Kubernetes The Hard Way.
Official Documentation: Get familiar with the source of truth for all components: Kubernetes.io Documentation.
Official GKE Documentation: Access the complete guide for Google Kubernetes Engine (GKE) Documentation.

Misadventures in Kubernetes: Creating Workers

Duncan — Sun, 10 May 2026 01:31:00 +0000

So we have built a control plan from scratch. That’s kinda useful and was also kinda hard to make? But, really what we want to make is a proper, functioning, multi-node cluster that can actually run our containerized applications. A control plane by itself is just the brain; we need the muscle (the worker nodes) to do the actual work.

When we set up the control node, setting up the Control Plane required us to manually configure every single detail of the seed VM, line-by-line. While that was an important exercise in understanding the components, trying to repeat those steps for 10, 50, or 100 worker nodes would quickly become a nightmare. This part of the series is about making the process scalable by creating a “Golden Image” from our pre-configured machine, allowing us to rapidly provision identical workers and efficiently join them to our existing cluster.

Creating a Golden Image

The core principle here is to stop doing repetitive manual configuration. We already spent the time when we set up the control node getting the operating system tuned exactly right. Remember all those steps? We loaded the overlay and br_netfilter kernel modules, tweaked the network bridge settings for iptables, and, critically, disabled swap entirely because Kubelet won’t run otherwise. We also installed containerd and configured it to use systemd as the cgroup driver.

That entire stack of prerequisites, the OS configuration, and the necessary Kubernetes binaries (kubeadm, kubelet, kubectl) are now perfectly baked into our k8s-seed VM. A Golden Image is simply a frozen, ready-to-use snapshot of that perfect setup. Instead of repeating those steps on every new node, we use this image as the template to rapidly spin up as many identical worker nodes as we need. This moves us from a manual, configuration-based process to an automated, image-based provisioning process, which is the foundational step toward true cloud-native infrastructure.

Step 1: Generalize the Seed Instance

Before we can use our k8s-seed VM as a template, we need to “seal” it. Every running Linux machine has unique identifiers — a machine ID and specific network logs. If we duplicate these across multiple nodes, the Kubernetes Control Plane will be confused, seeing multiple machines with the same identity. We need to clear these unique details so that when a new VM boots from the image, it generates its own fresh identity and network configuration.

SSH into the seed machine:

gcloud compute ssh k8s-seed --zone=us-central1-a

2. Clean up unique identifiers: Run the following command to remove the machine ID and reset cloud-init logs. This ensures that when new VMs boot from this image, they generate their own fresh identity.

sudo cloud-init clean --seed --logs --machine-id
exit

3. Stop the instance: Back on your local machine, stop the VM so we can snapshot its disk.

gcloud compute instances stop k8s-seed --zone=us-central1-a

Step 2: Create the Custom Image

With the seed machine prepped and stopped, we now create a Google Compute Engine image from that disk. This image, k8s-node-image-v1, will serve as the foundation for every node in our cluster, whether they eventually become a control plane member or a worker.

Run the following command to create the image:

gcloud compute images create k8s-node-image-v1 \
  --source-disk=k8s-seed \
  --source-disk-zone=us-central1-a \
  --family=k8s-node-family

Note: We added the — family flag so we can easily reference the “latest” version of this image later without needing to know the exact version name. This is a great cloud-native practice for image management!

Step 3: Provision Worker Nodes

Now for the easy part. Because the heavy lifting of configuration is done, spinning up new workers is instantaneous. We can create two worker nodes identically, pre-loaded with all the necessary Kubernetes prerequisites. This process takes minutes, a stark contrast to the hours spent manually configuring the seed VM in when we setup the control node.

Run these commands to create worker-1 and worker-2 using the image family we just created:

gcloud compute instances create worker-1 \
  --zone=us-central1-a \
  --image-family=k8s-node-family \
  --tags=k8s-worker

gcloud compute instances create worker-2 \
  --zone=us-central1-a \
  --image-family=k8s-node-family \
  --tags=k8s-worker

Step 4: Join the Cluster

Your worker VMs are running, but they are just generic machines with Kubernetes components installed; they don’t know they belong to a cluster yet. This is where the magic of the kubeadm join command comes in. The worker’s kubelet agent needs to securely authenticate and register with the Control Plane’s API server. The join command provides the token and the hash that validates the worker’s identity, effectively making it a true node.

Locate your join command: Find the kubeadm join command you saved from the output of kubeadm init when we set up the control node. It looks something like this:
Join the workers: SSH into each worker and run that command with sudo. For Worker 1:

kubeadm join 10.128.0.x:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

3. For Worker 2: Repeat the same steps for worker-2.

gcloud compute ssh worker-1 --zone=us-central1-a
sudo kubeadm join ... # Paste your command here
exit

Step 5: Verify the Cluster

Head back to your Control Plane to verify that everyone has checked in. This is the satisfying moment where all that hard work pays off and you see your multi-node cluster come alive.

kubectl get nodes

You should see output similar to this, with all nodes showing a Ready status:

NAME            STATUS   ROLES           AGE   VERSION
control-plane   Ready    control-plane   10m   v1.35.0
worker-1        Ready    <none>          2m    v1.35.0
worker-2        Ready    <none>          1m    v1.35.0

The Easy Way (GKE)

It’s always worth pausing to appreciate just how much effort we put into this. Throughout this series, we’ve manually provisioned VMs, installed binaries, configured systemd services, and joined nodes — all to get a basic Kubernetes cluster running.

If you were using Google Kubernetes Engine (GKE), this entire process (for both the control plane and worker nodes) could have been replaced by a single command:

gcloud container clusters create k8s-easy-cluster \
  --zone us-central1-a \
  --num-nodes 3 \
  --machine-type e2-medium

What’s Next?

You now have a multi-node cluster, which is a huge win. But there’s a big problem: your cluster is still static. If worker-1 crashes, it’s gone forever and your capacity is reduced until you manually replace it. If your application load increases, you have to manually provision worker-3, SSH in, and run the kubeadm join command. Future-proofing your infrastructure means moving beyond these manual interventions.

To achieve a truly “cloud-native” and resilient cluster, we need to automate the lifecycle of our worker nodes, ensuring they can be recreated and scaled without manual effort. This is something to think about, maybe this is something that we might touch on in the future possibly?

Go Deeper

The journey through “Kubernetes the Kinda Hard Way” is fundamentally about building foundational knowledge. Ready to solidify your understanding and tackle advanced concepts? Check out these resources:

The Original Blueprint: This entire series is inspired by Kelsey Hightower’s foundational guide, Kubernetes The Hard Way.
Official Documentation: Get familiar with the source of truth for all components: Kubernetes.io Documentation.

Misadventures in Kubernetes: Provisioning the Control Plane

Duncan — Sun, 10 May 2026 01:30:29 +0000

When using managed platforms like Google Kubernetes Engine (GKE), there are many things that you don’t have to worry about that it takes care of for you! But have you ever paused to consider what machinery operates beneath that simple surface? Perhaps wondered how you might be able to do it on your own, or how to at least go about trying to do it?

My personal motivation for this project was simple: I wanted to give this a try myself to truly understand Kubernetes at its core. Moving past the abstraction of managed services, I set out to peel back the layers and inspect the core components. This approach involves setting up standard Google Compute Engine (GCE) Virtual Machines, manually installing and configuring every component, and wiring the networking together ourselves. This deliberate, hands-on process helps build the foundational knowledge necessary to effectively troubleshoot, optimize, and better understand our own clusters.

Prerequisites

Before we dive in, ensure you have the following:

A Google Cloud Platform (GCP) account.
The gcloud CLI installed and authenticated on your local machine.

Step 1: Provision the Base Instance

We start by creating a “seed” or a base instance. This VM serves as our foundation where we install all necessary software, providing a clear template for how the Kubernetes nodes/VMs are constructed. The E2 series is often chosen because it offers the most cost-optimized VMs on GCP, providing a performance-to-cost balance suitable for a Kubernetes control plane. We are using the e2-standard-2 machine type because Kubernetes requires at least 2 vCPUs and 2GB of RAM to run comfortably.

Run the following command to create your VM:

gcloud compute instances create k8s-seed \
    --zone=us-central1-a \
    --machine-type=e2-standard-2 \
    --image-project=ubuntu-os-cloud \
    --image-family=ubuntu-2204-lts \
    --boot-disk-size=50GB

Once created, SSH into the machine:

gcloud compute ssh k8s-seed --zone=us-central1-a

Step 2: Configure the OS

Kubernetes has specific requirements for the underlying Linux OS. We need to load specific kernel modules and tweak network settings to allow Kubernetes to manipulate traffic correctly.

1. Load Kernel Modules

These modules allow Kubernetes to manipulate network traffic for Pods and Services.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

2. Network Bridge Settings

Ensure bridged traffic is passed to iptables for correct filtering.

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

3. Disable Swap

This is critical. The Kubelet will fail to start if swap is enabled.

sudo swapoff -a
# Edit fstab to prevent swap from turning on after reboot
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

Step 3: Install the Runtime (Containerd)

Kubernetes needs a container runtime to launch pods. We will use containerd.

We choose containerd because it is the industry-standard core container runtime and is lightweight. Kubernetes now defaults to containerd, and runtimes like Docker are being deprecated in modern Kubernetes versions.

1. Add Docker’s Apt Repository

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

2. Install Containerd

sudo apt-get update
sudo apt-get install -y containerd.io

3. Configure Systemd Cgroups

Kubernetes recommends using systemd as the cgroup driver.

sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

Step 4: Install Kubernetes Tools

Now we install the “big three”:

kubeadm: The bootstrapper.
kubelet: The node agent.
kubectl: The CLI.

# Add Kubernetes repo
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.35/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

# Install
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Note: We use apt-mark hold to prevent automatic updates from breaking the cluster.

Step 5: Initialize the Cluster

At this point, our “seed” machine is fully prepped. We will use it now to initialize the Control Plane.

1. Run Init

We specify a pod network CIDR that is compatible with our chosen networking plugin (Calico).

sudo kubeadm init --pod-network-cidr=192.168.0.0/16

2. Configure Kubectl

To run commands against your new cluster:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 6: Install Networking (Calico)

Nodes cannot communicate until a Container Network Interface (CNI) is installed. We’ll use Calico.

Calico offers high-performance networking by deploying without encapsulation or overlays, and provides a distributed firewall for flexible network policy enforcement.

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Run kubectl get nodes and within a minute, you should see your node transition to Ready.

What if I’d Just Used GKE?

To appreciate the work we just did, it’s worth noting that you could have achieved a fully managed, production-hardened equivalent with a single Google Kubernetes Engine (GKE) command:

gcloud container clusters create k8s-easy \
    --zone us-central1-a \
    --machine-type e2-standard-2 \
    --num-nodes 3

This single command provisions the control plane (managed by Google), creates worker nodes, configures networking, and sets up authentication. But by building part of it manually, you now understand how those components work together.

What’s Next?

Congratulations! You have successfully built a functional Kubernetes control plane from scratch.

By building the cluster “the kinda hard way,” you’ve moved beyond being a user of managed services to understanding more about the components that orchestrate modern containerized applications. This foundation is key to troubleshooting, optimizing, and scaling production-grade Kubernetes environments. But what if you want to use more than control nodes?? Maybe think about how you might be able to create worker nodes to work with your existing cluster.