DEV Community: Ashish Nair

Autoscaling in Openshift with Cluster Autoscaler and Machine Autoscaler

Ashish Nair — Mon, 27 Apr 2026 04:56:21 +0000

In my previous post, I covered how to scale compute (worker) nodes in OpenShift using a semi-automated approach. While OpenShift handled most of the heavy lifting—such as powering on the node via BMC, installing RHCOS, and joining the cluster—the scaling action itself still required manual intervention.

This document focuses on removing that manual step altogether. Specifically, it explores how Cluster Autoscaler and Machine Autoscaler work together to enable automatic, workload-driven scaling of compute nodes. While technologies like Cluster API and Machine API provide the underlying framework for managing machine lifecycles, the real decision-making around when to scale happens at the autoscaler layer.

Before I dive into it, I would like to explain in a line about the building blocks(So you don't think of me of a complete idiot!).

Cluster Autoscaler - responsible for observing the state of workloads in the cluster. It continuously monitors pending and unschedulable pods and determines whether adding (or removing) nodes would help satisfy resource requirements.

Machine Autoscaler - acts as the bridge between high-level scaling decisions and infrastructure changes.

Machinsets- Define how a worker node should be created

Cluster API: Cluster API is a Kubernetes project that provides a declarative way to create, manage, and scale Kubernetes clusters.

Machine API: Red Hat’s opinionated implementation of Cluster API concepts.

MachineSets: Equivalent of a ReplicaSet — but for nodes instead of pods.

The way these different component work together is:

The cluster Autoscaler watches for unschedulable pods.
Informs the Machine Autoscaler who updates the machineSets limits.Simply speaking, runs the "oc scale machineset/ replicas=2"
Our hardworking employee, Machine-API prepares the new node (of course, with help from BMH)
Pods get scheduled to the new worker node.

We will see how to perform the teeny-tiny set of 4 steps in Openshift:

Note: You can change to the "openshift-machine-api" namespace as most of the steps will be in that namespace:

oc project openshift-machine-api

Let's check our current nodes, machinsets and BMH(That's BareMetalHosts and not Jaspreet Bhumrah!)

oc get nodes
NAME      STATUS   ROLES                  AGE    VERSION
master    Ready    control-plane,master   8d     v1.27.10+28ed2d7
master2   Ready    control-plane,master   8d     v1.27.10+28ed2d7
master3   Ready    control-plane,master   8d     v1.27.10+28ed2d7
worker1   Ready    worker                 3d7h   v1.27.10+28ed2d7

oc get machinesets
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           8d

oc get bmh
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        8d
master2   externally provisioned   mycluster-7ln8n-master-1         true        8d
master3   externally provisioned   mycluster-7ln8n-master-2         true        8d
worker1   provisioned              mycluster-7ln8n-worker-0-qv4cn   true        3d10h

Let's create our cluster Autoscaler and Machineautoscaler manifests.

oc apply -f machineautoscaler.yaml 
machineautoscaler.autoscaling.openshift.io/worker-autoscaler created

oc apply -f clusautoscaler.yaml 
clusterautoscaler.autoscaling.openshift.io/default created

Note: You can construct the machine autoscaler and cluster autoscaler manifests from Redhat's official documentation

Let's add some load to our cluster. This is just a simple manifest that requests 1G memory and spawns ~20+ replicas.

oc apply -f dep.yaml 
deployment.apps/load-test created

This manifest has actually created someone load on the current worker node and needs probably another one to accommodate all the pods. Hence a lot of them are now in "Pending" state.

load-test-65777f99f7-458kp                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-4p72l                            1/1     Running   0          2m22s   10.128.2.43       worker1   <none>           <none>
load-test-65777f99f7-4v6hq                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-84n79                            1/1     Running   0          2m22s   10.128.2.39       worker1   <none>           <none>
load-test-65777f99f7-8955k                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-9h4j6                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>

Our Autoscalers have actually updated our machineset:

oc get machineset
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   2         2         1       1           8d

Which has started provisioning a new node for us:

oc get bmh
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        8d
master2   externally provisioned   mycluster-7ln8n-master-1         true        8d
master3   externally provisioned   mycluster-7ln8n-master-2         true        8d
worker2   provisioning             mycluster-7ln8n-worker-0-9vszs   true                          12m
worker1   provisioned              mycluster-7ln8n-worker-0-qv4cn   true                          3d10h

In no time(Ok, around 20 minutes) you should see your new worker node ready to take workload. The pending pods will move to this new worker gradually.

oc get nodes
NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   8d      v1.27.10+28ed2d7
master2   Ready    control-plane,master   8d      v1.27.10+28ed2d7
master3   Ready    control-plane,master   8d      v1.27.10+28ed2d7
worker1   Ready    worker                 3d8h    v1.27.10+28ed2d7
worker2   Ready    worker                 3m14s   v1.27.10+28ed2d7

The "oc get pods -o wide" will actually tell you that the workloads are actually moving to this new worker node:

load-test-65777f99f7-458kp                            1/1     Running   0          33m     10.131.0.5        worker2   <none>           
load-test-65777f99f7-4p72l                            1/1     Running   0          33m     10.128.2.43       worker1   <none> 
load-test-65777f99f7-84n79                            1/1     Running   0          33m     10.128.2.39       worker1   <none>           <none>
load-test-65777f99f7-9h4j6                            1/1     Running   0          33m     10.131.0.6        worker2   <none>           <none>
load-test-65777f99f7-ctvc7                            1/1     Running   0          33m     10.128.2.41       worker1   <none>           <none>

The autoscalers not only works well for scaling-out, it also works pretty well for scaling-in.

To test this out I deleted the deployment I had created earlier:

oc delete deployment/load-test
deployment.apps "load-test" deleted

We can also check the event logs using 'oc get events'. It actually tells us the entire flow:

24m                  Normal    Killing                  Pod/load-test-65777f99f7-ctvc7                     Stopping container stress
24m                  Normal    Killing                  Pod/load-test-65777f99f7-hgf2n                     Stopping container stress

13m                  Normal    ScaleDownEmpty           ConfigMap/cluster-autoscaler-status                Scale-down: removing empty node "worker2"
13m                  Normal    ScaleDownEmpty           ConfigMap/cluster-autoscaler-status                Scale-down: empty node worker2 removed
13m                  Normal    DrainProceeds            Machine/mycluster-7ln8n-worker-0-9vszs             Node drain proceeds
13m (x10 over 66m)   Normal    SuccessfulUpdate         MachineAutoscaler/worker-autoscaler                Updated MachineAutoscaler target: openshift-machine-api/mycluster-7ln8n-worker-0
13m                  Normal    Deleted                  Machine/mycluster-7ln8n-worker-0-9vszs             Node "worker2" drained
13m                  Normal    DrainSucceeded           Machine/mycluster-7ln8n-worker-0-9vszs             Node drain succeeded
13m                  Normal    DeprovisioningStarted    BareMetalHost/worker2                              Image deprovisioning started

8m34s                Normal    DeprovisioningComplete   BareMetalHost/worker2                              Image deprovisioning completed
8m33s                Normal    PowerOff                 BareMetalHost/worker2                              Host soft powered off

In a nutshell it has:

On deleting the deployment, the containers were killed.
It scaled down the workload, we can also check the machineset output.
Drained the node.
Deprovisioned and Power-off the node.

user@server1:~/test/manifests$ oc get machineset -n openshift-machine-api
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           8d

virsh list --all
 Id   Name      State
--------------------------
 1    master    running
 2    master2   running
 3    master3   running
 4    worker1   running
 -    worker2   shut off

Scaling worker nodes in openshift with machine-api

Ashish Nair — Mon, 13 Apr 2026 05:59:39 +0000

In my last post, I wrote about how we can run an Openshift IPI install on KVM. In this document(Which is relatively shorter), I'll talk about my experience in scaling a worker node (a semi-automated method).

We will be breaking this down into 2 steps:

The Preparation
The Scaling

The Preparation

The prerequisite to start this is obviously a server( A blank VM in KVM, in our case).
A DNS entry in our Antagonist's Database.
The sushy webserver listening on port 8000, this is our IDrac/iLo emulation. More on that here
And a bit of patience (did i pull-off a Robin Sharma here?)

The Scaling:

Before we start the actual scaling process let me walk through the actual process :

We create a VM in KVM (This is as good as having a spare Baremetal host)
We create a Manifest (actually two) and apply it. This manifest will remind you of the install-config.yaml we used here
Once the manifest is applied, the machine is booted as a result of our BMC magic (via sushy and Redfish) and Installation is kicked-off.
Once the install is completed the Machine(node) joins the cluster.

Gather some data:

All (okay, most) of the work will happen in the openshift-machine-api namespace.
Let's check the current nodes that we have

oc get nodes
NAME          STATUS   ROLES                  AGE     VERSION
master        Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
master2       Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
master3       Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
worker2.lab   Ready    worker                 2d22h   v1.27.10+28ed2d7

It's important to check the Baremetals we currently have.

oc get bmh -n openshift-machine-api
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        3d21h
master2   externally provisioned   mycluster-7ln8n-master-1         true        3d21h
master3   externally provisioned   mycluster-7ln8n-master-2         true        3d21h
worker2   provisioned              mycluster-7ln8n-worker-0-hpm5j   true        2d22h

A note on machinsets. Machine sets display the "group" of workernodes we have. They are used to scale compute(Workers). Make node of the name of the Machineset this is the one we will be scaling out.

oc get machinesets -n openshift-machine-api
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           3d21h

The manifests:

We need to 2 manifests here and it's largely inspired by the install-config.yaml we had earlier :

Snippet of install-config.yaml:

 - name: worker3
      role: worker
      bmc:
        address: redfish-virtualmedia+http://192.168.122.1:8000/redfish/v1/Systems/aa12a91d-56f9-41f4-b5ea-e5001dae179c
        username: admin
        password: password
      bootMACAddress: 52:54:00:fd:5e:1d
      rootDeviceHints:
         deviceName: /dev/vda

And our Manifests will do exactly what the above code we used while installation, but we split it into two:

Creates a BMH
Creates a secret to hold the console credentials.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: worker3
  namespace: openshift-machine-api
spec:
  online: true
  bootMACAddress: 52:54:00:c4:0e:6c
  bmc:
    address: "redfish-virtualmedia+http://192.168.122.1:8000/redfish/v1/Systems/d9fea2de-6fd9-4a44-99fc-95f32b610407"
    credentialsName: worker3-bmc-secret
  rootDeviceHints:
    deviceName: /dev/vda
apiVersion: v1
kind: Secret
metadata:
  name: worker3-bmc-secret
  namespace: openshift-machine-api
type: Opaque
stringData:
  username: admin
  password: password

Apply the manifests:

oc apply -f .

The wait (Part 1) :

After you apply the manifests, the node will transition into 4 states:

Registering
Inspecting
Available
Provisioned

For brevity we will jump to the "available" state, as this is the most interesting part. At this point if you login to the worker using 'ssh core@IP' you will see that our new worker is booted up using a CoreOS ISO by the Installer. This is a way of openshift telling us "The system is available, what do you want me to do?" (And this will also take us to the final part)

And we say we wanna scale, this is the machineset name I had asked you to make note of(If you haven't fallen asleep by now!). So the "oc scale" command uses the machineset name to scale the compute by 2 replicas in our case.

oc scale machineset/mycluster-7ln8n-worker-0 -n openshift-machine-api --replicas=2

Now, when you login to the worker machine you will see that the installation might have started:

all [-] coreos-installer: Read disk 868.2 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271
Apr 10 14:41:47 worker3 podman[1179]: 2026-04-10 14:41:47.517 1 DEBUG ironic_coreos_install [-] coreos-installer: Read disk 868.2 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271
Apr 10 14:41:48 worker3 ironic-agent[1191]: 2026-04-10 14:41:48.523 1 DEBUG ironic_coreos_install [-] coreos-installer: Read disk 883.6 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271

The wait (Final part, I promise!):

In about 15-20 minutes you will see that the machine state will have transitioned from "Available" to "Provisioned"

oc get bmh -n openshift-machine-api
NAME      STATE                    CONSUMER                         ONLINE   ERROR   AGE
master    externally provisioned   mycluster-7ln8n-master-0         true             4d6h
master2   externally provisioned   mycluster-7ln8n-master-1         true             4d6h
master3   externally provisioned   mycluster-7ln8n-master-2         true             4d6h
worker2   provisioned              mycluster-7ln8n-worker-0-hpm5j   true             3d7h
worker3   provisioned              mycluster-7ln8n-worker-0-k67h7   true             7h41m

The "get nodes" should also show it's available for use.

oc get nodes
NAME          STATUS   ROLES                  AGE     VERSION
master        Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
master2       Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
master3       Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
worker3       Ready    worker                 9h      v1.27.10+28ed2d7
worker2.lab   Ready    worker                 4d10h   v1.27.10+28ed2d7

That's all folks!!

While the procedure looks lengthy while explaining and documenting, it essentially is just creating the manifests and running oc scale command. If you have scaled a compute manually you will appreciate how much work this method cuts down!

Deploying OpenShift IPI on KVM (Baremetal Simulation with Redfish & Sushy)

Ashish Nair — Tue, 07 Apr 2026 19:04:42 +0000

If the article gave you a "Yet Another Openshift Setup Guide"(Sorry YAML, I stole some letters) feel, I don't blame you!(This is an indication of how much free time i have, lol!) While this isn't a typical How-To guide (I lied!) ,I'll tell you why this is different - Openshift doesn't support IPI method of installation on KVM (ironical huh? not supporting their own siblings) but there's a hack that allows you to do it(ofcourse! you can only use it for labs!!)

The Layout (okay! Architecture diagram)

The components

A KVM host(Our protagonist!)
DNSmasq built-in with KVM A.K.A our Antagonist!
Sushy (not the dish, this is a Redfish emulator)
VM's (The masters and the Workers)

A note on prerequisites and Hardware requirements

you can check the H/W requirements for openshift in Redhat's or OKD's official documentation here

1. Assembling components for the Virtual Machines

Adding disks to our Masters and Slaves.

qemu-img create -f qcow2 /var/lib/libvirt/images/master-1.qcow2 120G
qemu-img create -f qcow2 /var/lib/libvirt/images/master-2.qcow2 120G
qemu-img create -f qcow2 /var/lib/libvirt/images/master-3.qcow2 120G
qemu-img create -f qcow2 /var/lib/libvirt/images/worker-1.qcow2 120G

Configuring a Network in Libvirt

This is the most critical part of the setup. If this fails the install will fail and frustrate you to the core!
Save this into a file, probably default.xml

Apply it

virsh net-define default.xml
virsh net-start default
virsh net-autostart default

Creating the VM's

It's time to create the VM's using the KVM console but don't boot them yet! We will have our installer boot these machines for us via Redfish and Sushy. In other words, a poor man's iDrac/ILO but only for power management.

setting up Sushy

create a virtual Environment to install python module

python3 -m venv ~/sushy-env
source ~/sushy-env/bin/activate

Install the sushy module

pip install sushy-tools

start sushy

sushy-emulator -i 192.168.122.1 --port 8000 --libvirt-uri qemu:///system

Testing powering on/off the VM's using the tool we just installed.

validate the Redfish API

curl http://192.168.122.1:8000/redfish/v1/Systems

The output will look something like:

$ sushy-emulator -i 192.168.122.1 --port 8000 --libvirt-uri qemu:///system
 * Serving Flask app 'sushy_tools.emulator.main'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://192.168.122.1:8000
Press CTRL+C to quit

Validate power control. The curl command we ran above will give you an id assigned to every system that might look like 58ec1279-e393-4dba-a7b4-e8ea37c0d6da, replace that id in the below command.

curl -X POST http://192.168.122.1:8000/redfish/v1/Systems/<ID>/Actions/ComputerSystem.Reset \
  -H "Content-Type: application/json" \
  -d '{"ResetType": "On"}'

The system will be powered on if you check the KVM console. Power it off again!( I swear I'm not trying to irritate you!)

Redfish power operations require files under /usr/share/OVMF. My system had secure boot files missing so I had to create the below soft links.

sudo ln -s /usr/share/OVMF/OVMF_VARS_4M.fd /usr/share/OVMF/OVMF_VARS.fd
sudo ln -s /usr/share/OVMF/OVMF_CODE_4M.fd /usr/share/OVMF/OVMF_CODE.fd
sudo ln -s /usr/share/OVMF/OVMF_CODE_4M.secboot.fd /usr/share/OVMF/OVMF_CODE_4M.ms.fd
sudo ln -s /usr/share/OVMF/OVMF_CODE_4M.secboot.fd /usr/share/OVMF/OVMF_CODE.secboot.fd

2. Building the installer

Installing the tools to build the installer

sudo apt install golang git make gcc g++ libvirt-dev pkg-config

plaintext

clone the repo

git clone https://github.com/openshift/installer.git

shell

compile the installer with TAGS=libvirt hack/build.sh. A.K.A - The Hack

cd installer
make build

shell

copy the installer to /usr/local/bin

sudo cp bin/openshift-install /usr/local/bin/

markdown

3. The installation

Create directories for the install

mkdir ~/ocp-install
cd ~/ocp-install

yaml

create install-config.yaml

apiVersion: v1
baseDomain: lab
metadata:
  name: mycluster

controlPlane:
  name: master
  replicas: 3

compute:
- name: worker
  replicas: 2

networking:
  networkType: OVNKubernetes
  machineNetwork:
  - cidr: 192.168.122.0/24

platform:
  baremetal:
    externalBridge: "virbr0"
    apiVIP: 192.168.122.10
    ingressVIP: 192.168.122.11
    provisioningNetwork: "Disabled"

    hosts:
    - name: master
      role: master
      bmc:
        address: redfish-virtualmedia+http://192.168.122.1:8000/redfish/v1/Systems/<ID>
        username: admin
        password: password
      bootMACAddress: 52:54:00:3d:30:b5
      rootDeviceHints:
        deviceName: /dev/vda

plaintext
Note: populate fields for all you masters and workers and Add the pull-config(from the Redhat Portal) and sshKey(ssh public key from your home directory). fields and populate them. The ignition configs(RH core OS' Kickstart) will bake this into the RHCOS ISO and you should be able to login to your master/workers via ssh using core@master/worker using the key you entered above.

kick-off the installation (while you're inside ~/ocp-install which also houses your install-config.yaml)

openshift-install create cluster --dir . --log-level=DEBUG

plaintext
This will create a bootstrap node in KVM, then after the initial phase of install is complete it will remove it and boot your masters and workers(via sushy and Redfish) via RHCOS iso and continue with the install.

Note: you can change the --log-level parameter to INFO if detailing is not your thing(not judging you!)

The wait (And also the toughest part)

Yes, This is the toughest part as the install takes around 1 hour or even more to complete based on the resources you have on your system.
You can login to one of your nodes (core@master) and tail the bootkube logs and monitor the install.

Once the Install is complete you should see something like:

INFO Waiting up to 1h0m0s (until 5:21PM IST) for the cluster at https://api.mycluster.lab:6443 to initialize... 
INFO Checking to see if there is a route at openshift-console/console... 
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/home/user/test/ocp-install2/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.mycluster.lab 
INFO Login to the console with user: "kubeadmin", and password: "eI2ES-wtGQG-Lgwec-KUNum"

shell

export the kubeconfig file and access your cluster API's:

export KUBECONFIG=./auth/kubeconfig
oc get nodes

What a waste of my weekend!!