DEV Community: Ashish Nair

I Built Hybrid DNS Between AWS and My On-Prem Linux Server -- Here's What Finally Made Route 53 Resolver Click

Ashish Nair — Mon, 24 Aug 2026 17:13:50 +0000

Why I built this?

I wanted to actually build a hybrid DNS environment and see the DNS queries travel between an on-premises environment and AWS.

My lab ended up having two separate DNS requirements:

A public DNS name for my application: app.xcv.publicvm.com
A private DNS namespace inside AWS: aws.internal

These are the 2 hosted zones I created in Route53:

The interesting part was making my on-premises Linux DNS server — which is also my MariaDB server at 192.168.1.10 — resolve a private AWS RDS name: rds.aws.internal.

The Architecture

The DNS pieces and why they exist:

1.Public DNS - app.xcv.publicvm.com
In my Public hosted zone I created failover routing with 2 A records:
app.xcv.publicvm.com
- primary -> 13.206.252.123
- failover -> 13.233.70.186
This is the also the DNS record that my application lives on.

2.Private DNS - aws.internal
I created a private hosted zone: aws.internal. This zone isn't resolvable publicly. This simply has a CNAME that points to my RDS' endpoint.

3. On-premise DNS - onprem.example.com
My on-premises environment has a Linux server at 192.168.1.10. This is also my mariadb server (which is ofcourse not a good practice but laziness 😁)

Now, we have 2 DNS authorities:

On prem - onprem.example.com
Route53 - aws.internal

And the good part is neither needs to become authoritative for the other environment (Because my DNS is not greedy like humans 😁). Instead, we forward queries.

Now, The problem!
There has to be a problem for us to provide a solution! Imagine, an application that sits on-premise wants to resolve rds.aws.internal -> 192.168.1.10(A.K.A on-premise DNS) . This fails. Because our on-premise isn't authoritative for aws.internal(It's a private hosted zone on AWS).

You can beat me up and say - "Add a static record on the On-premise DNS server".
I'd say hold my beer, this is why my setup exists:

Adding a static entry is not a good hybrid architecture practice.
RDS is owned by AWS, what if it changes the IP?
And lastly, I'd say meet our friend - Route53 resolver Inbound Endpoint.

what is a Route53 resolver Inbound Endpoint ?
An Inbound Resolver Endpoint is used when DNS queries are coming into AWS from an external network(External is our on-premise network).

The flow (Roughly!) is:

The benefit : our on-prem DNS doesn't need to know how AWS works the magic! It just forwards anything *.aws.internal to the resolver.

And this is the validation that it worked:

Migrating on-premise block-level workloads to the cloud

Ashish Nair — Tue, 11 Aug 2026 17:20:53 +0000

Migrating on-premises block-based workloads to the cloud can be a daunting task, especially when the workload contains a large amount of data.

This is a follow-up to the site-to-site VPN setup I documented here https://dev.to/ashish_nair_d9b10ba4f8126/aws-site-to-site-vpn-using-strongswan-1olk .

For this lab, I wanted to combine the best of both worlds: keep the workload on-premises while using AWS as the destination for storage, backup and recovery — and keep the connectivity private using the VPN and AWS private endpoints.

The Goal:

My on-premise MariaDB server writes it's data to /var/lib/myql.

For this experiment, I wanted to explore two possibilities:

Keep a copy of the block-based workload in AWS and use EBS snapshots for backup/recovery.
Recover the block-based filesystem in AWS, attach it to an EC2 instance, and eventually move the compute workload to AWS.

I configured the Storage Gateway appliance to communicate with AWS through the private connectivity already established by my Site-to-Site VPN and used AWS private endpoints for the AWS-side connectivity.

This allowed me to keep the Storage Gateway traffic on private network paths rather than sending it over the public Internet.

The Final test:

Can I take the data that my on-premises MariaDB server was writing to, recover that storage in AWS, attach it to an EC2 instance, and verify that my database data is still there?

Why not just use mysqldump?

A very reasonable question is: Why not simply take a MariaDB/MySQL dump and restore it in AWS?

For a database migration, that can certainly be a much simpler approach.

However, that isn't what I wanted to experiment with here.

Let's assume instead that we have a very large block-based filesystem containing a large amount of data.

Rather than thinking about the contents of the filesystem as individual database objects or files, this experiment looks at the problem from a block-storage perspective.

That makes Storage Gateway a much more interesting technology to explore.

How does this work?

The interesting part of this setup is that MariaDB doesn't need to know that AWS is involved.

The Storage Gateway appliance exposes an iSCSI block device to the on-premises MariaDB server.

From the MariaDB server's perspective, it simply sees a block device

I scanned for the iSCSI target from the MariaDB server, logged into it, formatted the volume and mounted it as a normal filesystem.

The application therefore continues to interact with the filesystem normally.

Behind the scenes, the Storage Gateway handles the transfer of the data to AWS. For a cached volume, the gateway maintains local cache/upload-buffer storage and asynchronously transfers data to AWS.

This is what makes the experiment interesting: the application is working with a normal block device while the underlying storage is being integrated with AWS.

The Lab:

I first created a Storage Gateway in AWS and selected the appropriate gateway configuration for the experiment.

The next step required downloading the VMware appliance and starting it in my local VMware environment. The appliance provides the interface between my on-premises environment and AWS.

The gateway appliance console looked something like this:

I attached a 150G disk to the storage appliance. This storage is used by the gateway as part of the volume configuration and allows it to expose an iSCSI volume to the on-premises server.

From the on-premises MariaDB server, I scanned for the iSCSI target exposed by the Storage Gateway appliance and logged into it.

I formatted the newly discovered block device and mounted it on the MariaDB server.

Once the filesystem was mounted, I generated some entries in the database. This gave me some known data that I could later use to verify whether the migration/recovery actually worked.

After the volume was being used by the on-premises server, I could see the corresponding volume through the AWS Storage Gateway console.
I then created a snapshot of the Storage Gateway volume. This was the point where I could verify that AWS had successfully received the data. However, a successful snapshot alone isn't enough. I wanted to prove that the data could actually be recovered.

I used the snapshot to create an EBS volume and attached that volume to an EC2 instance.I then mounted the recovered filesystem on the EC2 instance.The final test was to inspect the MariaDB data.

The database entries that I had created while the workload was running on-premises were present in the recovered volume in AWS.

That gave me confidence that the complete path had worked.

This experiment demonstrated that I could take a block-based workload being accessed from my on-premises environment, expose it through Storage Gateway, create a recovery point in AWS, recover that storage as an EBS volume and verify the data from an EC2 instance.

The important part for me was that the storage abstraction was transparent to the application. MariaDB interacted with a normal filesystem and didn't need to know whether the underlying block storage was ultimately being integrated with AWS.

However, this experiment should not be interpreted as demonstrating an application-consistent MariaDB backup strategy. For production database workloads, database-native replication, AWS DMS, or an application-consistent backup procedure may be more appropriate depending on the RPO/RTO requirements.

AWS site-to-site VPN using StrongSwan

Ashish Nair — Mon, 03 Aug 2026 17:16:09 +0000

It's a Sunday and I get a call from one of my ex-colleagues and he said "site-to-site VPN does not work when I want to reach VM's on your laptop because we are behind our ISP's NAT". And immediately I was like "Devops engineers and their Networking pitfalls". I popped open my beer and started setting this up.

The setup is:

I have an EC2 instance (10.100.11.0/24)
I have 2 VM's on VMware Workstation.
One of them is a strongswan VM which will act as the on-premise router.(192.168.1.9)
Second One is a mariadb VM that the EC2 instance should be able to ping. (192.168.1.10)

Long-story-short version of system admins A.K.A - The architecture diagram

The easiest part of the setup - The AWS side of configuration

Create a Virtual Private Gateway and associate it with your VPC.
Identify your Public IP at the On-premise end and create the customer gateway in AWS. Enter your public ip here.
Create a site-to-site VPN, associate it with the Virtual Private Gateway and the customer gateway we created in step 1 and 2.
Download the configuration from the top-end of the page. This has the instructions to set up the tunnel interface on StrongSwan, along with other configurations.
Launch a Ec2 instance in your VPC .

The trickiest part of the setup - The on-premise configuration

Open the configuration we downloaded and make changes . We are actually making way for the IKE1 and IKE2 setups. I will be covering what are these in a separate document.
Make changes to the ipsec.conf and ipsec.secrets on the strongswan VM. These settings are in the configuration we downloaded from AWS when we created the VPN. The settings should be something like:

Create tunnel interface and assign IP's as mentioned in the downloaded file.

Add sysctl settings mentioned in the file.It should have something like:
net.ipv4.conf.Tunnel1.rp_filter=2 #This value allows the Linux kernel to handle asymmetric routing net.ipv4.conf.Tunnel1.disable_policy=1 #This value disables IPsec policy (SPD) for the interface net.ipv4.conf.<PHYSICAL INTERFACE>.disable_xfrm=1 #This value disables crypto transformations on the physical interface net.ipv4.conf.<PHYSICAL INTERFACE>.disable_policy=1 #This value disables IPsec policy (SPD) for the interface net.ipv4.ip_forward = 1
Flush routes:
ip route flush table 220 ip route flush cache
Add a route. Now, this is the CIDR where the EC2 instance lives. We are making this route so the packet destined for the ec2 instance goes via the tunnel interface.
ip route add 192.168.0.0/16 via 169.254.51.57 dev vti1
Add a route to the Ec2 instance's route table. This route tells the packet to travel via the Virtual Private Gateway if the destination network is our on-premise network.

And If we have everything in place our Ec2 should be able to ping our on-premise strongswan VM:

Now, this strongswan is the gateway at the on-premise end. So in order for the EC2 to be able to reach the mariadb instance we have to add a route. A return route to be specific.

And let's test the connectivity. Notice the difference in the number of hops when we ran traceroute to 192.168.1.9 and 192.168.1.10

And obviously, ping will also work:

While the configuration worked, I must admit it didn't work in the first try or even or the 4th one.

I will create a follow-up article how I troubleshot the AWS side and the On-premise end.

From Copy-Pasting YAML to Building My Own Kubernetes Manifest Studio

Ashish Nair — Sun, 28 Jun 2026 07:44:00 +0000

Over the last few months I've been spending a lot of time building for Kubernetes and OpenShift while working as a Platform Engineer. Almost every lab started the same way.

Create a Deployment.
Create a Service.
Create a ConfigMap.
Maybe a Secret.
Sometimes an Ingress.

Repeat.

Then do the same thing again for another application.

An idea stemmed out of a conversation I had with a developer friend who seemed visibly frustrated over this. To be honest, he had fair point - Remembering how to right a manifest to test their application than to concentrate on writing a better application. IDE's help up to an extent but I didn't want to argue. So I thought what if we can develop something that has a UI, asks for options and generates a YAML out of it. I had written something in perl more than a decade back to generate an Linux kickstart file which were on similar lines.(At that moment Perl was a thing!)

I thought why do something similar to that and write something in Python instead(Also, since I'm learning python it made sese to choose python). And something that can assimilate into the current GitOps flow.

The idea was never to replace a dashboard . Just provide a minimalistic approach to the developers and engineers whilst they concentrate on their niche instead of pulling their hair over kubernetes manifest syntax.

The final architecture:

The application currently uses Python's built-in HTTP server together with HTML, CSS, JavaScript and SQLite.
Also, it's debatable why i didn't chose Django. More on it some other day😁.

The code lives here: https://github.com/ashishnair86/mani

Since the application itself is intended for Kubernetes users, it made sense to package it as a Docker container.

The image can be found here: https://hub.docker.com/r/ashishnair/manifest-studio. As of now, a docker image does not make that much of sense as the app itself does not require anything more than the standard python modules. Or a moo-point as Joey Tribbiani would have said 😂

The first version intentionally focuses on Deployments and the resources commonly created alongside them.

Current features include:

Deployment generation
Namespace generation
Service generation
ConfigMap generation
Secret generation
Ingress generation
Horizontal Pod Autoscaler generation
Optional readiness and liveness probes
User authentication
Manifest history
YAML download
GitHub integration

Few words about the UI:

Whatever entries you make in the left pane will reflect immediately in the right pane.

Once you register, it also keep an history of your manifests if you select "Save History" :

Push to github allows to you do exactly what it says:

And if your CI/CD tool is configured poll SCM or If you configured webhooks your pipeline should kick-off after the commit:

I deliberately kept the scope small. Building one feature well teaches far more than trying to support every Kubernetes resource from day one.

From Metrics to Mitigation: Protecting my Application with NGINX Rate Limiting

Ashish Nair — Thu, 18 Jun 2026 17:02:35 +0000

In my previous article, I integrated Spring Boot Actuator with Prometheus and Grafana to gain visibility into application behavior. The goal was simple: expose metrics, scrape them with Prometheus, and visualize them in Grafana.

What I didn't expect was that the metrics would immediately influence architectural decisions.

After enabling metrics and exploring Prometheus, I started looking at request patterns using PromQL queries.

One metric in particular stood out:

sum by(uri)( http_server_requests_seconds_count )

This showed the total number of requests received by each endpoint.

The results were interesting(As an Admin not as someone who built this lab and send traffic on-purpose).

I thought (hypothetically), what if the Authentication and API endpoints were receiving significantly more traffic than expected? What if someone is sending more traffic than expected, what if someone is intentionally flooding the endpoints? (It could be my wife for not buying her gold last weekend 😂)

A login endpoint could be of the first targets during a brute-force attack, credential stuffing attempt, or denial-of-service event(Only a low esteemed hacker would be interested in my app 😂. But in production environments this is a real issue)

So the problem now is all legitimate and illegitimate requests are reaching my app and I'm wasting time processing the request, because the flow is:

Client -> Application -> MariaDB

So, I thought of introducing a new member to the family, tada - it's Nginx(Actually, reverse proxy). So now, the flow is :

client -> nginx -> App -> MariaDB

This architecture gave me several advantages:

A single entry point for all my services:
- Spring App
- Prometheus
- Grafana
Centralized routing. Meaning, all of the above will reach my reverse proxy which will forward the traffic to the correct endpoint.
Reverse proxy capabilities.
Rate-limiting (In Gen-z lingo, This is our main-character)

This is how i configured nginx for "Rate-limiting".

In simple words, this config:

allows 5 requests per minute.
Burst of 2 additional requests. (Because people make mistakes and having tolerations only helps)
Shouts a 429 when limits exceed ( Like a typical parent )

Let's test this out. Here I'm using a for-loop - A system admin's better half.

This started shouting at the 4th request:

But we configured nginx to tolerate 5 requests per minute right? Here's the math:

5 requests per minute = 1 request in 12 seconds.
Here 3 requests came in probably 3 seconds which means it allowed 1 request and already spent the 2 burst tokens it had it had since it happened within 3 seconds . Now, it did not have more burst tokens to assign it could not tolerate the 4th request and it blocked the request. It will block until a new token arrives every 12 seconds.

3 cheers to Nginx's Leaky Bucket Algorithm!! And 2 beers for me for breaking this down!!

The most important part is NGINX allowed the initial requests and began rejecting excess traffic before it ever reached our application.

With this setup the biggest learning is not the configuration itself but what we can improve on by enabling metrics, those dashboards have a meaning. In this case it forced an architectural change even if it's just adding one small component.

It's also important to know what endpoints to not limit. In my case, I only wanted to limit the /auth/register endpoint as only authenticated requests will reach my /students endpoint. If I limit my /students endpoint it will be like shooting myself in the foot. Also, I didn't want to limit my actuator endpoints as prometheus scrapes it every 15s.

You can configure dashboards in Grafana by connecting it to your prometheus instance to make you life easier.

Spring Boot + Prometheus: A Practical Introduction to Application Metrics

Ashish Nair — Wed, 10 Jun 2026 17:38:23 +0000

A developer spends a lot of time building features, but very little time asking an important question:

How healthy is our application when it is running? To be very specific we need to answer the below questions:

How many requests are we serving?
How much memory are we using?
Is our database connection pool exhausted?
Are users encountering errors?

Fortunately, Spring Boot and Prometheus make it surprisingly easy to answer these questions.

In this article, we'll:

Enable Prometheus metrics in a Spring Boot application
Configure Prometheus to scrape those metrics
Explore the most useful metrics exposed by Spring Boot

Enable Prometheus metrics in a Spring Boot application

Enabling prometheus metrics is a 2-step process:

Spring Boot uses Micrometer as its metrics facade. To expose metrics in a Prometheus-compatible format, add the following dependencies in pom.xml

The Actuator provides production-ready endpoints, while Micrometer translates metrics into a format Prometheus understands.

Expose the Prometheus Endpoint. Add the following properties to application.properties:

When you start your application, you should be able to see the metrics in the /actuator/prometheus endpoint.

Now our application is exposing operational data that we can scrape using Prometheus.

Configure Prometheus to scrape those metrics

configure prometheus.yml with the "target" we want to scrape, metric path and a scrape interval.

Explore the most useful metrics exposed by Spring Boot

At this point, Prometheus is scraping metrics every 15 seconds. Let's take a look at some important ones and query using PromQL:

jvm_memory_used_bytes

This tells us how much heap memory the JVM is currently using.

Let's check how many '403's we have(unauthorized requests):

How many requests are made to each endpoint in the last couple of hours?

Without observability such metrics often go unnoticed. Usually when there's a firefighting situation these queries are like gold.

Traditionally, developers(and lot of Admins like me) focus on logs. Observability encourages us to ask different questions:

Is the application healthy?
Are requests succeeding?
Are users experiencing failures?
Is the database becoming a bottleneck?
Is the JVM under stress?

Metrics provide the answers.

In the next article, we'll connect Prometheus to Grafana and build dashboards that allow us to visualize these metrics in real time.

From CI/CD to AI-Powered DevSecOps: Teaching a Local LLM to Analyze Security Reports

Ashish Nair — Tue, 26 May 2026 16:18:54 +0000

In the previous article, I built a complete CI pipeline around a Spring Boot monolith: GitHub → Jenkins → SonarQube → Trivy → Nexus → Docker. I introduced quality gates, security scanning, and artifact management.

But at the end of it, I had more questions than answers. The most important one being - What happens after Trivy generates the security report and uploads it to a nexus repository ? Do people actually go through the entire report?

In this article, we'll continue from the previous project and build an AI-powered DevSecOps workflow:

Jenkins generates Trivy security reports
Reports are uploaded into Nexus
Jenkins triggers an n8n webhook
n8n downloads the report
Local Ollama + phi3 analyzes vulnerabilities(smaller LLM because of compute restraints)
AI summarizes HIGH/CRITICAL findings
Email notifications are generated automatically

The Architecture:

The Flow:

Build application
SonarQube analysis
Quality Gate validation
Docker build
Trivy security scan
Upload reports to Nexus
Trigger AI workflow
Download report dynamically
Ask local LLM to analyze findings
Send AI-generated recommendations

The AI component in this setup answers the "What if's" questions that I had before. Before I move to the setup, I want to justify why I added the layer of AI . Let's be fair to the security guys here - Security reports can be long and noisy. It can have dozens of CVE's, dependent vulnerabilities and what not. Having them accountable for any miss sounds unfair. Here's where AI can help, It's like telling someone "read this report and give me the top 10 things i should be worried about in 10 seconds".

Sneak Peak of the setup:

Ran ollama locally using its docker image. Locally, because i wanted to avoid external API's and also because of poverty (😂)
Setup n8n also locally using its docker image. This is where we orchestrate the entire process using workflows.
- The jenkins job calls the webhook that n8n listens on.
- N8n kicks off the workflow by : Fetching the trivy report -> The next step is to prepare the AI prompt -> Send to Ollama and transform the response -> Send the report via email

This is how my workflow looks in n8n UI:

The transformed summary of the vulnerability report:

Further improvement is to train our LLM to be a junior SRE.

Building a Spring Boot Monolith Application and a DevSecOps Pipeline Around It

Ashish Nair — Wed, 20 May 2026 17:51:31 +0000

A lot of CI/CD tutorials show a simple “Hello World” app and a Jenkins job that prints Build Successful. That never felt close to how systems work in real environments. At least, they never helped me.

So I decided to build a small University Management Spring Boot monolith application with JWT authentication and protected APIs, and then build a complete pipeline around it — from source control to deployment, quality checks, security scans, artifact management and validation. Since I'm not a developer, the code itself isn't great it was just me exploring a new language along with some help from AI (Needless to say!)

The application itself is intentionally simple:

JWT-based authentication (/auth/login)
Protected APIs (/students)
Spring Security integration
Maven build and Docker packaging

The goal wasn't to show how to build an application (In that case, this would be the worst tutorial). Th goal is, to show a CI/CD(partly) flow to someone in less than 15 minutes.

So I created a mini DevSecOps pipeline that includes:

GitHub webhooks
Jenkins Controller + Agent
SonarQube quality gates
Docker image creation
Trivy security scans
Nexus artifact storage
Automated deployment using Docker Compose

Just incase, anyone's interested in the code, it stays here

By the end, a Git push automatically builds, scans, deploys and validates the application.

*The diagram: *

The App:

[root@server1 config]# curl -v -X POST "http://localhost:8081/auth/register?username=ram&password=redhat123"

Pass the credentials to the /auth/login API to login . The App will return a JWT token, which will then be used to interact with the API.

[root@server1 config]# curl -X POST \
"http://localhost:8081/auth/login?username=ram&password=redhat123"
eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJyYW0iLCJpYXQiOjE3NzkxNzA4MjIsImV4cCI6MTc3OTE3NDQyMn0.4AwQQqEa07f_-YId7LSlXYnctaWf7TdqG6xPYxTum_4

This step is actually not required, just a part of convenience.

[root@server1 config]# export TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJyYW0iLCJpYXQiOjE3NzkxNzA4MjIsImV4cCI6MTc3OTE3NDQyMn0.4AwQQqEa07f_-YId7LSlXYnctaWf7TdqG6xPYxTum_4

Create entries using the JWT token to authenticate using the /students API.

[root@server1 config]# curl -X POST \
http://localhost:8081/students \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name":"John",
"email":"john@test.com"
}'
{"id":1,"name":"John","email":"john@test.com"}

Query the API using the token.

[root@server1 config]# curl -H "Authorization: Bearer $TOKEN" http://localhost:8081/students;echo
[{"id":1,"name":"John","email":"john@test.com"}]

The pipeline:

We will make use of Jenkins to run the pipeline.
I have chosen to run the jobs off of a jenkins agent, simply to separate orchestration and execution

The very first stage of the pipeline after the developer pushes the code is to

checkout the code from source control
Build the application using Maven.

The build generates a jar file, which is quite famously referred to as the artifact.
Now, this artifact is scanned by our quality gates which checks it for any code smells, Bugs, Vulnerabilities, etc.

The sonarqube interface shows something like:

My 2 cents: Do not forget to add Jenkins call back in Sonarqube, else the job in Jenkins will wait to get the status from sonarqube and will eventually timeout.

If our code passes quality gates, we build the docker image and bake the artifact into the image.

Earlier, we scanned for vulnerabilities, bugs in the code. Now, it's time to scan for vulnerabilities in the docker image. I used Trivy for this. The ideal way to do this is to exit the job when there are vulnerabilities in the image. But for this lab I'm just running the scan as the image i pulled had loads of them.

Additionally, I also posted the scan results on a nexus repository with the job number appended.

This is the smallest CD part of the pipeline where we bring up the app using the image we built earlier using docker-compose.

Finally, keeping the artifact on the agent is not a good idea, so I moved it to a Nexus repo and ran a cleanup of the workspace to remove it from the agent.

Future enhancements: I must admit, this document is incomplete without :

converting this to a microservice
Moving the application to K8s or Openshift.
Adding monitoring for our App.

Integrating FreeIPA (LDAP) with OpenShift + Automated Group Sync Using CronJob

Ashish Nair — Thu, 14 May 2026 03:59:42 +0000

Managing users and groups centrally is critical in enterprise Kubernetes environments. In this guide, I’ll walk you through how I integrated FreeIPA (LDAP) with OpenShift and set up an automated CronJob to sync groups periodically.

This setup ensures:

Centralized authentication via FreeIPA
Automatic onboarding of users
Continuous group synchronization inside OpenShift

The Architecture:

And now, the subtitles for the Architecture!

User Login
- Developer logs in via OpenShift Console or CLI (oc login)
Authentication via LDAP/FreeIPA
- OpenShift OAuth server uses the configured LDAP Identity Provider
- Performs bind + search using the ldapbind service account
Group Sync Process
- oc adm groups sync queries LDAP periodically
- Fetches:
  - Users
  - Groups
  - Membership mapping
- Updates OpenShift Group objects
Automation
- CronJob runs the sync automatically every minute (in our setup)

The Integration:

Create a bind user (ldapbind in our case) to query LDAP/FreeIPA. Of course, you need to run this on the LDAP server. I have redacted some part of the output.

#ipa user-add ldapbind --first=ldap --last=bind --password
Password: 
Enter Password again to verify: 
---------------------
Added user "ldapbind"
---------------------
  User login: ldapbind
  First name: ldap
  Last name: bind
  Full name: ldap bind

Using the password you set, create a "secret" in openshift.

#oc create secret generic ldap-bind-password --from-literal=bindPassword='redhat' -n openshift-config

secret/ldap-bind-password created

Configure Oauth to use the LDAP/FreeIPA server. Basically, telling OAuth who's our LDAP server, where to search, the user who's allowed to search along with reference to the secret we created above, Yadda, Yadda, Yadda!! Here's the manifest and the command:

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
    - name: freeipa
      mappingMethod: claim
      type: LDAP
      ldap:
        url: "ldap://192.168.122.246/dc=example,dc=com?uid"
        bindDN: "uid=ldapbind,cn=users,cn=accounts,dc=example,dc=com"
        bindPassword:
          name: ldap-bind-password
        insecure: true
        attributes:
          id: [dn]
          name: [cn]
          email: [mail]
          preferredUsername: [uid]


#oc apply -f ldap.yaml 
oauth.config.openshift.io/cluster configured

Note: If you're too impatient, you can delete the pods in the "openshift-authentication" namespace so the integration is immediately picked up when the pods start back.

I created a user named "ashish" and Voila! I was able to login.

#oc login -u ashish api.snomaster.lab:6443 
Console URL: https://api.snomaster.lab:6443/console
Authentication required for https://api.snomaster.lab:6443 (openshift)
Username: ashish
Password: 
Login successful.

You don't have any projects. You can try to create a new project, by running

oc new-project <projectname>

Here's the tricky part. Whatever groups you create in LDAP/FreeIPA are not automatically imported by Openshift. This is where the second part of this article kicks in - _Automatic Group Sync. _

Automatic Group Sync - Part 1 ( The Manual way)

In this part we will create a manifest and run it to test if the manual sync works, if that works it's just a matter of creating a cron resource (Isolation is key!).This method is like a test that our manifest and group-sync works. So, here's the manifest

kind: LDAPSyncConfig
apiVersion: v1

url: ldap://192.168.122.246

bindDN: "uid=ldapbind,cn=users,cn=accounts,dc=example,dc=com"

bindPassword:
  file: "/etc/secrets/bindPassword"

insecure: true

rfc2307:
  groupsQuery:
    baseDN: "cn=groups,cn=accounts,dc=example,dc=com"
    scope: sub
    derefAliases: never
    filter: "(objectClass=groupofnames)"

  groupUIDAttribute: dn
  groupNameAttributes: [ cn ]

  groupMembershipAttributes: [ member ]

  usersQuery:
    baseDN: "cn=users,cn=accounts,dc=example,dc=com"
    scope: sub
    derefAliases: never

  userUIDAttribute: dn

  userNameAttributes: [ uid ]

Create a file that holds the "bind" password . These are the reasons why this is a bad method and should only be used for testing our sync. (These plain text passwords can give security guys an heart-attack!!)

#printf '%s' 'redhat' > /home/user/ldap-manifs/bindPassword

Time to test our group-sync. You might as well create some groups on your LDAP/FreeIPA server.

#oc adm groups sync --sync-config=group-sync.yaml --confirm
group/admins
group/ipausers
group/editors
group/trust admins
group/developers
group/platform-admins

We can also cross verify if they're actually visible in openshift.

#oc get groups
NAME              USERS
admins            admin
developers        bob
editors           
ipausers          ldapbind, ashish, bob
platform-admins   ashish
trust admins      admin

Why I call this automatic is, you can still add the oc adm groups sync command to cron and this will work just fine. The only bad part is the handling of password. There are ways to work around it, like ansible-ize your sync method and use ansible-vault to store the password or use Hashi Vault to store the password and fetch it using API's. We aren't going to touch base on those methods today. I will stick to the openshift-way of doing it. That brings us to part 2.

Automatic Group Sync - Part 2 ( The Openshift way)

The manifest remains the same, it's just that we will add the following resources to comply with production-grade standards:
We will create a secret to store the bind password. (Which we already did!).
Create a configMap that will store our group-sync manifest.
Create a CronJob Resource, this is the most interesting part.
Create a group-sync configMap
This is same group-sync manifest we used before, just we add it as a configMap resource this time. Here's the manifest

apiVersion: v1
kind: ConfigMap
metadata:
  name: ldap-group-sync-config
data:
  group-sync.yaml: |
    kind: LDAPSyncConfig
    apiVersion: v1

    url: ldap://192.168.122.246

    bindDN: "uid=ldapbind,cn=users,cn=accounts,dc=example,dc=com"

    bindPassword:
      file: "/etc/secrets/bindPassword"

    insecure: true

    rfc2307:
     groupsQuery:
       baseDN: "cn=groups,cn=accounts,dc=example,dc=com"
       scope: sub
       derefAliases: never
       filter: "(objectClass=groupofnames)"

     groupUIDAttribute: dn
     groupNameAttributes: [ cn ]

     groupMembershipAttributes: [ member ]

     usersQuery:
      baseDN: "cn=users,cn=accounts,dc=example,dc=com"
      scope: sub
      derefAliases: never

     userUIDAttribute: dn

     userNameAttributes: [ uid ]

The most interesting part of this process. We create a CronJob resource, this resource creates a temporary pod. This pod will mount our configMap and Secret inside the pod, run our job and terminate the pod. You're free to change the schedule.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ldap-group-sync
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
     template:
      spec:
       serviceAccountName: ldap-group-sync
       restartPolicy: OnFailure
       containers:
         - name: ldap-group-sync
           image: registry.redhat.io/openshift4/ose-cli
           command:
             - /bin/bash
             - -c 
             - |
             - oc adm groups sync --sync-config=/config/group-sync.yaml --confirm 
           volumeMounts:
             - name: sync-config
               mountPath: /config
           volumeMounts:
             - name: bind-secret
               mountPath: /etc/secrets
       volumes:
        - name: sync-config
          configMap:
           name: ldap-group-sync-config

        - name: bind-secret
          secret:
           secretName: ldap-bind-password

You can monitor the status under oc logs job/. You will see something to the tune of :

{"apiVersion":"v1","bindDN":"uid=ldapbind,cn=users,cn=accounts,dc=example,dc=com","bindPassword":{"file":"/etc/secrets/bindPassword"},"insecure":true,"kind":"LDAPSyncConfig","rfc2307":{"groupMembershipAttributes":["member"],"groupNameAttributes":["cn"],"groupUIDAttribute":"dn","groupsQuery":{"baseDN":"cn=groups,cn=accounts,dc=example,dc=com","derefAliases":"never","filter":"(objectClass=groupofnames)","scope":"sub"},"userNameAttributes":["uid"],"userUIDAttribute":"dn","usersQuery":{"baseDN":"cn=users,cn=accounts,dc=example,dc=com","derefAliases":"never","scope":"sub"}},"url":"ldap://192.168.122.246"}
group/admins
group/ipausers
group/editors
group/trust admins
group/developers
group/platform-admins
group/testers
group/hr

And just like that we have a fully automated Identity and Access management up and running.

OpenShift Upgrade Deep Dive: 4.19 4.20 (Step-by-Step + What Happens Under the Hood)

Ashish Nair — Thu, 07 May 2026 04:16:54 +0000

Upgrading Red Hat OpenShift isn’t just about running oc adm upgrade.
Behind the scenes, a carefully orchestrated process ensures your cluster stays available while everything gets updated.

In this guide, I’ll walk through:

Real Upgrade path 4.19 to 4.20
Handling AdminAckRequired blockers
Setting upgrade channels properly
Understanding how OpenShift upgrades nodes (masters + workers)
Why Pod Disruption Budgets (PDBs) can make or break your upgrade

Pre-Upgrade:

Check that none of the operators are "Degraded"
Check PDB's.
Check Failing Pods.
Run a must gather (oc adm must-gather).

1. Check current status

#oc adm upgrade

The output will look something like:

Cluster version is 4.19.27

Upgradeable=False

Reason: AdminAckRequired
Message: The admissionregistration.k8s.io/v1beta1 group version is deprecated in 4.19 and will be removed in 4.20.Any clients using the v1beta1 version of these resources must be updated to use the corresponding v1 version instead. See https://access.redhat.com/articles/7130599 for more information.

This essentially means our upgrade, usually Redhat will compliment the message with a knowledgebase as you can see in the above output.That knowledgebase will actually give you pointers on how to unblock.

2. Acknowledgement and Approval

The knowledgebase actually asked use to patch a configmap named admin-acks.

oc -n openshift-config patch cm admin-acks \
  --patch '{"data":{"ack-4.19-admissionregistration-v1beta1-api-removals-in-4.20":"true"}}' \
  --type=merge

3. Setting the upgrade channel

At the time when I ran this upgrade updates were only available in the fast channel but In production consider using stable or EUS channels.

oc adm upgrade channel fast-4.20

Verify by running the "oc adm upgrade" command again. Output should show something on these lines:

Channel: fast-4.20

VERSION     IMAGE
  4.20.18     quay.io/openshift-release-dev/ocp-release@sha256:2dab927fd20984e247301b2483083b71f942a1f550f5d8a1db42897edc042e39

If I could explain the relevance of channels in one word:

Fast channels - Latest features but mostly used for Dev/Test environments
Stable channels - Production grade upgrades
EUS channels - LTS upgrades

4. Trigger the upgrade

oc adm upgrade --to=4.20.18

Now, this is where most articles/guides stop. Because the next steps are taken care by openshift and most Engineers leave their desk for coffee breaks here. If you ask me, this part is equally important to understand, I will briefly walk though what openshift does "Under the Hood".

4.1 Upgrades the control planes first.

Suppose you have 3 masters, It will upgrade one master at a time, so that the cluster quorum is maintained. Each master goes through the below process:

Node gets cordoned.
Node gets drained.
Update is applied.
Node is rebooted.
API restored.
Uncordoned.
Moves on to the next Master.

4.2. Next it moves on the Worker nodes.

The worker nodes also go through a similar process:

Cordon
Drain
Update
Reboot
Uncordoned

Another topic that's less spoken about when an upgrade gets stuck is PDB's. If your PDB's are too strict your upgrade gets stuck and if it's too generous, the application is impacted. Be very careful while budgeting PDB's.

And lastly, Skip the coffee breaks and monitor the upgrade:

oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.19.27   True        True          11s     Working towards 4.20.18: 10 of 959 done (1% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18
version   4.19.27   True        True          11s     Working towards 4.20.18: 1 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 4 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 6 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 8 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 10 of 959 done (1% complete)

5. Verify completion

oc get clusterversion 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.18   True        False         10m     Cluster version is 4.20.18

My verdict on the upgrade process is, the upgrade is straight forward only when:

you handle any deprecations properly
Set correct channels.
Don't skip prechecks.

If this helped you, feel free to share or drop your upgrade experiences too!

Autoscaling in Openshift with Cluster Autoscaler and Machine Autoscaler

Ashish Nair — Mon, 27 Apr 2026 04:56:21 +0000

In my previous post, I covered how to scale compute (worker) nodes in OpenShift using a semi-automated approach. While OpenShift handled most of the heavy lifting—such as powering on the node via BMC, installing RHCOS, and joining the cluster—the scaling action itself still required manual intervention.

This document focuses on removing that manual step altogether. Specifically, it explores how Cluster Autoscaler and Machine Autoscaler work together to enable automatic, workload-driven scaling of compute nodes. While technologies like Cluster API and Machine API provide the underlying framework for managing machine lifecycles, the real decision-making around when to scale happens at the autoscaler layer.

Before I dive into it, I would like to explain in a line about the building blocks(So you don't think of me of a complete idiot!).

Cluster Autoscaler - responsible for observing the state of workloads in the cluster. It continuously monitors pending and unschedulable pods and determines whether adding (or removing) nodes would help satisfy resource requirements.

Machine Autoscaler - acts as the bridge between high-level scaling decisions and infrastructure changes.

Machinsets- Define how a worker node should be created

Cluster API: Cluster API is a Kubernetes project that provides a declarative way to create, manage, and scale Kubernetes clusters.

Machine API: Red Hat’s opinionated implementation of Cluster API concepts.

MachineSets: Equivalent of a ReplicaSet — but for nodes instead of pods.

The way these different component work together is:

The cluster Autoscaler watches for unschedulable pods.
Informs the Machine Autoscaler who updates the machineSets limits.Simply speaking, runs the "oc scale machineset/ replicas=2"
Our hardworking employee, Machine-API prepares the new node (of course, with help from BMH)
Pods get scheduled to the new worker node.

We will see how to perform the teeny-tiny set of 4 steps in Openshift:

Note: You can change to the "openshift-machine-api" namespace as most of the steps will be in that namespace:

oc project openshift-machine-api

Let's check our current nodes, machinsets and BMH(That's BareMetalHosts and not Jaspreet Bhumrah!)

oc get nodes
NAME      STATUS   ROLES                  AGE    VERSION
master    Ready    control-plane,master   8d     v1.27.10+28ed2d7
master2   Ready    control-plane,master   8d     v1.27.10+28ed2d7
master3   Ready    control-plane,master   8d     v1.27.10+28ed2d7
worker1   Ready    worker                 3d7h   v1.27.10+28ed2d7

oc get machinesets
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           8d

oc get bmh
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        8d
master2   externally provisioned   mycluster-7ln8n-master-1         true        8d
master3   externally provisioned   mycluster-7ln8n-master-2         true        8d
worker1   provisioned              mycluster-7ln8n-worker-0-qv4cn   true        3d10h

Let's create our cluster Autoscaler and Machineautoscaler manifests.

oc apply -f machineautoscaler.yaml 
machineautoscaler.autoscaling.openshift.io/worker-autoscaler created

oc apply -f clusautoscaler.yaml 
clusterautoscaler.autoscaling.openshift.io/default created

Note: You can construct the machine autoscaler and cluster autoscaler manifests from Redhat's official documentation

Let's add some load to our cluster. This is just a simple manifest that requests 1G memory and spawns ~20+ replicas.

oc apply -f dep.yaml 
deployment.apps/load-test created

This manifest has actually created someone load on the current worker node and needs probably another one to accommodate all the pods. Hence a lot of them are now in "Pending" state.

load-test-65777f99f7-458kp                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-4p72l                            1/1     Running   0          2m22s   10.128.2.43       worker1   <none>           <none>
load-test-65777f99f7-4v6hq                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-84n79                            1/1     Running   0          2m22s   10.128.2.39       worker1   <none>           <none>
load-test-65777f99f7-8955k                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>
load-test-65777f99f7-9h4j6                            0/1     Pending   0          2m22s   <none>            <none>    <none>           <none>

Our Autoscalers have actually updated our machineset:

oc get machineset
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   2         2         1       1           8d

Which has started provisioning a new node for us:

oc get bmh
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        8d
master2   externally provisioned   mycluster-7ln8n-master-1         true        8d
master3   externally provisioned   mycluster-7ln8n-master-2         true        8d
worker2   provisioning             mycluster-7ln8n-worker-0-9vszs   true                          12m
worker1   provisioned              mycluster-7ln8n-worker-0-qv4cn   true                          3d10h

In no time(Ok, around 20 minutes) you should see your new worker node ready to take workload. The pending pods will move to this new worker gradually.

oc get nodes
NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   8d      v1.27.10+28ed2d7
master2   Ready    control-plane,master   8d      v1.27.10+28ed2d7
master3   Ready    control-plane,master   8d      v1.27.10+28ed2d7
worker1   Ready    worker                 3d8h    v1.27.10+28ed2d7
worker2   Ready    worker                 3m14s   v1.27.10+28ed2d7

The "oc get pods -o wide" will actually tell you that the workloads are actually moving to this new worker node:

load-test-65777f99f7-458kp                            1/1     Running   0          33m     10.131.0.5        worker2   <none>           
load-test-65777f99f7-4p72l                            1/1     Running   0          33m     10.128.2.43       worker1   <none> 
load-test-65777f99f7-84n79                            1/1     Running   0          33m     10.128.2.39       worker1   <none>           <none>
load-test-65777f99f7-9h4j6                            1/1     Running   0          33m     10.131.0.6        worker2   <none>           <none>
load-test-65777f99f7-ctvc7                            1/1     Running   0          33m     10.128.2.41       worker1   <none>           <none>

The autoscalers not only works well for scaling-out, it also works pretty well for scaling-in.

To test this out I deleted the deployment I had created earlier:

oc delete deployment/load-test
deployment.apps "load-test" deleted

We can also check the event logs using 'oc get events'. It actually tells us the entire flow:

24m                  Normal    Killing                  Pod/load-test-65777f99f7-ctvc7                     Stopping container stress
24m                  Normal    Killing                  Pod/load-test-65777f99f7-hgf2n                     Stopping container stress

13m                  Normal    ScaleDownEmpty           ConfigMap/cluster-autoscaler-status                Scale-down: removing empty node "worker2"
13m                  Normal    ScaleDownEmpty           ConfigMap/cluster-autoscaler-status                Scale-down: empty node worker2 removed
13m                  Normal    DrainProceeds            Machine/mycluster-7ln8n-worker-0-9vszs             Node drain proceeds
13m (x10 over 66m)   Normal    SuccessfulUpdate         MachineAutoscaler/worker-autoscaler                Updated MachineAutoscaler target: openshift-machine-api/mycluster-7ln8n-worker-0
13m                  Normal    Deleted                  Machine/mycluster-7ln8n-worker-0-9vszs             Node "worker2" drained
13m                  Normal    DrainSucceeded           Machine/mycluster-7ln8n-worker-0-9vszs             Node drain succeeded
13m                  Normal    DeprovisioningStarted    BareMetalHost/worker2                              Image deprovisioning started

8m34s                Normal    DeprovisioningComplete   BareMetalHost/worker2                              Image deprovisioning completed
8m33s                Normal    PowerOff                 BareMetalHost/worker2                              Host soft powered off

In a nutshell it has:

On deleting the deployment, the containers were killed.
It scaled down the workload, we can also check the machineset output.
Drained the node.
Deprovisioned and Power-off the node.

user@server1:~/test/manifests$ oc get machineset -n openshift-machine-api
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           8d

virsh list --all
 Id   Name      State
--------------------------
 1    master    running
 2    master2   running
 3    master3   running
 4    worker1   running
 -    worker2   shut off

Scaling worker nodes in openshift with machine-api

Ashish Nair — Mon, 13 Apr 2026 05:59:39 +0000

In my last post, I wrote about how we can run an Openshift IPI install on KVM. In this document(Which is relatively shorter), I'll talk about my experience in scaling a worker node (a semi-automated method).

We will be breaking this down into 2 steps:

The Preparation
The Scaling

The Preparation

The prerequisite to start this is obviously a server( A blank VM in KVM, in our case).
A DNS entry in our Antagonist's Database.
The sushy webserver listening on port 8000, this is our IDrac/iLo emulation. More on that here
And a bit of patience (did i pull-off a Robin Sharma here?)

The Scaling:

Before we start the actual scaling process let me walk through the actual process :

We create a VM in KVM (This is as good as having a spare Baremetal host)
We create a Manifest (actually two) and apply it. This manifest will remind you of the install-config.yaml we used here
Once the manifest is applied, the machine is booted as a result of our BMC magic (via sushy and Redfish) and Installation is kicked-off.
Once the install is completed the Machine(node) joins the cluster.

Gather some data:

All (okay, most) of the work will happen in the openshift-machine-api namespace.
Let's check the current nodes that we have

oc get nodes
NAME          STATUS   ROLES                  AGE     VERSION
master        Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
master2       Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
master3       Ready    control-plane,master   3d20h   v1.27.10+28ed2d7
worker2.lab   Ready    worker                 2d22h   v1.27.10+28ed2d7

It's important to check the Baremetals we currently have.

oc get bmh -n openshift-machine-api
NAME      STATE                    CONSUMER                         ONLINE   ERROR                AGE
master    externally provisioned   mycluster-7ln8n-master-0         true        3d21h
master2   externally provisioned   mycluster-7ln8n-master-1         true        3d21h
master3   externally provisioned   mycluster-7ln8n-master-2         true        3d21h
worker2   provisioned              mycluster-7ln8n-worker-0-hpm5j   true        2d22h

A note on machinsets. Machine sets display the "group" of workernodes we have. They are used to scale compute(Workers). Make node of the name of the Machineset this is the one we will be scaling out.

oc get machinesets -n openshift-machine-api
NAME                       DESIRED   CURRENT   READY   AVAILABLE   AGE
mycluster-7ln8n-worker-0   1         1         1       1           3d21h

The manifests:

We need to 2 manifests here and it's largely inspired by the install-config.yaml we had earlier :

Snippet of install-config.yaml:

 - name: worker3
      role: worker
      bmc:
        address: redfish-virtualmedia+http://192.168.122.1:8000/redfish/v1/Systems/aa12a91d-56f9-41f4-b5ea-e5001dae179c
        username: admin
        password: password
      bootMACAddress: 52:54:00:fd:5e:1d
      rootDeviceHints:
         deviceName: /dev/vda

And our Manifests will do exactly what the above code we used while installation, but we split it into two:

Creates a BMH
Creates a secret to hold the console credentials.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: worker3
  namespace: openshift-machine-api
spec:
  online: true
  bootMACAddress: 52:54:00:c4:0e:6c
  bmc:
    address: "redfish-virtualmedia+http://192.168.122.1:8000/redfish/v1/Systems/d9fea2de-6fd9-4a44-99fc-95f32b610407"
    credentialsName: worker3-bmc-secret
  rootDeviceHints:
    deviceName: /dev/vda
apiVersion: v1
kind: Secret
metadata:
  name: worker3-bmc-secret
  namespace: openshift-machine-api
type: Opaque
stringData:
  username: admin
  password: password

Apply the manifests:

oc apply -f .

The wait (Part 1) :

After you apply the manifests, the node will transition into 4 states:

Registering
Inspecting
Available
Provisioned

For brevity we will jump to the "available" state, as this is the most interesting part. At this point if you login to the worker using 'ssh core@IP' you will see that our new worker is booted up using a CoreOS ISO by the Installer. This is a way of openshift telling us "The system is available, what do you want me to do?" (And this will also take us to the final part)

And we say we wanna scale, this is the machineset name I had asked you to make note of(If you haven't fallen asleep by now!). So the "oc scale" command uses the machineset name to scale the compute by 2 replicas in our case.

oc scale machineset/mycluster-7ln8n-worker-0 -n openshift-machine-api --replicas=2

Now, when you login to the worker machine you will see that the installation might have started:

all [-] coreos-installer: Read disk 868.2 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271
Apr 10 14:41:47 worker3 podman[1179]: 2026-04-10 14:41:47.517 1 DEBUG ironic_coreos_install [-] coreos-installer: Read disk 868.2 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271
Apr 10 14:41:48 worker3 ironic-agent[1191]: 2026-04-10 14:41:48.523 1 DEBUG ironic_coreos_install [-] coreos-installer: Read disk 883.6 MiB/2.4 GiB (35%) _run_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:271

The wait (Final part, I promise!):

In about 15-20 minutes you will see that the machine state will have transitioned from "Available" to "Provisioned"

oc get bmh -n openshift-machine-api
NAME      STATE                    CONSUMER                         ONLINE   ERROR   AGE
master    externally provisioned   mycluster-7ln8n-master-0         true             4d6h
master2   externally provisioned   mycluster-7ln8n-master-1         true             4d6h
master3   externally provisioned   mycluster-7ln8n-master-2         true             4d6h
worker2   provisioned              mycluster-7ln8n-worker-0-hpm5j   true             3d7h
worker3   provisioned              mycluster-7ln8n-worker-0-k67h7   true             7h41m

The "get nodes" should also show it's available for use.

oc get nodes
NAME          STATUS   ROLES                  AGE     VERSION
master        Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
master2       Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
master3       Ready    control-plane,master   5d8h    v1.27.10+28ed2d7
worker3       Ready    worker                 9h      v1.27.10+28ed2d7
worker2.lab   Ready    worker                 4d10h   v1.27.10+28ed2d7

That's all folks!!

While the procedure looks lengthy while explaining and documenting, it essentially is just creating the manifests and running oc scale command. If you have scaled a compute manually you will appreciate how much work this method cuts down!