DEV Community: Gaganpreet Singh

Autonomous Cyber Deception Grid: Designing Honeypots That Adapt

Gaganpreet Singh — Sat, 31 Oct 2026 14:00:00 +0000

A honeypot that never changes gets fingerprinted quickly. The Autonomous Cyber Deception Grid is my design for a multi-level deception system that starts as a set of static decoys and grows into an environment that adapts to the attacker in front of it, so defenders learn more and the attacker wastes more time.

Code: github.com/Willey2003/cyber-deception-grid

Four levels

Level 1: Static deception. Containerised mock services (SSH, HTTP and a MySQL-style database) that log the attacker’s source address, timestamps and every command they type.

Level 2: Behavioural adaptation. A local LLM generates believable command output, fake files and system errors, tailored to the attacker’s technique, so the decoy holds up beyond the first few commands.

Level 3: Autonomous landscape evolution. A reinforcement learning controller changes the fake network topology, moves high-value decoys, and expands the network footprint around aggressive actors.

Level 4: Threat intelligence. Captured payloads go to a malware sandbox, observed behaviour is mapped to MITRE ATT&CK, and the results feed a central dashboard.

Services

Service	Port	Role
ssh-honeypot	2222	Fake SSH server
http-honeypot	8080	Fake web server
db-honeypot	3306	Fake MySQL database
deception-api	8000	Orchestration API
llm-adapter	8001	Response generation
rl-controller	8002	Topology controller
intel-processor	8003	Threat intelligence
dashboard	3000	Web dashboard

What is built so far

Level 1 is the starting point. The SSH honeypot is written in Python with Paramiko and asyncio: it accepts password logins, records each credential pair and command in an attacker session, and hands the session to an orchestrator that later levels will use to decide how the environment responds. A CI workflow runs on every push.

Design principles

Containment first. Decoys run in isolated containers with no route to real systems. A honeypot that becomes a pivot point is worse than none.
Local models only. Attacker input never goes to a hosted AI service, and the model only ever generates fake output.
Everything maps to ATT&CK, so what the grid learns is usable by the SOC and not just interesting.

Next steps

HTTP and database decoys, the LLM adapter for Level 2, and a dashboard that shows live sessions and ATT&CK techniques. Run honeypots only on networks you own or are authorised to use, and keep them isolated from production.

Classifying Malware Variants with Graph Neural Networks

Gaganpreet Singh — Thu, 29 Oct 2026 14:00:00 +0000

Malware authors rarely write a new family from scratch. They recompile, repack and patch, so the bytes change while the program’s structure stays much the same. Hash-based detection misses those variants. Structure-based classification can catch them. This project is my build of that idea: a Graph Neural Network that classifies malware variants from their control flow graphs and call graphs.

Code: github.com/Willey2003/malware-variant-classifier

How it works

Feature extraction. Each binary is disassembled with Capstone, and its control flow graph and call graph are built with NetworkX. Each node gets a small feature vector describing its basic block.
Graph dataset. The NetworkX graphs are converted to PyTorch Geometric Data objects, labelled by family.
Model. A configurable GNN encoder (GCN, GAT or GraphSAGE, 3 layers, 128 hidden units, 64-dimensional embedding) with pooling and a classification head over 10 families.
Training. A training loop with validation split, checkpoints and MLflow experiment tracking.
Inference API. A FastAPI service with /health, /predict and /predict/batch endpoints that accept uploaded binaries and return the predicted family with a confidence score.

Deployment

The service ships with a Dockerfile, a docker-compose file and Kubernetes manifests, so the same image runs on a laptop or a cluster.

pip install -r requirements.txt
python src/training/train.py --config configs/train_config.yaml

docker-compose up --build     # local stack
kubectl apply -f k8s/         # Kubernetes

Why graphs

A repacked or recompiled variant changes its hash and many of its bytes, but its control flow tends to survive.
GNNs learn from the shape of the graph, so two samples with similar structure land close together in embedding space even when their bytes differ.
The embedding is also useful on its own: nearest-neighbour search over embeddings groups unknown samples with their closest known family.

Status and next steps

This is an early build. The pipeline, model, API and deployment files are in place. Next is training on a public labelled dataset, publishing accuracy per family, and comparing the three GNN architectures. Only handle live malware samples in an isolated analysis environment.

What Survives a VMware to OpenShift Migration (and What to Check)

Gaganpreet Singh — Tue, 27 Oct 2026 13:00:00 +0000

"Will it still be the same VM?" is the first question every application owner asks about a migration. Licences are keyed to serials, DHCP reservations to MACs, and fstab entries to disk IDs.

So after migrating a VM from vSphere to OpenShift Virtualization with MTV, I read the new VirtualMachine and traced every field back to the source.

What was kept

Field on the OpenShift VM	Value	Where it came from
`interfaces[net-0].macAddress`	`00:50:56:aa:00:01`	The source NIC's VMware MAC
`firmware.serial`	`VMware-42 00 00 …`	The VMware BIOS serial
`disks[vol-0].serial`	`6000C290-…`	The VMDK's UUID
`cpu`	2 sockets × 1 core	The source vCPU topology
`memory.guest`	`1Gi`	The source RAM
`firmware.bootloader`	`bios`	The source firmware (BIOS, not EFI)

(Values are documentation placeholders. In the lab, the real values matched the source exactly.)

Why each one matters:

MAC address: DHCP reservations and MAC-bound licences keep working.
BIOS serial: licence checks keyed to the hardware serial still pass.
Disk serial: /dev/disk/by-id paths inside the guest don't change, so fstab and app configs that use them keep working.

Check it on your own VM

VM=ubuntu-noble-24.04-cloudimg; NS=mtv-test
oc get vm $VM -n $NS -o jsonpath='{.spec.template.spec.domain.devices.interfaces[0].macAddress}{"\n"}'
oc get vm $VM -n $NS -o jsonpath='{.spec.template.spec.domain.firmware.serial}{"\n"}'
oc get vm $VM -n $NS -o jsonpath='{.spec.template.spec.domain.devices.disks[0].serial}{"\n"}'
oc get vm $VM -n $NS -o jsonpath='{.spec.template.spec.domain.cpu}{"\n"}'

Compare them with the source VM in the vSphere Client (network adapter MAC under Edit Settings) and, inside the guest before and after, with:

ip link                      # MAC
sudo dmidecode -s system-serial-number
ls -l /dev/disk/by-id/

What I did not verify

Static IPs. My test VM used DHCP, so I can't claim static IP preservation from this lab. MTV has a setting for it, but test it on one of your own VMs with a static address before you promise it to anyone.

The network path also changes. The VM now reaches its VLAN through a NetworkAttachmentDefinition on a Linux bridge, not a port group. Same VLAN, new plumbing, so test connectivity, not just the MAC.

Lesson

Prove it on one VM, write down the before and after, and attach that to your migration runbook. It turns "trust me" into evidence.

The full worked example, with every field of the migrated VM explained, is in the free Lab Manual. Next in this series: five gotchas from building the lab.

Modelling Worm Outbreaks in Python to See Which Defences Actually Help

Gaganpreet Singh — Sun, 25 Oct 2026 14:00:00 +0000

Every postmortem of a fast-moving outbreak ends with the same questions. Would segmentation have contained it? How much did the patch backlog matter? Would quarantining infected hosts at hour two have changed anything? In a real environment you cannot rerun the incident to find out. In a model, you can rerun it as many times as you like.

That is why I built the Worm Propagation Simulator. It is a small Python project that models how a self-replicating worm spreads across a network graph, and how defensive measures change the outcome. It is built for defensive analysis and teaching. It contains no exploit code, sends no network traffic and touches no real systems. Every “host” is a node in an in-memory graph, and every “infection” is a random number compared against a probability.

Repo: github.com/Willey2003/worm-propagation-simulator

How the model works

The simulation runs in ticks. The engine builds a network graph with NetworkX and gives every node a state: vulnerable, infected, patched or quarantined. Each node gets a security level of low, medium or high, which sets its chance of being compromised on a single attempt to 0.75, 0.5 or 0.25. A configurable share of nodes starts out already patched. One or more “patient zero” nodes start out infected.

On each tick, every infected node picks targets according to the chosen propagation model. For each target, the engine checks three things in order: whether the target is still susceptible, whether a firewall rule between their subnets blocks or rate-limits the attempt, and finally the probability roll. The engine records per-tick metrics as it goes: node counts by state, new infections, attempts, and attempts blocked by firewall rules.

What you can vary

Dimension	Options in the code
Topology	`random` (Erdős–Rényi), `scale_free` (Barabási–Albert), `small_world` (Watts–Strogatz), `hierarchical` (balanced tree), `datacenter` (fully meshed racks linked to each other)
Propagation model	`random_scan`, `hitlist`, `topological` (graph neighbours only), `permutation`, `local_preference` (biased towards the node’s own subnet)
Baseline defence	`patch_rate`: the fraction of nodes patched before the outbreak starts
Segmentation	Subnet-to-subnet firewall rules with `deny` or `rate_limit` (percentage of attempts allowed)
Mid-outbreak response	One scheduled intervention: `patch_all`, `patch_subnet`, `quarantine_infected` or `rate_limit`
Reproducibility	A `seed` value fixes the random state, so runs are repeatable

Nodes are assigned round-robin to subnets of about 20 nodes each. This is what gives the segmentation rules and the local-preference model something to act on.

The configs/ folder ships six YAML scenarios: five are loosely inspired by well-known worms (Code Red, SQL Slammer, Conficker, WannaCry, Mirai), plus a defensive_analysis scenario. Treat them as stylised parameter sets for comparison, not calibrated reproductions of history. A 500-node graph says nothing precise about the 2001 internet. What they do give you is a fair way to compare propagation behaviour and defensive choices against each other.

Running it

The core engine needs only NetworkX, NumPy and PyYAML. The test suite (17 tests covering nodes, topologies, propagation models, patching and firewall blocking) also runs in GitHub Actions on every push:

git clone https://github.com/Willey2003/worm-propagation-simulator.git
cd worm-propagation-simulator
pip install networkx numpy pyyaml pytest
python -m pytest tests/ -v

To run a scenario and print the summary:

import yaml
from src.simulator.engine import create_simulator_from_config

cfg = yaml.safe_load(open("configs/defensive_analysis.yaml"))
sim = create_simulator_from_config(cfg)
sim.run()
summary = sim.get_summary()
print({k: v for k, v in summary.items() if k != "infection_curve"})

The summary includes the peak infected count, the tick at which it peaked, total infections and attempts, and how many attempts were blocked. infection_curve holds the full per-tick history if you want to plot it.

The lesson that surprised me: timing beats thoroughness

The defensive_analysis scenario puts 500 nodes on a scale-free graph and 20% of them start patched. It applies two firewall rules between subnets and schedules a patch_all intervention at tick 30. With seed 42, that intervention never fires. The worm reaches every vulnerable node by tick 5 and the run ends, with 396 nodes infected.

Change one line, intervention_tick: 2, and the same patch_all caps the outbreak at 7 infected nodes out of 500.

The model is simple, but this is the same lesson real incidents keep teaching. Against a fast scanner, a complete response that arrives late is worth almost nothing. A response that arrives early is worth almost everything. That is the case for pre-approved containment playbooks and for patching exposed services before an outbreak, not during one.

Other entry points

The repo also includes:

A Typer CLI with run, compare and scenarios commands, using Rich tables for output
A FastAPI service to create simulations, step them, apply interventions mid-run, fetch state over REST, and stream per-tick metrics over a WebSocket
A Plotly visualisation module that writes the infection curve, network graph and a combined dashboard to HTML
Docker, Docker Compose and Kubernetes manifests for running the API in a container

From the command line, the same runs are one command each:

python -m src.simulator.main scenarios
python -m src.simulator.main run --scenario code_red
python -m src.simulator.main compare --scenarios code_red,slammer

The compare command prints a side-by-side table of infections, peak timing and blocked attempts for each scenario.

Ideas for extending it

Ideas I would like to explore next: per-node detection delay before quarantine, more than one intervention per run, and calibrating scan rates against published outbreak data. If you teach network security, the topology comparison on its own makes a good lab exercise. Run the same worm on a random graph and on a scale-free graph and look at what the highly connected hubs do to the curve.

Get it

The source, scenario files and tests are at github.com/Willey2003/worm-propagation-simulator. Clone it, change one parameter at a time, and see which defences change the curve. If you build a scenario that teaches something useful, open a pull request.

VDDK vs No VDDK: Why Your First MTV Migration Is Slow

Gaganpreet Singh — Fri, 23 Oct 2026 13:00:00 +0000

My first migration from VMware to OpenShift Virtualization worked. It was also painfully slow: one 10 GiB Ubuntu VM took 1 hour 24 minutes.

Step	Time
ImageConversion (virt-v2v)	23 min 26 s
DiskTransfer, about 2.8 MB/s	60 min 42 s
Total	1 h 24 min

At that rate a 500 GiB production disk would take more than two days. Here's why, and the fix.

The symptom

While the copy ran, the migration Pod's log said:

nbdkit: curl[4]: ...
virt-v2v monitoring: Progress update, completed 17 %

nbdkit curl means the disk was being read over HTTPS through vCenter (port 443). That path works anywhere, but it's slow, and it puts the load on vCenter.

The cause

The vSphere provider had no vddkInitImage in its settings:

oc get provider vcenter-lab -n openshift-mtv -o jsonpath='{.spec.settings}'

VDDK is VMware's Virtual Disk Development Kit, the same library backup products use. With it, MTV reads disk data straight from the ESXi host over NBD on port 902, without going through vCenter. Red Hat isn't allowed to redistribute VDDK, so MTV can't ship it. You download it from Broadcom and wrap it in a small container image yourself.

The fix: build the VDDK image

# on a machine with podman and access to your registry
mkdir -p ~/vddk
tar -xzf ~/VMware-vix-disklib-8.0.*.x86_64.tar.gz -C ~/vddk
cat > ~/vddk/Containerfile <<'EOT'
FROM registry.access.redhat.com/ubi9/ubi-minimal
USER 1001
COPY vmware-vix-disklib-distrib /vmware-vix-disklib-distrib
RUN mkdir -p /opt
ENTRYPOINT ["cp", "-r", "/vmware-vix-disklib-distrib", "/opt"]
EOT
podman build -t <REGISTRY>/openshift-mtv/vddk:8.0 ~/vddk
podman push <REGISTRY>/openshift-mtv/vddk:8.0

# point the provider at it
oc patch provider vcenter-lab -n openshift-mtv --type merge \
  -p '{"spec":{"settings":{"vddkInitImage":"<REGISTRY>/openshift-mtv/vddk:8.0"}}}'

Before your first wave, check

The OpenShift nodes can reach every ESXi host on TCP 902. A quick test from a node: timeout 4 bash -c "</dev/tcp/<esxi>/902" && echo open
The provider shows the VDDK image and is Ready: oc get provider -n openshift-mtv
You've timed one test VM with VDDK, so your wave plan uses your own throughput number, not a guess.

Lesson

Build the VDDK image before wave 1. Without it, MTV still works, which is exactly why it's easy to miss until a production disk takes days.

The full chapter, with the provider setup and real output, is in the free Lab Manual. Next in this series: what actually survives a migration.

Red Hat Astronaut 2026: a self-hosted OpenShift and Ansible practice platform

Gaganpreet Singh — Wed, 21 Oct 2026 14:00:00 +0000

I’m preparing for the OpenShift administration exam (DO280/EX280), and I kept running into the same problem. Red Hat exams are fully hands-on, but most free practice material is either a static PDF or a quiz with no explanations. I wanted something that would run against a real cluster and tell me, objectively, whether the thing I built actually works. So I wrote one and published it: Red Hat Astronaut 2026.

It is an independent practice tool. It is not affiliated with or endorsed by Red Hat. All questions are original material written to the published objectives. None of it is taken from, or claims to reproduce, real exam content.

Repo: github.com/Willey2003/redhat-astronaut-2026

What it is

The platform has four parts:

Engine. A Python web UI and API (the facilitator) plus a separate grader (the conductor), packaged with Docker Compose. Kubernetes manifests are included if you want to host it in a lab cluster.
Question banks. Plain YAML, one file per question, with an exam.yaml per bank for duration, pass mark and domain weights. A validator checks every bank, and it runs in CI on every push.
Curriculum. Six top-level modules, from containers and Linux up to capstones. There are also separate step-by-step tracks for the admin path (five modules) and the developer path (four modules).
Cluster tooling. Helper scripts for Red Hat OpenShift Local (CRC) and OKD, plus a preflight script that checks CPU, RAM, disk, virtualisation support and the oc client.

The banks

There are 15 study banks with 280 questions, plus a 4-question smoke-test bank. 25 questions are hands-on, all of them in the EX280 and EX288 banks. Everything else is single-answer multiple choice, with an explanation for every question.

Bank ID	What it covers	Questions	Hands-on
`ex280-admin`	OpenShift administration	20	12
`ex288-developer`	OpenShift application development	20	13
`ex229`	Core cluster admin: nodes, RBAC, SCCs, routes, storage	16	0
`ex282-developer`	Developer basics: projects, builds, ImageStreams, Tekton	18	0
`ex480-developer`	Advanced delivery: GitOps, serverless, mesh, rollouts	18	0
`ex380-admin`	Advanced admin: MachineConfig, autoscaling, remediation	18	0
`ex432`	Cluster installation (IPI/UPI, bootstrap, SNO)	16	0
`ex316`	OpenShift Virtualization	16	0
`ex370-storage`	OpenShift Data Foundation	16	0
`ex180`	Podman and container basics	18	0
`ex430-ansible`	Ansible and Automation Platform	16	0
`rhce`	Ansible fundamentals	20	0
`rhca`	Architecture and hybrid cloud concepts	20	0
`rhcoa`	OpenStack concepts	18	0
`knowledge`	Kubernetes and OpenShift concepts	30	0

A note on the bank IDs. Treat them as labels, not as official exam mappings. Red Hat renames and retires exams regularly, and a few of my labels don’t match the current catalogue. For example, ex432 holds installation material. Check the current objectives on Red Hat’s site and use the “What it covers” column to pick a bank.

Also, real Red Hat exams have no multiple-choice section. The multiple-choice banks here are concept checks to find gaps, not an imitation of the exam format. The hands-on banks are the closer rehearsal.

How grading works

Each hands-on question carries a list of checks. A check is an oc or shell command plus an expectation, such as:

equals
contains
a regex matches
exit-success
non-empty

The conductor runs the checks against your cluster and splits the points evenly across them, so a half-finished task earns partial credit. A Route task, for example, checks the host, the target service and the TLS termination type separately. Because the checks look at cluster state rather than at your YAML, any valid way of solving the task passes.

There are three modes:

Training shows the whole bank untimed, with solutions visible.
Mastery and Exam are timed and hide answers until grading.

Results include a per-domain score, weakest first.

There are also three written labs:

SCCs, RBAC and quotas for EX280
two application labs for EX288

Each lab ends with verification commands you can run yourself.

Running it

You need Docker with Compose. For hands-on work you also need an OpenShift cluster. CRC on a machine with at least 8 GB of RAM is the simplest option, and the helper script asks for your own Red Hat pull secret.

git clone https://github.com/Willey2003/redhat-astronaut-2026.git
cd redhat-astronaut-2026
./ga doctor                    # preflight
./cluster/preflight.sh         # can this host run CRC/OKD?
./ga up                        # build and start the platform

Then open http://127.0.0.1:8903. There is no separate install step: ga up builds the images and starts the stack. To start a timed attempt from the terminal instead, run ./ga exam ex280-admin.

The conductor mounts your ~/.kube read-only and runs oc and kubectl from inside its container. Use a practice cluster only, and make sure its API is reachable from that container. The platform has no login, so keep it bound to localhost unless you are on a private network.

Status

This is v0.1 and it is not finished. Most banks are knowledge-only, and the roadmap lists what is planned but not built yet:

setup steps that reset a task’s starting state before each attempt
multi-answer questions
a grading backend that talks to the OpenShift API instead of a shell
progress tracking

Get it

Repo: github.com/Willey2003/redhat-astronaut-2026
Licence: Apache-2.0
Independent project, not affiliated with or endorsed by Red Hat. Product and certification names are trademarks of their owners.

If you’re working towards an OpenShift certification and want to add hands-on tasks for another bank, pull requests are welcome. The question format is documented in docs/bank-spec.md.

VMware Concepts Mapped to OpenShift Virtualization: VMDK, Port Groups, vMotion and More

Gaganpreet Singh — Tue, 20 Oct 2026 13:00:00 +0000

If you've run vSphere for years, OpenShift Virtualization looks foreign for about a day. Then you notice that almost every VMware concept has a direct counterpart. It just has a Kubernetes name and lives in YAML instead of a vCenter database.

This is the translation table I wish I'd had on day one, taken from the lab I built for The Lab Manual: Migration Edition.

The core table

vSphere	OpenShift Virtualization	What to know
ESXi host	Node with `/dev/kvm`	KVM is built into the RHCOS kernel
VM (the `.vmx` definition)	`VirtualMachine` (VM)	Exists even when powered off
Running VM process	`VirtualMachineInstance` (VMI) + `virt-launcher` Pod	Exists only while the VM runs
VMDK	PVC / DataVolume	The disk is a PersistentVolumeClaim
Datastore	StorageClass	Backed by a CSI driver (NFS, FC, iSCSI, Ceph…)
Port group	NetworkAttachmentDefinition (NAD)	Carries the VLAN ID
vSwitch / VDS	Linux bridge built by NMState	Declared once, applied on every node
vMotion	Live migration (`VirtualMachineInstanceMigration`)	Needs RWX storage
VM template / content library	Instance types + preferences + boot sources	"How big", "which OS", "which image"
VMware Tools	QEMU guest agent + virtio drivers	MTV installs them during conversion
vCenter	The OpenShift API + the console's Virtualization pages

The three that trip people up

1. VM vs VMI. In vSphere a VM is one thing. Here it's two: the VirtualMachine is the definition (like the .vmx), and the VirtualMachineInstance only exists while it's running. Powering off deletes the VMI and its Pod. The VM object stays.

oc get vm,vmi -n mtv-test

2. Port groups become NADs. A NAD is a small JSON config: which bridge, which VLAN. Put shared networks in the default namespace and any project can use them as default/vlan-110.

oc get net-attach-def -A
oc get net-attach-def vlan-110 -n default -o jsonpath="{.spec.config}"

3. vMotion needs RWX. Live migration moves the VM's memory to another node, but both nodes need the disk at the same time. That means a StorageClass with ReadWriteMany. In my lab, OpenShift Virtualization didn't recognise the NFS StorageClass and would have created RWO disks. One StorageProfile patch fixed it:

oc patch storageprofile nfs-csi --type merge \
  -p '{"spec":{"claimPropertySets":[{"accessModes":["ReadWriteMany"],"volumeMode":"Filesystem"}],"cloneStrategy":"copy"}}'

Where vSwitches went

On each node, an NMState policy (NodeNetworkConfigurationPolicy) builds a Linux bridge, br-vm, on a second NIC that carries a VLAN trunk. NADs then tag each VM's traffic with its VLAN on that bridge. It's the same idea as a distributed switch with port groups, declared as code.

oc get nncp           # the policy
oc get nnce           # did it apply on each node?

The architecture diagram shows all of these objects in one picture, and the free manual walks through creating each one with real output. Next in this series: why my first migration ran at 2.8 MB/s.

Golden Kubestronaut 2026: a self-hosted practice simulator for the CNCF path

Gaganpreet Singh — Sun, 18 Oct 2026 14:00:00 +0000

After passing the CKA again this year, I wanted one place to practise for the rest of the CNCF list without juggling a dozen browser tabs and paid mock-exam sites. So I built a small, self-hosted simulator and put it on GitHub: Golden Kubestronaut 2026.

It is a practice tool, not a shortcut. Every question is original practice material written against the published exam objectives, and the project is independent: it is not affiliated with or endorsed by the Linux Foundation or the CNCF, and the questions are not taken from, and do not claim to mirror, the real exams.

Repo: github.com/Willey2003/golden-kubestronaut

What is in the box

The project has four parts:

Engine. A small Python service split into a facilitator (web UI and JSON API) and a conductor (the grader), plus a bank loader, a validator and a ga command-line wrapper.
Question banks. One folder per certification: an exam.yaml with duration, pass mark and domain weights, and one YAML file per question.
Curriculum. Seven Markdown modules, from foundations to networking and cost, with exercises to run on your own cluster.
Deployment. A Docker Compose file for running it locally, and Kubernetes manifests if you prefer to host it in a lab cluster.

The banks

There are 16 banks and 274 questions in total. 48 of them are hands-on tasks; the rest are single-answer multiple choice with a written explanation for every option.

Bank	Focus	Questions	Hands-on
`cka`	Cluster administration	16	10
`ckad`	Application development	16	11
`cks`	Security	16	11
`kcna`	Cloud native fundamentals	24	0
`kcsa`	Security concepts	16	4
`lfcs`	Linux system administration	18	4
`pca`	Prometheus and PromQL	18	0
`otca`	OpenTelemetry	18	0
`ica`	Istio service mesh	18	0
`capa`	Argo CD, Workflows, Events, Rollouts	16	0
`cgoa`	GitOps principles and practice	16	0
`cba`, `cnpa`, `cnpe`	Cilium networking, basic to advanced	48	8
`cca`	General cloud concepts	18	0
`kca`	Kubernetes cost and FinOps	16	0

Known gaps

This is version one and I would rather be upfront about it. Several bank IDs do not yet match the content of the certification that carries that code:

The Cilium material currently lives under cba, cnpa and cnpe. The real Cilium certification is CCA, but the cca bank holds general cloud-computing questions instead.
There is no Backstage (CBA), Kyverno (KCA) or platform engineering (CNPA/CNPE) content yet. The kca bank covers Kubernetes cost allocation instead.
The labs/ folder is empty. The hands-on practice currently lives in the curriculum exercises and in the hands-on bank questions.

So today, the CKA, CKAD, CKS, KCNA, KCSA, LFCS, PCA, OTCA, ICA, CAPA and CGOA banks line up with their syllabi, and Cilium is well covered under the wrong labels. Fixing the labels and writing the missing banks is next on my list.

How grading works

Multiple-choice questions are marked right or wrong against the answer key.

Hands-on questions are where the project earns its keep. Each one has a list of checks: a shell command plus an expected result (equals, contains, a regex matches, exit-success and so on). The conductor runs those commands against your cluster and splits the points evenly across the checks, so you get partial credit. For example, the etcd backup task checks that the snapshot file exists at the required path and is not empty. It doesn’t look at what you typed.

There are three modes:

Training shows every question in the bank with no timer, and reveals answers and solutions straight away.
Mastery and Exam draw a timed attempt and hide answers until grading.

After each attempt you get a per-domain breakdown, weakest domain first, which tells you what to study next.

Running it

You need Docker with Compose. For local validation you also need Python 3 and PyYAML.

git clone https://github.com/Willey2003/golden-kubestronaut.git
cd golden-kubestronaut
./ga validate      # check every bank parses and is well-formed
./ga doctor        # preflight (python, docker, oc, kubectl, RAM)
./ga up            # build and start the containers

Then open http://127.0.0.1:8902. By default it binds to localhost only. There is no authentication, so if you use ./ga expose to reach it from your LAN, keep it on a private network.

For hands-on grading, the conductor container mounts your ~/.kube read-only (override with KUBECONFIG_DIR) and runs kubectl from inside the container. Two practical notes:

Point it at a throwaway practice cluster, never a real one, because the checks execute commands with that kubeconfig.
The cluster’s API address must be reachable from inside the container.

One rough edge: doctor shares preflight code with my OpenShift project, so it flags a missing oc client and less than 8 GB of RAM. Ignore those on a plain Kubernetes setup.

The smaller sibling: kubestronaut

There is also an earlier repo, Willey2003/kubestronaut. It runs the same engine and shares six identical banks with this one: CKA, CKAD, CKS, KCNA, KCSA and PCA. It adds two banks you won’t find here:

a Grafana bank (dashboards, data sources, alerting, provisioning)
a combined Prometheus-plus-Grafana monitoring track (cpna)

Neither of those is a Linux Foundation certification; they are skill tracks. Use it if you want that Grafana practice. For everything else, use the golden repo.

Get it

Repo: github.com/Willey2003/golden-kubestronaut
Licence: Apache-2.0
Not affiliated with or endorsed by the Linux Foundation or CNCF. Certification names are trademarks of their owners.

If you spot a wrong answer or want to help write the missing banks, open an issue or a pull request. The bank format is plain YAML, and ./ga validate runs in CI on every push.

Zero to Senior Cloud-Native Engineer: A Free, Hands-On Lab Path on GitHub

Gaganpreet Singh — Fri, 16 Oct 2026 14:00:00 +0000

People regularly ask me where to start in Kubernetes, DevOps or platform engineering with no Linux, networking or coding background. A list of links and course names doesn’t help much when you don’t yet know what a process, a subnet or a YAML file is.

So I wrote the path down properly, as a GitHub repo you can clone and work through in order: zero-to-senior-cloud-native-engineer. It covers a roadmap, a certification order, numbered labs, on-call style break/fix tickets and a capstone project spec. This post explains what is in it, how it is meant to be used, and which parts are still unfinished.

Who it is for

The repo assumes you are starting from nothing. Each lab follows the same structure:

Why this matters: what the topic is and where you will meet it at work
Concepts: the few ideas you need before typing anything
Hands-on steps: exact commands and what you should see
Real-world scenario: a broken system to fix, written like an on-call ticket
Check yourself: questions and tasks that show you actually learned it
Cert mapping: which exam objectives the lab covers

The README’s golden rules: type every command yourself, break things on purpose, spend two hours in the lab for every hour of reading, and explain each topic out loud.

The roadmap: 30 months, about 25 hours a week

ROADMAP.md lays out 20 phases (0 to 19) over roughly 30 months, assuming about 25 hours a week. The order goes from fundamentals to specialisation:

Phase	Topic	Exam at the end
1-4	Linux, networking, Bash/Git/Python, RHEL admin	RHCSA
5-6	Docker, AWS, Terraform, CI/CD	AWS SAA, Terraform Associate (optional)
7-10	Kubernetes from fundamentals to security	KCNA, CKAD, CKA, KCSA, CKS
11-15	Ansible, observability, GitOps, service mesh, platform engineering	RHCE, PCA, OTCA, CGOA, CAPA, ICA, CCA, KCA, CBA, CNPA, CNPE
16-18	OpenShift, cloud security, AI/MLOps	EX280, EX316, AWS Security Specialty

The roadmap also tells you to start applying for your first job around month 8 to 10 instead of waiting until the end, and to keep studying while you work.

It is also direct about salary. Certifications alone will not get you to the top pay bands. Skill you can show live in an interview, production and on-call experience, public visibility and a specialisation will. The career ladder gives indicative India ranges per stage and makes clear they vary widely.

The labs

The labs/ folder has 19 numbered labs, from 00-setup to 18-ai-security. Lab 01 is Linux from absolute zero. Lab 15 is OpenShift and OpenShift Virtualization. Most labs come with real files, not just notes:

02-networking: netns-lab.sh builds container networking by hand with two network namespaces, a bridge, veth pairs and NAT, plus a subnet quiz in Python.
03-bash-git-python: a health-check script, a log analyser, and a small FastAPI service (lab-api) with pytest tests. Later labs reuse it for Docker, Kubernetes, Istio and observability.
07-kubernetes-core: a 3-node kind cluster config, a hardened lab-api.yaml, and a kubeadm install guide for CKA practice.
09-kubernetes-security: audit policy, encryption-at-rest config, read-only RBAC, Falco custom rules and Kyverno policies.
10-redhat-ansible: an Ansible role and a mock RHCSA exam.
18-ai-security: a prompt-injection lab that needs no real model. A naive bot leaks a secret, and your job is to write a safer version that blocks every listed attack.

To start, set up a local cluster with the lab’s own kind config:

git clone https://github.com/Willey2003/zero-to-senior-cloud-native-engineer.git
cd zero-to-senior-cloud-native-engineer/labs/07-kubernetes-core
kind create cluster --config manifests/kind-3node.yaml --name lab
kubectl get nodes -o wide

On-call scenarios

My years in VMware support taught me that you learn troubleshooting under pressure, not from slides. So scenarios/README.md lists 30 tickets across Linux, networking, Docker, cloud, Kubernetes, security, observability, GitOps, service mesh, OpenShift and AI. Examples include “Disk full but files not visible”, “Service has no endpoints”, “etcd data loss” and “mTLS broke legacy client”.

To be clear about what exists today: five of the Kubernetes tickets (14, 15, 16, 20 and 21) have a script that breaks a kind cluster on purpose. The other 25 are written ticket descriptions with the skills they test, and you set up the failure yourself.

kubectl apply -f labs/07-kubernetes-core/manifests/lab-api.yaml
./scenarios/k8s/break.sh 16   # "curl api.shop hangs but pods are Running"

Set a 30-minute timer, write down your hypothesis before each command, and finish with the post-mortem template in the same file.

Certifications and the capstone

CERTIFICATIONS.md lists 24 exams in a suggested order, from RHCSA through Kubestronaut and Golden Kubestronaut to RHCA, with a short reason for each position. It also includes exam-day tips for the hands-on exams. Formats and prices change, so check the official page before you book.

capstone/README.md describes “ShopPlatform”, an internal platform project: Terraform or kubeadm infrastructure, Argo CD app-of-apps, Kyverno and Falco, Istio canaries with automatic rollback, an SLO dashboard, a Backstage template and a RAG assistant with tenant isolation. It is a specification with eight milestones, not a finished build. It is meant to be the project you build yourself and walk interviewers through.

What is still in progress

This is an early, single-commit release, so here is where it stands:

The roadmap, certification guide, scenarios list and labs 00 to 18 are written.
The later labs (11 onwards) are shorter and point you to more self-directed work than the early ones.
The weekly journal/ has a template and a stub for week 1. It is meant to be filled in by whoever follows the path.
Only five scenarios are scripted so far.

I plan to add more scripted scenarios and expand the later labs. If you spot a mistake or want a specific scenario, open an issue.

Get it

Repo: github.com/Willey2003/zero-to-senior-cloud-native-engineer

Read ROADMAP.md once from start to finish, then start at labs/00-setup and go in number order. Commit after every lab, because your Git history becomes your proof of work. If it helps, star the repo and share it with someone just starting out.

Highly Available HAProxy and keepalived Load Balancers for OpenShift UPI: A Lab Repo

Gaganpreet Singh — Wed, 14 Oct 2026 14:00:00 +0000

If you install OpenShift with UPI, or agent-based on platform none, you provide the load balancer. Most guides show a single HAProxy VM and stop there. That VM then becomes the one thing that can take down your API, node ignition and every route at once.

I wanted a lab that treats the load balancer as a real HA component. That means two nodes, a floating VIP, proper health checks, and a way to prove failover works instead of assuming it does. The result is ocp-haproxy-ha-lab. It covers prep, load balancers, ingress placement, failover testing and monitoring, in that order.

The topology

The lab targets a cluster with 3 masters and 2 infra nodes. Two RHEL 9 VMs (lb1, lb2) run identical HAProxy configs and share one virtual IP through keepalived (VRRP). DNS for api, api-int and *.apps all points at the VIP, never at a single LB.

Port	Frontend	Backends	Health check
6443	Kubernetes API	3 masters (+ bootstrap at install)	HTTPS `GET /readyz`, expects 200
22623	Machine Config Server	3 masters (+ bootstrap)	TCP connect
443 / 80	Ingress	2 infra nodes	`GET /healthz/ready` on router port 1936
9000	Stats page	local	`/stats` (basic auth), `/healthz` for keepalived
8405	Prometheus metrics	local	`/metrics`

Every address in the repo is a placeholder on one lab subnet, and prep/inventory.env is the single list of them. Change it once and the prep scripts follow.

Design choices that matter

Most of the value is in the details, and each one is commented in the config:

/readyz over TLS for the API, not a TCP check. The kube-apiserver starts failing /readyz before it shuts down, so HAProxy drains a master that is rebooting gracefully. A plain TCP check keeps sending requests to a dying API server.
Router health on port 1936. A TCP check on 443 stays green when the node is up but the router pod is gone.
keepalived checks HAProxy, not just the process. check_haproxy.sh curls the local /healthz monitor URI, which catches a hung HAProxy as well as a dead one. With interval 2 and fall 2, the VIP moves in about 4 seconds.
Unicast VRRP for networks that filter multicast, and periodic gratuitous ARP so switches don’t keep a stale MAC for the VIP.
TLS passthrough with source affinity on ingress, and 1-hour tunnel timeouts so oc watches and websockets are not cut.
SELinux stays enforcing. The prep scripts label ports and scripts rather than disabling it.

What is in the repo

Directory	What it does
`prep/`	Inventory, BIND zone templates for api/api-int/*.apps with A and PTR records, firewalld rules, sysctl (`ip_nonlocal_bind`), SELinux settings, and DNS and host verification scripts
`haproxy/`	`haproxy.cfg`, keepalived configs for lb1 (MASTER) and lb2 (BACKUP), check and notify scripts, offline validation
`ingress/`	Infra MachineConfigPool, IngressController placement patch, setup, verify and rollback scripts, plus an install-time manifest
`failover/`	Failover test runbook and scripts, LB metrics, alert rules, and two Prometheus setups

Before copying anything to /etc, you can syntax-check every config from a workstation:

git clone https://github.com/Willey2003/ocp-haproxy-ha-lab.git
cd ocp-haproxy-ha-lab
./haproxy/validate.sh    # haproxy -c plus keepalived -t on both LB configs

The ingress setup script follows a safe order. The infra taint includes NoExecute, so the routers are moved onto the infra nodes and rolled out with tolerations before the taint is applied. Run it with DRY_RUN=true first to preview every command.

Proving failover: ten tests

From a support background, my view is simple: an HA design you haven’t broken on purpose is just a diagram. failover/RUNBOOK.md defines ten tests (T0 to T9). Each one breaks a single thing, measures what a client sees through the VIP, and puts it back:

T0 baseline, zero failed requests for 60 seconds
T1 to T3 HAProxy stopped, keepalived stopped, and a hard crash of the VIP holder
T4 VRRP blocked between the LBs to force split brain
T5 to T8 master and infra node loss, graceful and hard
T9 router pod deleted

scripts/failover-test.sh runs each test end to end. It prechecks the cluster and refuses to run on a degraded one, starts a probe pinned to the VIP with curl --resolve, breaks something, restores it, and grades the longest outage per endpoint against configurable limits (10 seconds by default). Results are saved as a raw probe.csv plus a summary.

cp failover/lab.env.example failover/lab.env   # edit for your lab
failover/scripts/failover-test.sh baseline
failover/scripts/failover-test.sh lb-haproxy-stop

The runbook lists the outage I expect from each setting, for example about 4 to 6 seconds for T1 and 0 seconds for a graceful master reboot. These are design targets. The results table in the runbook is a blank template for you to fill in from your own runs, and the repo does not publish measured numbers.

Monitoring and alerts

failover/monitoring/ adds node_exporter and a VRRP-state hook on each LB, then defines alerts including KeepalivedNoMaster, KeepalivedSplitBrain, HAProxyBackendDown, LBServiceNotActive and VIPEndpointDown. A “live MASTER” only counts while keepalived.service is actually active, so a stale state file from a killed keepalived cannot hide a missing VIP holder.

There are two Prometheus setups on purpose. One uses OpenShift user workload monitoring, so LB alerts show up in the console. But it sits behind the VIP it is watching. The other runs Prometheus, Alertmanager and blackbox_exporter in podman on a bastion, probing from outside, so it keeps alerting when the cluster is unreachable. For a lab, one is enough. For production, the repo recommends both.

Limits to know

There are no dedicated worker nodes, so with infra nodes tainted, application pods land on the masters. Fine for a lab. For real workloads, add workers or set TAINT=false.
Replace every placeholder before deploying, including the VRRP password and stats credentials, which are marked CHANGE ME.
It assumes OpenShift 4.x with a user-managed load balancer and DNS you control.

Get it

Repo: github.com/Willey2003/ocp-haproxy-ha-lab

Start with prep/inventory.env, then work through prep/, haproxy/, ingress/ and failover/ in order. If you run the failover tests in your own lab, I would like to see your numbers. Open an issue with your results table.

VMware to OpenShift Virtualization Migration SOP (Free)

Gaganpreet Singh — Tue, 13 Oct 2026 13:10:00 +0000

Every VMware shop I talk to is asking the same question this year: where do our VMs go next? For many of them, the answer is Red Hat OpenShift Virtualization. It runs virtual machines on Kubernetes using KubeVirt, and Red Hat ships a tool, the Migration Toolkit for Virtualization (MTV), that moves them across from vSphere.

The documentation is good, but it is written for people who already think in Kubernetes. A VMware engineer opening it for the first time meets Pods, PVCs, NADs and operators all at once. So I built a lab, migrated a VM end to end, and wrote down everything, including the parts that went wrong.

The concepts translate

Most of what you know still applies. It just has a new name.

vSphere	OpenShift Virtualization
VMDK on a datastore	PersistentVolumeClaim (PVC) on a StorageClass
Port group	NetworkAttachmentDefinition (NAD)
vSwitch / VDS	Linux bridge built by NMState
vMotion	Live migration (needs RWX storage)
VMware Tools	QEMU guest agent

Once that table clicks, the rest is detail.

What MTV actually does

When you press Start on a migration plan, MTV runs a short pipeline. It validates the plan, creates the target disks, converts the guest with virt-v2v (which installs virtio drivers and the guest agent), copies the blocks, and creates a VirtualMachine with the same CPU, memory and MAC address. A cold migration only reads from the source, so the original VM stays where it is as your rollback.

What I learned the hard way

The read path decides your speed. Without VDDK, MTV reads disks over HTTPS through vCenter on port 443. My 10 GiB test VM took 1 h 24 min, most of it disk transfer at about 2.8 MB/s. With VDDK, it reads from the ESXi hosts over NBD on port 902, which is much faster. Build the VDDK image before your first real wave.
DNS is everything. The agent-based installer, the console and MTV's connection to vCenter all fail in confusing ways when a record is missing.
Storage needs RWX for live migration. If your StorageClass is ReadWriteOnce only, VMs work but cannot move between nodes.
Check what was kept. In my test the VM kept its MAC address, disk serial and BIOS serial, so licensing and DHCP reservations survived. Verify this on your own VMs before you promise it to anyone.

The commands I used most

oc get providers,plan,migration -n openshift-mtv     # is MTV connected, what is running
oc get migration -n openshift-mtv -w                  # watch a run move through the pipeline
oc get vm,vmi -n mtv-test                             # the migrated VM, defined and running
oc get dv,pvc -n mtv-test                             # disk import progress and the claim
virtctl console ubuntu-noble-24.04-cloudimg -n mtv-test # serial console into the guest

The full 53-command cheat sheet is in the repo as a PDF.

What's in the manual

The Lab Manual: Migration Edition has ten chapters, from Linux and networking basics through the agent-based install, storage, VM networking and MTV, to an end-to-end migration and a troubleshooting chapter. Every command block shows real output from the lab, with hostnames and IPs replaced by placeholders. There is also a 53-command cheat sheet and an exam map for EX316.

It is MIT-licensed and free: github.com/Willey2003/openshift-vmware-migration-sop

If you are planning a migration, I'd like to hear what you are running into. Corrections and production lessons are welcome as issues or pull requests.

From Single-Master Lab to Production-Style Kubernetes: HA, Backups, Active Directory and Gateway API

Gaganpreet Singh — Sun, 11 Oct 2026 14:00:00 +0000

Our on-prem lab cluster was a typical kubeadm build on VMware: one control-plane node, no backups, local accounts on every host, and clocks drifting by up to six minutes. It worked until it didn’t. Over a few weeks I rebuilt it into a production-style platform that a whole team can use safely, on plain VMware VMs with no cloud load balancers.

Every step is scripted and published here: github.com/Willey2003/k8s-ha-ad-gateway-lab.

What was built

Area	Outcome
High availability	3 control-plane nodes behind a floating API VIP (kube-vip, ARP mode). The running cluster was migrated from a single-node endpoint to the VIP, with automatic rollback. Measured failover: about 1 second.
External etcd	3-member TLS etcd cluster with weekly snapshots from the bastion.
Backups and DR	Velero (including volume data) to a self-hosted S3 store, plus etcd snapshots, on a weekly systemd timer. A scripted restore drill proves the backups actually restore.
Time and DNS	All hosts sync time from the domain controllers; A/PTR records and an `*.apps` wildcard.
Active Directory for Linux	Every node joined to AD (realmd/SSSD), AD-group-based SSH and sudo, Kerberos SSO between nodes.
Active Directory for kubectl	LDAPS on the domain controllers, Dex as the OIDC provider, and the API server’s structured authentication config (CEL claim mappings). RBAC comes from AD groups.
Self-service	Every AD user can read the cluster and deploy into a quota-limited `playground` namespace (Pod Security baseline, no NodePorts or LoadBalancers).
Ingress	Ingress-NGINX is retired, so: MetalLB (L2) plus HAProxy Unified Gateway with Gateway API. One shared Gateway, wildcard TLS, and users publish apps with a single `HTTPRoute`.

Stack: Kubernetes 1.32 (kubeadm), CRI-O, Calico, kube-vip, etcd 3.5, Velero 1.18, Dex 2.45, MetalLB 0.16, HAProxy Unified Gateway 1.0, Gateway API 1.3, Active Directory (SSSD, Kerberos, LDAPS), chrony.

Access model

Role	Who	SSH	Kubernetes
Lab user	Every enabled AD account	Bastion only, no sudo	`view` cluster-wide (no Secrets) and `edit` in `playground`
Lab administrator	AD group `K8s-Admins`	All hosts with sudo	`cluster-admin`
Emergency	Local accounts	All hosts	`admin.conf` on the control plane

For a user it is simple: SSH to the bastion with an AD account, and kubectl is already configured. The first kubectl command asks for the AD password once, and the token refreshes for up to seven days.

Run order

01-access: a least-privilege deployer ServiceAccount and kubeconfig.
02-ha-control-plane: prepare the new managers, migrate to the VIP in three steps, test failover, roll back if needed.
03-time-sync: chrony against the domain controllers.
04-backups: backup disk, S3 gateway, Velero, weekly timer and the restore drill.
05-dns: PowerShell for A and PTR records, with conflict detection and -WhatIf.
06-active-directory: DNS through the DCs, realm join, Kerberos SSH, then Dex, the API-server auth config and RBAC for kubectl.
07-gateway: MetalLB, HAProxy Unified Gateway, the shared Gateway and a demo app with route RBAC.

Lessons learned: the things that actually bit

Check the router’s subnet, not just the hosts’. Every VM used a /24, but the gateway routed only a /27. Load-balancer IPs above .31 worked from inside the subnet and silently timed out from everywhere else.
Validate API-server auth config before a rolling restart. claims.groups.map(...) fails CEL type-checking; it needs dyn(claims.groups). A local kube-apiserver binary reproduces the error in seconds, before two managers crash-loop.
A health check must prove the new container is healthy. Waiting for /readyz right after editing a static pod manifest can hit the old process. Wait for a new container ID, then require sustained health.
HAProxy sizes its memory budget from the container memory limit. Set the limit too low and TLS workers refuse to start with the default maxconn.
Helm post-install CRD jobs can race the controller. Restart the controller once after the first install.
Dex expands $VAR in config values. A $ in the bind password silently broke LDAP; set DEX_EXPAND_ENV=false.
realm join with a short hostname registers only short SPNs, so Kerberos SSO fails with “Server not found in Kerberos database”. Fix /etc/hosts and register HOST/<fqdn>.
Never make a repair tool depend on the thing it repairs. The Dex password rotation writes through the control plane’s local admin config, not through Dex-based kubectl.
Prove the restore, not the backup. My first restore drill “passed” for the wrong reason: the proof string lived in the pod spec. Writing it only into the volume made the test honest.

Get it

All IPs, hostnames and the domain in the repository are example values, and no secrets are stored: keys and passwords are generated at run time. Read the scripts before you run them, and run them in a lab first: github.com/Willey2003/k8s-ha-ad-gateway-lab.