DEV Community: Gaganpreet Singh

Would your Kubernetes cluster survive losing one ESXi host? I built a read-only tool to find out

Gaganpreet Singh — Sun, 11 Oct 2026 01:09:34 +0000

Kubernetes sees nodes. vSphere sees VMs on hosts. Neither tells you when DRS has put two of your three etcd VMs on the same ESXi host. From then on, one host failure takes your control plane down, and every dashboard still says green.

I spent nearly seven years in VMware support, and I couldn't find a tool that joins those two views and simulates host failures against Kubernetes semantics. So I built kube-hostfail, then wrapped it in a working observability and service-mesh stack so you can see it in action.

Repo: https://github.com/Willey2003/k8s-vsphere-ha-cockpit (Apache-2.0)

What it checks

kube-hostfail is read-only. It joins node and etcd-member data from the Kubernetes API with VM and host data from vCenter (pyvmomi), then simulates every single-host failure, and optionally every two-host failure:

etcd quorum: external etcd (from --etcd-servers on kube-apiserver) or stacked etcd pods, matched to VMs by name, guest hostname or guest IP
API availability: control-plane nodes left, quorum, and which host holds the kube-vip VIP lease
Workloads: Deployments and StatefulSets losing every ready replica; PodDisruptionBudget violations
Capacity: can the surviving workers absorb the evicted pods' requests?
DRS: is each etcd and control-plane VM covered by an enabled VM-VM anti-affinity rule?

Verdicts are SAFE, DEGRADED, APP OUTAGE and CLUSTER DOWN. For missing DRS rules it prints the govc or PowerCLI command and never applies it.

Try it in 30 seconds, no vSphere needed

git clone https://github.com/Willey2003/k8s-vsphere-ha-cockpit.git
cd k8s-vsphere-ha-cockpit/kube-hostfail && pip install .
kube-hostfail report --snapshot examples/lab-drifted.json --depth 2

The bundled snapshot is a lab whose etcd members have drifted onto one host:

[DOWN] esx002.corp.example   CLUSTER DOWN  etcd 1/3  apiservers 1
       - etcd loses quorum: 1/3 members left, 2 needed (members on esx002: etcd-1, etcd-2)
DRS ANTI-AFFINITY
  etcd   MISSING   fix (review first): govc cluster.rule.create -cluster 'LAB-CL01' -name k8s-lab-etcd-anti-affinity -enable -anti-affinity etcd-1 etcd-2 etcd-3
RESULT: NOT highly available - losing esx002.corp.example takes the cluster down.

kube-hostfail check exits with code 2 when any single host is a single point of failure, so you can put it in a pipeline or a cron job.

The cockpit around it

One script per stage, all idempotent:

scripts/00-preflight.sh      # read-only: what exists, what will be created, what could block you
scripts/10-observability.sh  # kube-prometheus-stack tuned for kubeadm
scripts/20-mesh.sh           # Istio, Jaeger, Kiali, Gateway
scripts/30-apps.sh           # kube-hostfail, launcher, demo shop, routes
scripts/40-verify.sh

Prometheus + Grafana with no permanently red targets, an emptyDir fallback when there's no StorageClass, opt-in scraping of external etcd, and a kube-hostfail dashboard with alert rules.
Istio + Kiali + Jaeger with a three-service demo shop (one slow, flaky pricing version) and a load generator, so the graph and the traces are never empty.
One Gateway API Gateway, path-routed: /grafana, /kiali, /jaeger, /hostfail, /shop. A single address or a single SSH tunnel reaches everything.
An app launcher that builds its tiles from your HTTPRoutes, with a live health dot per tile. Publish an app, a tile appears.

What a real cluster taught me

Unit tests and a vCenter simulator passed, so I installed it on a real kubeadm 1.32 cluster (3 managers, 4 workers, CRI-O, Calico, MetalLB, external etcd) that already ran another Prometheus stack. The first run broke in three places:

Port clash. The existing stack's node-exporter used host port 9100 on the same nodes, so four of my pods stayed Pending ("didn't have free ports"). Mine now uses 9101.
Grafana scrape. Serving Grafana from /grafana means /metrics redirects to root_url, so the target went DOWN. The ServiceMonitor now scrapes /grafana/metrics.
Uppercase GitHub owner. My CI built the image tag from github.repository_owner, and container image names must be lowercase.

After fixing those, the Gateway came up with a MetalLB address, Kiali reported Prometheus, Grafana and Jaeger reachable, and Jaeger held traces for all three demo services. Your existing cluster isn't modified: everything lands in new namespaces, and sidecars are injected only into the demo namespace.

I also left one honest finding in the README: two managers block the node-exporter port at the host firewall, so those two targets show DOWN. That's a host-level change I wasn't going to make on a cluster I don't own.

What is and isn't verified

Verified: the simulation logic (20 unit tests), an end-to-end test against govmomi's vcsim, chart and manifest validation in CI, and a full install on a real cluster. Not yet verified: live mode against a production vCenter. The simulator isn't byte-for-byte vCenter, so I'd welcome reports from real environments.

Not modelled yet: datastore/PV placement, network partitions, DaemonSets and vSphere HA admission control.

Try it and tell me what breaks

If you run Kubernetes on vSphere, run kube-hostfail report against a read-only vCenter account and see what it says about your etcd members. Issues and PRs are welcome.

Repo: https://github.com/Willey2003/k8s-vsphere-ha-cockpit

Private AI SOC Triage Lab: Can a Small Local LLM Triage Alerts Safely?

Gaganpreet Singh — Sat, 10 Oct 2026 14:00:00 +0000

Can a small, local LLM take tier-1 alert triage off a SOC team without missing attacks, and without being talked out of an escalation by the attacker? I built a lab to measure it instead of guessing. The code is on GitHub: github.com/Willey2003/soc-triage-lab.

Three designs, same alerts

The same 120 labelled alerts (60 malicious, 60 benign look-alikes) go through three designs:

Rules: a classic SIEM threshold.
LLM only: the model decides everything.
Hybrid: rules at both ends, the model only in the grey zone, with a prompt-injection guard and fail-safe defaults.

Everything runs on one VM with no GPU, using qwen2.5:1.5b through Ollama, and no alert data leaves the box.

feeder -> API -> enrich -> rule score -> hybrid gate -> (grey zone) -> Ollama qwen2.5:1.5b
                                |                            |
                           SQLite cases <--------------------+    Prometheus <- /metrics -> Grafana

Results

Metric	Rules	LLM only	Hybrid
Accuracy	67%	48%	81%
Recall (escalate)	100%	88%	100%
Missed attacks	0	7	0
False positives	40	55	23
Escalated to an analyst	100	108	83
Prompt-injection success	0%	10%	0%
Model calls	0	120	43

The hybrid design cut the analyst queue by 17% and false positives by 43%, and missed no attacks. Letting the model decide everything was the worst design: it missed 7 attacks, closed foreign logins from new devices, and one injected alert talked it into closing.

The guardrails that mattered

Rule floor. Any alert scoring 70 or above is escalated before the model sees it. This handled 54 of 120 alerts, including an injection written to evade the pattern list.
Close veto. The model may close an alert only when a trusted CMDB or allowlist fact explains it (VPN egress, signed internal tool, approved scanner). Without it, the hybrid closed 3 malicious foreign logins.
Injection guard. Alert text is untrusted: command lines, user agents and DNS labels are attacker-controlled. The guard strips instruction-like text and the prompt marks the alert as evidence only.
Fail-safe on invalid output. 12% of model answers still failed validation, and every one was escalated rather than guessed.

What the model is actually good for

Reading business context the rule does not have. Its wins were VPN logins (10/10 correctly closed in hybrid mode) and corporate-network password typos. It still did not trust an allowlist for telemetry DNS or the automation-host flag for SCCM PowerShell. A 1.5B model often restates the rule instead of weighing the context.

Latency was about 10 seconds per alert on CPU (p95 18 s). That is fine for a steady queue and not fine for a burst, which is one more reason to keep the model in the grey zone only.

Run it yourself

The lab runs rootless with podman (tested on RHEL 9 with SELinux enforcing), and there is a Helm chart for Kubernetes with a NetworkPolicy so only the API can reach the model.

git clone https://github.com/Willey2003/soc-triage-lab && cd soc-triage-lab
scripts/lab.sh up        # build image, start the pod, pull the model
scripts/lab.sh eval 120  # rules vs llm vs hybrid on 120 labelled alerts
scripts/lab.sh test      # unit tests
scripts/lab.sh down

Limits

The alerts are synthetic with clean labels, and the indicators use RFC 5737 documentation IP ranges and invented hashes and domains. It is one model, one seed, temperature 0. Larger models are the obvious next run. Treat the model’s output as a suggestion that a human reviews, and keep high-risk alerts on deterministic rules.

From One VM to a Whole Application: Planning VMware to OpenShift Migration Waves

Gaganpreet Singh — Fri, 09 Oct 2026 13:23:41 +0000

After I shared my VMware to OpenShift Virtualization lab, a consultant asked me a question I keep thinking about: “Is this about moving a VM, or about moving application stacks? It’s a lab, so what’s the overall context?”

It’s a fair challenge. My lab proves the mechanics on one VM: networking, storage, the Migration Toolkit for Virtualization (MTV) mappings and the cutover. But nobody migrates “a VM”. You migrate a payroll system, an order platform, a reporting stack, each made of several VMs that talk to each other, to databases, to load balancers and to things nobody has written down.

In my years in VMware support, the escalations that hurt most were rarely “the tool failed”. They were “we moved half of an application and the other half stopped working”. So this post is about the part the lab doesn’t show: how to go from one migrated VM to a migration programme, organised by application, in waves, with a way back.

One VM vs. a programme

	One-VM lab	A real migration programme
Unit of work	A VM	An application (all of its tiers)
Main risk	Wrong mapping, ports, storage class	Broken dependencies, missed change windows, no rollback
Downtime	Doesn’t matter	Agreed per application with its owner
Network	One test VLAN	Keep IPs or re-IP, firewall rules, load balancers, DNS
Success	The VM boots and pings	The business process works end to end, and monitoring, backup and CMDB know about it

The tooling is the same. What changes is the planning around it.

Step 1: Inventory and dependencies

Start with facts, not spreadsheets people remember.

Inventory: export every VM with its CPU, memory, disks, datastore, port groups, guest OS, snapshots and VMware Tools state. MTV’s own inventory, RVTools or a PowerCLI export all work.
Dependencies: who talks to whom. Use what you already have: network flow data from your monitoring or firewall logs, application owner interviews, and the CMDB as a starting point (not as truth). On a few critical VMs, ss -tunp (Linux) or netstat -ano (Windows) over a business day is surprisingly revealing.
Output: a list of application groups. Each group is the set of VMs that must move together, plus its external dependencies (shared databases, AD, file shares, licence servers, SMTP relays).

A rule I use: if two VMs exchange traffic every few seconds, they belong in the same wave. Splitting them puts latency, and sometimes a firewall, between them during the migration.

Step 2: Decide what happens to each VM

Not every VM should go through MTV. Give each one a disposition:

Disposition	When
Rehost with MTV	Supported guest OS, standard disks and NICs. Most VMs.
Rehost later	Needs fixing first: unsupported OS, BitLocker, Windows Fast Startup, old snapshots, CBT off
Special handling	Shared-disk clusters, RDMs, very large or very busy databases, appliances with vendor support rules
Replatform	Stateless apps you plan to containerise anyway; do it once, not twice
Retire	Nobody owns it, nobody uses it. Every migration finds these.
Stay on vSphere (for now)	Licence-bound or vendor-certified only on VMware

The readiness checks per VM are the boring part that saves weekends: a supported guest OS, VMware Tools running, no leftover snapshots, CBT enabled for warm migration, BitLocker suspended, Windows Fast Startup off, and firmware (BIOS or UEFI) recorded so the target VM matches.

Step 3: Network: keep the IPs or re-IP?

This decision drives everything else, so make it early.

Keep the IPs (recommended for the first waves). Present the same VLANs to OpenShift with NMState bridges and NetworkAttachmentDefinitions, so a migrated VM lands on the same subnet. MTV keeps the NIC MAC addresses, so DHCP reservations and MAC-bound licences keep working. Firewall rules, load-balancer pools and DNS don’t change.
Re-IP. Only when you’re redesigning the network anyway. Every re-IP touches DNS, firewall rules, load balancers, application configs and sometimes certificates, so it multiplies the testing.

The most common first-wave surprise I’ve seen in labs and in support is routing, not MTV. A VLAN that “exists” on OpenShift but isn’t trunked to every node, or a subnet mask that doesn’t match what the router actually routes. Test the network path with a throwaway VM on every target VLAN before wave 1.

Step 4: Build the waves

The SOP I published uses this structure:

Wave	What goes in	Why
0	Test VMs: one Linux and one Windows, one per storage protocol	Prove each protocol, network and the runbook
1	Low-risk, stateless VMs (web servers, jump hosts)	Build confidence and real timing data
2..n	Application groups, all tiers of one application together	Keep dependencies intact
Last	Databases, shared-disk clusters, very large VMs	Most planning, warm migration, longest windows

Three rules make waves predictable:

Group by application, not by datastore. Datastores are how vSphere stores things; applications are how the business notices outages.
Keep each MTV plan to roughly 10–20 VMs. When something fails, you want to know which VM and why, not search through 200.
Size the window from measured numbers. After wave 0 and 1 you know your real throughput. Then: window ≥ (total GB to copy ÷ measured GB per hour) + conversion + validation + rollback buffer

For example, if wave 1 showed 300 GB per hour and an application has 1.2 TB of disks, a cold copy alone is about 4 hours, before conversion, testing and a rollback buffer. That’s usually the moment the application owner chooses warm migration.

Step 5: Cold or warm, per tier

	Cold	Warm
Source VM during copy	Powered off	Keeps running
Downtime	Full copy + conversion	Final delta + conversion + boot
Needs	Nothing extra	CBT on, VDDK strongly recommended
Good for	Small VMs, test VMs, tiers that can be off for hours	Production tiers with short change windows

In one application you’ll often mix them. The web tier can go cold on a Saturday morning; the database goes warm, with precopy running for days and a cutover scheduled in the change window. With warm migration, the downtime you agree with the business is the final delta, not the full disk size.

Step 6: Runbook and rollback

Every wave gets the same runbook. A shortened version of the one in the SOP:

T-5 days: change approved; application owner and rollback criteria agreed. Readiness checks on every VM.
T-3 days: backups of the source VMs verified (restored, not just “job succeeded”).
T-2 days: MTV plan created and Ready, all critical concerns cleared; warm precopy started.
T-1 hour: cluster and storage health checks, capacity checked.
T-0: application owner stops the service if needed; cutover.
T+: watch the pipeline, validate each VM, run the application’s smoke tests, then go / no-go.
T+1 day: update the CMDB, monitoring and backup jobs for the new VMs.
T+14 days: hypercare ends; only now delete the source VMs.

Rollback is the reason this works. MTV copies the disks and leaves the source VM in vCenter. If the go/no-go fails, you power off the new VM on OpenShift, power the source VM back on, and the application is where it was (same IPs if you kept them). Write the rollback steps down per application, and decide in advance who says no-go and by when.

The things that get forgotten

These aren’t migration steps, but each one has caused a “successful” migration to fail a week later:

Backup: the vSphere backup jobs don’t follow the VM. Set up backup for OpenShift Virtualization VMs (OADP or your vendor’s tool) and test a restore before the source VMs are deleted.
Monitoring and alerting: agents and dashboards that keyed on vCenter objects.
Licensing: check guest OS and application licensing on the new platform with your vendors before wave 1, not after.
Operations: your team now runs VMs as Kubernetes objects. Live migration, node maintenance and upgrades work differently, so train people before the first production wave, not during it.
Storage for production: the lab used NFS. Production needs a CSI driver certified for OpenShift Virtualization, with RWX volumes so VMs can live-migrate during node maintenance.

Where my lab stands, honestly

The lab validated the mechanics on a single Ubuntu VM with cold migration. The wave structure, runbook and rollback above come from the production part of my SOP and from what I saw in VMware support escalations. They’re not yet proven on a multi-tier application in my lab.

That’s the next build: a three-tier application (web, app, database) moved as one wave, with warm migration on the database, a timed cutover and a deliberate rollback test. I’ll publish the results, including what goes wrong.

Your checklist

Inventory every VM and map dependencies into application groups.
Give each VM a disposition; fix the “rehost later” ones early.
Decide keep-IP vs re-IP; test every target VLAN with a throwaway VM.
Wave 0 with one VM per OS and storage protocol; measure throughput.
Build waves by application, 10–20 VMs per plan, windows sized from real numbers.
Choose cold or warm per tier; enable CBT and VDDK for warm.
Run the same runbook every wave, with a written rollback and a named go/no-go owner.
Move backup, monitoring, CMDB and licensing with the VMs.

The 10-part lab series, from Linux basics to a migrated VM: start with Part 1, or jump to Part 8: MTV and Part 10: Troubleshooting and going to production.
The full SOP (including FC, iSCSI and NFS storage, wave planning and the runbook): github.com/Willey2003/openshift-vmware-migration-sop

If you’re planning a VMware exit, I’d like to hear how you’re grouping applications into waves, and what surprised you. Leave a comment here or message me on LinkedIn.