DEV Community: Muhammad Adel

Part 6: Day-2 Operations, Live Migration & Production Troubleshooting

Muhammad Adel — Sun, 08 Nov 2026 23:09:46 +0000

Recent licensing shifts across enterprise virtualization—most notably VMware by Broadcom—have accelerated a massive industry migration: moving enterprise virtual machines to Red Hat OpenShift Virtualization.

Rebuilding hundreds or thousands of mission-critical VMs from scratch is simply not feasible. Enterprises need an automated, scalable migration pipeline that can:

Discover existing VMware vSphere clusters, datastores, and port groups.
Convert proprietary virtual disks (.vmdk) into cloud-native persistent storage volumes.
Inject native VirtIO drivers for KVM performance.
Cut downtime to minutes using continuous background pre-copying.

This is the exact purpose of MTV (Migration Toolkit for Virtualization), powered by the open-source Forklift project.

In this sixth and final installment of our OpenShift Virtualization (CNV) Zero to Hero series, we explore how to perform large-scale VM migrations from VMware vSphere to OpenShift with near-zero downtime.

🏗️ Architecture: Migration Toolkit for Virtualization (MTV)

The MTV migration engine orchestrates data movement between legacy hypervisors and OpenShift:

The MTV architecture centers on five core declarative primitives:

Providers: Define source (e.g., VMware vCenter) and target (OpenShift Virtualization) endpoints.
Network Map: Maps source VMware Port Groups / VLANs to target OpenShift Pod Networks or Multus CNI bridges.
Storage Map: Maps source VMware Datastores (VMFS/vSAN) to target OpenShift StorageClasses (ODF Ceph or LVM).
Migration Plan: Specifies the inventory of VMs to migrate, associated mappings, and cutover windows.
Conversion Pod (virt-v2v): A transient pod that mounts the incoming disk, installs VirtIO drivers, and prepares the OS to boot on KVM.

⚡ Cold Migration vs. Warm Migration

Choosing the right migration strategy directly impacts application downtime:

🔹 Cold Migration (Direct Cutover)

How it works: The source VM on VMware is powered off immediately. MTV copies 100% of the virtual disk across the network to OpenShift, runs virt-v2v conversion, and boots the new VM.
Downtime: Proportional to the size of the disk (e.g., a 500GB disk might require 1 to 2 hours of downtime).
Best for: Non-production environments, batch workloads, or systems that can tolerate scheduled maintenance windows.

🔹 Warm Migration (Pre-Copy with CBT) — The Enterprise Gold Standard

How it works: The source VM remains online and actively serving users on VMware! MTV leverages VMware CBT (Changed Block Tracking) to take periodic snapshots, pre-copying 95%+ of the disk data in the background over days or weeks.
The Cutover: When you are ready for the maintenance window, you trigger the final cutover. The source VM is powered down, only the tiny incremental delta of dirty blocks is transferred (taking just 2 to 5 minutes!), and the VM boots up in OpenShift.
Best for: Production databases, customer-facing applications, and business-critical systems.

🛠️ Step-by-Step: Migrating from VMware vCenter

1. Install the MTV Operator

Install the Migration Toolkit for Virtualization Operator from OperatorHub inside the openshift-mtv namespace.

2. Connect the VMware vCenter Provider

Store vCenter credentials in a secret, then declare the vSphere Provider:

apiVersion: v1
kind: Secret
metadata:
  name: vcenter-credentials
  namespace: openshift-mtv
type: Opaque
stringData:
  user: "administrator@vsphere.local"
  password: "YourSuperSecretPassword"
  insecureSkipVerify: "true"

---
apiVersion: forklift.konveyor.io/v1beta1
kind: Provider
metadata:
  name: vmware-source
  namespace: openshift-mtv
spec:
  type: vsphere
  url: "https://vcenter.corp.local/sdk"
  secret:
    name: vcenter-credentials
    namespace: openshift-mtv

3. Create Network and Storage Mappings

These mappings tell OpenShift how to translate VMware infrastructure into Kubernetes equivalents:

# 1. Network Mapping: VMware Port Group -> Multus Bridge
apiVersion: forklift.konveyor.io/v1beta1
kind: NetworkMap
metadata:
  name: vcenter-network-map
  namespace: openshift-mtv
spec:
  map:
    - destination:
        type: multus
        name: vlan100-net
        namespace: vm-workloads
      source:
        id: "network-12"  # VMware Port Group ID (or name: "VM Network")
  provider:
    source:
      name: vmware-source
    destination:
      name: host

---
# 2. Storage Mapping: VMware Datastore -> OpenShift StorageClass
apiVersion: forklift.konveyor.io/v1beta1
kind: StorageMap
metadata:
  name: vcenter-storage-map
  namespace: openshift-mtv
spec:
  map:
    - destination:
        storageClass: odf-ceph-rbd
        accessMode: ReadWriteMany
        volumeMode: Block
      source:
        id: "datastore-34"  # VMware Datastore ID (or name: "vsanDatastore")
  provider:
    source:
      name: vmware-source
    destination:
      name: host

4. Declare the Migration Plan (Warm Migration)

Now we group the target VMs into a migration plan and enable warm: true:

apiVersion: forklift.konveyor.io/v1beta1
kind: Plan
metadata:
  name: production-db-migration-plan
  namespace: openshift-mtv
spec:
  warm: true  # <--- Enables background pre-copy with CBT
  provider:
    source:
      name: vmware-source
    destination:
      name: host
  map:
    network:
      name: vcenter-network-map
    storage:
      name: vcenter-storage-map
  targetNamespace: vm-workloads
  vms:
    - name: "prod-oracle-db-01"
    - name: "prod-app-server-02"

5. Execute the Migration & Final Cutover

Initiate the background pre-copy:

apiVersion: forklift.konveyor.io/v1beta1
kind: Migration
metadata:
  name: run-db-migration
  namespace: openshift-mtv
spec:
  plan:
    name: production-db-migration-plan

MTV will begin streaming snapshots in the background while the VMs stay active on VMware.

When your maintenance window arrives, apply the final cutover timestamp:

oc patch plan production-db-migration-plan -n openshift-mtv --type='merge' \
  -p '{"spec":{"cutover":"2026-11-10T22:00:00Z"}}'

MTV powers off the VMware VMs, transfers the final delta, injects VirtIO drivers via virt-v2v, and boots the virtual machines in OpenShift!

⚠️ Battle-Tested Gotchas & Troubleshooting

Common Challenge	Root Cause	Proven Fix
Warm migration fails with CBT error	Changed Block Tracking is disabled on the source VM.	Enable CBT in VMware (`ctkEnabled = TRUE`) or power-cycle the VM once in vSphere.
Windows VM Blue Screens (BSOD) on boot	Missing KVM VirtIO storage/network drivers in Windows.	MTV automatically injects the `virtio-win` driver ISO. Ensure the `virtio-win` container is available on OpenShift.
Slow transfer speeds (throttling)	Data is passing through default ingress or routed networks.	Designate a dedicated, high-speed migration network in the `HyperConverged` CR configuration.
MAC Address changes after import	VM network adapter regenerated during import.	Check "Preserve static MAC addresses" in MTV so KubeMacPool retains the original VMware MACs.

🏁 Series Grand Finale: You Are Now a CNV Hero!

Over the course of this 6-part journey, we built an end-to-end enterprise virtualization practice on Kubernetes:

Part 1: Decoded the KubeVirt engine (virt-operator, virt-api, and virt-launcher).
Part 2: Mastered Block vs. Filesystem storage and navigated the TopoLVM extent trap.
Part 3: Built automated golden OS pipelines with containerDisk and DataImportCron.
Part 4: Connected VMs to physical enterprise VLANs using Multus and KubeMacPool.
Part 5: Built rock-solid DR strategies with OADP, NooBaa, and fsfreeze hooks.
Part 6: Automated large-scale hypervisor migrations from VMware to OpenShift with MTV.

OpenShift Virtualization proves that containers and virtual machines are no longer rival architectures—they are collaborative primitives running under one unified, cloud-native control plane.

Thank you for following along with this series! 🚀

💬 Are you actively planning or currently executing a VMware-to-OpenShift migration? What are your biggest hurdles? Share your story in the comments below!

Part 5: Enterprise Backup & Disaster Recovery with OADP & NooBaa

Muhammad Adel — Sun, 08 Nov 2026 23:05:39 +0000

Backing up stateless microservices in Kubernetes is relatively straightforward: export your GitOps manifests, and redeploy.

Virtual Machines, however, are massive stateful engines.

If you take a naive snapshot of a running virtual machine while an active database is processing write transactions in RAM, your restored VM will likely boot into a corrupted filesystem, broken journal logs, or lost transactions (the dreaded "Crash-Consistent" state).

To achieve true enterprise-grade Application Consistency, we must coordinate three distinct layers:

The guest operating system's internal filesystem memory buffers.
The hypervisor virtualization engine (KubeVirt).
The underlying storage array CSI snapshot controller.

In this fifth part of our series, we construct an end-to-end Enterprise Disaster Recovery strategy using OADP (OpenShift API for Data Protection / Velero) and local NooBaa (S3 Object Storage).

🏗️ Architecture: The Application-Consistent Backup Workflow

The coordinated backup flow operates in five discrete phases:

Pre-Backup Hook: OADP communicates through KubeVirt to the qemu-guest-agent inside the VM, executing fsfreeze to flush all cached data from RAM to disk and freeze write operations.
CSI Volume Snapshots: TopoLVM or ODF instantly captures point-in-time storage snapshots of all associated disks (PVCs).
Post-Backup Hook: OADP signals KubeVirt to issue fsthaw, immediately unfreezing guest OS writes with zero perceptible application disruption.
Metadata & Archive Streaming: VM definitions, cloud-init configurations, secrets, and metadata are bundled and encrypted into an S3-compatible object bucket.
Cross-Namespace Disaster Recovery: The VM can be restored seamlessly to another namespace or an entirely separate recovery cluster.

🛠️ Step 1: Provision Local S3 Storage via NooBaa

Instead of routing gigabytes of backup data across public cloud networks, we can use NooBaa (Multicloud Object Gateway) to create a high-speed, local S3 bucket directly inside the cluster:

cat << 'EOF' | oc apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: oadp-backup-bucket
  namespace: openshift-adp
spec:
  generateBucketName: oadp-backup
  storageClassName: openshift-storage.noobaa.io
EOF

Extract Bucket Credentials

BUCKET_NAME=$(oc get obc oadp-backup-bucket -n openshift-adp -o jsonpath='{.spec.bucketName}')
ACCESS_KEY=$(oc get secret oadp-backup-bucket -n openshift-adp -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
SECRET_KEY=$(oc get secret oadp-backup-bucket -n openshift-adp -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)

cat << EOF > /tmp/credentials-velero
[default]
aws_access_key_id = ${ACCESS_KEY}
aws_secret_access_key = ${SECRET_KEY}
EOF

oc create secret generic cloud-credentials -n openshift-adp \
  --from-file=cloud=/tmp/credentials-velero \
  --dry-run=client -o yaml | oc apply -f -
rm -f /tmp/credentials-velero

⚙️ Step 2: Configure the DataProtectionApplication (DPA)

To support OpenShift Virtualization, your OADP deployment must include 4 mandatory plugins:

openshift: Handles OpenShift-specific resources (Routes, ImageStreams).
aws: Interfaces with NooBaa's S3-compatible API.
kubevirt: Orchestrates VM states and QEMU guest agent hooks.
csi: Interacts with storage drivers for instantaneous volume snapshots.

cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: oadp-dpa
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift
        - aws
        - kubevirt
        - csi
    nodeAgent:
      enable: false
      uploaderType: kopia
    backupLocations:
      - velero:
          provider: aws
          default: true
          objectStorage:
            bucket: ${BUCKET_NAME}
            prefix: velero
          config:
            profile: default
            region: noobaa
            s3Url: https://s3.openshift-storage.svc:443
            s3ForcePathStyle: "true"
            insecureSkipTLSVerify: "true"
          credential:
            key: cloud
            name: cloud-credentials
EOF

Wait until the backup location is ready:

oc get backupstoragelocation -n openshift-adp
# Output must show: PHASE: Available

🧊 Step 3: Enable Application-Consistent Hooks

Inside your virtual machines (RHEL, CentOS, Ubuntu, Windows), ensure the QEMU Guest Agent is installed and running:

# On RHEL/CentOS guest:
sudo dnf install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

With the kubevirt plugin installed in your DPA, KubeVirt automatically communicates with the QEMU Guest Agent over an internal virtio-serial socket to trigger fsfreeze before snapshots and fsthaw immediately after!

📦 Step 4: Execute the VM Namespace Backup

The most reliable strategy is backing up the entire namespace hosting your VM workloads. This guarantees all accompanying secrets, configmaps, data volumes, and network definitions are preserved:

cat << 'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: prod-vm-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - vm-workloads
  snapshotVolumes: true
  storageLocation: oadp-dpa-1
EOF

Monitor progress until completion:

oc get backup prod-vm-backup -n openshift-adp -w
# Wait for Phase: Completed

🔄 Step 5: Disaster Recovery & The KubeMacPool Bypass

When restoring a VM into a new namespace (e.g., during disaster recovery drills or staging clone tests), KubeMacPool will reject the restore because the original VM's MAC address already exists in the cluster!

1. Create the Recovery Namespace and Bypass KubeMacPool

oc new-project vm-workloads-restored --skip-config-write

# Instruct KubeMacPool to ignore this namespace during restoration:
oc label namespace vm-workloads-restored \
  mutatevirtualmachines.kubemacpool.io=ignore --overwrite

2. Execute the Restore CR

cat << 'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: prod-vm-restore
  namespace: openshift-adp
spec:
  backupName: prod-vm-backup
  restorePVs: true
  namespaceMapping:
    vm-workloads: vm-workloads-restored
EOF

3. Power On the Restored VM

Once the restore reaches Completed, boot up the recovered machine:

virtctl start production-db-vm -n vm-workloads-restored

As soon as the VM boots, any pending PVCs will bind immediately, bringing your application back online with zero data corruption!

🎯 Key Takeaways

Never rely on crash-consistent backups for databases or production VMs. Always run the qemu-guest-agent.
Ensure your DPA includes the kubevirt and csi plugins to automate pre/post snapshot hooks.
Use the mutatevirtualmachines.kubemacpool.io=ignore namespace label to prevent webhook rejections during cross-namespace DR restores.

⏭️ Next Up: Day-2 Operations, Live Migration & Troubleshooting

Now that we have covered architecture, storage, golden images, networking, and enterprise backups, it's time to master day-to-day cluster operations!

In Part 6 (The Grand Finale), we explore:

Performing zero-downtime Live Migrations during node kernel patching.
Draining worker nodes (oc adm drain) without dropping VM user sessions.
The top production debugging commands every OpenShift Virtualization administrator must know.

💬 What is your current RPO and RTO target for virtual machines? Let us know in the comments below!

Part 4: Advanced VM Networking & MAC Address Governance

Muhammad Adel — Sun, 08 Nov 2026 23:02:43 +0000

Containers thrive on ephemeral, dynamic IP addresses. If a container dies, Kubernetes replaces it with a new pod, a new IP, and services automatically route traffic via kube-proxy or OVN-Kubernetes.

Virtual Machines, however, come from an entirely different enterprise heritage.

Legacy enterprise workloads running inside VMs often require:

Static, predictable Layer-2 IP addresses.
Direct connectivity to external corporate VLANs without NAT or Ingress proxies.
Multiple Network Interface Cards (NICs) separating management, storage, and application traffic.
Persistent MAC addresses that survive reboots, live migrations, and cluster restarts.

In this fourth part of our series, we dive deep into OpenShift Virtualization Networking, uncovering how Multus CNI bridges the gap between Kubernetes and physical networks, and how KubeMacPool prevents devastating Layer-2 MAC address collisions.

🏗️ Dual-Layer Architecture: Pod Network vs. Secondary Networks

OpenShift Virtualization solves the networking dilemma by offering a flexible, dual-layer networking model:

The two networking layers operate simultaneously:

The Default Pod Network (Primary): Connects the VM to the internal cluster SDN (OVN-Kubernetes) using Masquerade (NAT) mode. The VM can reach Kubernetes Services, DNS, and Ingress routers.
Secondary Physical Networks (Multus CNI): Bypasses the cluster SDN entirely, plugging the VM directly into an external VLAN via a Linux Bridge or Open vSwitch (OVS).

🔌 Connecting VMs Directly to External VLANs

To attach a virtual machine directly to an existing physical corporate network (e.g., VLAN 100 on subnet 192.168.10.0/24), we follow a three-step configuration:

1. Configure the Physical Bridge on Worker Nodes (NMState)

Using the Kubernetes NMState Operator, we declare a NodeNetworkConfigurationPolicy (NNCP) that creates a Linux Bridge on the physical NIC of worker nodes:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br-vlan100-policy
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
  desiredState:
    interfaces:
      - name: br-vlan100
        type: linux-bridge
        state: up
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: eth1

2. Define the `NetworkAttachmentDefinition` (NAD)

Multus CNI requires a NetworkAttachmentDefinition in the workload namespace to make the bridge available:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan100-net
  namespace: vm-workloads
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "vlan100-net",
    "type": "cnv-bridge",
    "bridge": "br-vlan100",
    "macspoofchk": true
  }'

3. Attach Dual NICs to the VirtualMachine

Now, in your VirtualMachine manifest, declare both interfaces under devices.interfaces and map them under networks:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: dual-nic-enterprise-vm
  namespace: vm-workloads
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          interfaces:
            - name: default-pod-net
              masquerade: {}        # Cluster SDN access
            - name: physical-vlan   # Direct external VLAN access
              bridge: {}
        resources:
          requests:
            memory: 4Gi
            cpu: 2
      networks:
        - name: default-pod-net
          pod: {}
        - name: physical-vlan
          multus:
            networkName: vlan100-net

Inside the guest OS, you will now see two physical network adapters (eth0 and eth1).

🏷️ MAC Address Governance: Enter KubeMacPool

In a standard Kubernetes cluster, pods receive random MAC addresses upon startup because Layer-2 MAC uniqueness rarely matters across routed overlay networks.

However, when multiple VMs connect directly to an external Layer-2 bridge:

If two VMs accidentally receive the same MAC address, an enterprise network collision occurs, dropping packets across the physical switch.
If a VM loses its MAC address after a live migration or restart, external DHCP reservations and ARP tables break.

How KubeMacPool Works

KubeMacPool is an intelligent admission controller deployed automatically with OpenShift Virtualization:

It manages a dedicated, configured pool of unique IEEE MAC addresses (e.g., 02:00:00:00:00:00 to 02:FF:FF:FF:FF:FF).
When a VirtualMachine is submitted, KubeMacPool's mutating webhook intercepts the creation request and injects a guaranteed-unique static MAC address into the manifest.
It tracks allocated MAC addresses in real-time, preventing duplicate assignments across all namespaces.

⚠️ Battle-Tested Gotchas & Production Traps

1. The Backup/Restore MAC Collision Trap

When restoring a VM backup (via OADP/Velero) into a new namespace for disaster recovery testing, KubeMacPool will intercept the restore request and block it with a webhook validation error:
MAC address already allocated to another VM.

🛠️ The Solution:

Temporarily instruct KubeMacPool to ignore the target restore namespace using an exclusion label:

oc label namespace vm-workloads-restored \
  mutatevirtualmachines.kubemacpool.io=ignore --overwrite

2. Promiscuous Mode on Nested Hypervisors

If your OpenShift cluster is running inside a nested virtualization platform (such as VMware vSphere or OpenStack):

The physical Linux bridge needs to pass foreign MAC addresses through the underlying vSwitch.
Ensure Promiscuous Mode and MAC Address Changes are set to "Accept" on the host port group / security group, otherwise incoming traffic will be silently discarded by the hypervisor host.

3. MTU Mismatch

If your cluster SDN uses an MTU of 1450 (Geneve encapsulation) but your external VLAN uses standard Ethernet MTU 1500 or Jumbo Frames 9000:

Ensure your NetworkAttachmentDefinition specifies the correct MTU matching the physical switch port to avoid silent TCP packet fragmentation and packet drops.

🎯 Key Takeaways

Default Pod Network (masquerade) is ideal for Kubernetes-internal communications and standard Ingress routing.
Multus CNI (cnv-bridge) allows seamless, direct connection to physical enterprise VLANs and legacy subnets.
KubeMacPool ensures carrier-grade Layer-2 stability by preventing duplicate MAC addresses across your cluster fleet.

⏭️ Next Up: Enterprise Backup & Disaster Recovery (DR)

Now that our VMs have reliable compute, high-performance storage, and enterprise networking, how do we protect them against catastrophic failures?

In Part 5, we will construct an end-to-end Enterprise Disaster Recovery strategy using OADP (Velero) and NooBaa (S3)—including how to configure QEMU Guest Agent hooks (fsfreeze / fsthaw) for 100% application-consistent snapshots.

💬 How are you handling VM VLANs and IP management in your container platform? Let us know in the comments below!

Part 3: Automated Boot Sources & Custom Golden Images with CDI

Muhammad Adel — Sun, 08 Nov 2026 22:53:03 +0000

In a traditional virtualization world, provisioning a virtual machine often meant downloading a 15GB .iso file, mounting it through an administrative console, clicking through an OS installer wizard, and running post-install scripts.

In a cloud-native platform like OpenShift Virtualization, that manual approach is unacceptable.

We need VMs to spin up in seconds—just like containers—using pre-baked, customized Golden Images (Boot Sources) that update automatically on a schedule.

In this third part of our series, we will walk through the end-to-end production workflow of building custom golden OS images, pushing them to the OpenShift Internal Registry, and automating recurring imports with CDI (Containerized Data Importer) and DataImportCron.

🏗️ Architecture: The Automated Boot Source Pipeline

The automated pipeline consists of three core stages:

Packaging: Wrapping a raw/qcow2 virtual disk into an ultra-lightweight containerDisk.
Distribution: Storing the image inside the secured OpenShift Internal Registry.
Import & Consumption: CDI continuously pulls updates, populates a DataVolume, and registers a ready-to-use DataSource for instant VM cloning.

🛠️ Stage 1: Build the containerDisk from Scratch

A containerDisk is simply an OCI container image containing a disk image located at /disk/. To keep it minimal, secure, and disconnected-friendly, we build it directly FROM scratch without pulling heavy base images like UBI.

1. Prepare File Permissions & Containerfile

Before building, ensure the host qcow2 file has read-only permissions (0440), and assign ownership to UID 107:107 (the internal QEMU user inside KubeVirt pods):

chmod 0440 custom-disk.qcow2

cat << 'EOF' > Containerfile
FROM scratch
ADD --chown=107:107 custom-disk.qcow2 /disk/
EOF

🚀 Stage 2: Push to the OpenShift Internal Registry

To push images without relying on external registries, we leverage OpenShift's built-in registry.

1. Expose the Internal Registry Route

oc patch configs.imageregistry.operator.openshift.io/cluster \
  --type=merge -p '{"spec":{"defaultRoute":true}}'

export REGISTRY_ROUTE=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
echo "Internal Registry: $REGISTRY_ROUTE"

2. Login and Push

podman login --tls-verify=false -u kubeadmin -p $(oc whoami -t) $REGISTRY_ROUTE

podman build -t $REGISTRY_ROUTE/openshift-virtualization-os-images/custom-os-disk:latest .
podman push --tls-verify=false $REGISTRY_ROUTE/openshift-virtualization-os-images/custom-os-disk:latest

🔐 Stage 3: Configure Authentication & TLS Certificates

When the CDI importer pod pulls the image from the internal service address (image-registry.openshift-image-registry.svc:5000), it requires credentials and cluster trust:

1. Create the Pull Secret

oc create secret generic kubeadmin-reg-secret \
  --from-literal=accessKeyId=kubeadmin \
  --from-literal=secretKey=$(oc whoami -t) \
  -n openshift-virtualization-os-images

2. Extract Cluster CA for TLS Verification

To prevent x509 certificate errors:

oc extract configmap/openshift-service-ca.crt \
  -n openshift-virtualization-os-images --to=- > service-ca.crt

oc create configmap internal-registry-ca \
  --from-file=ca.crt=service-ca.crt \
  -n openshift-virtualization-os-images

⏰ Stage 4: Deploy the Automated DataImportCron

Now we wire CDI to monitor the registry and automatically update a DataSource.

Notice the crucial annotation cdi.kubevirt.io/storage.bind.immediate.requested: "true", which forces storage allocation immediately rather than getting stuck in WaitForFirstConsumer:

cat << 'EOF' | oc apply -f -
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
  name: custom-os-image-cron
  namespace: openshift-virtualization-os-images
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: "true"
spec:
  garbageCollect: Outdated
  importsToKeep: 2
  managedDataSource: custom-os-datasource
  retentionPolicy: None
  schedule: "*/15 * * * *"
  template:
    spec:
      source:
        registry:
          pullMethod: pod
          url: "docker://image-registry.openshift-image-registry.svc:5000/openshift-virtualization-os-images/custom-os-disk:latest"
          secretRef: kubeadmin-reg-secret
          certConfigMap: internal-registry-ca
      storage:
        resources:
          requests:
            storage: 15Gi
EOF

Verify Importer Status

oc get dataimportcron,dv,pvc,datasource -n openshift-virtualization-os-images

Wait until the DataSource reports READY: True.

🚀 Stage 5: Spin Up a VM in 5 Seconds!

With our managed DataSource active, developers can spin up a production VM with a single CLI command using fast CSI cloning:

virtctl create vm --name production-rhel-vm \
  --volume-datasource src:openshift-virtualization-os-images/custom-os-datasource \
  | oc apply -f -

⚠️ Battle-Tested Gotchas & Troubleshooting Guide

Here are the top 5 errors engineers face when working with custom boot sources and CDI:

Common Error	Root Cause	Proven Fix
error: token is not available	Relying on stale KUBECONFIG without active login.	Always run `oc login -u kubeadmin` before generating tokens.
Permission denied on VM boot	Disk file inside container not owned by user 107.	Set `chmod 0440` and `ADD --chown=107:107` in `Containerfile`.
x509: certificate signed by unknown authority	Importer pod does not trust internal service CA.	Extract `openshift-service-ca.crt` and bind via `certConfigMap`.
pullMethod: node ignores secretRef	Node pull method uses kubelet service accounts, not user secrets.	Use `pullMethod: pod` when pulling with custom secrets.
Stuck in WaitForFirstConsumer	StorageClass waits for VM scheduling before binding PVC.	Add annotation `cdi.kubevirt.io/storage.bind.immediate.requested: "true"`.

🎯 Key Takeaways

Use FROM scratch and assign ownership to 107:107 for lightweight, secure containerDisk images.
Automate lifecycle updates with DataImportCron so developers always provision from the latest patched OS baseline.
Don't forget immediate binding annotations to eliminate PVC provisioning delays.

⏭️ Next Up: Advanced VM Networking & KubeMacPool

Now that our VMs can boot in seconds, how do we connect them to enterprise networks, VLANs, and external subnets?

In Part 4, we dive deep into:

Pod Networking (Masquerade) vs. Bridged Secondary Networks with Multus CNI.
Managing MAC address allocation and avoiding duplicate MAC clashes with KubeMacPool.

💬 How do you currently build and distribute OS golden images in your infrastructure? Share your workflow in the comments below!

Part 2: Storage Deep Dive — Block vs Filesystem & TopoLVM vs ODF

Muhammad Adel — Sun, 08 Nov 2026 22:44:50 +0000

Storage is often the make-or-break factor when migrating traditional virtual machines to Kubernetes.

In traditional environments (like VMware ESXi or Nutanix), storage is handled by centralized SAN/NAS arrays or vSAN. But in OpenShift Virtualization, storage is managed entirely through the Kubernetes CSI (Container Storage Interface).

If you misconfigure your storage layers, you risk sluggish I/O latency, broken live migrations, or failing CSI snapshot backups.

In this second part of our series, we demystify the two most critical storage decisions in OpenShift Virtualization:

Block vs. Filesystem Volume Modes
LVM Storage (TopoLVM) vs. OpenShift Data Foundation (ODF)
The infamous TopoLVM extent mismatch trap (and how to fix it).

🏗️ Storage Architectural Overview

http://googleusercontent.com/image_generation_content/10_28

Let's dissect these architectures to understand where each shines.

1️⃣ Volume Modes: Block vs. Filesystem

When creating a PersistentVolumeClaim (PVC) for a VM disk, Kubernetes lets you declare a volumeMode. Choosing the wrong one can degrade write throughput by up to 25–40%.

🔹 Filesystem Mode (`volumeMode: Filesystem`)

How it works: A standard Linux filesystem (such as ext4 or xfs) is formatted onto the underlying storage volume. The VM's virtual disk is stored as an image file (e.g., disk.img or .qcow2) inside that filesystem.
The Penalty: Double filesystem overhead! The guest OS inside the VM writes to its own filesystem, which writes to an image file, which in turn writes to the host node's filesystem.
When to use: Great for simple scratch disks, file sharing, or legacy CSI drivers that do not support raw block volumes.

🔹 Block Mode (`volumeMode: Block`) — The Gold Standard for VMs

How it works: The raw block device is passed directly into the virt-launcher pod and mapped straight to the QEMU/KVM process.
The Advantage: Zero host filesystem overhead, direct I/O execution, and lower CPU usage.
When to use: Production enterprise VMs, high-throughput databases (PostgreSQL, Oracle, SQL Server), and I/O-intensive workloads.

Example: Production Block Mode PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-disk-pvc
  namespace: vm-workloads
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block  # <--- Bypasses host filesystem overhead
  resources:
    requests:
      storage: 50Gi
  storageClassName: odf-ceph-rbd

2️⃣ Storage Engines: TopoLVM (LVMS) vs. ODF (Ceph)

Choosing the right storage backend depends on your cluster topology and business requirements:

Feature / Criterion	LVM Storage (TopoLVM / LVMS)	OpenShift Data Foundation (ODF)
Backend Technology	Local Linux LVM Volume Groups	Distributed Ceph (RBD & CephFS)
Ideal Deployment	Single Node OpenShift (SNO), Edge	Multi-node HA Clusters
I/O Latency	Ultra-low (Direct NVMe speeds)	Low to moderate (Network traversal)
Live Migration	❌ No (Disks pinned to node)	✅ Yes (Shared network storage)

⚠️ Why does the TopoLVM Extent bug happen?

When TopoLVM provisions a Logical Volume (LV) from a physical volume group, it rounds the requested disk size up to the nearest Physical Extent (PE) boundary.

If your snapshot hangs, patch the PVC request size upward by 1Gi:

oc patch pvc <pvc-name> -n <vm-namespace> --type='json' \
  -p='[{"op": "replace", "path": "/spec/resources/requests/storage", "value": "36Gi"}]'

🎯 Key Takeaways

Always prefer volumeMode: Block for production VM disks to achieve native bare-metal I/O performance.
Use ODF if live migration is a business requirement; use TopoLVM if raw NVMe speed on Single-Node is your priority.

Part 1: Architecture Under the Hood

Muhammad Adel — Sun, 08 Nov 2026 22:40:59 +0000

When engineers first hear about OpenShift Virtualization (formerly CNV / Container-Native Virtualization), their immediate reaction is often skepticism:

"Wait... are you putting an entire virtual machine inside a container? Isn't that anti-pattern virtualization-ception?"

The short answer: No.

OpenShift Virtualization does not run a hypervisor inside a container. Instead, it leverages the open-source KubeVirt technology to treat Virtual Machines as native Kubernetes workloads—giving VMs the same declarative deployment, GitOps pipelines, software-defined networking, and storage capabilities as regular containers.

In this first part of our series, we're popping the hood to see how KubeVirt actually coordinates compute, memory, and kernel primitives across an OpenShift cluster.

🏗️ The Big Picture Architecture

At its core, a virtual machine on Linux is simply a QEMU process running in user space, accelerated by the KVM kernel module.

KubeVirt encapsulates that QEMU process inside standard Linux namespaces (cgroups, network, PID) through a specialized pod called virt-launcher.

Here is how the control plane and worker nodes interact:

http://googleusercontent.com/image_generation_content/8_25

Let's break down each component shown above.

🧠 The Control Plane Components

The control plane orchestrates the lifecycle of virtual machines using standard Kubernetes controller patterns:

1. `virt-operator`

The operator responsible for deploying, upgrading, and managing the lifecycle of all KubeVirt custom resource definitions (CRDs) and services across the cluster. It ensures your virtualization stack stays healthy and up-to-date.

2. `virt-api`

Acts as the entry door for virtualization requests. It provides:

Admission Webhooks: Validates and mutates VM specifications before they hit etcd.
Subresource Endpoints: Powers interactive features like VNC graphical consoles, serial consoles, and power actions (start, stop, pause, restart) via virtctl.

3. `virt-controller`

The brain behind state reconciliation. When you request a virtual machine:

It reads your VirtualMachine manifest.
It coordinates with the Kubernetes scheduler by generating a matching Pod specification.
It tracks VM lifecycle transitions (e.g., Scheduling ➔ Starting ➔ Running ➔ Stopping).

⚙️ The Node-Level Components

Down on the worker nodes where compute happens, two key components make virtualization possible:

1. `virt-handler` (DaemonSet)

Runs as a privileged pod on every worker node eligible to run virtual machines. Its responsibilities include:

Communicating with the local node's libvirt daemon.
Interfacing with host hardware devices (like /dev/kvm).
Executing node-specific operations triggered by the virt-controller.

2. `virt-launcher` Pod

This is the pod scheduled by Kubernetes to host your virtual machine.

It creates the cgroup and namespace boundaries.
It executes libvirt and boots the qemu-kvm process.
If your VM crashes or shuts down, the virt-launcher pod terminates cleanly.

📄 The Two Essential CRDs: VM vs. VMI

One of the most common early confusions in KubeVirt is the difference between:

Custom Resource	Equivalent in K8s	Purpose
`VirtualMachine` (VM)	`Deployment` / `StatefulSet`	Declarative definition of the VM (CPU, memory, disks). Maintains desired running state even across reboots.
`VirtualMachineInstance` (VMI)	`Pod`	The active, ephemeral running instance of a VM. Created dynamically when a `VM` is started (`running: true`).


yaml`
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: fedora-workload
spec:
  running: true  # <--- Setting this creates the active VMI
  template:
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
        resources:
          requests:
            memory: 2Gi
            cpu: 2
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/fedora:latest

🛠️ Step-by-Step: Enabling OpenShift Virtualization

To get started on your OpenShift cluster via CLI:
1. Subscribe to the Operator
Bash

cat <<EOF | oc apply -f -
apiVersion: [operators.coreos.com/v1alpha1](https://operators.coreos.com/v1alpha1)
kind: Subscription
metadata:
  name: hco-operatorhub
  namespace: openshift-cnv
spec:
  channel: stable
  name: kubevirt-hyperconverged
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

2. Deploy the HyperConverged Cluster (HCO) CR

Once the operator CSV reports Succeeded, initialize the control plane:
Bash

cat <<EOF | oc apply -f -
apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec: {}
EOF

3. Verify Health

Verify that all control plane and node daemon pods are running:
Bash

oc get pods -n openshift-cnv

💡 Battle-Tested Gotchas & Pro-Tips
⚠️ Gotcha: Is Hardware Virtualization Enabled?

If your worker nodes run inside a public cloud or nested lab environment, KVM hardware acceleration (/dev/kvm) might not be exposed by default.

Verify your node labels:
Bash

oc get nodes -l kubevirt.io/schedulable=true

If you see no nodes matching this label, check if nested virtualization is enabled on the hypervisor host:
Bash

# On your node terminal:
ls -l /dev/kvm
egrep -c '(vmx|svm)' /proc/cpuinfo

For dev/lab testing only: You can enable software emulation inside the HyperConverged CR under spec.configuration.developerConfiguration.useEmulation: true, though it is strictly not supported for production.
⏭️ What's Next?

Now that the control plane is live and you understand the anatomy of a virt-launcher pod, the next critical hurdle is Storage.

In Part 2, we will dissect:

    Block vs. Filesystem volume modes.

    TopoLVM / LVM Storage vs. ODF.

    Avoiding the dangerous extent size calculation bug when creating CSI snapshots.

💬 Question for you: Are you currently planning a migration from legacy hypervisors to OpenShift Virtualization, or experimenting in labs? Drop your thoughts in the comments!

OpenShift Virtualization (CNV) Zero to Hero: Roadmap

Muhammad Adel — Sun, 08 Nov 2026 22:34:11 +0000

🚀 Welcome to the Series!

Virtualization isn't dead—it's evolving. As enterprise infrastructure shifts toward unified, cloud-native platforms, running Virtual Machines (VMs) alongside containerized microservices has become a major game changer.

Whether you're looking into modernizing legacy infrastructure, evaluating alternatives to traditional hypervisors (like VMware ESXi), or managing high-performance stateful workloads, OpenShift Virtualization (powered by open-source KubeVirt) brings VMs into the Kubernetes control plane as first-class citizens.

In this hands-on, battle-tested series, we will bridge the gap between theory and production realities—complete with copy-paste YAML manifests, CLI commands (oc and virtctl), architecture breakdowns, and troubleshooting guides for real-world edge cases.

🎯 Who Is This Series For?

DevOps & Platform Engineers responsible for managing hybrid workloads.
Systems Administrators transitioning from traditional virtualization (VMware / Proxmox / Hyper-V) to cloud-native platforms.
Kubernetes Practitioners & SREs looking to master KubeVirt, storage provisioning, and disaster recovery.

📑 The Master Index (Table of Contents)

Bookmark this page! As each part goes live, links will be updated here:

🔹 Part 1: The Foundations & Architecture Under the Hood

Status: Coming Soon
What we cover:
- Why run VMs on Kubernetes?
- KubeVirt architecture explained: virt-operator, virt-api, virt-controller.
- Node-level mechanics: virt-handler and the virt-launcher pod.
- Installing the OpenShift Virtualization Operator step-by-step.

🔹 Part 2: Storage Architecture & The Volume Extent Trap

Status: Coming Soon
What we cover:
- Persistent Storage for VMs: Block vs. Filesystem mode.
- Working with LVM Storage (TopoLVM) and OpenShift Data Foundation (ODF).
- StorageClasses and VolumeSnapshotClasses configuration.
- Production Gotcha: Solving the infamous TopoLVM extent mismatch (OutOfRange: requested size is smaller than source logical volume).

🔹 Part 3: Automated Boot Sources & Custom Golden Images

Status: Coming Soon
What we cover:
- Understanding CDI (Containerized Data Importer) and DataVolume CRDs.
- Building minimal containerDisk images from scratch using Containerfile.
- Pushing custom disks to the OpenShift Internal Registry with kubeadmin.
- Scheduling automated gold image updates using DataImportCron and DataSource.

🔹 Part 4: Advanced VM Networking & MAC Address Governance

Status: Coming Soon
What we cover:
- Default Pod Network (Masquerade) vs. Secondary Networks.
- Bridged networking with Multus CNI.
- Managing Layer-2 MAC addresses seamlessly with KubeMacPool.
- Handling MAC address collisions during workload cloning and migration.

🔹 Part 5: Enterprise Backup & Disaster Recovery (DR)

Status: Coming Soon
What we cover:
- Backup & Restore architecture using OADP (Velero) and local NooBaa (S3).
- Application consistency: Configuring QEMU Guest Agent and hooks (fsfreeze / fsthaw).
- End-to-end VM backup and cross-namespace restore.
- Bypassing mutating webhooks and snapshot binding errors during recovery.

🔹 Part 6: Day-2 Operations, Live Migration & Troubleshooting

Status: Coming Soon
What we cover:
- Zero-downtime Live Migration across worker nodes.
- Node evacuation (oc adm drain) with active VM workloads.
- Top 5 production failure modes and how to debug them (virtctl, crash loops, and stuck PVCs).

🛠️ Recommended Lab Environment

To follow along with the hands-on labs in this series, you will need:

An OpenShift 4.12+ cluster (SNO / Single Node OpenShift or a standard multi-node cluster).
Bare-metal or nested virtualization enabled on worker nodes (/dev/kvm).
Access to oc and virtctl CLI tools.
Cluster-admin privileges.

💬 Stay Connected

Follow my profile on DEV.to and save this post to your reading list. If there is a specific OpenShift Virtualization challenge you're currently facing in your environment, drop a comment below and we might include it in the series!

Next Up: Part 1: The Foundations & Architecture Under the Hood. Stay tuned!

DEV Community: Muhammad Adel

Part 6: Day-2 Operations, Live Migration & Production Troubleshooting

🏗️ Architecture: Migration Toolkit for Virtualization (MTV)

⚡ Cold Migration vs. Warm Migration

🔹 Cold Migration (Direct Cutover)

🔹 Warm Migration (Pre-Copy with CBT) — The Enterprise Gold Standard

🛠️ Step-by-Step: Migrating from VMware vCenter

1. Install the MTV Operator

2. Connect the VMware vCenter Provider

3. Create Network and Storage Mappings

4. Declare the Migration Plan (Warm Migration)

5. Execute the Migration & Final Cutover

⚠️ Battle-Tested Gotchas & Troubleshooting

🏁 Series Grand Finale: You Are Now a CNV Hero!

Part 5: Enterprise Backup & Disaster Recovery with OADP & NooBaa

🏗️ Architecture: The Application-Consistent Backup Workflow

🛠️ Step 1: Provision Local S3 Storage via NooBaa

Extract Bucket Credentials

⚙️ Step 2: Configure the DataProtectionApplication (DPA)

🧊 Step 3: Enable Application-Consistent Hooks

📦 Step 4: Execute the VM Namespace Backup

🔄 Step 5: Disaster Recovery & The KubeMacPool Bypass

1. Create the Recovery Namespace and Bypass KubeMacPool

2. Execute the Restore CR

3. Power On the Restored VM

🎯 Key Takeaways

⏭️ Next Up: Day-2 Operations, Live Migration & Troubleshooting

Part 4: Advanced VM Networking & MAC Address Governance

🏗️ Dual-Layer Architecture: Pod Network vs. Secondary Networks

🔌 Connecting VMs Directly to External VLANs

1. Configure the Physical Bridge on Worker Nodes (NMState)

2. Define the NetworkAttachmentDefinition (NAD)

3. Attach Dual NICs to the VirtualMachine

🏷️ MAC Address Governance: Enter KubeMacPool

How KubeMacPool Works

⚠️ Battle-Tested Gotchas & Production Traps

1. The Backup/Restore MAC Collision Trap

🛠️ The Solution:

2. Promiscuous Mode on Nested Hypervisors

3. MTU Mismatch

🎯 Key Takeaways

⏭️ Next Up: Enterprise Backup & Disaster Recovery (DR)

Part 3: Automated Boot Sources & Custom Golden Images with CDI

🏗️ Architecture: The Automated Boot Source Pipeline

🛠️ Stage 1: Build the containerDisk from Scratch

1. Prepare File Permissions & Containerfile

🚀 Stage 2: Push to the OpenShift Internal Registry

1. Expose the Internal Registry Route

2. Login and Push

🔐 Stage 3: Configure Authentication & TLS Certificates

1. Create the Pull Secret

2. Extract Cluster CA for TLS Verification

⏰ Stage 4: Deploy the Automated DataImportCron

Verify Importer Status

🚀 Stage 5: Spin Up a VM in 5 Seconds!

⚠️ Battle-Tested Gotchas & Troubleshooting Guide

🎯 Key Takeaways

⏭️ Next Up: Advanced VM Networking & KubeMacPool

Part 2: Storage Deep Dive — Block vs Filesystem & TopoLVM vs ODF

🏗️ Storage Architectural Overview

1️⃣ Volume Modes: Block vs. Filesystem

🔹 Filesystem Mode (volumeMode: Filesystem)

🔹 Block Mode (volumeMode: Block) — The Gold Standard for VMs

Example: Production Block Mode PVC

2️⃣ Storage Engines: TopoLVM (LVMS) vs. ODF (Ceph)

⚠️ Why does the TopoLVM Extent bug happen?

🎯 Key Takeaways

Part 1: Architecture Under the Hood

🏗️ The Big Picture Architecture

🧠 The Control Plane Components

1. virt-operator

2. virt-api

3. virt-controller

⚙️ The Node-Level Components

1. virt-handler (DaemonSet)

2. virt-launcher Pod

📄 The Two Essential CRDs: VM vs. VMI

OpenShift Virtualization (CNV) Zero to Hero: Roadmap

🚀 Welcome to the Series!

🎯 Who Is This Series For?

2. Define the `NetworkAttachmentDefinition` (NAD)

🔹 Filesystem Mode (`volumeMode: Filesystem`)

🔹 Block Mode (`volumeMode: Block`) — The Gold Standard for VMs

1. `virt-operator`

2. `virt-api`

3. `virt-controller`

1. `virt-handler` (DaemonSet)

2. `virt-launcher` Pod