Muhammad Adel

Posted on Nov 8 Edited on Nov 8

Part 6: Day-2 Operations, Live Migration & Production Troubleshooting

#devops #infrastructure #kubernetes #sre

Recent licensing shifts across enterprise virtualization—most notably VMware by Broadcom—have accelerated a massive industry migration: moving enterprise virtual machines to Red Hat OpenShift Virtualization.

Rebuilding hundreds or thousands of mission-critical VMs from scratch is simply not feasible. Enterprises need an automated, scalable migration pipeline that can:

Discover existing VMware vSphere clusters, datastores, and port groups.
Convert proprietary virtual disks (.vmdk) into cloud-native persistent storage volumes.
Inject native VirtIO drivers for KVM performance.
Cut downtime to minutes using continuous background pre-copying.

This is the exact purpose of MTV (Migration Toolkit for Virtualization), powered by the open-source Forklift project.

In this sixth and final installment of our OpenShift Virtualization (CNV) Zero to Hero series, we explore how to perform large-scale VM migrations from VMware vSphere to OpenShift with near-zero downtime.

🏗️ Architecture: Migration Toolkit for Virtualization (MTV)

The MTV migration engine orchestrates data movement between legacy hypervisors and OpenShift:

The MTV architecture centers on five core declarative primitives:

Providers: Define source (e.g., VMware vCenter) and target (OpenShift Virtualization) endpoints.
Network Map: Maps source VMware Port Groups / VLANs to target OpenShift Pod Networks or Multus CNI bridges.
Storage Map: Maps source VMware Datastores (VMFS/vSAN) to target OpenShift StorageClasses (ODF Ceph or LVM).
Migration Plan: Specifies the inventory of VMs to migrate, associated mappings, and cutover windows.
Conversion Pod (virt-v2v): A transient pod that mounts the incoming disk, installs VirtIO drivers, and prepares the OS to boot on KVM.

⚡ Cold Migration vs. Warm Migration

Choosing the right migration strategy directly impacts application downtime:

🔹 Cold Migration (Direct Cutover)

How it works: The source VM on VMware is powered off immediately. MTV copies 100% of the virtual disk across the network to OpenShift, runs virt-v2v conversion, and boots the new VM.
Downtime: Proportional to the size of the disk (e.g., a 500GB disk might require 1 to 2 hours of downtime).
Best for: Non-production environments, batch workloads, or systems that can tolerate scheduled maintenance windows.

🔹 Warm Migration (Pre-Copy with CBT) — The Enterprise Gold Standard

How it works: The source VM remains online and actively serving users on VMware! MTV leverages VMware CBT (Changed Block Tracking) to take periodic snapshots, pre-copying 95%+ of the disk data in the background over days or weeks.
The Cutover: When you are ready for the maintenance window, you trigger the final cutover. The source VM is powered down, only the tiny incremental delta of dirty blocks is transferred (taking just 2 to 5 minutes!), and the VM boots up in OpenShift.
Best for: Production databases, customer-facing applications, and business-critical systems.

🛠️ Step-by-Step: Migrating from VMware vCenter

1. Install the MTV Operator

Install the Migration Toolkit for Virtualization Operator from OperatorHub inside the openshift-mtv namespace.

2. Connect the VMware vCenter Provider

Store vCenter credentials in a secret, then declare the vSphere Provider:

apiVersion: v1
kind: Secret
metadata:
  name: vcenter-credentials
  namespace: openshift-mtv
type: Opaque
stringData:
  user: "administrator@vsphere.local"
  password: "YourSuperSecretPassword"
  insecureSkipVerify: "true"

---
apiVersion: forklift.konveyor.io/v1beta1
kind: Provider
metadata:
  name: vmware-source
  namespace: openshift-mtv
spec:
  type: vsphere
  url: "https://vcenter.corp.local/sdk"
  secret:
    name: vcenter-credentials
    namespace: openshift-mtv

3. Create Network and Storage Mappings

These mappings tell OpenShift how to translate VMware infrastructure into Kubernetes equivalents:

# 1. Network Mapping: VMware Port Group -> Multus Bridge
apiVersion: forklift.konveyor.io/v1beta1
kind: NetworkMap
metadata:
  name: vcenter-network-map
  namespace: openshift-mtv
spec:
  map:
    - destination:
        type: multus
        name: vlan100-net
        namespace: vm-workloads
      source:
        id: "network-12"  # VMware Port Group ID (or name: "VM Network")
  provider:
    source:
      name: vmware-source
    destination:
      name: host

---
# 2. Storage Mapping: VMware Datastore -> OpenShift StorageClass
apiVersion: forklift.konveyor.io/v1beta1
kind: StorageMap
metadata:
  name: vcenter-storage-map
  namespace: openshift-mtv
spec:
  map:
    - destination:
        storageClass: odf-ceph-rbd
        accessMode: ReadWriteMany
        volumeMode: Block
      source:
        id: "datastore-34"  # VMware Datastore ID (or name: "vsanDatastore")
  provider:
    source:
      name: vmware-source
    destination:
      name: host

4. Declare the Migration Plan (Warm Migration)

Now we group the target VMs into a migration plan and enable warm: true:

apiVersion: forklift.konveyor.io/v1beta1
kind: Plan
metadata:
  name: production-db-migration-plan
  namespace: openshift-mtv
spec:
  warm: true  # <--- Enables background pre-copy with CBT
  provider:
    source:
      name: vmware-source
    destination:
      name: host
  map:
    network:
      name: vcenter-network-map
    storage:
      name: vcenter-storage-map
  targetNamespace: vm-workloads
  vms:
    - name: "prod-oracle-db-01"
    - name: "prod-app-server-02"

5. Execute the Migration & Final Cutover

Initiate the background pre-copy:

apiVersion: forklift.konveyor.io/v1beta1
kind: Migration
metadata:
  name: run-db-migration
  namespace: openshift-mtv
spec:
  plan:
    name: production-db-migration-plan

MTV will begin streaming snapshots in the background while the VMs stay active on VMware.

When your maintenance window arrives, apply the final cutover timestamp:

oc patch plan production-db-migration-plan -n openshift-mtv --type='merge' \
  -p '{"spec":{"cutover":"2026-11-10T22:00:00Z"}}'

MTV powers off the VMware VMs, transfers the final delta, injects VirtIO drivers via virt-v2v, and boots the virtual machines in OpenShift!

⚠️ Battle-Tested Gotchas & Troubleshooting

Common Challenge	Root Cause	Proven Fix
Warm migration fails with CBT error	Changed Block Tracking is disabled on the source VM.	Enable CBT in VMware (`ctkEnabled = TRUE`) or power-cycle the VM once in vSphere.
Windows VM Blue Screens (BSOD) on boot	Missing KVM VirtIO storage/network drivers in Windows.	MTV automatically injects the `virtio-win` driver ISO. Ensure the `virtio-win` container is available on OpenShift.
Slow transfer speeds (throttling)	Data is passing through default ingress or routed networks.	Designate a dedicated, high-speed migration network in the `HyperConverged` CR configuration.
MAC Address changes after import	VM network adapter regenerated during import.	Check "Preserve static MAC addresses" in MTV so KubeMacPool retains the original VMware MACs.

🏁 Series Grand Finale: You Are Now a CNV Hero!

Over the course of this 6-part journey, we built an end-to-end enterprise virtualization practice on Kubernetes:

Part 1: Decoded the KubeVirt engine (virt-operator, virt-api, and virt-launcher).
Part 2: Mastered Block vs. Filesystem storage and navigated the TopoLVM extent trap.
Part 3: Built automated golden OS pipelines with containerDisk and DataImportCron.
Part 4: Connected VMs to physical enterprise VLANs using Multus and KubeMacPool.
Part 5: Built rock-solid DR strategies with OADP, NooBaa, and fsfreeze hooks.
Part 6: Automated large-scale hypervisor migrations from VMware to OpenShift with MTV.

OpenShift Virtualization proves that containers and virtual machines are no longer rival architectures—they are collaborative primitives running under one unified, cloud-native control plane.

Thank you for following along with this series! 🚀

💬 Are you actively planning or currently executing a VMware-to-OpenShift migration? What are your biggest hurdles? Share your story in the comments below!

DEV Community