OpenShift Virtualization: Running VMs and Pods on the Same Cluster Without Breaking Anything

Every platform team eventually hits the same wall: 80% of the workload is containerized, and the remaining 20% is a set of VMs nobody wants to touch — an Oracle client that only works on RHEL 7, a Windows appliance with a vendor license tied to a MAC address, a middleware box whose build pipeline left with the contractor. The business case for OpenShift Virtualization (KubeVirt packaged, supported and integrated by Red Hat) is simple: stop paying for two infrastructures and two operating models.

The technical case is harder. A VM is a pod that cannot be killed, rescheduled in 300 ms, or restarted on a whim. Dropping VMs onto a cluster tuned for stateless pods is how you end up with 200 GB of RAM pinned on a node you need to patch, a live migration that never converges, and an etcd member starved of CPU because someone gave a VM 32 vCPUs on a control-plane-adjacent node.

This is a field report on the three things that actually decide whether the coexistence works: storage, live migration, and secondary networking with Multus. Then we'll cover node maintenance, because that's where the design gets validated.

Context: what changes when a VM lands on your cluster

A VirtualMachine object is reconciled into a VirtualMachineInstance (VMI), which is backed by a virt-launcher pod running qemu-kvm inside the container. From the scheduler's point of view it's just a pod with a big memory request and a devices.kubevirt.io/kvm resource. From the operations point of view, three invariants break:

1. Eviction is no longer free. Deleting a virt-launcher pod means hard-powering off a VM. Drains must trigger migrations, not deletions.
2. Memory is not elastic. The guest RAM is a hard allocation plus QEMU overhead (roughly 200–300 MiB per VM plus page tables). Overcommit on a node that also runs latency-sensitive pods is a liability.
3. Storage access mode becomes a scheduling constraint. RWO means the VM cannot live-migrate. Period.

Start by deciding whether VMs and pods share nodes. In most retrofits, they shouldn't — not for isolation reasons, but for capacity predictability.

oc label node worker-07 node-role.kubernetes.io/virt=""
oc adm taint node worker-07 kubevirt.io/workload=true:NoSchedule

Then pin the virtualization workloads via the HyperConverged CR, which is the single source of truth for the whole stack:

apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: openshift-cnv
spec:
  workloads:
    nodePlacement:
      nodeSelector:
        node-role.kubernetes.io/virt: ""
      tolerations:
        - key: kubevirt.io/workload
          operator: Equal
          value: "true"
          effect: NoSchedule
  liveMigrationConfig:
    parallelMigrationsPerCluster: 5
    parallelOutboundMigrationsPerNode: 2
    completionTimeoutPerGiB: 150
    progressTimeout: 150
    network: migration-net
  featureGates:
    enableCommonBootImageImport: true

liveMigrationConfig.network is worth calling out early: it points to a NetworkAttachmentDefinition so migration traffic uses a dedicated NIC instead of saturating the pod network. On a 10 Gb link, a 32 GiB VM with a busy working set will happily eat the whole interface for two minutes.

Storage: the decision that makes or breaks live migration

Everything downstream depends on one property: can the PVC backing the root disk be attached to two nodes at once? Live migration requires ReadWriteMany (or RWX block). If your CSI driver can't do it, your VMs are pets forever.

Check what CDI thinks your storage can do — StorageProfile objects are auto-populated per StorageClass and drive the defaults for DataVolumes:

oc get storageprofile
oc get storageprofile ocs-storagecluster-ceph-rbd -o jsonpath='{.status.claimPropertySets}' | jq

For Ceph RBD you want volumeMode: Block + accessModes: [ReadWriteMany]. Block mode skips the filesystem layer, gives you a measurable IOPS gain and avoids the .img file-on-CephFS pattern.

If a profile reports nothing useful, override it rather than patching every DataVolume:

apiVersion: cdi.kubevirt.io/v1beta1
kind: StorageProfile
metadata:
  name: ocs-storagecluster-ceph-rbd
spec:
  claimPropertySets:
    - accessModes: [ReadWriteMany]
      volumeMode: Block
  cloneStrategy: csi-clone

Where HostPathProvisioner still fits

HostPathProvisioner (HPP) is part of the OpenShift Virtualization bundle and provisions node-local directories. It's RWO by definition, so any VM on HPP is non-migratable. We still use it, deliberately, for two cases: scratch/swap disks and throughput-bound workloads sitting on local NVMe where the VM is already in an application-level HA pair.

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner
spec:
  imagePullPolicy: IfNotPresent
  storagePools:
    - name: local-nvme
      path: /var/hpvolumes
  workload:
    nodeSelector:
      node-role.kubernetes.io/virt: ""

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-csi-local-nvme
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagePool: local-nvme

WaitForFirstConsumer is mandatory here — otherwise the PV is bound on a random node and the VM becomes unschedulable. And make sure /var/hpvolumes is a real mount point with its own device, created through a MachineConfig, not a directory carved out of the root filesystem.

Deploying a VM you can actually operate

Golden images live in openshift-virtualization-os-images as DataSource objects when enableCommonBootImageImport is on. Reference them with sourceRef so image updates don't require editing every VM:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: legacy-app-01
  namespace: legacy-apps
  labels:
    app: legacy-app
spec:
  runStrategy: Always
  dataVolumeTemplates:
    - metadata:
        name: legacy-app-01-rootdisk
      spec:
        sourceRef:
          kind: DataSource
          name: rhel9
          namespace: openshift-virtualization-os-images
        storage:
          accessModes: [ReadWriteMany]
          volumeMode: Block
          storageClassName: ocs-storagecluster-ceph-rbd
          resources:
            requests:
              storage: 60Gi
  template:
    metadata:
      labels:
        app: legacy-app
    spec:
      evictionStrategy: LiveMigrate
      terminationGracePeriodSeconds: 180
      domain:
        cpu:
          sockets: 1
          cores: 4
          threads: 1
          model: host-model
        memory:
          guest: 8Gi
        firmware:
          bootloader:
            efi:
              secureBoot: true
        features:
          smm:
            enabled: true
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}
              model: virtio
            - name: vlan120
              bridge: {}
          networkInterfaceMultiqueue: true
          rng: {}
      networks:
        - name: default
          pod: {}
        - name: vlan120
          multus:
            networkName: legacy-apps/vlan120-bridge
      volumes:
        - name: rootdisk
          dataVolume:
            name: legacy-app-01-rootdisk
        - name: cloudinitdisk
          cloudInitNoCloud:
            secretRef:
              name: legacy-app-01-cloudinit

Three details that matter more than they look:

• cpu.model: host-model gives the guest the host CPU features but restricts migration to nodes with a compatible CPU. On heterogeneous hardware, either use a named baseline model (Nehalem, Skylake-Server) or group nodes by generation and let the scheduler handle the cpu-model node labels published by virt-handler.
• terminationGracePeriodSeconds: 180 gives migration time to complete before anything forceful happens.
• Cloud-init goes through a Secret, never inline in the VM spec — the VM object is readable by anyone with get virtualmachines.

oc create secret generic legacy-app-01-cloudinit -n legacy-apps \
  --from-file=userdata=./user-data.yaml
oc apply -f legacy-app-01.yaml
oc get vmi -n legacy-apps -w
virtctl console legacy-app-01 -n legacy-apps

virtctl is downloadable straight from the cluster, which keeps versions aligned:

oc get consoleclidownload virtctl -o jsonpath='{.spec.links[*].href}{"\n"}'

Exposing the VM's service to the rest of the cluster

With masquerade binding on the pod network, the virt-launcher pod has a normal cluster IP and the VM sits behind NAT inside it. That means plain Kubernetes Services work. KubeVirt labels the launcher pod with vm.kubevirt.io/name, so:

apiVersion: v1
kind: Service
metadata:
  name: legacy-app
  namespace: legacy-apps
spec:
  selector:
    vm.kubevirt.io/name: legacy-app-01
  ports:
    - name: http
      port: 8080
      targetPort: 8080

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: legacy-app
  namespace: legacy-apps
spec:
  to:
    kind: Service
    name: legacy-app
  port:
    targetPort: http
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

For quick checks, virtctl expose vm legacy-app-01 --name=legacy-app --port=8080 --target-port=8080 generates the equivalent Service — fine for a lab, but keep the declarative manifest in Git for anything real.

Note that the guest keeps the same IP across a live migration on the pod network only because the masquerade NAT lives in the launcher pod and the pod IP is re-advertised; connections that are not reestablished will see a sub-second blip. For TCP sessions that must survive, use the secondary network with a stable L2 address.

Multus: giving legacy VMs the network they expect

Legacy VMs routinely need to be on a specific VLAN, with a specific IP, reachable by appliances that don't speak Kubernetes. That's Multus territory. Build the host bridge with NMState first:

apiVersion: nmstate.io/v1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: br-vlan120
spec:
  nodeSelector:
    node-role.kubernetes.io/virt: ""
  desiredState:
    interfaces:
      - name: bond0.120
        type: vlan
        state: up
        vlan:
          base-iface: bond0
          id: 120
      - name: br-vlan120
        type: linux-bridge
        state: up
        ipv4:
          enabled: false
        ipv6:
          enabled: false
        bridge:
          options:
            stp:
              enabled: false
          port:
            - name: bond0.120

oc get nncp br-vlan120
oc get nnce | grep br-vlan120

Then the attachment, in the VM's namespace:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: vlan120-bridge
  namespace: legacy-apps
  annotations:
    k8s.v1.cni.cncf.io/resourceName: bridge.network.kubevirt.io/br-vlan120
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "name": "vlan120-bridge",
      "type": "bridge",
      "bridge": "br-vlan120",
      "macspoofchk": true,
      "ipam": {},
      "preserveDefaultVlan": false
    }

"ipam": {} is intentional: the guest gets its address from the VLAN's existing DHCP server or a static config, exactly like it did on vSphere. macspoofchk: true prevents a compromised guest from impersonating another MAC on a shared L2 — turn it off only for VMs that legitimately need nested MACs.

If you're on OVN-Kubernetes and want to stay away from host bridges, the localnet topology of ovn-k8s-cni-overlay is the modern equivalent and is the direction Red Hat is pushing; the VM-side spec (multus.networkName) is identical.

Pin MAC addresses for license-bound guests with spec.template.spec.domain.devices.interfaces[].macAddress, and make sure that MAC is unique — nothing in the cluster will check it for you across namespaces.

Live migration and planned node maintenance

Trigger and observe a migration explicitly before you trust a drain:

virtctl migrate legacy-app-01 -n legacy-apps
oc get vmim -n legacy-apps
oc get vmi legacy-app-01 -n legacy-apps \
  -o jsonpath='{.status.migrationState}' | jq

For VMs that dirty memory faster than the link can transfer it (databases, JVMs with large heaps), plain pre-copy migration never converges. Use a MigrationPolicy scoped by namespace or VMI labels rather than tuning each VM:

apiVersion: migrations.kubevirt.io/v1alpha1
kind: MigrationPolicy
metadata:
  name: legacy-heavy-writers
spec:
  selectors:
    namespaceSelector:
      migration-tier: heavy
  allowAutoConverge: true
  allowPostCopy: false
  completionTimeoutPerGiB: 300
  bandwidthPerMigration: 2Gi

allowAutoConverge throttles the guest vCPUs to force convergence — a controlled slowdown instead of an aborted migration. Post-copy converges almost always but makes the VM dependent on the source node mid-flight; if that node dies, the VM dies. We keep it off outside of carefully chosen cases.

Draining a node that hosts VMs

KubeVirt automatically creates a PodDisruptionBudget for every VMI with evictionStrategy: LiveMigrate, which blocks the eviction API and lets virt-controller convert the disruption into a migration.

oc adm cordon worker-07
oc adm drain worker-07 \
  --delete-emptydir-data --ignore-daemonsets --force --timeout=1800s

Watch the migrations in parallel, and keep in mind parallelOutboundMigrationsPerNode — with 20 VMs at 2 concurrent migrations, a drain is a 20-minute operation, not a 30-second one. Size your MachineConfigPool maxUnavailable: 1 accordingly and budget cluster upgrades in hours per pool.

Audit before every maintenance window — any VM without LiveMigrate will be shut down by the drain:

oc get vmi -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,NODE:.status.nodeName,EVICT:.spec.evictionStrategy'

Set a cluster-wide default so nobody forgets: spec.evictionStrategy: LiveMigrate in the HyperConverged CR. For VMs that genuinely cannot migrate (HPP disks, GPU passthrough), use evictionStrategy: External and handle them with an explicit, scheduled shutdown — at least the drain won't surprise you.

The NodeMaintenance CRD from the Node Maintenance Operator gives you the same thing as a declarative, auditable object:

apiVersion: nodemaintenance.medik8s.io/v1beta1
kind: NodeMaintenance
metadata:
  name: worker-07-firmware
spec:
  nodeName: worker-07
  reason: "BIOS 2.14.1 rollout - change CHG-4821"

Security best practices

• Keep virt-launcher unprivileged. Modern OpenShift Virtualization runs QEMU as non-root under the restricted-v2-like SCC with a dedicated SELinux type. Never grant privileged SCC to the VM service accounts to "fix" a device issue — fix the device plugin instead.
• Scope RBAC on subresources. virtualmachineinstances/console and /vnc are full keyboard access to the guest. Bind kubevirt.io:view broadly, but grant console access through a dedicated role and a named group.
• Treat secondary networks as outside the cluster. NetworkPolicy only applies to the pod network. Traffic on a Multus bridge bypasses it entirely. Either enforce at the physical fabric/VLAN level, or enable MultiNetworkPolicy support in the cluster network operator and write explicit policies per attachment.
• Always enable macspoofchk, and prefer explicit, inventoried MAC addresses over generated ones on shared L2 segments.
• No credentials in the VM spec. cloud-init user data goes in a Secret referenced by secretRef; SSH keys go through spec.template.spec.accessCredentials with a qemuGuestAgent propagation method so you can rotate them without rebooting.
• UEFI Secure Boot + vTPM for Windows guests (efi.secureBoot: true with smm.enabled: true); it's a prerequisite for BitLocker and Credential Guard, and a hard requirement for Windows 11/Server 2025.
• Encrypt at rest at the storage layer (Ceph/LUKS or the array), and additionally inside the guest for regulated data. Disk images are plain blocks in a PVC — anyone with exec on the right namespace can read them.
• Quota the VM namespaces. A ResourceQuota on requests.memory and a LimitRange prevent a single team from allocating a node's worth of RAM, which is the most common way VM workloads starve collocated pods.
• Isolate migration traffic. Live migration carries unencrypted guest memory. Put it on a dedicated, non-routed network via liveMigrationConfig.network.

Key takeaways

Running VMs next to pods is a solved problem, provided you accept that the VM dictates the constraints, not the other way around.

• Storage decides everything. RWX + block, validated through StorageProfile, is the entry ticket to live migration. HostPathProvisioner is a deliberate, documented exception — never a default.
• evictionStrategy: LiveMigrate is not optional. Set it cluster-wide in the HyperConverged CR and audit for exceptions before every maintenance window.
• Multus plus NMState reproduces the legacy L2 the VMs expect, at the cost of losing NetworkPolicy — compensate with macspoofchk, fabric-level filtering, or MultiNetworkPolicy.
• Standard Kubernetes primitives still apply to VM services: a Service selecting vm.kubevirt.io/name plus a Route is all you need to expose a legacy app, which is exactly the point of converging the platforms.
• Drains become slow operations. Re-plan your upgrade windows around parallelOutboundMigrationsPerNode and the size of your guests, and use MigrationPolicy with auto-converge for the heavy writers.

Get those five right and OpenShift Virtualization stops being a risk and becomes what it should be: one control plane, one RBAC model, one GitOps pipeline, for both halves of your estate.