DEV Community

Muhammad Adel
Muhammad Adel

Posted on Edited on

Part 2: Storage Deep Dive — Block vs Filesystem & TopoLVM vs ODF

#architecture #infrastructure #kubernetes

Storage is often the make-or-break factor when migrating traditional virtual machines to Kubernetes.

In traditional environments (like VMware ESXi or Nutanix), storage is handled by centralized SAN/NAS arrays or vSAN. But in OpenShift Virtualization, storage is managed entirely through the Kubernetes CSI (Container Storage Interface).

If you misconfigure your storage layers, you risk sluggish I/O latency, broken live migrations, or failing CSI snapshot backups.

In this second part of our series, we demystify the two most critical storage decisions in OpenShift Virtualization:

  1. Block vs. Filesystem Volume Modes
  2. LVM Storage (TopoLVM) vs. OpenShift Data Foundation (ODF)
  3. The infamous TopoLVM extent mismatch trap (and how to fix it).

🏗️ Storage Architectural Overview

http://googleusercontent.com/image_generation_content/10_28

Let's dissect these architectures to understand where each shines.

1️⃣ Volume Modes: Block vs. Filesystem

When creating a PersistentVolumeClaim (PVC) for a VM disk, Kubernetes lets you declare a volumeMode. Choosing the wrong one can degrade write throughput by up to 25–40%.

🔹 Filesystem Mode (volumeMode: Filesystem)

  • How it works: A standard Linux filesystem (such as ext4 or xfs) is formatted onto the underlying storage volume. The VM's virtual disk is stored as an image file (e.g., disk.img or .qcow2) inside that filesystem.
  • The Penalty: Double filesystem overhead! The guest OS inside the VM writes to its own filesystem, which writes to an image file, which in turn writes to the host node's filesystem.
  • When to use: Great for simple scratch disks, file sharing, or legacy CSI drivers that do not support raw block volumes.

🔹 Block Mode (volumeMode: Block) — The Gold Standard for VMs

  • How it works: The raw block device is passed directly into the virt-launcher pod and mapped straight to the QEMU/KVM process.
  • The Advantage: Zero host filesystem overhead, direct I/O execution, and lower CPU usage.
  • When to use: Production enterprise VMs, high-throughput databases (PostgreSQL, Oracle, SQL Server), and I/O-intensive workloads.

Example: Production Block Mode PVC 

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-disk-pvc
  namespace: vm-workloads
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block  # <--- Bypasses host filesystem overhead
  resources:
    requests:
      storage: 50Gi
  storageClassName: odf-ceph-rbd
Enter fullscreen mode Exit fullscreen mode

2️⃣ Storage Engines: TopoLVM (LVMS) vs. ODF (Ceph)

Choosing the right storage backend depends on your cluster topology and business requirements:

Feature / Criterion LVM Storage (TopoLVM / LVMS) OpenShift Data Foundation (ODF)
Backend Technology Local Linux LVM Volume Groups Distributed Ceph (RBD & CephFS)
Ideal Deployment Single Node OpenShift (SNO), Edge Multi-node HA Clusters
I/O Latency Ultra-low (Direct NVMe speeds) Low to moderate (Network traversal)
Live Migration ❌ No (Disks pinned to node) ✅ Yes (Shared network storage)

⚠️ Why does the TopoLVM Extent bug happen?

When TopoLVM provisions a Logical Volume (LV) from a physical volume group, it rounds the requested disk size up to the nearest Physical Extent (PE) boundary.

If your snapshot hangs, patch the PVC request size upward by 1Gi:

oc patch pvc <pvc-name> -n <vm-namespace> --type='json' \
  -p='[{"op": "replace", "path": "/spec/resources/requests/storage", "value": "36Gi"}]'
Enter fullscreen mode Exit fullscreen mode

🎯 Key Takeaways

  • Always prefer volumeMode: Block for production VM disks to achieve native bare-metal I/O performance.
  • Use ODF if live migration is a business requirement; use TopoLVM if raw NVMe speed on Single-Node is your priority.

Top comments (0)