DEV Community

Muhammad Adel
Muhammad Adel

Posted on Edited on

Part 5: Enterprise Backup & Disaster Recovery with OADP & NooBaa

#cloud #devops #infrastructure #kubernetes


Backing up stateless microservices in Kubernetes is relatively straightforward: export your GitOps manifests, and redeploy.

Virtual Machines, however, are massive stateful engines.

If you take a naive snapshot of a running virtual machine while an active database is processing write transactions in RAM, your restored VM will likely boot into a corrupted filesystem, broken journal logs, or lost transactions (the dreaded "Crash-Consistent" state).

To achieve true enterprise-grade Application Consistency, we must coordinate three distinct layers:

  1. The guest operating system's internal filesystem memory buffers.
  2. The hypervisor virtualization engine (KubeVirt).
  3. The underlying storage array CSI snapshot controller.

In this fifth part of our series, we construct an end-to-end Enterprise Disaster Recovery strategy using OADP (OpenShift API for Data Protection / Velero) and local NooBaa (S3 Object Storage).

🏗️ Architecture: The Application-Consistent Backup Workflow

The coordinated backup flow operates in five discrete phases:

  1. Pre-Backup Hook: OADP communicates through KubeVirt to the qemu-guest-agent inside the VM, executing fsfreeze to flush all cached data from RAM to disk and freeze write operations.
  2. CSI Volume Snapshots: TopoLVM or ODF instantly captures point-in-time storage snapshots of all associated disks (PVCs).
  3. Post-Backup Hook: OADP signals KubeVirt to issue fsthaw, immediately unfreezing guest OS writes with zero perceptible application disruption.
  4. Metadata & Archive Streaming: VM definitions, cloud-init configurations, secrets, and metadata are bundled and encrypted into an S3-compatible object bucket.
  5. Cross-Namespace Disaster Recovery: The VM can be restored seamlessly to another namespace or an entirely separate recovery cluster.

🛠️ Step 1: Provision Local S3 Storage via NooBaa

Instead of routing gigabytes of backup data across public cloud networks, we can use NooBaa (Multicloud Object Gateway) to create a high-speed, local S3 bucket directly inside the cluster:

cat << 'EOF' | oc apply -f -
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: oadp-backup-bucket
  namespace: openshift-adp
spec:
  generateBucketName: oadp-backup
  storageClassName: openshift-storage.noobaa.io
EOF
Enter fullscreen mode Exit fullscreen mode

Extract Bucket Credentials 

BUCKET_NAME=$(oc get obc oadp-backup-bucket -n openshift-adp -o jsonpath='{.spec.bucketName}')
ACCESS_KEY=$(oc get secret oadp-backup-bucket -n openshift-adp -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
SECRET_KEY=$(oc get secret oadp-backup-bucket -n openshift-adp -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)

cat << EOF > /tmp/credentials-velero
[default]
aws_access_key_id = ${ACCESS_KEY}
aws_secret_access_key = ${SECRET_KEY}
EOF

oc create secret generic cloud-credentials -n openshift-adp \
  --from-file=cloud=/tmp/credentials-velero \
  --dry-run=client -o yaml | oc apply -f -
rm -f /tmp/credentials-velero
Enter fullscreen mode Exit fullscreen mode

⚙️ Step 2: Configure the DataProtectionApplication (DPA)

To support OpenShift Virtualization, your OADP deployment must include 4 mandatory plugins:

  1. openshift: Handles OpenShift-specific resources (Routes, ImageStreams).
  2. aws: Interfaces with NooBaa's S3-compatible API.
  3. kubevirt: Orchestrates VM states and QEMU guest agent hooks.
  4. csi: Interacts with storage drivers for instantaneous volume snapshots. 
cat << EOF | oc apply -f -
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: oadp-dpa
  namespace: openshift-adp
spec:
  configuration:
    velero:
      defaultPlugins:
        - openshift
        - aws
        - kubevirt
        - csi
    nodeAgent:
      enable: false
      uploaderType: kopia
    backupLocations:
      - velero:
          provider: aws
          default: true
          objectStorage:
            bucket: ${BUCKET_NAME}
            prefix: velero
          config:
            profile: default
            region: noobaa
            s3Url: https://s3.openshift-storage.svc:443
            s3ForcePathStyle: "true"
            insecureSkipTLSVerify: "true"
          credential:
            key: cloud
            name: cloud-credentials
EOF
Enter fullscreen mode Exit fullscreen mode

Wait until the backup location is ready:

oc get backupstoragelocation -n openshift-adp
# Output must show: PHASE: Available
Enter fullscreen mode Exit fullscreen mode

🧊 Step 3: Enable Application-Consistent Hooks

Inside your virtual machines (RHEL, CentOS, Ubuntu, Windows), ensure the QEMU Guest Agent is installed and running:

# On RHEL/CentOS guest:
sudo dnf install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent
Enter fullscreen mode Exit fullscreen mode

With the kubevirt plugin installed in your DPA, KubeVirt automatically communicates with the QEMU Guest Agent over an internal virtio-serial socket to trigger fsfreeze before snapshots and fsthaw immediately after!

📦 Step 4: Execute the VM Namespace Backup

The most reliable strategy is backing up the entire namespace hosting your VM workloads. This guarantees all accompanying secrets, configmaps, data volumes, and network definitions are preserved:

cat << 'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: prod-vm-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - vm-workloads
  snapshotVolumes: true
  storageLocation: oadp-dpa-1
EOF
Enter fullscreen mode Exit fullscreen mode

Monitor progress until completion:

oc get backup prod-vm-backup -n openshift-adp -w
# Wait for Phase: Completed
Enter fullscreen mode Exit fullscreen mode

🔄 Step 5: Disaster Recovery & The KubeMacPool Bypass

When restoring a VM into a new namespace (e.g., during disaster recovery drills or staging clone tests), KubeMacPool will reject the restore because the original VM's MAC address already exists in the cluster!

1. Create the Recovery Namespace and Bypass KubeMacPool 

oc new-project vm-workloads-restored --skip-config-write

# Instruct KubeMacPool to ignore this namespace during restoration:
oc label namespace vm-workloads-restored \
  mutatevirtualmachines.kubemacpool.io=ignore --overwrite
Enter fullscreen mode Exit fullscreen mode

2. Execute the Restore CR 

cat << 'EOF' | oc apply -f -
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: prod-vm-restore
  namespace: openshift-adp
spec:
  backupName: prod-vm-backup
  restorePVs: true
  namespaceMapping:
    vm-workloads: vm-workloads-restored
EOF
Enter fullscreen mode Exit fullscreen mode

3. Power On the Restored VM

Once the restore reaches Completed, boot up the recovered machine:

virtctl start production-db-vm -n vm-workloads-restored
Enter fullscreen mode Exit fullscreen mode

As soon as the VM boots, any pending PVCs will bind immediately, bringing your application back online with zero data corruption!

🎯 Key Takeaways

  1. Never rely on crash-consistent backups for databases or production VMs. Always run the qemu-guest-agent.
  2. Ensure your DPA includes the kubevirt and csi plugins to automate pre/post snapshot hooks.
  3. Use the mutatevirtualmachines.kubemacpool.io=ignore namespace label to prevent webhook rejections during cross-namespace DR restores.

⏭️ Next Up: Day-2 Operations, Live Migration & Troubleshooting

Now that we have covered architecture, storage, golden images, networking, and enterprise backups, it's time to master day-to-day cluster operations!

In Part 6 (The Grand Finale), we explore:

  • Performing zero-downtime Live Migrations during node kernel patching.
  • Draining worker nodes (oc adm drain) without dropping VM user sessions.
  • The top production debugging commands every OpenShift Virtualization administrator must know.

💬 What is your current RPO and RTO target for virtual machines? Let us know in the comments below!

Top comments (0)