Illusory Cloud Backups: Actually Testing Your DR Plan with Velero and a Timed Restore

Your backup job has been green for 14 months. Your Grafana dashboard shows a reassuring velero_backup_success_total counter climbing steadily. Your compliance auditor ticked the box. And yet, if your production cluster disappeared tonight, you have no idea whether you could bring the platform back — nor how long it would take.

This is not a hypothetical. The failure mode is boringly consistent: backups succeed, restores fail. Custom Resource Definitions come back after the Custom Resources that depend on them. PersistentVolumes restore empty because the CSI driver was swapped. Secrets are present but the operator that consumes them crash-loops because an owner reference points at a UID that no longer exists. The Completed status in velero backup get tells you a tarball landed in an S3 bucket. It tells you nothing about recoverability.

This article builds a complete, timed restore scenario — PVs, CRDs, secrets — on OpenShift/Kubernetes with Velero, and turns it into an automated test you can run on a schedule. The goal is simple: replace your declared RTO/RPO with measured RTO/RPO.

Context: why "backup success" is a meaningless metric

RTO (Recovery Time Objective) and RPO (Recovery Point Objective) are contractual numbers. In most organisations they were written in a Word document by someone who never ran velero restore create.

• RPO is bounded by your backup schedule and by the consistency of what was captured. A backup taken every 6 hours with no filesystem quiescing gives you a 6-hour RPO on paper, and an unknown RPO in practice if your database was mid-write.
• RTO is the full wall-clock time from "we decide to restore" to "the service answers requests correctly". That includes cluster provisioning, Velero installation, CRD reconciliation, volume rehydration from object storage, operator convergence, and DNS/ingress propagation. The velero restore command itself is often the fastest part.

Velero's architecture explains where the time goes. Kubernetes object manifests are tiny JSON blobs in an S3 bucket — restoring 5,000 objects takes seconds. Volume data is the expensive part, and the mechanism matters enormously:

Mechanism	Backup target	Restore speed	Cross-cluster / cross-region
CSI snapshots (VolumeSnapshotClass)	Storage backend	Very fast (CoW clone)	Usually no, unless the snapshot is replicated
File System Backup (restic / Kopia)	S3 bucket	Slow, data-size bound	Yes
CSI snapshot + data movement	S3 bucket	Medium	Yes

A team that benchmarked restores using local CSI snapshots and then has to rebuild in another region from restic will discover a 20x gap in RTO at the worst possible moment.

Setting up Velero correctly for a testable DR

Install with the right plugins

On OpenShift, prefer the OADP operator (OpenShift API for Data Protection) which packages Velero. On vanilla Kubernetes, the Velero CLI is fine. The key decision is enabling both CSI snapshots and File System Backup, so you can measure both paths.

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0,velero/velero-plugin-for-csi:v0.7.1 \
  --bucket velero-dr-eu-west-3 \
  --prefix prod-cluster \
  --backup-location-config region=eu-west-3 \
  --snapshot-location-config region=eu-west-3 \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --uploader-type kopia \
  --features=EnableCSI \
  --namespace velero

--use-node-agent deploys the DaemonSet that performs File System Backup. --uploader-type kopia is the modern replacement for restic; restic remains supported and the CLI flags are identical (--default-volumes-to-fs-backup). If you are on an older Velero, substitute --uploader-type restic.

On OpenShift, the node agent needs a privileged SCC to read pod volumes from the host:

oc adm policy add-scc-to-user privileged -z velero -n openshift-adp
oc adm policy add-scc-to-user privileged -z node-agent -n openshift-adp

Declare a VolumeSnapshotClass Velero can use

Velero only picks up a VolumeSnapshotClass that carries the right label. Without it, CSI backups silently fall back to nothing.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-gce-pd-snapclass
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: pd.csi.storage.gke.io
deletionPolicy: Retain

deletionPolicy: Retain matters: with Delete, removing the VolumeSnapshot object during cleanup destroys the underlying storage snapshot and your restore point with it.

A schedule that encodes your RPO

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: prod-apps-hourly
  namespace: velero
spec:
  schedule: "0 * * * *"
  useOwnerReferencesInBackup: false
  template:
    includedNamespaces:
      - payments
      - identity
    excludedResources:
      - events
      - events.events.k8s.io
    snapshotVolumes: true
    defaultVolumesToFsBackup: false
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 168h
    hooks:
      resources:
        - name: pg-consistent-snapshot
          includedNamespaces:
            - payments
          labelSelector:
            matchLabels:
              app: postgresql
          pre:
            - exec:
                container: postgresql
                command:
                  - /bin/bash
                  - -c
                  - "psql -U postgres -c \"SELECT pg_backup_start('velero');\""
                onError: Fail
                timeout: 3m
          post:
            - exec:
                container: postgresql
                command:
                  - /bin/bash
                  - -c
                  - "psql -U postgres -c \"SELECT pg_backup_stop();\""
                onError: Continue
                timeout: 3m

The hooks are what turn a crash-consistent snapshot into an application-consistent one. onError: Fail on the pre-hook is deliberate — a backup you cannot trust is worse than a missing backup, because it gives false confidence.

Building the timed restore scenario

The test must run against a separate cluster (or at minimum a separate namespace with a rewritten namespace mapping). Restoring into the live namespace proves nothing and risks production.

Step 1: define what "restored" means

Before touching Velero, write the assertions. A restore is successful when:

1. All Deployments/StatefulSets in the target namespace are Available with full replica count.
2. The PVC contains a known canary file or database row written just before the backup.
3. The application answers a functional HTTP probe (not just a liveness probe).
4. CRs reconciled by operators reach their expected status.phase.

Plant the canary before the backup so you can verify the actual data, not just the volume's existence:

oc -n payments exec deploy/ledger -- \
  sh -c 'echo "canary-$(date -u +%Y%m%dT%H%M%SZ)" > /data/dr-canary.txt'

Step 2: the restore manifest

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: dr-drill-20240612
  namespace: velero
spec:
  backupName: prod-apps-hourly-20240612010014
  includedNamespaces:
    - payments
  namespaceMapping:
    payments: payments-dr
  restorePVs: true
  existingResourcePolicy: none
  includeClusterResources: true
  hooks:
    resources:
      - name: wait-for-db
        includedNamespaces:
          - payments-dr
        labelSelector:
          matchLabels:
            app: postgresql
        postHooks:
          - exec:
              container: postgresql
              command:
                - /bin/bash
                - -c
                - "until pg_isready -U postgres; do sleep 2; done"
              execTimeout: 5m
              waitTimeout: 10m

includeClusterResources: true pulls in CRDs, ClusterRoles and PVs. This is the setting most often forgotten — without it, a restore into a fresh cluster produces namespaces full of CRs whose CRDs do not exist, and Velero will log error restoring ...: no matches for kind.

Step 3: the timing harness

Here is the core of the drill: a script that measures each phase and emits structured results.

#!/usr/bin/env bash
set -euo pipefail

NS_SRC="payments"
NS_DST="payments-dr"
RESTORE="dr-drill-$(date -u +%Y%m%d%H%M%S)"
BACKUP="$(velero backup get -o json \
  | jq -r '.items[] | select(.status.phase=="Completed")
           | [.metadata.name, .status.completionTimestamp] | @tsv' \
  | sort -k2 | tail -1 | cut -f1)"

echo "Using backup: ${BACKUP}"

# --- RPO: age of the restore point at the moment the drill starts
BACKUP_TS=$(velero backup get "${BACKUP}" -o json | jq -r '.status.completionTimestamp')
RPO_SECONDS=$(( $(date -u +%s) - $(date -u -d "${BACKUP_TS}" +%s) ))

T0=$(date -u +%s)

velero restore create "${RESTORE}" \
  --from-backup "${BACKUP}" \
  --namespace-mappings "${NS_SRC}:${NS_DST}" \
  --include-cluster-resources=true \
  --wait

T_VELERO=$(date -u +%s)

# --- Wait for workloads to converge
oc -n "${NS_DST}" wait --for=condition=Available deployment --all --timeout=15m
oc -n "${NS_DST}" rollout status statefulset/postgresql --timeout=15m

T_WORKLOAD=$(date -u +%s)

# --- Functional assertion: canary file present on the restored PV
CANARY=$(oc -n "${NS_DST}" exec deploy/ledger -- cat /data/dr-canary.txt)
[[ -n "${CANARY}" ]] || { echo "FAIL: canary missing"; exit 1; }

# --- Functional assertion: application answers correctly
oc -n "${NS_DST}" run dr-probe --rm -i --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal:latest -- \
  curl -sf --max-time 10 http://ledger:8080/healthz/ready > /dev/null

T_READY=$(date -u +%s)

cat <<EOF
{
  "restore": "${RESTORE}",
  "backup": "${BACKUP}",
  "rpo_seconds": ${RPO_SECONDS},
  "velero_restore_seconds": $((T_VELERO - T0)),
  "workload_convergence_seconds": $((T_WORKLOAD - T_VELERO)),
  "functional_ready_seconds": $((T_READY - T_WORKLOAD)),
  "rto_seconds": $((T_READY - T0)),
  "canary": "${CANARY}"
}
EOF

Three measurements, not one. If your RTO regresses, the breakdown tells you whether the cause is Velero (more objects, slower S3), workload convergence (operator reconcile loops, image pulls), or application warm-up (cache rebuild, JIT).

Step 4: automate the drill

Run it weekly as a Kubernetes CronJob in the DR cluster, with a ServiceAccount scoped tightly:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: dr-drill
  namespace: velero
spec:
  schedule: "0 3 * * 1"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 10
  jobTemplate:
    spec:
      backoffLimit: 0
      activeDeadlineSeconds: 5400
      template:
        spec:
          serviceAccountName: dr-drill
          restartPolicy: Never
          containers:
            - name: drill
              image: quay.io/myorg/dr-drill:1.4.0
              command: ["/usr/local/bin/dr-drill.sh"]
              env:
                - name: PUSHGATEWAY_URL
                  value: http://prometheus-pushgateway.monitoring:9091
              resources:
                requests:
                  cpu: 100m
                  memory: 256Mi
                limits:
                  memory: 512Mi
              securityContext:
                allowPrivilegeEscalation: false
                runAsNonRoot: true
                capabilities:
                  drop: ["ALL"]
                seccompProfile:
                  type: RuntimeDefault

Push the JSON output to Prometheus Pushgateway and alert on it. The alert that matters is not "backup failed" but:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: dr-objectives
  namespace: monitoring
spec:
  groups:
    - name: dr
      rules:
        - alert: RTOBudgetExceeded
          expr: dr_drill_rto_seconds > 1800
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Measured RTO ({{ $value }}s) exceeds the 30 min objective"
        - alert: DRDrillStale
          expr: time() - dr_drill_last_success_timestamp > 10 * 24 * 3600
          labels:
            severity: critical
          annotations:
            summary: "No successful DR drill in the last 10 days"

DRDrillStale is the single most valuable alert in this article. A DR plan that is not exercised decays silently.

Handling the hard parts

CRDs and ordering

Velero restores resources in a built-in priority order (namespaces, then CRDs, then cluster-scoped, then namespaced). It does not wait for operators to reconcile. If your CR requires a webhook served by a Deployment that is itself being restored, the API server will reject the CR.

Two mitigations: split the restore into two passes (infrastructure first, workloads second), or use --existing-resource-policy=update on the second pass. For OLM-managed operators, restoring Subscription and ClusterServiceVersion objects is usually a mistake — let OLM reinstall the operator from the catalog and restore only the CRs.

velero restore create dr-infra --from-backup "${BACKUP}" \
  --include-resources crds,namespaces,storageclasses,volumesnapshotclasses \
  --wait

velero restore create dr-apps --from-backup "${BACKUP}" \
  --exclude-resources crds \
  --namespace-mappings payments:payments-dr --wait

Secrets that cannot be restored as-is

Service account tokens, kubernetes.io/service-account-token secrets and anything bound to a cluster UID must be excluded — they will be regenerated. Externally managed secrets (External Secrets Operator, Vault Agent) should also be excluded and resynced from source of truth.

velero backup create prod-apps \
  --include-namespaces payments \
  --exclude-resources 'secrets.bootstrap.kubernetes.io' \
  --selector 'velero.io/exclude-from-backup notin (true)'

Label the secrets you do not want captured:

oc -n payments label secret vault-synced-db-creds velero.io/exclude-from-backup=true

StorageClass mismatch across clusters

A restore into a different cloud region or provider will fail if the PVC references a StorageClass that does not exist. Use a Velero ConfigMap to remap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  gp3-encrypted: standard-csi
  premium-rwo: standard-csi

Security best practices

• Dedicated bucket, dedicated credentials. The Velero IAM principal needs s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket scoped to one prefix — never s3:* on *. Prefer IRSA (EKS), Workload Identity (GKE) or OpenShift's CloudCredential operator over long-lived access keys.
• Object Lock in compliance mode. Enable S3 Object Lock with a retention period at least equal to your backup TTL. This is the only control that survives an attacker with cluster-admin who runs velero backup delete --all. Note that Velero's own deletion of expired backups will fail against locked objects; manage expiry with S3 lifecycle rules instead.
• Encrypt at rest and in transit. SSE-KMS on the bucket with a customer-managed key, and a key policy that denies the Velero role kms:ScheduleKeyDeletion. Kopia/restic repositories are additionally encrypted with a repository password — store it in a separate secret and back it up out-of-band, because without it your S3 data is unrecoverable.
• Treat backups as production data for compliance. A backup of a namespace containing PII is PII. Same retention rules, same region constraints, same access logging (s3:GetObject events in CloudTrail with alerting on unusual read volume).
• Isolate the DR cluster's RBAC. The drill ServiceAccount needs read on Velero CRs and write on the target namespace only. It should never hold cluster-admin, even though the restore operation itself requires broad permissions performed by the Velero controller.
• Never restore into production by accident. Guard the drill script with an explicit check on the cluster's API server URL or a well-known ConfigMap flag, and refuse to run if the target namespace lacks a dr-drill=true label.

Key takeaways

• A green backup is a necessary condition for recovery, never a sufficient one. The only evidence of recoverability is a successful restore with functional assertions.
• Measure RTO in three segments — Velero restore, workload convergence, application readiness — so regressions are attributable.
• Measure real RPO as the age of the restore point at drill time, not as the cron interval.
• includeClusterResources: true plus a two-pass restore (CRDs first, workloads second) solves the large majority of restore failures in operator-heavy clusters.
• CSI snapshots and File System Backup have radically different restore times. Benchmark the path you will actually use in a disaster, which is usually the slow one.
• Alert on drill staleness, not just drill failure. An untested plan degrades to zero value within months.

Start small: pick one namespace with a PV, plant a canary, run the timed restore manually this week, and write down the number. Whatever it is, it will be more useful than the number currently in your DR document.