DEV Community

giveitatry
giveitatry

Posted on

Backup and Restore of a K3s Cluster Using Velero

#kubernetes #velero

This guide provides a practical, production-oriented walkthrough of installing, configuring, and troubleshooting Velero for backing up a K3s cluster. It focuses on reliability and reproducibility rather than theory.

1. Overview

Velero is an open-source tool designed to:

  • Back up Kubernetes resources (Deployments, Services, CRDs, etc.)
  • Optionally back up persistent volumes
  • Restore workloads to the same or a different cluster

In K3s environments, Velero is commonly used for:

  • Disaster recovery
  • Cluster migration
  • CI/CD environment replication

2. Installing Velero CLI on Linux

2.1 Download

You already identified the correct approach. Use:

curl -L https://github.com/velero-io/velero/releases/download/v1.18.0/velero-v1.18.0-linux-amd64.tar.gz | tar xz
Enter fullscreen mode Exit fullscreen mode

This downloads and extracts:

velero-v1.18.0-linux-amd64/
Enter fullscreen mode Exit fullscreen mode

2.2 Install binary system-wide 

cd velero-v1.18.0-linux-amd64
sudo mv velero /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

2.3 Verify installation 

velero version
Enter fullscreen mode Exit fullscreen mode

Expected output:

Client:
  Version: v1.18.0
Enter fullscreen mode Exit fullscreen mode

At this stage, the server part is not yet installed — that comes next.

3. Preparing Storage Backend

Velero requires object storage (S3-compatible).

Supported options:

  • AWS S3
  • MinIO (recommended for on-prem)
  • Hetzner Object Storage
  • Azure Blob / GCP

3.1 Create credentials file

Example:

cat <<EOF > credentials-velero
[default]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>
EOF
Enter fullscreen mode Exit fullscreen mode

4. Installing Velero in K3s

Velero runs inside the cluster as a controller.

4.1 Installation command

Example (S3-compatible storage):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket velero \
  --secret-file ./credentials-velero \
  --use-volume-snapshots=false \
  --backup-location-config \
    region=hel1,s3ForcePathStyle="true",s3Url=https://<OBJECT_STORAGE_ENDPOINT>:<PORT-IF-NEEDED>
Enter fullscreen mode Exit fullscreen mode

Key parameters explained

  • --provider aws
    Required even for S3-compatible systems

  • --plugins
    Enables S3 integration

  • --bucket
    Must already exist in storage

  • region=minio
    Required (even if not AWS) - any region will work

  • s3ForcePathStyle="true"
    Required for most non-AWS providers

  • s3Url
    Must be reachable from inside the cluster

4.2 Verify installation 

kubectl get pods -n velero
Enter fullscreen mode Exit fullscreen mode

Expected:

velero-xxxxx   Running
Enter fullscreen mode Exit fullscreen mode

4.3 Validate storage 

velero backup-location get
Enter fullscreen mode Exit fullscreen mode

Expected:

PHASE: Available
Enter fullscreen mode Exit fullscreen mode

This is the most important health indicator.

5. Creating Backups

5.1 Full cluster backup 

velero backup create full-backup
Enter fullscreen mode Exit fullscreen mode

5.2 Check status 

velero backup get
Enter fullscreen mode Exit fullscreen mode

Expected:

STATUS: Completed
Enter fullscreen mode Exit fullscreen mode

5.3 Detailed inspection 

velero backup describe full-backup --details
Enter fullscreen mode Exit fullscreen mode

6. Restoring from Backup 

velero restore create --from-backup full-backup
Enter fullscreen mode Exit fullscreen mode

This will:

  • recreate resources
  • restore metadata
  • optionally restore volumes (if configured)

7. Automating Backups

Example: daily backup at 02:00

velero schedule create daily-backup \
  --schedule="0 2 * * *"
Enter fullscreen mode Exit fullscreen mode

8. Persistent Volume Backups (Important)

By default (as in your setup):

--use-volume-snapshots=false
Enter fullscreen mode Exit fullscreen mode

This means:

  • Kubernetes objects are backed up
  • Persistent data is NOT

Options:

Option A — CSI snapshots (preferred)

Requires storage support

Option B — Node-agent (Restic) 

--use-node-agent
Enter fullscreen mode Exit fullscreen mode

Without this, your backups are incomplete for stateful workloads.

9. Troubleshooting

9.1 CLI works but server not found

Error:

no matches for velero.io/v1
Enter fullscreen mode Exit fullscreen mode

Cause:

  • Velero not installed in cluster

Fix:

  • run velero install

9.2 Backup fails with FailedValidation

Common causes:

  • missing region
  • invalid storage config
  • bucket not reachable

Fix:
Ensure:

region: minio
Enter fullscreen mode Exit fullscreen mode

9.3 MissingRegion error 

MissingRegion: could not find region configuration
Enter fullscreen mode Exit fullscreen mode

Fix:
Add:

region=minio
Enter fullscreen mode Exit fullscreen mode

9.4 Storage unavailable 

BackupStorageLocation is in unavailable state
Enter fullscreen mode Exit fullscreen mode

Check:

velero backup-location get
Enter fullscreen mode Exit fullscreen mode

If not Available, verify:

  • endpoint URL (must be reachable from cluster)
  • credentials
  • bucket existence

9.5 Connection timeout 

dial tcp ... i/o timeout
Enter fullscreen mode Exit fullscreen mode

Cause:

  • cluster cannot reach storage

Fix:

  • verify network access from pod: 
kubectl run test --rm -it --image=busybox -- sh
wget -O- https://<endpoint>
Enter fullscreen mode Exit fullscreen mode

9.6 Incorrect protocol (HTTP vs HTTPS)

Many providers require HTTPS.

Incorrect:

http://...
Enter fullscreen mode Exit fullscreen mode

Correct:

https://...
Enter fullscreen mode Exit fullscreen mode

9.7 Plugin process exited (not an error) 

plugin process exited
Enter fullscreen mode Exit fullscreen mode

This is normal behavior:

  • plugin runs per request
  • exits after execution

No action required.

9.8 Inconsistent logs (0/0/1 state) 

available/unavailable/unknown: 0/0/1
Enter fullscreen mode Exit fullscreen mode

Occurs during startup.

Ignore if:

velero backup-location get
Enter fullscreen mode Exit fullscreen mode

shows:

Available
Enter fullscreen mode Exit fullscreen mode

10. Recovery Strategy for K3s

To restore a cluster on a new server:

  1. Install K3s
  2. Install Velero
  3. Configure same storage
  4. Run: 
velero restore create --from-backup <backup-name>
Enter fullscreen mode Exit fullscreen mode

This reconstructs:

  • workloads
  • configs
  • namespaces

11. Best Practices

  • Always verify backups with test restores
  • Use external object storage (not same node)
  • Automate backups (schedules)
  • Enable volume backup for production workloads
  • Version control your Velero installation config

Top comments (0)