DEV Community: Aleksey Zhukov

Backup Kubernetes PVC with Restic, and ketchup(k8up)

Aleksey Zhukov — Thu, 20 Jun 2024 06:58:04 +0000

I am using Bitwarden as a password manager with Vaultwarden as the server implementation. When migrating this valuable data into my homelab Kubernetes setup, I decided to implement proper disaster recovery. As part of the migration, I planned to use a backup/restore procedure to facilitate data movement and validate the recovery process.

Before diving into the implementation details, let's review our goals and briefly describe the tooling.

Vaultwarden is running as a pod in a Kubernetes cluster. ArgoCD is responsible for provisioning all Kubernetes resources from a single source of truth. Data is stored within a Persistent Volume Claim (PVC). I'm running this homelab on a Virtual Private Cloud (VPC), so it lacks all the "cloud" features like persistence on EBS or similar services. To ensure peace of mind, I need to be sure that all my passwords are safe in case of an emergency. Therefore, I decided to go with a slightly modified version of the 3-2-1 backup schema: one backup to a local MinIO deployment, and another backup to remote S3-compatible storage.

Setting up MinIO is beyond the scope of this article, but you can check this Gist

k8up 101

K8up is a Kubernetes Backup Operator. It's a CNCF Sandbox Open Source project that uses other brilliant open-source software like Restic for handling backups and working with remote storage. It uses custom resources to define backup, restore, and scheduling tasks.

K8up scans namespaces for matching Persistent Volume Claims (PVCs), creates backup jobs, and mounts the PVCs for Restic to back up to the configured endpoint.

To create a backup with k8up, you define a Backup object in YAML, specifying details such as the backend storage and credentials. This configuration is then applied to your Kubernetes cluster using kubectl apply. For regular backups, you create a Schedule object, which outlines the frequency and other parameters for backup, prune, and check jobs.

Installation instruction can be found on the official site

Steps

Create an empty deployment: This will produce dummy data for the initial backup.
Create a Backup object: This will produce a Restic repository on the backend of your choice.
Backup existing data with Restic: This will create a new Restic snapshot and place all data into the backup store.
Clean up the deployment before restoring (optional: you can restore into a new PVC and point the deployment to it).
Create a Restore object: This will move data from the backup snapshot into the production PVC.
Create a Schedule object: This will automate the regular creation of backups.

Create an empty deployment

I'll skip this section because your use case will probably be different from mine.

Create a Backup object

K8up Backup object and Secret object with credentials for the backend.

apiVersion: v1
kind: Secret
metadata:
  name: local-backup-repo
type: Opaque
stringData:
# password: change_to_strong_password_or_passhprase

apiVersion: v1
kind: Secret
metadata:
  name: local-backup-creds
type: Opaque
stringData:
  username: minio
  password: minio123

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-dummy
spec:
    repoPasswordSecretRef:                  # (1)
      name: local-backup-repo
      key: password
    s3:                                     # (2)
      endpoint: http://minio.minio.svc:9000 # (3)
      bucket: backups/vaultwarden           # (4)
      accessKeyIDSecretRef:                 # (5)
        name: local-backup-creds
        key: username
      secretAccessKeySecretRef:             # (5)
        name: local-backup-creds
        key: password

Key parts of the manifest with environment variables used by Restic:

Restic repository password reference used to encrypt the backup: RESTIC_PASSWORD
Restic storage backend type.
Storage backend endpoint.
Backup path within the storage backend.
Credentials reference for the backend.

RESTIC_REPOSITORY could be constracted from (2),(3),(4) as such: s3:http://minio.minio.svc:9000/backups/vaultwarden

Applying these manifests will trigger the Backup procedure by the K8up controller. You have several means to check the backup status. You should definitely do so because of a current issue #910 with K8up which incorrectly reports status into the Backup object itself.

Also, a Snapshot object in Kubernetes will be created as a mirror of the Restic snapshot.

After the issue is fixed, this will be the proper way to check the Backup status.

kubectl -n vaultwarden describe backups.k8up.io backup-dummy

Name:         backup-dummy
Namespace:    vaultwarden
...
Status:
  Conditions:
    Last Transition Time:  2024-06-19T08:21:37Z
    Message:               no container definitions found
    Reason:                NoPreBackupPodsFound
    Status:                True
    Type:                  PreBackupPodReady
    Last Transition Time:  2024-06-19T08:21:47Z
    Message:               job 'backup_backup-dummy' completed successfully
    Reason:                Finished
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2024-06-19T08:21:47Z
    Message:               "backup_backup-dummy" has 1 succeeded, 0 failed, and 0 started jobs
    Reason:                Succeeded
    Status:                True
    Type:                  Completed
    Last Transition Time:  2024-06-19T08:21:47Z
    Message:               Deleted 2 resources
    Reason:                Succeeded
    Status:                True
    Type:                  Scrubbed
  Finished:                true
  Started:                 true

But for now, you must also check and parse Restic logs from the appropriate pod.

kubectl -n vaultwarden get po | grep dummy

kubectl -n vaultwarden logs <pod_name>

You can go deeper and list actual files within the snapshot with the Restic CLI. In this example, you need to get access to the backup storage backend.

Port-forward the MinIO port:

kubectl -n minio port-forward svc/minio 9000:9000

Prepare the environment for Restic:

export RESTIC_PASSWORD='<password@local-backup-repo>'
export RESTIC_REPOSITORY='s3:http://127.0.0.1:9000/backups/vaultwarden'
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123

Get a list of all snapshots:

restic snapshots

List files in the backup storage:

restic ls -l <snapshot_id>

The last step will show you how files are stored within the backup. We need this information to mimic K8up backup with the Restic CLI in the following steps. We need to know what common path prefix the files in the backup have. It should be constructed as such: /data/<PVC_NAME>. In my case, it was /data/vaultwarden.

Backup existing data with Restic

The most tricky part of this step is to produce a correct Restic snapshot that can be restored by K8up. It requires properly forged paths for backed-up files and access to the storage backend.

Initially, I was trying to achieve this with Docker. But Restic failed with strange IO errors on backup. So, I switched to Podman. Let's assume that files are stored in ~/backup/vaultwarden on my host. Here are the steps for creating a Restic snapshot with a file structure suited for K8up from local files.

Get access to the storage backend:

kubectl -n minio port-forward svc/minio 9000:9000

Set the environment for Restic as described earlier, altering the repository:

export RESTIC_PASSWORD='<password@local-backup-repo>'
export RESTIC_REPOSITORY='s3:http://host.containers.internal:9000/backups/vaultwarden'
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123

The trick here is to use host.containers.internal as the hostname. This name is provided for each Podman container and points to the host IP address. Docker has another name for that purpose: host.docker.internal.

Run the container with the proper mount and environments:

cd ~/backup/vaultwarden
podman run -it --rm --env-file <(env | egrep '^AWS_|RESTIC_') -v "$(pwd)":/data/vaultwarden --entrypoint=sh restic/restic

Backup files using the Restic CLI:

restic backup /data/vaultwarden

Get the snapshot ID

In output of previous command you should get created snapshot id. Or you can get all snapshot with:

restic snapshots

The output should have two snapshots. One with dummy data created by K8up and another one with actual data created with the help of Restic by you just now.

Create a Restore object

By now, we have managed to put data into the backup storage. Let's create a Restore object to get this data into the PVC for production use.

Restore object mirrors backup and additionally specifies the restore method, which could be either a folder or S3.

apiVersion: k8up.io/v1
kind: Restore
metadata:
  name: restore
spec:
  snapshot: <SNAPTHOST_ID>
  restoreMethod:
    folder:
      claimName: vaultwarden
  backend:
    repoPasswordSecretRef:
      name: local-backup-repo
      key: password
    s3:
      endpoint: http://minio.minio.svc:9000
      bucket: backups/vaultwarden
      accessKeyIDSecretRef:
        name: local-backup-creds
        key: username
      secretAccessKeySecretRef:
        name: local-backup-creds
        key: password

Create a Schedule object

The final step in our journey will be setting up scheduling, which will combine and automate the following actions:

backing up data frequently
checking the integrity of backup storage
maintaining an appropriate number of backup versions over time

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: local
spec:
  backend:
    s3:
      endpoint: http://minio.minio.svc:9000
      bucket: backups/vaultwarden
      accessKeyIDSecretRef:
        name: local-backup-creds
        key: username
      secretAccessKeySecretRef:
        name: local-backup-creds
        key: password
    repoPasswordSecretRef:
      name: local-backup-repo
      key: password
  backup:
    schedule: '3 * * * *'
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
  check:
    schedule: '33 2 * * 1'
  prune:
    schedule: '33 2 * * *'
    retention:
      keepHourly: 14
      keepDaily: 14
      keepMonthly: 3

In conclusion, implementing a robust backup and recovery strategy for Vaultwarden in a Kubernetes setup is crucial for ensuring the security and availability of your password data. By leveraging the powerful capabilities of K8up and Restic, we can create a reliable and automated backup process that mitigates the risks associated with data loss.

Unfork with ArgoCD

Aleksey Zhukov — Sat, 13 Jan 2024 18:24:27 +0000

With the help of existing freely available software, we can build a personal or company software stack without starting from scratch, but rather by standing on the shoulders of giants. This eliminates the need to constantly reinvent most parts of our systems. However, sometimes existing off-the-shelf solutions don't provide enough customization to achieve the required goals. In such cases, we face a dilemma: to fork or not to fork. There are reasons for either choice, but today we will explore the "unfork" approach. We will investigate available options with examples and compare them afterward.

All examples are avaliable in companion repo.

Assumptions regarding the environment and the goal

Kubernetes - extensible API server as universal control plane
ArgoCD - GitOPS controller with monitoring Git repos and apply objects to k8s
Any third-party off-the-shelf software

Goal: We need to add a specific Kubernetes (k8s) resource to the ArgoCD application when the source of the application is managed by a third party, without managing a fork of that software.

In all the described cases, you can either add or overwrite the entire resource. More granular patching is not always available; I'll note this later.

Flavors of software distribution

Here is a list of ways to distribute software that occur in the wild, along with corresponding examples.

plain kubernetes manifests: ArgoCD
kustomize Kube Prometheus
helm chart Traefik Ingress

ArgoCD has its own set of supported sources for applications as such:

Kustomize applications
Helm charts
A directory of manifests

How does Renovate fit in here

It is a good practice to keep software up to date. To track changes in upstream software, we can utilize automatic dependency tracking systems such as Dependabot or Renovate. This is a broad topic and requires a separate article to be covered. If you would like to read about it, please vote in the comments section below.

Outline

Multiple Sources for an Application
Umbrella chart
Kustomize them all

Multiple Sources for an Application

Recent version (2.6) of ArgoCD supports spec.sources (plural) on an Application instead of spec.source. This allows you to specify multiple sources, and if they produce the same resource (same group, kind, name, and namespace), the last source to produce the resource will take precedence.

Pros:

Because this feature is part of the ArgoCD Application definition, it supports all available ArgoCD application sources out of the box

Cons:

Only complete resource overwrites are possible
This is a beta feature. The UI and CLI still generally behave as if only the first source is specified
Rollback is not supported for applications with multiple sources

Here is an example of using a Helm Chart from upstream with our custom values.yaml from Git, along with resources from plain manifests overwriting on top.

$ cat apps/_installed/multiple-sources.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: multiple-sources
  namespace: argocd
spec:
  project: default
  sources:
    - chart: authentik
      repoURL: https://charts.goauthentik.io
      targetRevision: 2023.10.5
      helm:
        valueFiles:
          - $values/apps/multiple-sources/values.yaml
    - repoURL: https://github.com/alezkv/unfork-with-argocd
      targetRevision: HEAD
      ref: values
    - repoURL: https://github.com/alezkv/unfork-with-argocd
      path: apps/multiple-sources/resources/
      targetRevision: HEAD
  destination:
    server: "https://kubernetes.default.svc"
    namespace: multiple-sources

You can combine any sources supported by ArgoCD. For example, you could obtain external software from Kustomize, then apply a local Helm Chart, and finally apply a couple of plain manifests on top.

Umbrella chart

This technique utilizes the dependency feature of Helm Chart. You can use multiple charts as dependencies and also embed their configurations within an umbrella chart.

You need to have the Application and the umbrella chart.

$ tree apps

apps
├── _installed
│   └── chart-umbrella.yaml
└── chart-umbrella
    ├── Chart.yaml
    ├── templates
    │   └── configmap.yaml
    └── values.yaml

$ cat apps/_installed/chart-umbrella.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: chart-umbrella
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://github.com/alezkv/unfork-with-argocd
      path: apps/chart-umbrella/
      targetRevision: HEAD
  destination:
    server: "https://kubernetes.default.svc"
    namespace: chart-umbrella

$ cat apps/chart-umbrella/Chart.yaml

apiVersion: v1
version: 1.0.0
name: my-podinfo

dependencies:
  - name: podinfo
    version: 6.5.4
    repository: https://stefanprodan.github.io/podinfo

$ cat apps/chart-umbrella/values.yaml

podinfo:  # Values for the sub-chart must be under its dependency name key
  replicaCount: 2

This setup will use the upstream chart, apply any specified values to it, and, on top of that, add resources from the umbrella chart's template directory.

Kustomize them all

Kustomize alone deserves a dedicated series; let's try to stick with the unforked approach for now. The whole nature of it is to add, remove, or update configuration options without forking.

Hydrate chart with Kustomize

It's possible to render a Helm Chart with Kustomize. This approach allows for additional and highly tunable ways to handle Helm Charts. However, this feature isn't enabled by default and requires custom configuration. You can find more details on this matter here and here

To use Kustomize, you'll need to create a kustomization.yaml file and point the ArgoCD Application to its directory.

$ tree apps

apps
├── _installed
│   └── kustomize.yaml
└── kustomize
    ├── cloudflare-api-token.yaml
    ├── cloudflare-issuer.yaml
    └── kustomization.yaml

$ cat apps/_installed/kustomize.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kustomize
  namespace: argocd
spec:
  project: default
  source:
    path: apps/kustomize
    repoURL: https://github.com/alezkv/unfork-with-argocd
    targetRevision: HEAD
  destination:
    namespace: kustomize
    server: https://kubernetes.default.svc

$ cat apps/kustomize/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io
    releaseName: cert-manager
    namespace: kustomize
    version: v1.13.3
    includeCRDs: true
    valuesInline:
      installCRDs: true

resources:
  - cloudflare-api-token.yaml
  - cloudflare-issuer.yaml

In this setup, you can add or replace resources in an existing chart using resources:. This will work the same way as with multiple application sources, overwriting the 'same' resources. However, you can also utilize the full transformation capabilities of Kustomize for more precise resource manipulation. This could include replacement, patches, or others methods

Don't forget that the source for resources in Kustomize could be remote Git repositories with their own Kustomize or plain Kubernetes manifests

What to pick

Multiple sources in ArgoCD are great for merging separate configurations, best for complete resource overwrites. Umbrella charts, using Helm's dependencies, offer structured management of complex deployments, ideal for hierarchical configuration integration. Kustomize, with its detailed customization capabilities, excels in precise resource manipulation for nuanced adjustments.

Final Thoughts

Kubernetes and ArgoCD offer tremendous flexibility and power for managing containerized applications, but this comes with the responsibility of having a well-defined strategy and vision. Without a clear direction, the complexity of these tools can become a hindrance rather than an advantage. Therefore, it's essential to strike a balance between leveraging the flexibility they provide and maintaining a clear and purposeful approach to application deployment and management