DEV Community

Cover image for Getting Started on MOCO Part 2: Backup and Restore to MinIO (and the davfs2 Bug That Broke It)
Mucheru Maina
Mucheru Maina

Posted on

Getting Started on MOCO Part 2: Backup and Restore to MinIO (and the davfs2 Bug That Broke It)

#kubernetes #devops #mysql #moco

A 74GB dump finished in 1h23m without a single warning. Then the upload to MinIO died with IncompleteBody, every time, on every retry. Not once did the dump itself fail. It was always the same spot, right after "Dumping data - done."

Part 1 covered what MOCO is and how to stand up a cluster. This part covers wiring that cluster to a real backup target: MinIO backed by a Hetzner storagebox, and the bug that took down backups on two production clusters at once until we found it.

Standing up MinIO on the storagebox

We don't run MinIO against local disk, and we don't run the official MinIO Helm chart or Operator either. It's a plain Kubernetes Deployment running a custom image: Ubuntu base, with davfs2, the minio server binary, and the mc client all installed via a Dockerfile, plus an entrypoint script that mounts the Hetzner storagebox over WebDAV/FUSE before starting the MinIO server on top of that mount. The image lives in a private registry, referenced in the Deployment spec like any other container image.

That custom-image choice is the whole reason the davfs2 bug below exists in the first place. The official MinIO Operator expects to own real block storage per pod. Ours hands MinIO a FUSE-mounted network filesystem instead, and FUSE doesn't know or care about MinIO's multipart upload semantics sitting on top of it.

Stripped down, the Dockerfile looks like this:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install -y davfs2 ca-certificates wget && \
    rm -rf /var/lib/apt/lists/*

RUN wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /usr/bin/minio && \
    chmod +x /usr/bin/minio

RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/bin/mc && \
    chmod +x /usr/bin/mc

COPY mount-webdav.sh /mount-webdav.sh
RUN chmod +x /mount-webdav.sh
CMD ["/mount-webdav.sh"]
Enter fullscreen mode Exit fullscreen mode

mount-webdav.sh does the actual work at container startup: mounts the storagebox over WebDAV with davfs2, then execs the MinIO server against that mount point. Note there's no version pinning on either the minio or mc binary, so a rebuild today pulls whatever's current at dl.min.io that day, not necessarily what was running yesterday. That's a separate footgun from the one below, but it's bitten us too.

The Dockerfile, the entrypoint script, and the davfs2 tuning from the incident below now all live in one place: github.com/muchezz/minio-mounter. If you're running MOCO against a Hetzner storagebox (or any other WebDAV-only backend) rather than building your own version of this from scratch, start there. The image built from that repo is what the Deployment below actually references.

One caveat before the manifests: we manage our whole fleet with Ansible, so that's what deploys this custom image here. MOCO doesn't care how MinIO gets onto the cluster. The Helm chart, the official MinIO Operator, a raw StatefulSet, any of it works the same from MOCO's side, since all that matters to a BackupPolicy is the resulting S3-compatible endpoint. Swap the Ansible role below for whatever your team already uses to deploy workloads.

Deployment is an Ansible role. Add a minio_storagebox block to the k8s variables file, for example group_vars/k8s_cluster/k8s_cluster.yml:

minio_storagebox:
  storagebox: "u000000" # storagebox where to keep the data
  storagebox_password: "{{ vault_storagebox.u000000 }}" # storagebox password
  directory: "minio-data" # folder where data is stored
  minio_root_password: "{{ vault_relevant_var }}" # root password from minio
  minio_kms_secret_key: "{{ vault_relevant_var }}" # encryption key for at-rest encryption. Generate with `openssl rand -base64 32`, then set as encryption-key:generated_secret_key
Enter fullscreen mode Exit fullscreen mode

Sensitive values go in the vault, not in plain group_vars. The custom mounter image also needs a pull token, which lives in defaults/main.yml:dockerconfigjson_b64. If it ever expires, that's a separate runbook (renewing GitLab deploy tokens on k8s).

The role's tasks do five things, in order:

  1. Pre-create the remote directory over SFTP, delegated to localhost, before anything touches Kubernetes. WebDAV mounting a directory that doesn't exist yet on the storagebox fails, so this runs first and is allowed to fail quietly if the directory's already there.
  2. Create the namespace, minio-storagebox by default.
  3. Create a secret holding minio_root_user, minio_root_password, minio_kms_secret_key, and a combined secrets string (the WebDAV URL plus storagebox credentials, which the mount script reads on startup).
  4. Create a dockerconfigjson pull secret named regcred, since the mounter image sits in a private registry.
  5. Create the Deployment and Service.

The Deployment template is the part that actually matters for how MinIO ends up sitting on FUSE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ name }}-deployment"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "{{ name }}"
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: "{{ name }}"
    spec:
      containers:
        - name: minioclient
          image: ghcr.io/muchezz/minio-mounter:latest
          securityContext:
            privileged: true
          env:
            - name: WEBDAV_URL
              value: "{{ webdav_url }}"
            - name: MOUNT_POINT
              value: "/data"
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_root_user
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_root_password
            - name: MINIO_KMS_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_kms_secret_key
          volumeMounts:
            - name: data
              mountPath: /data
            - name: secrets
              mountPath: /secrets
              subPath: secrets
            - name: fuse
              mountPath: /dev/fuse
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: data
          emptyDir: {}
        - name: secrets
          secret:
            secretName: "{{ secretname }}"
        - name: fuse
          hostPath:
            path: /dev/fuse
Enter fullscreen mode Exit fullscreen mode

Two lines here are load-bearing and easy to miss on a skim: securityContext.privileged: true, and the fuse volume mounting the node's /dev/fuse device straight into the container via hostPath. davfs2 mounts a FUSE filesystem, and FUSE needs access to that device node plus enough privilege to actually create the mount from inside the container. Without both, the entrypoint script's mount -t davfs call just fails at startup. /data itself is a plain emptyDir; nothing is persisted there directly, it's the mount point the WebDAV filesystem gets attached to once the container starts.

The Service is unremarkable by comparison, just the S3 API port and the console port:

apiVersion: v1
kind: Service
metadata:
  name: "{{ name }}-service"
spec:
  selector:
    app: "{{ name }}"
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9000
      name: s3
    - protocol: TCP
      port: 9001
      targetPort: 9001
      name: console
  type: ClusterIP
Enter fullscreen mode Exit fullscreen mode

Port 80 on the Service forwards to MinIO's S3 API on 9000 inside the pod, which is why the endpoint_url MOCO talks to later is just http://minio-storagebox-service.minio-storagebox.svc with no port. Console traffic on 9001 stays separate, and we only ever reach it through a port-forward, never a Service exposed outside the cluster.

Add the role to the relevant playbook:

- hosts: [ k8s_cluster ]
  strategy: linear
  roles:
    - { role: some/other-role, tags: some-other-tag }
    - { role: k8s/minio-storagebox, tags: k8s-minio-storagebox }
Enter fullscreen mode Exit fullscreen mode

Then run it against the target cluster:

ansible-playbook k8s_cluster.yml -i inventory.k8s_cluster -t k8s-minio-storagebox -D
Enter fullscreen mode Exit fullscreen mode

That deploys MinIO. Next, create the bucket MOCO will actually write to. Port-forward the console instead of exposing it:

kubectl -n minio-storagebox port-forward svc/minio-storagebox-service 9001
Enter fullscreen mode Exit fullscreen mode

Log in with minio / the root password from the vault, create a bucket, and turn on encryption: SSE-KMS with the preconfigured encryption-key. Once that's done, the bucket behaves like any S3-compatible target:

endpoint_url: http://minio-storagebox-service.minio-storagebox.svc
aws_access_key_id: minio
aws_secret_access_key: minio_root_password
bucket: whateveryoucreated
Enter fullscreen mode Exit fullscreen mode

Pointing MOCO at the bucket

MOCO doesn't take a bucket config on the cluster spec directly. It goes through a BackupPolicy CR, referenced from MySQLCluster.spec.backupPolicyName. One BackupPolicy can be shared across multiple clusters, since MOCO prefixes object keys with moco/ so they don't collide.

apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  namespace: app-db
  name: daily
spec:
  schedule: "0 3 * * *"
  jobConfig:
    serviceAccountName: backup-owner
    env:
      - name: AWS_ACCESS_KEY_ID
        value: minio
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: minio-backup-creds
            key: secret-access-key
    bucketConfig:
      bucketName: app-db-backups
      region: us-east-1
      endpointURL: http://minio-storagebox-service.minio-storagebox.svc
      usePathStyle: true
Enter fullscreen mode Exit fullscreen mode

usePathStyle: true matters here. MinIO expects endpoint/bucket/key, not the virtual-hosted bucket.endpoint/key style S3 defaults to, and skipping that flag means every request 404s before it even gets to the davfs2 layer.

Reference the policy from the cluster:

spec:
  backupPolicyName: daily
Enter fullscreen mode Exit fullscreen mode

moco-controller turns that into a CronJob. Each run dumps the instance with MySQL Shell's dump utility, tars it, pushes it to the bucket, then separately ships binlogs since the last backup for point-in-time recovery. Two objects per run, not one.

Restoring

Restore isn't a flag on the existing cluster. You stand up a new MySQLCluster with a spec.restore block pointing at where the old backup lives:

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  namespace: app-db
  name: app-db-restored
spec:
  restore:
    sourceNamespace: app-db
    sourceName: app-db
    restorePoint: "2026-07-28T02:15:00Z"
    jobConfig:
      serviceAccountName: backup-owner
      env:
        - name: AWS_ACCESS_KEY_ID
          value: minio
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-backup-creds
              key: secret-access-key
      bucketConfig:
        bucketName: app-db-backups
        region: us-east-1
        endpointURL: http://minio-storagebox-service.minio-storagebox.svc
        usePathStyle: true
    workVolume:
      emptyDir: {}
Enter fullscreen mode Exit fullscreen mode

sourceNamespace/sourceName don't need the original cluster to still exist. They're just how MOCO locates the right prefix in the bucket. restorePoint in RFC3339 gives you PITR: MOCO restores the last full backup before that timestamp, then replays binlogs up to it.

The upload that never finished

Backups ran clean for weeks. Then app-db's nightly job started failing at the same point every night: dump completes, upload starts, upload dies.

IncompleteBody: The request body terminated unexpectedly
Enter fullscreen mode Exit fullscreen mode

Same error, same stage, every retry. That ruled out a one-off network blip; something structural was cutting the stream short partway through the multipart upload.

The mounter pod runs davfs2 with defaults, and nobody had ever had a reason to look at them:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- \
  cat /etc/davfs2/davfs2.conf | grep -E 'cache_size|buf_size|delay_upload'
Enter fullscreen mode Exit fullscreen mode 
buf_size        16                 # KiByte  (commented out, default)
cache_size      50                # MiByte  (commented out, default)
delay_upload    10                (commented out, default)
Enter fullscreen mode Exit fullscreen mode

All three commented out, all three running on the default. A 50MB cache trying to buffer a 74GB multipart upload. MOCO's backup client declares a Content-Length per part; davfs2 was flushing its cache mid-part and handing MinIO fewer bytes than promised. MinIO had no choice but to reject it.

Fix, applied live inside the running pod first to confirm it worked before baking it into the image:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- bash -c '
cat >> /etc/davfs2/davfs2.conf << EOF

cache_size      4096
buf_size        64
delay_upload    0
use_locks       0
EOF
'
Enter fullscreen mode Exit fullscreen mode

davfs2 only reads its config at mount time, so the change doesn't do anything until you remount:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- bash -c '
umount -l /data && sleep 5 && mount -t davfs $WEBDAV_URL /data
'
Enter fullscreen mode Exit fullscreen mode

Clear the incomplete upload MinIO was still holding onto, then re-run the job:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- \
  mc rm --incomplete --recursive --force local/backups/moco/app-db/

kubectl create job moco-backup-app-db-manual-$(date +%s) \
  --from=cronjob/moco-backup-app-db -n app-db
Enter fullscreen mode Exit fullscreen mode

Same 74.1GB dump, same duration, but this time the upload went through. delay_upload 0 turned out to matter almost as much as the bigger cache. With the default 10-second delay, davfs2 was batching writes in a way that made the mid-stream flush worse, not better, once the upload actually got large.

That fix lived in the running pod, which meant it wouldn't survive a restart. The permanent version went into the custom mounter image so every fresh pod starts with sane davfs2 settings instead of relying on someone remembering to patch it again at 3am.

What we watch now

Two things came out of chasing this. First, df -h on the mount mid-upload is worth watching before a backup fails, not after. Cache pressure shows up there before MinIO ever throws an error. Second, we alert on backup job duration now, not just success/failure. A job that silently starts taking longer, or one where the upload phase drags relative to the dump phase, is usually the cache filling up again before it's bad enough to fail outright.

MOCO itself never lied to us here. The CR status, the CronJob, the job logs all pointed at the right layer once we looked. The failure was one level down, in infrastructure MOCO doesn't know exists and has no reason to. Worth remembering when the operator's own status says everything's fine and the backup still isn't landing.

Ref - https://cybozu-go.github.io/moco/backup.html

Top comments (0)