DEV Community: Mucheru Maina

Getting Started on MOCO Part 2: Backup and Restore to MinIO (and the davfs2 Bug That Broke It)

Mucheru Maina — Fri, 31 Jul 2026 12:39:22 +0000

A 74GB dump finished in 1h23m without a single warning. Then the upload to MinIO died with IncompleteBody, every time, on every retry. Not once did the dump itself fail. It was always the same spot, right after "Dumping data - done."

Part 1 covered what MOCO is and how to stand up a cluster. This part covers wiring that cluster to a real backup target: MinIO backed by a Hetzner storagebox, and the bug that took down backups on two production clusters at once until we found it.

Standing up MinIO on the storagebox

We don't run MinIO against local disk, and we don't run the official MinIO Helm chart or Operator either. It's a plain Kubernetes Deployment running a custom image: Ubuntu base, with davfs2, the minio server binary, and the mc client all installed via a Dockerfile, plus an entrypoint script that mounts the Hetzner storagebox over WebDAV/FUSE before starting the MinIO server on top of that mount. The image lives in a private registry, referenced in the Deployment spec like any other container image.

That custom-image choice is the whole reason the davfs2 bug below exists in the first place. The official MinIO Operator expects to own real block storage per pod. Ours hands MinIO a FUSE-mounted network filesystem instead, and FUSE doesn't know or care about MinIO's multipart upload semantics sitting on top of it.

Stripped down, the Dockerfile looks like this:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install -y davfs2 ca-certificates wget && \
    rm -rf /var/lib/apt/lists/*

RUN wget https://dl.min.io/server/minio/release/linux-amd64/minio -O /usr/bin/minio && \
    chmod +x /usr/bin/minio

RUN wget https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/bin/mc && \
    chmod +x /usr/bin/mc

COPY mount-webdav.sh /mount-webdav.sh
RUN chmod +x /mount-webdav.sh
CMD ["/mount-webdav.sh"]

mount-webdav.sh does the actual work at container startup: mounts the storagebox over WebDAV with davfs2, then execs the MinIO server against that mount point. Note there's no version pinning on either the minio or mc binary, so a rebuild today pulls whatever's current at dl.min.io that day, not necessarily what was running yesterday. That's a separate footgun from the one below, but it's bitten us too.

The Dockerfile, the entrypoint script, and the davfs2 tuning from the incident below now all live in one place: github.com/muchezz/minio-mounter. If you're running MOCO against a Hetzner storagebox (or any other WebDAV-only backend) rather than building your own version of this from scratch, start there. The image built from that repo is what the Deployment below actually references.

One caveat before the manifests: we manage our whole fleet with Ansible, so that's what deploys this custom image here. MOCO doesn't care how MinIO gets onto the cluster. The Helm chart, the official MinIO Operator, a raw StatefulSet, any of it works the same from MOCO's side, since all that matters to a BackupPolicy is the resulting S3-compatible endpoint. Swap the Ansible role below for whatever your team already uses to deploy workloads.

Deployment is an Ansible role. Add a minio_storagebox block to the k8s variables file, for example group_vars/k8s_cluster/k8s_cluster.yml:

minio_storagebox:
  storagebox: "u000000" # storagebox where to keep the data
  storagebox_password: "{{ vault_storagebox.u000000 }}" # storagebox password
  directory: "minio-data" # folder where data is stored
  minio_root_password: "{{ vault_relevant_var }}" # root password from minio
  minio_kms_secret_key: "{{ vault_relevant_var }}" # encryption key for at-rest encryption. Generate with `openssl rand -base64 32`, then set as encryption-key:generated_secret_key

Sensitive values go in the vault, not in plain group_vars. The custom mounter image also needs a pull token, which lives in defaults/main.yml:dockerconfigjson_b64. If it ever expires, that's a separate runbook (renewing GitLab deploy tokens on k8s).

The role's tasks do five things, in order:

Pre-create the remote directory over SFTP, delegated to localhost, before anything touches Kubernetes. WebDAV mounting a directory that doesn't exist yet on the storagebox fails, so this runs first and is allowed to fail quietly if the directory's already there.
Create the namespace, minio-storagebox by default.
Create a secret holding minio_root_user, minio_root_password, minio_kms_secret_key, and a combined secrets string (the WebDAV URL plus storagebox credentials, which the mount script reads on startup).
Create a dockerconfigjson pull secret named regcred, since the mounter image sits in a private registry.
Create the Deployment and Service.

The Deployment template is the part that actually matters for how MinIO ends up sitting on FUSE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ name }}-deployment"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "{{ name }}"
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: "{{ name }}"
    spec:
      containers:
        - name: minioclient
          image: ghcr.io/muchezz/minio-mounter:latest
          securityContext:
            privileged: true
          env:
            - name: WEBDAV_URL
              value: "{{ webdav_url }}"
            - name: MOUNT_POINT
              value: "/data"
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_root_user
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_root_password
            - name: MINIO_KMS_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: "{{ secretname }}"
                  key: minio_kms_secret_key
          volumeMounts:
            - name: data
              mountPath: /data
            - name: secrets
              mountPath: /secrets
              subPath: secrets
            - name: fuse
              mountPath: /dev/fuse
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: data
          emptyDir: {}
        - name: secrets
          secret:
            secretName: "{{ secretname }}"
        - name: fuse
          hostPath:
            path: /dev/fuse

Two lines here are load-bearing and easy to miss on a skim: securityContext.privileged: true, and the fuse volume mounting the node's /dev/fuse device straight into the container via hostPath. davfs2 mounts a FUSE filesystem, and FUSE needs access to that device node plus enough privilege to actually create the mount from inside the container. Without both, the entrypoint script's mount -t davfs call just fails at startup. /data itself is a plain emptyDir; nothing is persisted there directly, it's the mount point the WebDAV filesystem gets attached to once the container starts.

The Service is unremarkable by comparison, just the S3 API port and the console port:

apiVersion: v1
kind: Service
metadata:
  name: "{{ name }}-service"
spec:
  selector:
    app: "{{ name }}"
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9000
      name: s3
    - protocol: TCP
      port: 9001
      targetPort: 9001
      name: console
  type: ClusterIP

Port 80 on the Service forwards to MinIO's S3 API on 9000 inside the pod, which is why the endpoint_url MOCO talks to later is just http://minio-storagebox-service.minio-storagebox.svc with no port. Console traffic on 9001 stays separate, and we only ever reach it through a port-forward, never a Service exposed outside the cluster.

Add the role to the relevant playbook:

- hosts: [ k8s_cluster ]
  strategy: linear
  roles:
    - { role: some/other-role, tags: some-other-tag }
    - { role: k8s/minio-storagebox, tags: k8s-minio-storagebox }

Then run it against the target cluster:

ansible-playbook k8s_cluster.yml -i inventory.k8s_cluster -t k8s-minio-storagebox -D

That deploys MinIO. Next, create the bucket MOCO will actually write to. Port-forward the console instead of exposing it:

kubectl -n minio-storagebox port-forward svc/minio-storagebox-service 9001

Log in with minio / the root password from the vault, create a bucket, and turn on encryption: SSE-KMS with the preconfigured encryption-key. Once that's done, the bucket behaves like any S3-compatible target:

endpoint_url: http://minio-storagebox-service.minio-storagebox.svc
aws_access_key_id: minio
aws_secret_access_key: minio_root_password
bucket: whateveryoucreated

Pointing MOCO at the bucket

MOCO doesn't take a bucket config on the cluster spec directly. It goes through a BackupPolicy CR, referenced from MySQLCluster.spec.backupPolicyName. One BackupPolicy can be shared across multiple clusters, since MOCO prefixes object keys with moco/ so they don't collide.

apiVersion: moco.cybozu.com/v1beta2
kind: BackupPolicy
metadata:
  namespace: app-db
  name: daily
spec:
  schedule: "0 3 * * *"
  jobConfig:
    serviceAccountName: backup-owner
    env:
      - name: AWS_ACCESS_KEY_ID
        value: minio
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: minio-backup-creds
            key: secret-access-key
    bucketConfig:
      bucketName: app-db-backups
      region: us-east-1
      endpointURL: http://minio-storagebox-service.minio-storagebox.svc
      usePathStyle: true

usePathStyle: true matters here. MinIO expects endpoint/bucket/key, not the virtual-hosted bucket.endpoint/key style S3 defaults to, and skipping that flag means every request 404s before it even gets to the davfs2 layer.

Reference the policy from the cluster:

spec:
  backupPolicyName: daily

moco-controller turns that into a CronJob. Each run dumps the instance with MySQL Shell's dump utility, tars it, pushes it to the bucket, then separately ships binlogs since the last backup for point-in-time recovery. Two objects per run, not one.

Restoring

Restore isn't a flag on the existing cluster. You stand up a new MySQLCluster with a spec.restore block pointing at where the old backup lives:

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  namespace: app-db
  name: app-db-restored
spec:
  restore:
    sourceNamespace: app-db
    sourceName: app-db
    restorePoint: "2026-07-28T02:15:00Z"
    jobConfig:
      serviceAccountName: backup-owner
      env:
        - name: AWS_ACCESS_KEY_ID
          value: minio
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-backup-creds
              key: secret-access-key
      bucketConfig:
        bucketName: app-db-backups
        region: us-east-1
        endpointURL: http://minio-storagebox-service.minio-storagebox.svc
        usePathStyle: true
    workVolume:
      emptyDir: {}

sourceNamespace/sourceName don't need the original cluster to still exist. They're just how MOCO locates the right prefix in the bucket. restorePoint in RFC3339 gives you PITR: MOCO restores the last full backup before that timestamp, then replays binlogs up to it.

The upload that never finished

Backups ran clean for weeks. Then app-db's nightly job started failing at the same point every night: dump completes, upload starts, upload dies.

IncompleteBody: The request body terminated unexpectedly

Same error, same stage, every retry. That ruled out a one-off network blip; something structural was cutting the stream short partway through the multipart upload.

The mounter pod runs davfs2 with defaults, and nobody had ever had a reason to look at them:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- \
  cat /etc/davfs2/davfs2.conf | grep -E 'cache_size|buf_size|delay_upload'

buf_size        16                 # KiByte  (commented out, default)
cache_size      50                # MiByte  (commented out, default)
delay_upload    10                (commented out, default)

All three commented out, all three running on the default. A 50MB cache trying to buffer a 74GB multipart upload. MOCO's backup client declares a Content-Length per part; davfs2 was flushing its cache mid-part and handing MinIO fewer bytes than promised. MinIO had no choice but to reject it.

Fix, applied live inside the running pod first to confirm it worked before baking it into the image:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- bash -c '
cat >> /etc/davfs2/davfs2.conf << EOF

cache_size      4096
buf_size        64
delay_upload    0
use_locks       0
EOF
'

davfs2 only reads its config at mount time, so the change doesn't do anything until you remount:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- bash -c '
umount -l /data && sleep 5 && mount -t davfs $WEBDAV_URL /data
'

Clear the incomplete upload MinIO was still holding onto, then re-run the job:

kubectl exec -n minio-storagebox <minio-mounter-pod> -- \
  mc rm --incomplete --recursive --force local/backups/moco/app-db/

kubectl create job moco-backup-app-db-manual-$(date +%s) \
  --from=cronjob/moco-backup-app-db -n app-db

Same 74.1GB dump, same duration, but this time the upload went through. delay_upload 0 turned out to matter almost as much as the bigger cache. With the default 10-second delay, davfs2 was batching writes in a way that made the mid-stream flush worse, not better, once the upload actually got large.

That fix lived in the running pod, which meant it wouldn't survive a restart. The permanent version went into the custom mounter image so every fresh pod starts with sane davfs2 settings instead of relying on someone remembering to patch it again at 3am.

What we watch now

Two things came out of chasing this. First, df -h on the mount mid-upload is worth watching before a backup fails, not after. Cache pressure shows up there before MinIO ever throws an error. Second, we alert on backup job duration now, not just success/failure. A job that silently starts taking longer, or one where the upload phase drags relative to the dump phase, is usually the cache filling up again before it's bad enough to fail outright.

MOCO itself never lied to us here. The CR status, the CronJob, the job logs all pointed at the right layer once we looked. The failure was one level down, in infrastructure MOCO doesn't know exists and has no reason to. Worth remembering when the operator's own status says everything's fine and the backup still isn't landing.

Ref - https://cybozu-go.github.io/moco/backup.html

Getting started on MOCO, the MySQL Operator for Kubernetes Part 1

Mucheru Maina — Tue, 14 Jan 2025 05:59:49 +0000

MOCO (MySQL Operator for Kubernetes) is a robust, cloud-native solution designed to simplify the management of MySQL clusters in Kubernetes environments. It automates the provisioning, scaling, backup, and maintenance of MySQL instances while ensuring high availability and reliability. MOCO leverages Kubernetes resources to create, monitor, and manage MySQL clusters.

MOCO supports specific versions of MySQL and Kubernetes. As of the latest information, it supports MySQL versions 8.0.28, 8.0.37, 8.0.39, 8.0.40, and 8.4.3, and Kubernetes versions 1.29, 1.30, and 1.31

How MOCO Works

Provisioning Clusters

MOCO provisions MySQL clusters by creating Kubernetes StatefulSets for each cluster. The process involves:
Defining a MySQLCluster custom resource (CR) with desired configurations.
MOCO controller creates StatefulSets and persistent volume claims (PVCs) for the cluster nodes.
Configuring MySQL instances with semi-synchronous replication for high availability.

Deployment Services

MOCO deploys the following services:

Primary Service: Routes traffic to the primary node.
Replica Service: Routes traffic to replica nodes for read queries.
Backup Service: Handles backups through sidecars integrated into the StatefulSet.

Types of Deployment

Single Primary with Replicas: Default mode with one primary and multiple replicas.
Multi-Region Clusters: For cross-region replication.

Backup and restore

MOCO can take full and incremental backups regularly. The backup data are stored in Amazon S3 compatible object storages.
MOCO supports:

Scheduled backups to object storage (e.g., S3).
On-demand backups triggered through the Kubernetes API.
Restorations from backups via simple CR updates.
Point-in-Time Recovery: Ensures robust data protection in disaster recovery scenarios.

Object storage bucket
Bucket is a management unit of objects in S3. MOCO stores backups in a specified bucket.

MOCO does not remove backups. To remove old backups automatically, you can set a lifecycle configuration to the bucket.

Quick setup

You can choose between two installation methods.

MOCO depends on cert-manager. If cert-manager is not installed on your cluster, install it as follows:

Install using raw manifests:
$ curl -fsLO https://github.com/cybozu-go/moco/releases/latest/download/moco.yaml $ kubectl apply -f moco.yaml

Install using Helm chart:
$ helm repo add moco https://cybozu-go.github.io/moco/ $ helm repo update $ helm install --create-namespace --namespace moco-system moco moco/moco

Customize manifests
If you want to edit the manifest, config/ directory contains the source YAML for kustomize.

Creating a Cluster

An empty cluster always has a writable instance called the primary. All other instances are called replicas. Replicas are read-only and replicate data from the primary.

The following YAML is to create a three-instance cluster. It has an anti-affinity for Pods so that all instances will be scheduled to different Nodes. It also sets the limits for memory and CPU to make the Pod Guaranteed.

apiVersion: moco.cybozu.com/v1beta2
kind: MySQLCluster
metadata:
  namespace: default
  name: test
spec:
  # replicas is the number of mysqld Pods.  The default is 1.
  replicas: 3
  podTemplate:
    spec:
      # Make the data directory writable. If moco-init fails with "Permission denied", uncomment the following settings.
      # securityContext:
      #   fsGroup: 10000
      #   fsGroupChangePolicy: "OnRootMismatch"  # available since k8s 1.20
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                - mysql
              - key: app.kubernetes.io/instance
                operator: In
                values:
                - test
            topologyKey: "kubernetes.io/hostname"
      containers:
      # At least a container named "mysqld" must be defined.
      - name: mysqld
        image: ghcr.io/cybozu-go/moco/mysql:8.4.3
        # By limiting CPU and memory, Pods will have Guaranteed QoS class.
        # requests can be omitted; it will be set to the same value as limits.
        resources:
          limits:
            cpu: "10"
            memory: "10Gi"
  volumeClaimTemplates:
  # At least a PVC named "mysql-data" must be defined.
  - metadata:
      name: mysql-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

By default, MOCO uses preferredDuringSchedulingIgnoredDuringExecution to prevent Pods from being placed on the same Node.
There are other example manifests in thier examples directory.

Using the cluster

kubectl moco
From outside of your Kubernetes cluster, you can access MOCO MySQL instances using kubectl-moco. kubectl-moco is a plugin for kubectl. Pre-built binaries are available on GitHub releases.

The following is an example to run mysql command interactively to access the primary instance of test MySQLCluster in foo namespace.

$ kubectl moco -n foo mysql -it test

Connecting to mysqld over network
MOCO prepares two Services for each MySQLCluster. For example, a MySQLCluster named test in foo Namespace has the following Services.

Service Name DNS Name Description moco-test-primary moco-test-primary.foo.svc Connect to the primary instance. moco-test-replica moco-test-replica.foo.svc Connect to replica instances.

Cluster status

You can see the health and availability status of MySQLCluster as follows:

$ kubectl get mysqlcluster NAME AVAILABLE HEALTHY PRIMARY SYNCED REPLICAS ERRANT REPLICAS test True True 0 3

The cluster is available when the primary Pod is running and ready.
The cluster is healthy when there is no problems.
PRIMARY is the index of the current primary instance Pod.
SYNCED REPLICAS is the number of ready Pods.
ERRANT REPLICAS is the number of instances having errant transactions. You can also use kubectl describe mysqlclusterto see the recent events on the cluster

Logs

Error logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 mysqld
Slow logs from mysqld can be viewed as follows:

$ kubectl logs moco-test-0 slow-log

Switchover
Switchover is an operation to change the live primary to one of the replicas.

MOCO automatically switch the primary when the Pod of the primary instance is to be deleted.

Users can manually trigger a switchover with kubectl moco switchover CLUSTER_NAME.

Failover
Failover is an operation to replace the dead primary with the most advanced replica. MOCO automatically does this as soon as it detects that the primary is down.

The most advanced replica is a replica who has retrieved the most up-to-date transaction from the dead primary. Since MOCO configures loss-less semi-synchronous replication, the failover is guaranteed not to lose any user data.

Re-initializing an errant replica
Delete the PVC and Pod of the errant replica, like this:

$ kubectl delete --wait=false pvc mysql-data-moco-test-0 $ kubectl delete --grace-period=1 pods moco-test-0
Depending on your Kubernetes version, StatefulSet controller may create a pending Pod before PVC gets deleted. Delete such pending Pods until PVC is actually removed.

Ref - https://cybozu-go.github.io/moco/index.html

What is Kured (KUbernetes REboot Daemon) in k8s?

Mucheru Maina — Fri, 01 Mar 2024 11:34:08 +0000

Defination from the official page states that kured is a Kubernetes daemonset that performs safe automatic node reboots when the need to do so is indicated by the package management system of the underlying OS.

By periodically rebooting nodes, Kured ensures that any pending updates or configuration changes take effect, resulting in a more efficient and reliable cluster.

Here are the key points to note:

Kured monitors the operating system for security patches, kernel updates, and system-level changes in Kubernetes nodes. It proactively identifies the need for reboots to keep the cluster secure and up-to-date.
When a reboot is required, Kured gracefully cordons the node, marking it as unschedulable for new pods without disrupting existing ones. It then proceeds to drain the node, evicting existing pods in a controlled manner to ensure a smooth reboot process.
Kured includes built-in safety mechanisms to prevent unnecessary reboots and allows users to define maintenance windows for avoiding disruptions.

The continuous monitoring by Kured ensures that the Kubernetes cluster operates with the latest updates, enhancing performance, security, and stability.
Organizations can leverage Kubernetes clusters more effectively while minimizing risks associated with outdated software and configurations.

Setting up Kured is a straightforward process that involves deploying it as a DaemonSet in the Kubernetes cluster. This deployment strategy ensures that Kured runs on every node within the cluster, effectively monitoring and managing the rebooting process for each individual node.

Here is how you can do that

# ClusterRole for kured
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kured
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "delete", "get"]
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]

# ClusterRoleBinding for kured
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kured
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kured
subjects:
- kind: ServiceAccount
  name: kured
  namespace: kube-system

# Role for kured in kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: kube-system
  name: kured
rules:
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  resourceNames: ["kured"]
  verbs: ["update"]

# RoleBinding for kured in kube-system namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-system
  name: kured
subjects:
- kind: ServiceAccount
  namespace: kube-system
  name: kured
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kured

# ServiceAccount for kured
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kured
  namespace: kube-system

# DaemonSet for kured
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kured
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: kured
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: kured
    spec:
      serviceAccountName: kured
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: "node-role.kubernetes.io/mysql"
        operator: "Equal"
        effect: "NoSchedule"
      hostPID: true
      restartPolicy: Always
      containers:
      - name: kured
        image: ghcr.io/kubereboot/kured:{{ kured_version }}
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        env:
        - name: KURED_NODE_ID
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        command:
        - /usr/bin/kured
        - --reboot-days=mon,tue,wed,thu
        - --reboot-delay=90s
        - --start-time=3am
        - --end-time=5am
        - --time-zone=UTC
        - --prometheus-url={{ prometheus_url }}
        - --alert-filter-regexp=^Watchdog$
        - --period=15m

Explanation:
This part allows you to define maintenance windows for avoiding disruptions.

  command:
        - /usr/bin/kured
        - --reboot-days=mon,tue,wed,thu
        - --reboot-delay=90s
        - --start-time=3am
        - --end-time=5am
        - --time-zone=UTC
        - --prometheus-url={{ prometheus_url }}
        - --alert-filter-regexp=^Watchdog$
        - --period=15m

Karpenter: The Better Autoscaling Solution for Kubernetes- Part 1

Mucheru Maina — Sat, 25 Feb 2023 04:33:48 +0000

If you're running Kubernetes, you're likely familiar with the standard cluster autoscaler. While it's a useful tool, it has its limitations. Enter Karpenter, an open-source autoscaling solution that offers many advantages over the standard cluster autoscaler. It is a flexible, high-performance Kubernetes cluster autoscaler built with AWS.

How Karpenter Works

Karpenter takes a different approach to autoscaling than the standard cluster autoscaler. Instead of adding or removing nodes based on demand, Karpenter provisions nodes based on application requirements. This means that it can optimize resource utilization and reduce costs.

Karpenter works by creating custom Kubernetes resources called "provisioners." Provisioners are used to define the resources that Karpenter should provision, such as nodes or virtual machines. When an application needs more resources, Karpenter checks the provisioners to see if any need to be created. If so, Karpenter will create the new resources and add them to the cluster.

Image Courtesy.

Comparing Karpenter to Cluster Autoscaler

Karpenter offers several advantages over the standard cluster autoscaler as detailed below.

Optimal Resource Utilization

One of the most significant advantages of Karpenter is its ability to optimize resource utilization. It can do this by automatically provisioning nodes based on application needs. This means that you can avoid overprovisioning, which can lead to wasted resources and increased costs.

Customizable Scaling

Karpenter offers the ability to customize scaling behaviors based on your specific needs. You can configure scaling based on metrics such as CPU or memory usage, or you can use your own custom metrics.

Cost Savings

Because Karpenter optimizes resource utilization, it can lead to significant cost savings. By avoiding overprovisioning, you can reduce the number of nodes required to run your applications, which can result in lower cloud bills.

Ease of Use

Karpenter is easy to use and deploy. It can be installed using Helm, and it integrates seamlessly with Kubernetes.

The standard cluster autoscaler can be more challenging to set up, and it may require more manual configuration.

How to Get Started with Karpenter

There are different ways to get started with Karpenter. This article will just highlight the steps. (Watch out for part 2 with a step-by-step guide on how to install and configure Karpenter) We will use Helm Chart to install Karpernter. Here are some steps before having it operational:

Create the KarpenterNode IAM Role - Instances launched by Karpenter must run with an InstanceProfile that grants permissions necessary to run containers and configure networking.
Create the IAM role for Karpenter Controller - Associate the Kubernetes Service Account and the IAM role using IRSA
Update aws-auth ConfigMap - to allow the nodes that use the KarpenterRole IAM Role to join the cluster
Deploy Karpenter Helm Chart:

helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION}

Create a default Provisioner, (example)

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    intent: apps
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
    - key: karpenter.k8s.aws/instance-size
      operator: NotIn
      values: [nano, micro, small, medium, large]
  limits:
    resources:
      cpu: 1000
      memory: 1000Gi
  ttlSecondsAfterEmpty: 30
  ttlSecondsUntilExpired: 2592000
  providerRef:
    name: default
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  securityGroupSelector:
    alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
  tags:
    KarpenerProvisionerName: "default"
    NodeType: "karpenter-workshop"
    IntentLabel: "apps"
EOF

Once you've installed Karpenter, you can begin using it to optimize your Kubernetes cluster's resource utilization and reduce costs.

Provisioner configuration

Karpenter configuration comes in the form of a Provisioner CRD (Custom Resource Definition). A single Karpenter provisioner is capable of handling many different pod shapes. You can play with it to suit whatever needs you have. For example

One can limit Karpenter to use either on-demand or spot instances, you can use the spot field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  labels:
    type: karpenter
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
        # - key: karpenter.sh/capacity-type
    #   operator: In
    #   values: ["spot"]
    - key: "node.kubernetes.io/instance-type"
      operator: In
      values: ["c5.large", "m5.large", "m5.xlarge"]

In this example we're setting the karpenter.sh/capacity-typeto initially limit Karpenter to provisioning On-Demand instances, and karpenter.k8s.aws/instance-typeto limit to specific instance types.

One can also limit Karpenter to specific instance types, regions and zones you can use the instanceTypes field in the provisioner definition. Here's an example:

apiVersion: karpenter.sh/v1alpha1
kind: Provisioner
metadata:
  name: example-provisioner
spec:
  constraints:
    - type: "awsec2"
      region: "us-west-2"
      zones:
        - "a"
      instanceTypes:
        - "t2.micro"

In this example, the instanceTypes field is set to t2.micro, which means that the provisioner will only use t2.micro instances. You can add additional instance types to the list if you want to allow for more flexibility.

Limitations

While Karpenter offers many advantages over the standard cluster autoscaler, it also has some limitations. Some of the key limitations include:

Limited support for custom metrics: While Karpenter does offer support for custom metrics, it is more limited than some other solutions. This can make it challenging to implement certain types of custom scaling behaviors.
Lack of integration with some cloud providers: Karpenter is designed to work with Kubernetes, but it may not integrate seamlessly with all cloud providers. This can make it more challenging to deploy Karpenter in certain environments.
Complexity: Karpenter is a powerful tool, but it can also be complex to configure and use. It may require more expertise and resources than some other scaling solutions.

Despite these limitations, Karpenter is still an excellent choice for many Kubernetes users. Its ability to optimize resource utilization, customize scaling behaviors, and reduce costs make it a compelling option for many organizations.

Conclusion

Karpenter is a powerful autoscaling solution that offers many advantages over the standard cluster autoscaler. With its ability to optimize resource utilization, customized scaling, and ease of use, Karpenter is a must-have tool for Kubernetes users. Follow the steps above to get started with Karpenter today!

Stay tuned for Part 2 of this blog, where we'll provide a step-by-step guide on how to install and configure Karpenter. Part 2 will be more technical, so if you're interested in getting started with Karpenter, be sure to check it out!

References

Follow me at LinkedIn and Twitter.