When your mysql image refuses to start on Kubernetes

One of the most common deployments of services in k8s has always been a simple mysql-service. You google and you find a thousand examples out there.

But recently I started experimenting with debezium, and it requires a mysql with a custom my.cnf file in order to enable full binary logs.

First I deployed this service on my minikube with no issues at all:

apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  clusterIP: None
  selector:
    app: mysql
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
  labels:
    app: mysql
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - image: debezium/example-mysql:0.10      
        name: mysql
        env:
          # Use secret in real usage
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc

But to my surprise, it would not start on GKE. Instead I would get this:

> [ERROR] --initialize specified but the data directory has files in it. Aborting.

What? Isn't Kubernetes meant to be the fabric on which we could easily port our workloads across different providers with no issues?

After a good amount of googling, it turns out that GKE Persistence Volume provider mounts the /var/lib/mysql volume with a lost+found directory that is not empty. That does not happen on minikube. And mysql does not like that (not when you are not running with default my.cnf at least).

So to fix this, you need to pass --ignore-db-dir=lost+found to the command line arguments of the container as in this snippet:

spec:
      containers:
      - image: debezium/example-mysql:0.10
        args:
          - "--ignore-db-dir=lost+found"
        name: mysql

Since this took me a while to find, I thought it would be worth sharing for others that may bump into this.

Happy coding

Comments

[Noah Williams]

Dude! Thank you! This was such a pain in the backend! Of course, it makes perfect sense that a Linux filesystem would have a lost+found folder. But I sometimes forget we're dealing with Linux thanks to all the K8s abstraction and distraction!