Day 66: MySQL and a Private Container

Day 66 of DevOps, Day 16 of Azure. Both tasks today came down to a name being exactly right. I created a PersistentVolumeClaim under the wrong name, and the pod never got scheduled, with no container and no logs to look at. And an Azure storage account name has to be unique across the whole of Azure, because it becomes a hostname.

One Kubernetes task, one Azure task. Run MySQL on a PersistentVolume with its credentials in Secrets, then create a private blob container. The tasks come from the KodeKloud Engineer platform.

Task 1: MySQL on a PersistentVolume

The task: a 250Mi PersistentVolume and a claim, three Secrets for the root password, the application user and password, and the database name, a MySQL Deployment reading all four from those Secrets, and a NodePort Service on 30007.

Step 1: The volume and the claim

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 250Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dba"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250Mi

kubectl apply -f pv.yaml
kubectl get pv

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS
mysql-pv   250Mi      RWO            Retain           Available           manual

Retain is the documented default for a volume created by hand, as opposed to Delete for a dynamically provisioned one.

Step 2: Three Secrets, split by what rotates together

kubectl create secret generic mysql-user-pass \
  --from-literal='username=kodekloud_tim' --from-literal='password=8FmzjvFU6S'
kubectl create secret generic mysql-root-pass --from-literal='password=YUIidhb667'
kubectl create secret generic mysql-db-url --from-literal='database=kodekloud_db6'
kubectl get secrets

NAME              TYPE     DATA   AGE
mysql-db-url      Opaque   1      5s
mysql-root-pass   Opaque   1      42s
mysql-user-pass   Opaque   2      81s

The user Secret holds two keys, so its DATA column reads 2. Rotating the application password then touches one object rather than three.

Step 3: The Deployment reads all four from Secrets

      containers:
      - image: mysql:8.0
        name: mysql-container
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-root-pass
              key: password
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: mysql-db-url
              key: database
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: username
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: password
        volumeMounts:
        - name: mysql-pvc
          mountPath: /var/lib/mysql

/var/lib/mysql is not an arbitrary choice. The official image documents it as "where MySQL by default will write its data files".

Step 4: The pod that would not schedule

Warning  FailedScheduling  0/1 nodes are available: persistentvolumeclaim "mysql-pv-claim" not found.
Warning  FailedScheduling  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.

I had created the claim under the volume's name rather than its own, so kubectl get pvc listed mysql-pv while the Deployment asked for mysql-pv-claim.

The useful part is where that showed up. There was no container, so no logs. There was no image pull to fail, so nothing in the pod's state. The pod simply never got scheduled, and the only account of why is in kubectl describe pod under Events. That is a different debugging path from a container that starts and crashes, and the first place to look is different too.

The two messages are also distinct. The first means the claim does not exist. The second means a claim exists and has not bound to anything yet, which is what you would see if the class or the capacity did not match.

Step 5: Watch the entrypoint build the database

kubectl logs <pod>

[Note] [Entrypoint]: Initializing database files
[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Creating database kodekloud_db6
[Note] [Entrypoint]: Creating user kodekloud_tim
[Note] [Entrypoint]: Giving user kodekloud_tim access to schema kodekloud_db6
[Note] [Entrypoint]: Stopping temporary server
[Note] [Entrypoint]: MySQL init process done. Ready for start up.

The image starts a temporary server, applies the environment variables, shuts it down, then starts the real one. That only happens on an empty data directory, so pointing a second pod at the same claim runs none of it.

kubectl exec -it <pod> -- mysql -u root -p
kubectl exec -it <pod> -- mysql -u kodekloud_tim -p

Both logins work, which tests the credentials rather than the pod's status field.

The Secret that is no longer a secret

kubectl describe pod is careful:

MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'mysql-root-pass'>  Optional: false

The container is not:

kubectl exec -it <pod> -- env | grep MYSQL

MYSQL_PASSWORD=8FmzjvFU6S
MYSQL_ROOT_PASSWORD=YUIidhb667
MYSQL_DATABASE=kodekloud_db6
MYSQL_USER=kodekloud_tim

Putting a Secret in an environment variable means anything that can run a command in that container can read it, and so can anything that reads /proc/<pid>/environ. The Secret object did its job in transit and stopped protecting anything on arrival.

Day 62 mounted a Secret as a file instead, which keeps the value out of the environment and lets the kubelet update it in place. The environment variable form has the opposite property: a change "will not be seen by the container unless it is restarted".

Note too that MYSQL_MAJOR and MYSQL_VERSION are in that same output and came from the image. A grep across the environment does not distinguish what you injected from what was already there.

What the docs recommend

Set storageClassName on both objects, and understand why omitting it is not the same as setting it empty. The docs separate three cases. storageClassName: "" on a PVC "is always interpreted to be requesting a PV with no class". A PVC with no storageClassName at all "is treated differently by the cluster, depending on whether the DefaultStorageClass admission plugin is turned on", and where a default class exists, "All PVCs that have no storageClassName can be bound only to PVs of that default".

So on a cluster with a default StorageClass, a hand-made class-less PV and a PVC that just leaves the field out will never bind to each other. The claim gets dynamically provisioned somewhere else, and the PV sits Available forever. Naming manual on both, which is what the task asked for, avoids the question.

This matters on k3s in particular, which ships the Local Path Provisioner as a packaged component deployed by default. Whether its local-path class is annotated as the default is not something the k3s documentation states, and its own example sets the class explicitly rather than relying on one. So I can say the provisioner is there, and I cannot say from the docs that it is the default.

Treat hostPath as a lab-only arrangement. The example in the Kubernetes docs carries the warning in a code comment, "for single node testing only; WILL NOT WORK in a multi-node cluster", and the Volumes page is blunter about the type: "Using the hostPath volume type presents many security risks. If you can avoid using a hostPath volume, you should."

Give a database some resources. This pod is BestEffort, because nothing sets requests or limits, which makes it the first thing evicted when the node comes under memory pressure.

And check what the Service actually reaches. Mine listens on 80, targets 80, and describe svc shows Endpoints: 10.22.0.9:80, while MySQL is bound to 3306. The task specified those numbers, so it passes, and the Service routes to a closed port. That is Day 64's fault from the other direction: there I found a wrong targetPort by fixing the thing that hid it, and here nothing hides it at all.

Task 2: A private blob container

The task: a storage account xfusionst744158879, and a private container xfusion-blob-744158879 inside it.

Step 1: Two names, two rule sets

	Storage account	Container
Uniqueness	Across Azure	Within the account
Length	3 to 24	3 to 63
Characters	Lowercase letters and digits	Lowercase, digits, hyphens
Hyphens	Not allowed	Allowed, not consecutive, each between alphanumerics

Microsoft: "Storage account names must be between 3 and 24 characters in length. They can contain only numbers and lowercase letters", and "Your storage account name must be unique within Azure."

The two names the task gave me look inconsistent and each is the only legal form for its resource.

Uniqueness across Azure is a DNS requirement, because the account name is a subdomain:

xfusionst744158879.blob.core.windows.net

Which is why the lab appended a random-looking number. xfusionst alone would collide with somebody else's account somewhere in the world, in a subscription you cannot see.

I had been calling that failure StorageAccountAlreadyTaken. It is not an error code Microsoft documents. Account creation is a control-plane operation, where the name check returns a reason of AlreadyExists; the data-plane error table has AccountAlreadyExists with a 409. Neither is spelled the way I had it.

Step 2: Create the account and the container

az storage account create -g $RG -n $SA --sku Standard_LRS --kind StorageV2

KEY=$(az storage account keys list -g $RG -n $SA --query "[0].value" -o tsv)
az storage container create --name $CONTAINER --account-name $SA --account-key $KEY

Step 3: Confirm it is private on both layers

az storage account show -g $RG -n $SA --query "{AllowBlobPublic:allowBlobPublicAccess}"
az storage container show --name $CONTAINER --account-name $SA --account-key $KEY \
  --query "{PublicAccess:properties.publicAccess}"

"AllowBlobPublic": false
"PublicAccess": null

Two independent switches, and either one blocking is enough. The account-level default is documented as "The default interpretation is false for this property", and the word interpretation is doing real work there: the stored value can legitimately be null, and Microsoft's own remediation script treats null and true alike.

At the container level, private is the absence of a value rather than a value named private: "If this header isn't returned in the response, the container is private to the account owner."

Worth knowing that one state has five spellings across Azure's own surfaces. The portal calls it Private, PowerShell calls it Off, the CLI accepts blob, container, off, the data plane omits the header, and Azure Resource Manager returns the string "None". The one thing consistently true is that no API has a value called "private", so a check for publicAccess == 'private' matches nothing anywhere.

The permission system that is two permission systems

I created the container with the account key rather than my signed-in identity, and that was not laziness. Microsoft: "Built-in roles such as Owner, Contributor, and Storage Account Contributor permit a security principal to manage a storage account, but don't provide access to the blob data within that account via Microsoft Entra ID."

So Owner can create the account, read its keys and change its configuration, and still be refused when listing blobs with --auth-mode login. Data access needs its own roles: Storage Blob Data Owner, Contributor or Reader.

There is a caveat that spoils the tidy version, and I had missed it. Owner and Contributor include Microsoft.Storage/storageAccounts/listKeys/action, and Microsoft spells out what follows: a user with such a role "can access data in the storage account via Shared Key authorisation by using the account access keys". The separation is real at the authorisation layer and leaky in practice, unless shared key access is switched off.

Role assignments also take time, and Microsoft publishes more than one number for it: "up to 30 minutes to propagate" in one callout, "up to 10 minutes for changes to take effect" in another on the same page, and up to 12 hours for data actions at management group scope. Any figure quoted for this needs its source named.

What the docs recommend

Prefer identity to keys. The account key is documented as 512-bit, there are two, so one can rotate while the other is live, and "Storage account access keys provide full access to the storage account data and the ability to generate SAS tokens". No expiry, no identity, so the access log records the operation and not the person. Microsoft's order of preference is Entra ID with data roles, then a user delegation SAS, then keys.

If you disable keys, know what else stops working. Setting allowSharedKeyAccess to false means "Azure Storage rejects all subsequent requests to that account that are authorised with the account access keys", and that includes service SAS and account SAS, since both are signed with the shared key. A user delegation SAS keeps working, because it is signed with an Entra ID key. The default is permissive: "The default value is null, which is equivalent to true."

Turn on soft delete before anything real goes in, and check the default rather than assuming it. Blob soft delete "is enabled by default when you create a new storage account with the Azure portal", and Microsoft says just as directly that it "isn't enabled when you create a new storage account with Azure CLI". This account was made with the CLI, so it has none. Container soft delete's default is not documented either way.

One correction to my own redundancy table. I had written LRS as three copies in one data centre. The managed disk documentation does say three copies, but the storage account redundancy page no longer gives a count for LRS: it says "a single physical datacenter located in the primary region of your choice" and states durability as "at least 99.999999999% (11 9s)" instead. ZRS is "three or more Azure availability zones", not exactly three. If a post states several copies for a storage account, it is ahead of the documentation.

What I learned

The two halves disagreed about how much a name is worth. In Kubernetes, a wrong name costs you a pod that never schedules, with a clear message waiting in the events. In Azure, a storage account name is a hostname in a namespace shared with everyone, so a name that is merely sensible is a name somebody else already has.

The other thing I am taking from today is the environment variable output. The Secret objects worked exactly as designed, and three seconds with env inside the container printed every value back in plain text. The protection was in the transport, not in the destination.

So here is the Day 66 question. Where in your system does a secret stop being handled as a secret, and does anything in your tooling tell you where that line is?

Day 66 down. Thirty-four to go.