DEV Community

Cover image for Day 62: Secrets and Tags
Nnamdi Felix Ibe
Nnamdi Felix Ibe

Posted on AI-assisted

Day 62: Secrets and Tags

#devops #kubernetes #azure #containers

Day 62 of DevOps, Day 12 of Azure. Both tasks today turned on a name I did not choose. Kubernetes named the file inside my Secret after the file on disk, and Azure's tag commands disagree among themselves about what setting a tag means.

One Kubernetes task, one Azure task. Mount a Secret into a pod as a file, then add a single tag to a VM. The tasks come from the KodeKloud Engineer platform.

Task 1: Mount a Secret as a file

The task: create a generic Secret called official from /opt/official.txt, then run a pod, secret-xfusion, with one container, secret-container-xfusion, on debian:latest, that mounts the Secret at /opt/demo.

Step 1: Look at what is going in 

cat /opt/official.txt
Enter fullscreen mode Exit fullscreen mode 
5ecur3
Enter fullscreen mode Exit fullscreen mode

Step 2: Create the Secret from the file 

kubectl create secret generic official --from-file=/opt/official.txt
Enter fullscreen mode Exit fullscreen mode 
secret/official created
Enter fullscreen mode Exit fullscreen mode

generic is the subcommand for arbitrary data, and it produces a Secret of type Opaque, which the docs describe as "arbitrary user-defined data". The other types are for specific jobs: registry credentials, TLS material, SSH keys, service account tokens.

Step 3: Find out what the key is called 

kubectl get secret
kubectl describe secret
Enter fullscreen mode Exit fullscreen mode 
NAME       TYPE     DATA   AGE
official   Opaque   1      4s

Data
====
official.txt:  7 bytes
Enter fullscreen mode Exit fullscreen mode

This is the detail the whole task rests on. The key is official.txt, not official. The docs: "The default key name is the file name." It matters because the key is what becomes the filename inside the container, so the path you mount at is only half the address.

Seven bytes is 5ecur3 plus the trailing newline. Note what describe does not print: the value. It gives you the key and a size, and nothing else.

Step 4: Mount it in a pod 

apiVersion: v1
kind: Pod
metadata:
  name: secret-xfusion
spec:
  volumes:
    - name: secret-volume
      secret:
        secretName: official
  containers:
    - name: secret-container-xfusion
      image: debian:latest
      command: ['sleep','6000']
      volumeMounts:
        - name: secret-volume
          readOnly: true
          mountPath: "/opt/demo"
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f po.yaml
kubectl get po
Enter fullscreen mode Exit fullscreen mode 
NAME             READY   STATUS    RESTARTS   AGE
secret-xfusion   1/1     Running   0          4s
Enter fullscreen mode Exit fullscreen mode

sleep 6000 is there because debian:latest has nothing to run. Without it the container exits immediately and there is nothing to inspect.

Step 5: Read it from inside the container 

kubectl exec -it secret-xfusion -c secret-container-xfusion -- bash
ls /opt/demo/
cat /opt/demo/official.txt
Enter fullscreen mode Exit fullscreen mode 
official.txt
5ecur3
Enter fullscreen mode Exit fullscreen mode

One key, one file. The docs: "Each key in the Secret data map becomes a file name in this directory." A Secret with four keys would have put four files in /opt/demo.

The read-only flag that is not doing the work

kubectl describe pod shows the mount like this:

Mounts:
  /opt/demo from secret-volume (ro)
Enter fullscreen mode Exit fullscreen mode

I had written readOnly: true in the manifest, so the ro looks like my doing. It is not. Kubernetes mounts Secret volumes read-only whatever the manifest says, and the volume plugin in the Kubernetes source reports ReadOnly: true unconditionally.

The documentation does not settle this for Secrets in either direction. What it does say, on the ConfigMap page, is to set the flag, which is probably where most of us picked up the habit. So the line is worth keeping as a statement of intent. It just is not the thing producing the result.

What the docs recommend

Prefer a volume to an environment variable. Both work, and they behave differently when the Secret changes. Mounted files are updated by the kubelet on their own. A Secret consumed as an environment variable "will not be seen by the container unless it is restarted". Day 66 uses the environment variable form, and that is the trade it makes.

Do not promise a one-minute update. The figure that gets repeated is the kubelet sync period, and the docs are careful to make it a sum: the total delay "can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type". The sync period defaults to one minute. The second term has no documented default, so there is no documented ceiling.

Know what the storage actually is. The caution is blunt: "Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd." Encryption at rest and tight RBAC are the two steps the docs put first.

Keep them small. "Individual Secrets are limited to 1MiB in size. This is to discourage creation of very large Secrets that could exhaust the API server and kubelet memory."

And watch out for the update that is not an update. This looks like it refreshes the Secret in place:

kubectl create secret generic official --from-file=/tmp/new-official.txt \
  --dry-run=client -o yaml | kubectl apply -f -
Enter fullscreen mode Exit fullscreen mode

It does not. The key still defaults to the base name, so the manifest carries new-official.txt, and data is a map that apply merges key by key. The pod ends up with a second file while /opt/demo/official.txt keeps its old contents. The form that updates in place names the key explicitly:

kubectl create secret generic official --from-file=official.txt=/tmp/new-official.txt \
  --dry-run=client -o yaml | kubectl apply -f -
Enter fullscreen mode Exit fullscreen mode

I did not run the first version, so that outcome is reasoning from the documented key naming and merge rules rather than something I watched break.

Task 2: Add one tag to a VM

The task: put Environment=dev on the VM devops-vm. One tag, one command, and a choice of command that matters more than the tag does.

Step 1: Check what is already there 

az vm show -g $RG -n devops-vm --query "tags"
Enter fullscreen mode Exit fullscreen mode 
{}
Enter fullscreen mode Exit fullscreen mode

An empty object. Not null, not an empty string. That is the fourth distinct way these labs have spelled "nothing" so far, after a missing field, an unset field and a computed empty string, and it decides how you test for it: length(tags) returns 0 for {}, while a null check matches nothing.

Step 2: Choose the command that says what it means 

az tag update --resource-id $VM_ID --operation Merge --tags Environment=dev
Enter fullscreen mode Exit fullscreen mode

Azure has several ways to set a tag and they do not mean the same thing. az tag update --operation Merge "allows adding tags with new names and updating the values of tags with existing names". Replace "replaces the entire set of existing tags with a new set". Delete removes the ones you name.

The one to be careful with is az resource tag, which reads like "add this tag" and whose reference describes it only as "Tag a resource."

Step 3: Verify 

az vm show -g $RG -n devops-vm --query "{Tags:tags,Power:powerState}" -d
Enter fullscreen mode Exit fullscreen mode 
{ "Tags": { "Environment": "dev" }, "Power": "VM running" }
Enter fullscreen mode Exit fullscreen mode

VM running before and after. Nothing restarted.

I went looking for a Microsoft sentence confirming that tagging never restarts a VM and could not find one, so that is an observation from this run rather than a guarantee. The closest related statement points the other way: "you can only update tags on virtual machine extensions when the virtual machine is running."

The warning the docs decline to give

I wanted to write that az resource tag replaces all tags unless you pass --is-incremental. The reference will not quite support it.

What the reference actually contains is the description of the flag, "The option to add tags incrementally without deleting the original tags", and an example captioned "It doesn't empty the existing tags." Both sentences are only meaningful if the default does delete and does empty. The behaviour is a sound inference. It is not a documented statement, and on a VM carrying Owner, CostCentre and Project the difference between those two things is three tags.

That is enough reason to use the command that names its operation. It is not enough to assert the other command's behaviour as fact without running it.

It also made no difference here, which is worth admitting. The VM had no tags, so Merge and Replace would have produced identical output. The command has to be chosen before you know whether it matters.

What the docs recommend

Expect no inheritance. "Resources don't inherit the tags you apply to a resource group or a subscription." Tag a VM and its disks, NIC and public IP stay untagged, which quietly splits a cost report: compute under dev, storage under nothing.

Azure Policy is the fix, and the policy names are close enough to misread. "Inherit a tag from the resource group" uses a modify effect and "Adds or replaces the specified tag and value from the parent resource group". "Inherit a tag from the resource group if missing" leaves an existing value alone. "Append a tag and its value from the resource group" does not touch resources created before the assignment. Picking the wrong one either overwrites deliberate values or silently skips everything that already exists.

Standardise the casing of values. "Tag names are case-insensitive for operations", but "Tag values are case-sensitive". So Environment and environment reach the same tag, while Dev and dev are two different answers in a cost report.

Know the limits before designing a tagging scheme: 50 tag name-value pairs per resource, names up to 512 characters, values up to 256. Storage accounts cap names at 128. Five resource types allow only 15 tags, Azure Automation and the DNS services among them. And tag names cannot contain <, >, %, &, \, ? or /.

Query them server-side, since that is the entire point of having them:

az resource list --tag Environment=dev --query "[].{Name:name,Type:type,RG:resourceGroup}" -o table
Enter fullscreen mode Exit fullscreen mode

Note that --tag will not combine with --resource-group, and the CLI's source shows it also refuses --namespace, --resource-type, --name and --location. The error can even fire on a filter you did not type, because a configured CLI default counts.

Compare with AWS, which solves this by having one command per intent. aws ec2 create-tags "Adds or overwrites only the specified tags for the specified Amazon EC2 resource or resources", and removing tags is a separate delete-tags call. There is no flag to get wrong.

What I learned

Both tasks hid their real behaviour behind a name. The Secret's key came from the filename, so the path inside the container was decided by something I typed on the host. The tag command's behaviour came from which of four commands I reached for, and the documentation describes the safe one clearly while leaving the dangerous one's default to be inferred.

The second case is the one I will remember, because the inference is almost certainly right and I still could not write it down as a fact.

So here is the Day 62 question. When a tool's documentation describes the safe option carefully and says nothing about the default, which one do you assume you are using?

Day 62 down. Thirty-eight to go.

Top comments (1)

Collapse
 
merbayerp profile image
Mustafa ERBAY •

Nice one, Nnamdi! 😄

I really liked the distinction between what the documentation guarantees and what we simply infer from it.

My rule is pretty simple: if a command can modify existing state and its default behavior isn’t clear, I don’t trust the default. I verify it first.

Especially in production, “probably safe” isn’t quite good enough. 😅

38 days to go! Keep it up, my friend! 🚀