DEV Community: Nnamdi Felix Ibe

Day 67: A Guest Book and a Public Container

Nnamdi Felix Ibe — Mon, 19 Oct 2026 17:00:00 +0000

Day 67 of DevOps, Day 17 of Azure. Both tasks today turned on exactly which word sat in one field. In Kubernetes, redis-master is not a label I chose, it is a hostname compiled into the application. In Azure, the difference between blob and container decides whether a stranger can list your files or merely read the ones they can already name.

One Kubernetes task, one Azure task. Deploy a three-tier Guest Book, then create a storage account and a deliberately public container. The tasks come from the KodeKloud Engineer platform.

Task 1: A Guest Book in three tiers

The task: a redis-master Deployment with one replica behind a ClusterIP Service, a redis-slave Deployment with two replicas behind a ClusterIP Service, and a frontend Deployment with three replicas behind a NodePort Service on 30009. Clicking the app button before any of it existed returned 502.

Step 1: The write tier

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: redis-master
  name: redis-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-master
  strategy: {}
  template:
    metadata:
      labels:
        app: redis-master
    spec:
      containers:
      - image: redis:alpine
        name: master-redis-devops
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
        - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: redis-master
spec:
  selector:
    app: redis-master
  ports:
    - targetPort: 6379
      port: 6379

No type on the Service, which makes it ClusterIP: "the default that is used if you don't explicitly specify a type for a Service", and one that "makes the Service only reachable from within the cluster". That is the entire isolation story for the Redis tier, and it is a default rather than a decision.

Step 2: The read tier, two replicas

      containers:
      - image: gcr.io/google_samples/gb-redisslave:v3
        name: slave-redis-devops
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
        - containerPort: 6379
        env:
        - name: GET_HOSTS_FROM
          value: "dns"

Step 3: The frontend, three replicas, on a node port

      containers:
      - image: gcr.io/google-samples/gb-frontend@sha256:a908df84...
        name: php-redis-devops
        env:
        - name: "GET_HOSTS_FROM"
          value: "dns"
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  type: NodePort
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30009

30009 sits in the static band of the node port range, 30000 to 30085, which is where a hand-picked number belongs. Automatic assignment draws from the dynamic band, 30086 to 32767, first.

Step 4: Apply and check

kubectl apply -f backend-deploy.yaml
kubectl apply -f backend-svc.yaml
kubectl apply -f redis-slave-deploy.yaml
kubectl apply -f redis-slave-svc.yaml
kubectl apply -f frontend-deploy.yaml
kubectl apply -f frontend-svc.yaml

kubectl get deploy

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
frontend       3/3     3            3           41s
redis-master   1/1     1            1           2m18s
redis-slave    2/2     2            2           90s

Why the Service names are not mine to choose

GET_HOSTS_FROM: dns looks like boilerplate. It is the switch that decides how the frontend finds Redis, and the guestbook PHP shows what the other side of it looks like:

$host = 'redis-master';
if (getenv('GET_HOSTS_FROM') == 'env') {
  $host = getenv('REDIS_MASTER_SERVICE_HOST');
}

The hostnames are hardcoded. Setting the variable to anything other than env leaves them as literal strings, and the application relies on cluster DNS to resolve them. A Service gets a DNS record "with a name of the form my-svc.my-namespace.svc.cluster-domain.example", and "DNS queries that don't specify a namespace are limited to the Pod's namespace", so a bare redis-master resolves inside the pod's own namespace.

Which means a Service name here is an address, not a label. Rename the Deployment to anything you like and nothing breaks. Rename the Service and the application stops finding its database, with every object still reporting itself healthy.

The alternative branch is worth noticing too. REDIS_MASTER_SERVICE_HOST is the Docker-link-style variable the kubelet injects, and it only exists for Services created before the pod. That is the ordering trap from Day 62's notes, and dns is how you avoid it.

The 502 I cannot explain

The app returned 502 before the backend existed, and that is where my account stops. Nothing in the Kubernetes documentation describes what a client sees when a Service's selector matches no pods. The EndpointSlice page covers what EndpointSlices contain; the debugging guide goes as far as "If the ENDPOINTS column is <none>, you should check that the spec.selector field".

A 502 is a gateway error, so it came from whatever proxy fronted the lab's app button, not from kube-proxy or the Service. I am recording it as something I saw rather than something Kubernetes does.

What the docs recommend

Know that these images are retired. The current guestbook manifests use us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2 and gb-frontend:v5. The gcr.io paths above match what the Kubernetes docs shipped around v1.19.

Do not assume the underscore is a typo. google_samples with an underscore and google-samples with a hyphen both appear upstream, in the same manifest set. It is not even a stable per-image rule: at the v1.26 docs branch, the frontend was also gcr.io/google_samples. Both images have used both spellings at different times.

And the tutorial this descends from has moved on. kubernetes.io still publishes a guestbook, and since the v1.21 docs branch, its objects are redis-leader and redis-follower, with the prose reading "Start up a Redis leader." The master and slave naming belongs to the older branch and to this lab.

Redis made the same change, and only halfway. Its SLAVEOF page says the command "is regarded as deprecated" as of 5.0.0 and that "the Redis project no longer uses the word slave", pointing to REPLICAOF. The current replication page still writes "master" for the other side, so replica replaced slave while primary did not replace master.

One resource note. Every container sets requests and no limits, which makes each pod Burstable. The class belongs to the pod, not the container: Burstable means the pod "does not meet the criteria for QoS class Guaranteed" while "At least one Container in the Pod has a memory or CPU request or limit". Day 63 was the mirror image, where limits and no requests produced Guaranteed.

Task 2: A deliberately public container

The task: a storage account and a container with anonymous read access for containers and blobs. Day 16 inverted.

Step 1: Set the account flag at creation

az storage account create -g $RG -n $SA --sku Standard_LRS --kind StorageV2 \
  --allow-blob-public-access true

Order matters here. A new account defaults the flag to false, and while it is false, Microsoft says "you can't configure the access setting for a container to permit anonymous access". Doing it the other way round costs a failed command and an az storage account update before retrying.

Step 2: Choose the level deliberately

KEY=$(az storage account keys list -g $RG -n $SA --query "[0].value" -o tsv)
az storage container create --name $CONTAINER --account-name $SA --account-key $KEY \
  --public-access container

Three values, and the requirement's wording picked one:

Value	Read a known blob	List the container
`off`	No	No
`blob`	Yes	No
`container`	Yes	Yes

Microsoft's wording for the two public levels: "Blobs within the container can be read by anonymous request, but container data isn't available anonymously", against "Container and blob data can be read by anonymous request".

The task said "anonymous read access for containers and blobs", naming both, which is container. blob would have satisfied half the requirement while looking correct in the portal. The distinction is not cosmetic: with blob, a caller has to know a blob's exact name, and with container they can enumerate everything and take the lot.

Step 3: Prove it from outside

curl -s -o /dev/null -w "http_code=%{http_code}\n" \
  "https://$SA.blob.core.windows.net/$CONTAINER?restype=container&comp=list"

http_code=200

No key, no login, no SAS token. ?restype=container&comp=list is the Blob REST List Blobs operation, so this tests the exact capability the requirement names without uploading anything first. A configuration read tells you the settings are right. This tells you the behaviour is correct.

The failure code I had wrong

I have been repeating that a private container returns 404 rather than 403, and that Azure does this deliberately so as not to confirm the container exists. Checking it, neither half survives.

Microsoft documents three outcomes keyed to the client's service version. For "a service version that supports the bearer challenge", the service "returns error code 401 (Unauthorized)". For an older version where anonymous access is disallowed on the account, 409 Conflict. For an older version where it is allowed, 404 Not Found. And separately: "After you disallow anonymous access for a storage account, clients that use the anonymous bearer challenge find that Azure Storage returns a 403 error (Forbidden) rather than a 401 error."

So 401, 403, 404 and 409 are all documented, depending on which version the client negotiates and what the account permits. The "will not confirm it exists" rationale appears nowhere in Microsoft's documentation at all. It is a tidy story that explains one of the four codes, and I had been treating it as the rule.

What the docs recommend

Understand exactly how far anonymous access reaches, because it is narrower than it sounds and wider than it feels. The documented operation list is all reads, and clients "can enumerate blobs within the container via anonymous request, but can't enumerate containers within the storage account". No writes, no deletes, no visibility into other containers.

Then understand who pays. Egress leaves the owner holding the bill, and Microsoft's bluntest statement of it is in the SAS guidance: "If you've given them read access as well, they may choose to download it 10 times, incurring 2 TB in egress costs for you." Every anonymous download is also a read transaction, and cool or archive tiers add a retrieval fee on top.

Expect to be flagged for it. Microsoft Defender for Cloud carries a recommendation named "Storage account public access should be disallowed", describing anonymous read access as "a convenient way to share data, but might present security risks."

And know that you cannot qualify it. An S3 bucket policy can condition public read on a prefix, a source IP or a referrer, through s3:prefix, aws:SourceIp or aws:Referer. Azure's container access is a single enum with no condition syntax: on for the whole container, or off. The only qualifier that still applies is the storage firewall, since "any firewall rules that you configure for the storage account remain in effect".

Which is why the production answer is a shared access signature rather than a public container. Microsoft notes a SAS "can be issued on a container or a blob regardless of whether container data is available for anonymous read access" and "provides a greater measure of control", and recommends a user delegation SAS "when possible for superior security", because it is signed with Entra credentials rather than the account key.

What I learned

The Kubernetes half was a lesson in which names are addresses. A Deployment name is a label for humans. A Service name is a hostname the application has compiled into it, so the one field that looks most like a free choice is the one that is not.

The Azure half was a lesson about a story I had been repeating. The 404 explanation was plausible, it matched something I had seen, and it turned out to be one of four documented codes with none of the reasoning I had attached to it.

So here is the Day 67 question. Which of your configuration values is a free label, and which one is something else quietly depending on?

Day 67 down. Thirty-three to go.

Day 66: MySQL and a Private Container

Nnamdi Felix Ibe — Sun, 18 Oct 2026 16:00:00 +0000

Day 66 of DevOps, Day 16 of Azure. Both tasks today came down to a name being exactly right. I created a PersistentVolumeClaim under the wrong name, and the pod never got scheduled, with no container and no logs to look at. And an Azure storage account name has to be unique across the whole of Azure, because it becomes a hostname.

One Kubernetes task, one Azure task. Run MySQL on a PersistentVolume with its credentials in Secrets, then create a private blob container. The tasks come from the KodeKloud Engineer platform.

Task 1: MySQL on a PersistentVolume

The task: a 250Mi PersistentVolume and a claim, three Secrets for the root password, the application user and password, and the database name, a MySQL Deployment reading all four from those Secrets, and a NodePort Service on 30007.

Step 1: The volume and the claim

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 250Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/dba"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250Mi

kubectl apply -f pv.yaml
kubectl get pv

NAME       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS
mysql-pv   250Mi      RWO            Retain           Available           manual

Retain is the documented default for a volume created by hand, as opposed to Delete for a dynamically provisioned one.

Step 2: Three Secrets, split by what rotates together

kubectl create secret generic mysql-user-pass \
  --from-literal='username=kodekloud_tim' --from-literal='password=8FmzjvFU6S'
kubectl create secret generic mysql-root-pass --from-literal='password=YUIidhb667'
kubectl create secret generic mysql-db-url --from-literal='database=kodekloud_db6'
kubectl get secrets

NAME              TYPE     DATA   AGE
mysql-db-url      Opaque   1      5s
mysql-root-pass   Opaque   1      42s
mysql-user-pass   Opaque   2      81s

The user Secret holds two keys, so its DATA column reads 2. Rotating the application password then touches one object rather than three.

Step 3: The Deployment reads all four from Secrets

      containers:
      - image: mysql:8.0
        name: mysql-container
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-root-pass
              key: password
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: mysql-db-url
              key: database
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: username
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-user-pass
              key: password
        volumeMounts:
        - name: mysql-pvc
          mountPath: /var/lib/mysql

/var/lib/mysql is not an arbitrary choice. The official image documents it as "where MySQL by default will write its data files".

Step 4: The pod that would not schedule

Warning  FailedScheduling  0/1 nodes are available: persistentvolumeclaim "mysql-pv-claim" not found.
Warning  FailedScheduling  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.

I had created the claim under the volume's name rather than its own, so kubectl get pvc listed mysql-pv while the Deployment asked for mysql-pv-claim.

The useful part is where that showed up. There was no container, so no logs. There was no image pull to fail, so nothing in the pod's state. The pod simply never got scheduled, and the only account of why is in kubectl describe pod under Events. That is a different debugging path from a container that starts and crashes, and the first place to look is different too.

The two messages are also distinct. The first means the claim does not exist. The second means a claim exists and has not bound to anything yet, which is what you would see if the class or the capacity did not match.

Step 5: Watch the entrypoint build the database

kubectl logs <pod>

[Note] [Entrypoint]: Initializing database files
[Note] [Entrypoint]: Starting temporary server
[Note] [Entrypoint]: Creating database kodekloud_db6
[Note] [Entrypoint]: Creating user kodekloud_tim
[Note] [Entrypoint]: Giving user kodekloud_tim access to schema kodekloud_db6
[Note] [Entrypoint]: Stopping temporary server
[Note] [Entrypoint]: MySQL init process done. Ready for start up.

The image starts a temporary server, applies the environment variables, shuts it down, then starts the real one. That only happens on an empty data directory, so pointing a second pod at the same claim runs none of it.

kubectl exec -it <pod> -- mysql -u root -p
kubectl exec -it <pod> -- mysql -u kodekloud_tim -p

Both logins work, which tests the credentials rather than the pod's status field.

The Secret that is no longer a secret

kubectl describe pod is careful:

MYSQL_ROOT_PASSWORD:  <set to the key 'password' in secret 'mysql-root-pass'>  Optional: false

The container is not:

kubectl exec -it <pod> -- env | grep MYSQL

MYSQL_PASSWORD=8FmzjvFU6S
MYSQL_ROOT_PASSWORD=YUIidhb667
MYSQL_DATABASE=kodekloud_db6
MYSQL_USER=kodekloud_tim

Putting a Secret in an environment variable means anything that can run a command in that container can read it, and so can anything that reads /proc/<pid>/environ. The Secret object did its job in transit and stopped protecting anything on arrival.

Day 62 mounted a Secret as a file instead, which keeps the value out of the environment and lets the kubelet update it in place. The environment variable form has the opposite property: a change "will not be seen by the container unless it is restarted".

Note too that MYSQL_MAJOR and MYSQL_VERSION are in that same output and came from the image. A grep across the environment does not distinguish what you injected from what was already there.

What the docs recommend

Set storageClassName on both objects, and understand why omitting it is not the same as setting it empty. The docs separate three cases. storageClassName: "" on a PVC "is always interpreted to be requesting a PV with no class". A PVC with no storageClassName at all "is treated differently by the cluster, depending on whether the DefaultStorageClass admission plugin is turned on", and where a default class exists, "All PVCs that have no storageClassName can be bound only to PVs of that default".

So on a cluster with a default StorageClass, a hand-made class-less PV and a PVC that just leaves the field out will never bind to each other. The claim gets dynamically provisioned somewhere else, and the PV sits Available forever. Naming manual on both, which is what the task asked for, avoids the question.

This matters on k3s in particular, which ships the Local Path Provisioner as a packaged component deployed by default. Whether its local-path class is annotated as the default is not something the k3s documentation states, and its own example sets the class explicitly rather than relying on one. So I can say the provisioner is there, and I cannot say from the docs that it is the default.

Treat hostPath as a lab-only arrangement. The example in the Kubernetes docs carries the warning in a code comment, "for single node testing only; WILL NOT WORK in a multi-node cluster", and the Volumes page is blunter about the type: "Using the hostPath volume type presents many security risks. If you can avoid using a hostPath volume, you should."

Give a database some resources. This pod is BestEffort, because nothing sets requests or limits, which makes it the first thing evicted when the node comes under memory pressure.

And check what the Service actually reaches. Mine listens on 80, targets 80, and describe svc shows Endpoints: 10.22.0.9:80, while MySQL is bound to 3306. The task specified those numbers, so it passes, and the Service routes to a closed port. That is Day 64's fault from the other direction: there I found a wrong targetPort by fixing the thing that hid it, and here nothing hides it at all.

Task 2: A private blob container

The task: a storage account xfusionst744158879, and a private container xfusion-blob-744158879 inside it.

Step 1: Two names, two rule sets

	Storage account	Container
Uniqueness	Across Azure	Within the account
Length	3 to 24	3 to 63
Characters	Lowercase letters and digits	Lowercase, digits, hyphens
Hyphens	Not allowed	Allowed, not consecutive, each between alphanumerics

Microsoft: "Storage account names must be between 3 and 24 characters in length. They can contain only numbers and lowercase letters", and "Your storage account name must be unique within Azure."

The two names the task gave me look inconsistent and each is the only legal form for its resource.

Uniqueness across Azure is a DNS requirement, because the account name is a subdomain:

xfusionst744158879.blob.core.windows.net

Which is why the lab appended a random-looking number. xfusionst alone would collide with somebody else's account somewhere in the world, in a subscription you cannot see.

I had been calling that failure StorageAccountAlreadyTaken. It is not an error code Microsoft documents. Account creation is a control-plane operation, where the name check returns a reason of AlreadyExists; the data-plane error table has AccountAlreadyExists with a 409. Neither is spelled the way I had it.

Step 2: Create the account and the container

az storage account create -g $RG -n $SA --sku Standard_LRS --kind StorageV2

KEY=$(az storage account keys list -g $RG -n $SA --query "[0].value" -o tsv)
az storage container create --name $CONTAINER --account-name $SA --account-key $KEY

Step 3: Confirm it is private on both layers

az storage account show -g $RG -n $SA --query "{AllowBlobPublic:allowBlobPublicAccess}"
az storage container show --name $CONTAINER --account-name $SA --account-key $KEY \
  --query "{PublicAccess:properties.publicAccess}"

"AllowBlobPublic": false
"PublicAccess": null

Two independent switches, and either one blocking is enough. The account-level default is documented as "The default interpretation is false for this property", and the word interpretation is doing real work there: the stored value can legitimately be null, and Microsoft's own remediation script treats null and true alike.

At the container level, private is the absence of a value rather than a value named private: "If this header isn't returned in the response, the container is private to the account owner."

Worth knowing that one state has five spellings across Azure's own surfaces. The portal calls it Private, PowerShell calls it Off, the CLI accepts blob, container, off, the data plane omits the header, and Azure Resource Manager returns the string "None". The one thing consistently true is that no API has a value called "private", so a check for publicAccess == 'private' matches nothing anywhere.

The permission system that is two permission systems

I created the container with the account key rather than my signed-in identity, and that was not laziness. Microsoft: "Built-in roles such as Owner, Contributor, and Storage Account Contributor permit a security principal to manage a storage account, but don't provide access to the blob data within that account via Microsoft Entra ID."

So Owner can create the account, read its keys and change its configuration, and still be refused when listing blobs with --auth-mode login. Data access needs its own roles: Storage Blob Data Owner, Contributor or Reader.

There is a caveat that spoils the tidy version, and I had missed it. Owner and Contributor include Microsoft.Storage/storageAccounts/listKeys/action, and Microsoft spells out what follows: a user with such a role "can access data in the storage account via Shared Key authorisation by using the account access keys". The separation is real at the authorisation layer and leaky in practice, unless shared key access is switched off.

Role assignments also take time, and Microsoft publishes more than one number for it: "up to 30 minutes to propagate" in one callout, "up to 10 minutes for changes to take effect" in another on the same page, and up to 12 hours for data actions at management group scope. Any figure quoted for this needs its source named.

What the docs recommend

Prefer identity to keys. The account key is documented as 512-bit, there are two, so one can rotate while the other is live, and "Storage account access keys provide full access to the storage account data and the ability to generate SAS tokens". No expiry, no identity, so the access log records the operation and not the person. Microsoft's order of preference is Entra ID with data roles, then a user delegation SAS, then keys.

If you disable keys, know what else stops working. Setting allowSharedKeyAccess to false means "Azure Storage rejects all subsequent requests to that account that are authorised with the account access keys", and that includes service SAS and account SAS, since both are signed with the shared key. A user delegation SAS keeps working, because it is signed with an Entra ID key. The default is permissive: "The default value is null, which is equivalent to true."

Turn on soft delete before anything real goes in, and check the default rather than assuming it. Blob soft delete "is enabled by default when you create a new storage account with the Azure portal", and Microsoft says just as directly that it "isn't enabled when you create a new storage account with Azure CLI". This account was made with the CLI, so it has none. Container soft delete's default is not documented either way.

One correction to my own redundancy table. I had written LRS as three copies in one data centre. The managed disk documentation does say three copies, but the storage account redundancy page no longer gives a count for LRS: it says "a single physical datacenter located in the primary region of your choice" and states durability as "at least 99.999999999% (11 9s)" instead. ZRS is "three or more Azure availability zones", not exactly three. If a post states several copies for a storage account, it is ahead of the documentation.

What I learned

The two halves disagreed about how much a name is worth. In Kubernetes, a wrong name costs you a pod that never schedules, with a clear message waiting in the events. In Azure, a storage account name is a hostname in a namespace shared with everyone, so a name that is merely sensible is a name somebody else already has.

The other thing I am taking from today is the environment variable output. The Secret objects worked exactly as designed, and three seconds with env inside the container printed every value back in plain text. The protection was in the transport, not in the destination.

So here is the Day 66 question. Where in your system does a secret stop being handled as a secret, and does anything in your tooling tell you where that line is?

Day 66 down. Thirty-four to go.

Day 65: An Unread ConfigMap and an Unattached NSG

Nnamdi Felix Ibe — Sat, 17 Oct 2026 17:00:00 +0000

Day 65 of DevOps, Day 15 of Azure. Both tasks today produced an object that looks configured and enforces nothing. A ConfigMap mounted into a Redis container that Redis never opens, and a network security group with two carefully ordered rules and nothing attached to apply them to.

Both passed. Neither does what its shape suggests.

One Kubernetes task, one Azure task. Deploy Redis with a ConfigMap, then create an NSG with two inbound rules. The tasks come from the KodeKloud Engineer platform.

Task 1: Redis with a ConfigMap

The task: a ConfigMap my-redis-config with the key redis-config carrying a maxmemory setting, and a Deployment redis-deployment running redis:alpine with one replica, port 6379, a CPU request of 1, an emptyDir at /redis-master-data, and the ConfigMap mounted at /redis-master.

Step 1: The ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-redis-config
data:
  redis-config: |
    maxmemory=2mb

Step 2: Generate the Deployment, then fill it in

kubectl create deploy redis-deployment --replicas=1 --image=redis:alpine \
  --dry-run=client -o yaml > deployment.yaml

    spec:
      volumes:
        - name: data
          emptyDir: {}
        - name: redis-config
          configMap:
            name: my-redis-config
      containers:
      - image: redis:alpine
        name: redis-container
        ports:
        - containerPort: 6379
        resources:
          requests:
            cpu: "1"
        volumeMounts:
        - name: data
          mountPath: /redis-master-data
        - name: redis-config
          mountPath: /redis-master

Step 3: Apply and check

kubectl apply -f config.yaml
kubectl apply -f deployment.yaml
kubectl get deploy
kubectl get po

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
redis-deployment   1/1     1            1           21s

NAME                                READY   STATUS    RESTARTS   AGE
redis-deployment-6b4f7dfc48-c629j   1/1     Running   0          33s

Every requirement is met. describe deploy lists the CPU request, both mounts and the ConfigMap volume. describe po shows QoS Class: Burstable, which follows from setting a request and no limits.

The part that does not work

Nothing in this manifest passes the mounted file to Redis.

There is no command and no args, so the container runs the image's entrypoint, which starts redis-server with no configuration file argument. Redis is explicit about what that requires: "Note that to read the configuration file, Redis must be started with the file path as first argument." And about what happens otherwise: "Redis is able to start without a configuration file using a built-in default configuration, however this setup is only recommended for testing and development purposes."

So maxmemory keeps its default, which the docs give as zero: "Set maxmemory to zero to specify that you don't want to limit the memory for the dataset. This is the default behavior for 64-bit systems."

I did not run CONFIG GET maxmemory on this pod, so that is reasoning from three documented facts rather than something I watched. The Kubernetes tutorial for this exact setup publishes the transcript, and it shows maxmemory as "0" and maxmemory-policy as "noeviction" before it recreates the pod with a config-reading command, then "2097152" and "allkeys-lru" afterwards.

There is a second problem waiting behind the first. A ConfigMap volume names each file after its key, so the mount produces /redis-master/redis-config. Anyone who adds a command copied from the tutorial would point Redis at /redis-master/redis.conf, which does not exist here.

And a third. redis.conf directives are space-separated. The value in my ConfigMap reads maxmemory=2mb with an equals sign, which is not the directive syntax. A grader can match that string; Redis cannot parse it.

What the official example does differently

command:
  - redis-server
  - "/redis-master/redis.conf"
volumes:
- name: config
  configMap:
    name: example-redis-config
    items:
    - key: redis-config
      path: redis.conf

Two additions, both necessary. The command hands Redis the path. The items block renames the key to redis.conf so that path exists. Drop either one and the file goes unread.

What the docs recommend

If a container takes a config file, give it the path. The mount is only half of it, and nothing in Kubernetes will tell you the other half is missing, because from the cluster's point of view the volume mounted correctly and the container is running.

Get the units right while you are there. redis.conf's table: "1mb => 1024*1024 bytes", so 2mb is 2097152. 2m would be 2000000, since "1m => 1000000 bytes". The suffix carries the difference between binary and decimal.

Know what allkeys-lru is and is not. The docs define it as "Evict the least recently used (LRU) keys" and call it "a good default option if you have no reason to prefer any others". That is advice, not the default. The default policy is noeviction, so an unconfigured Redis under memory pressure returns errors on writes rather than evicting anything.

Do not reach for CONFIG SET as a fix. It changes the running server and not the file: "modifying the configuration on the fly does not affect the redis.conf and redis-full.conf files, so at the next restart of Redis, the old configuration will be used instead." The command that persists it is CONFIG REWRITE, and on this pod it would fail, because "if the server was started without a configuration file at all, the CONFIG REWRITE will just return an error."

Size the request against the node. A CPU request of 1 is a whole core, since "1 CPU unit is equivalent to 1 physical CPU core". On a single-node lab cluster, that is most of what exists. Worth noting the docs never actually write that 1 equals 1000m; they give the fractional form as 0.1 equaling 100m and leave the rest to arithmetic.

Task 2: An NSG with two rules

The task: create devops-nsg with Allow-HTTP on port 80 and Allow-SSH on port 22, both from 0.0.0.0/0.

Step 1: A failure that described the wrong problem

My first attempt skipped creating the NSG and went straight to the rule:

(RequestDisallowedByPolicy) Resource 'Allow-HTTP' was disallowed by policy.
Reasons: 'Global: This resource location is not allowed. Please use one of the
approved locations: eastus, southcentralus, centralus, westus.'

Nothing in that message is about the real problem, which is that the parent did not exist. The fix was az network nsg create.

I had an explanation for the shape of the error: a security rule is a child resource with no location of its own, so with nothing to inherit from, ARM evaluated it as Global, and the subscription's Azure Policy rejects that.

It fits, and it is not documented. What the documentation supports is narrower. The ARM reference for securityRules lists four top-level properties and no location, while the parent network security group does have one. So the schema omits location for the child. No Microsoft page states that a child inherits its parent's location, and the general guidance says you "must provide a location for each resource". Microsoft's own page on this error attributes it entirely to policy and never mentions child resources or missing parents.

So the lesson is the observation: create the parent first, and a Global policy error on a child is what a missing parent looked like. The mechanism is my reconstruction.

Step 2: Create the NSG, then the rules

az network nsg create -g $RG -n devops-nsg

az network nsg rule create -g $RG --nsg-name devops-nsg -n Allow-HTTP \
  --priority 100 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes 0.0.0.0/0 --destination-port-ranges 80

az network nsg rule create -g $RG --nsg-name devops-nsg -n Allow-SSH \
  --priority 110 --direction Inbound --access Allow --protocol Tcp \
  --source-address-prefixes 0.0.0.0/0 --destination-port-ranges 22

Priority has no default, and the CLI lists it as required. Microsoft: "A number between 100 and 4096. Rules are processed in priority order, with lower numbers processed before higher numbers because lower numbers have higher priority. Once traffic matches a rule, processing stops."

100 and 110 rather than 100 and 101, so a rule can be inserted between them later without renumbering everything.

Tcp rather than *, because * would also open UDP on 80 and 22, and neither HTTP nor SSH uses it.

Step 3: List the rules, including the ones you did not write

az network nsg rule list -g $RG --nsg-name devops-nsg --include-default -o table

Every NSG ships with defaults that custom rules have to beat:

Priority	Inbound rule	Effect
65000	`AllowVnetInBound`	Traffic within the VNet
65001	`AllowAzureLoadBalancerInBound`	Azure health probes
65500	`DenyAllInBound`	Everything else

DenyAllInBound at 65500 is why an NSG with no custom rules blocks everything from outside the VNet. The defaults sit outside the 100 to 4096 custom range on purpose, so your rules always run first, and "You can't remove the default rules, but you can override them by creating rules with higher priorities."

They are also in a different place in the API. securityRules holds yours; defaultSecurityRules holds these. A query against the first returns two rules and looks complete.

The source that is not what I thought

"Src": "0.0.0.0/0"

Azure stored exactly what I sent rather than normalising it, which matters for a graded task where a checker may be matching the literal string.

* would have been identical: "0.0.0.0/0 in the Source and Destination columns represents all IP addresses. Clients like the Azure portal, Azure CLI, or PowerShell can use * or Any for this expression."

I had been treating the service tag Internet as a third spelling of the same thing. It is not. Microsoft defines it as "The IP address space that's outside the virtual network and reachable by the public internet." Outside is the operative word: Internet excludes the virtual network's own address space, and 0.0.0.0/0 does not. Substituting one for the other changes which traffic matches.

What the docs recommend

Remember that an unattached NSG enforces nothing. This one is associated with no NIC and no subnet, so its two rules are inert configuration. The task asked only for the NSG and its rules, so I left it that way, and the moment it is attached to anything, Allow-SSH from 0.0.0.0/0 exposes SSH to the whole internet.

Mind the ordering when adding rules later, because this is where Azure differs from AWS in a way that bites. An AWS security group cannot contradict itself: "You can specify allow rules, but not deny rules", and it "Evaluates all rules before deciding whether to allow traffic". An Azure NSG has Deny rules and stops at the first match, so a new Deny at priority 100 silently disables an Allow at 200.

AWS does have an equivalent construct, and it is the network ACL rather than the security group: "Each rule has a number from 1 to 32766. We evaluate the rules in order, starting with the lowest numbered rule... If the traffic matches a rule, the rule is applied and we do not evaluate any additional rules." That is much closer to an NSG, with the difference that network ACLs are stateless.

One correction to something I had written about the API. Azure keeps singular and plural properties for source and port, sourceAddressPrefix and sourceAddressPrefixes, and I had said that exactly one is populated while the other returns an empty array. The REST reference does not say that, and its own sample response omits the plural keys entirely rather than returning []. The empty array is what the Azure CLI renders for an unset list. The practical warning survives and now points at the right layer: a query written against one form breaks when someone adds a second CIDR.

What I learned

Both halves ended with an object that satisfies its requirements and changes nothing. The ConfigMap is mounted at the right path with the right key, and Redis never opens it. The NSG has the right rules in the right order and is attached to nothing.

What makes the first one worse is that every signal available inside Kubernetes says it worked. The volume mounted, the container is running, describe lists everything the task asked for. The only way to find out is to ask Redis what its configuration actually is, which is a step outside the system that was doing the reporting.

So here is the Day 65 question. What in your setup is mounted, loaded, attached or deployed, and has never been asked whether it is actually in effect?

Day 65 down. Thirty-five to go.

Day 64: A Bad Image and a Small Disk

Nnamdi Felix Ibe — Fri, 16 Oct 2026 17:00:00 +0000

Day 64 of DevOps, Day 14 of Azure. Today I had to withdraw an explanation. The Kubernetes half went fine, but the Azure half rested on a tidy theory I had built from one observation on Day 8, and when I checked it against the source, it was wrong.

One Kubernetes task, one Azure task. Fix a Flask app whose pod will not start and whose Service points at the wrong port, then create a managed disk. The tasks come from the KodeKloud Engineer platform.

Task 1: Two faults, one application

The task: python-deployment-devops is not running, and python-service-devops does not reach it. Find out why and fix both.

Step 1: Ask the Deployment what it thinks it is running

kubectl get deploy
kubectl describe deployments.apps

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-devops   0/1     1            0           4m35s

Containers:
 python-container-devops:
  Image:         poroko/flask-app-demo
  Port:          5000/TCP

Two facts worth holding on to: the image name, and that the container declares port 5000.

Step 2: Ask the pod what went wrong

kubectl get po
kubectl describe po

NAME                                        READY   STATUS         RESTARTS   AGE
python-deployment-devops-7dd8f6ddf8-4z6qd   0/1     ErrImagePull   0          5m52s

State:          Waiting
  Reason:       ImagePullBackOff

Events:
  Warning  Failed   Failed to pull image "poroko/flask-app-demo": failed to resolve reference
           "docker.io/poroko/flask-app-demo:latest": pull access denied, repository does not
           exist or may require authorisation: server message: insufficient_scope
  Warning  Failed   Error: ErrImagePull
  Warning  Failed   Error: ImagePullBackOff
  Normal   BackOff  Back-off pulling image "poroko/flask-app-demo"

The existing image is poroko/flask-demo-app. The Deployment asks for poroko/flask-app-demo. Two words transposed.

The registry error offers three explanations and only one applies: "pull access denied, repository does not exist or may require authorisation". A registry will not tell an anonymous client whether a private repository exists, so a typo and a missing credential produce the same message. That is why the docs' first check is the dull one: "Make sure that you have the name of the image correct."

Step 3: Fix the image in place

kubectl edit deployment

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-devops   1/1     1            1           12m

kubectl edit is enough here, and for a specific documented reason: "A Deployment's rollout is triggered if and only if the Deployment's Pod template (that is, .spec.template) is changed, for example if the labels or container images of the template are updated." The image is in the template, so editing it rolls out new pods by itself.

The alternative some walkthroughs take, deleting the Deployment and recreating it from an exported file, also deletes its ReplicaSets, and with them the revision history kubectl rollout undo needs. On an app that is already down, it costs nothing. On a live one, it is an outage.

Step 4: Now the second fault becomes visible

kubectl get svc
kubectl describe svc python-service-devops

NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
python-service-devops   NodePort   10.43.140.241   <none>        8080:32345/TCP   13m

Port:        <unset>  8080/TCP
TargetPort:  8080/TCP
NodePort:    <unset>  32345/TCP
Endpoints:   10.22.0.10:8080

The node port is 32345, which the task required. The target port is 8080, and the application listens on 5000. The Endpoints line is pointing at a port nothing is bound to.

This fault was there the whole time and could not be seen, because a Service with no running pods and a Service aimed at the wrong port look the same from outside. Fixing the image is what made it observable.

Step 5: Fix the target port

kubectl edit svc python-service-devops

Changing targetPort from 8080 to 5000 took effect without touching the pod. Whether that is guaranteed is not something the docs say; the Service page ties EndpointSlice updates to the set of pods changing, not to a port change. What I can report is that it worked, which is consistent with a Service being a routing rule rather than part of the pod.

The status kubectl shows you that nobody documented

kubectl get po said ErrImagePull. kubectl describe po said Reason: ImagePullBackOff. Same container, seconds apart.

They are two stages of one failure: the pull is attempted and fails, then the kubelet waits before trying again. ImagePullBackOff is documented, including the fact that "Kubernetes raises the delay between each attempt until it reaches a compiled-in limit, which is 300 seconds (5 minutes)".

ErrImagePull is not. I checked the Images page, the Pod Lifecycle page and the Debug Pods page, and the string appears on none of them. It exists in the kubelet source as ErrImagePull = errors.New("ErrImagePull") and nowhere in the documentation. So one of the two statuses you will see most often during an image failure cannot be looked up.

What the docs recommend

Read the port the container declares rather than guessing the framework default. The Deployment said Port: 5000/TCP, which is the containerPort field, and that field is informational: "Not specifying a port here DOES NOT prevent that port from being exposed." Declaring it opens nothing and omitting it closes nothing. It is documentation that happens to be machine-readable, and on this task it was the fastest route to the answer.

5000 is also Flask's default, and the default is conditional. Flask's reference gives the port as "the port of the webserver. Defaults to 5000 or the port defined in the SERVER_NAME config variable if present", with the host defaulting to 127.0.0.1 on the same condition. Two conditions are one more than most people remember, which is an argument for reading the manifest.

Work through the Waiting checklist in order. The debugging guide: a pod stuck in Waiting "has been scheduled to a worker node, but it can't run on that machine", and "The most common cause of Waiting pods is a failure to pull the image." Its three checks are the image name, whether the image was pushed, and whether you can pull it by hand.

Task 2: Create a managed disk

The task: create xfusion-disk, Standard_LRS, 2 GiB. The title says "Create and Attach Managed Disks", and the numbered requirements mention no VM.

Step 1: Check whether there is anything to attach to

az vm list -g $RG --query "[].{Name:name}" -o table

(empty)

No VM exists in the resource group, so attaching was not merely unrequested, it was impossible. The requirements are the contract and the title describes a topic area. Day 11 said "Using Console" and was done with the CLI for the same reason.

Step 2: Create it

az disk create -g $RG -n xfusion-disk --size-gb 2 --sku Standard_LRS

"diskSizeGB": 2, "diskSizeBytes": 2147483648, "diskState": "Unattached"

2147483648 is 2 × 1024³, which confirms the Day 8 finding that Azure labels disk sizes GB and provisions GiB.

The explanation I had to withdraw

Querying the size gave two different answers:

az disk create ... --query "{GB:diskSizeGB}"   ->  "GB": null
az disk show   ... --query "{GB:diskSizeGB}"   ->  "GB": 2

On Day 8 I had found that az disk show prints diskSizeGB while az vm show printed diskSizeGb, and I generalised: the response shape belongs to the command, not the resource type, and create and show disagree even within az disk.

It is a neat rule, and it is not true. Both commands are generated from the same schema, and in both generated files the line is identical:

properties.disk_size_gb = AAZIntType(serialized_name="diskSizeGB",)

So the null was not caused by the two commands naming the field differently. The likeliest remaining explanations are the shape of the object my query ran against, since create returns the result of a long-running operation, or a nesting difference between the top level and properties. I did not work out which, and I am not going to publish a second tidy theory to replace the first one.

What survives is the habit, which never depended on the theory being right:

az disk show -g $RG -n xfusion-disk -o json | grep -i disksize

grep -i matches whatever the casing and whatever the nesting. When a value you know exists comes back null, look at the raw object rather than reasoning about why.

2 GiB costs the same as 32

This is the part that actually matters for a 2 GiB disk, and Microsoft states it plainly: "managed disks are billed according to their provisioned size. Azure maps the provisioned size (rounded up) to the nearest offered disk size." The disks FAQ works an example: "A standard managed disk created from an 80-GiB VHD is treated as the next available standard disk size, which is an S10 disk."

The Standard HDD tiers start at S4:

Tier	S4	S6	S10	S15	S20	S30
Size in GiB	32	64	128	256	512	1,024

A 2 GiB disk lands on S4 and is billed as 32 GiB, with 30 GiB of unused space you have already paid for. Asking for less than the smallest tier saves nothing.

It is specific to Standard HDD. Premium SSD starts at P1 with 4 GiB and Standard SSD at E1 with 4 GiB, so the floor is much lower there.

What the docs recommend

Find the orphans. The disk exists, is fully provisioned, is fully billed and is attached to nothing:

az disk list --query "[?diskState=='Unattached'].{Name:name,RG:resourceGroup,GB:diskSizeGB,Sku:sku.name}" -o table

Microsoft's warning is about what survives a VM: "After a VM is deleted, you will continue to pay for unattached disks." It is the storage equivalent of the unattached public IP from Day 7.

Pass the region explicitly when it matters. Leaving --location off worked because the task named no region, and the CLI documents the fallback per command: "If location is not specified and no default location specified, location will be automatically set as same as the resource group." That is a statement about az disk create, not a general Resource Manager rule.

Snapshot before deleting. "Deleting an unattached managed disk is permanent. You can't recover its data after deletion." Azure Advisor's version is the one to act on: "Deleting a disk is irreversible. Create a snapshot before deletion and confirm the data is no longer needed."

And know that Standard_LRS means different things in different places. On a disk, it is "Standard HDD locally redundant storage", with LRS replicating "your data three times within a single data centre in the selected region". On a storage account, which is Day 66, the same string describes a performance tier and a redundancy option with no HDD involved.

What I learned

The Kubernetes half was ordinary: two typos in two objects, one hiding the other, found by reading what the system already said about itself. The Azure half was the useful one, because it made me take a rule out of my notes rather than add one.

The rule was plausible, it explained something I had actually seen, and it was wrong. What saved the note was that the practical advice sitting on top of it, look at the raw object when a query returns null, works regardless of the cause.

So here is the Day 64 question. Which rule of thumb are you carrying that explains something you really observed, and has never been checked against the source?

Day 64 down. Thirty-six to go.

Day 63: Two Tiers and a Root Key

Nnamdi Felix Ibe — Thu, 15 Oct 2026 17:00:00 +0000

Day 63 of DevOps, Day 13 of Azure. Both tasks today had something already written before I arrived. Kubernetes filled in resource requests I never typed, which changed the pod's QoS class. And cloud-init had already put a line in root's authorized_keys on the Azure VM, which is why adding my key to it did not work on its own.

One Kubernetes task, one Azure task. Deploy a frontend and a database into their own namespace, then install a root SSH key on a VM with no SSH access to start from. The tasks come from the KodeKloud Engineer platform.

Task 1: A two-tier app in its own namespace

The task: a namespace, iron-namespace-nautilus, holding a frontend Deployment, a MariaDB Deployment, a ClusterIP Service for the database on 3306, and a NodePort Service for the frontend on 32678.

Step 1: Create the namespace

kubectl create ns iron-namespace-nautilus
kubectl get ns

NAME                      STATUS   AGE
default                   Active   96m
iron-namespace-nautilus   Active   23m
kube-node-lease           Active   96m
kube-public               Active   96m
kube-system               Active   96m

The docs: "namespaces provide a mechanism for isolating groups of resources within a single cluster", and "Names of resources need to be unique within a namespace, but not across namespaces."

Step 2: The frontend Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iron-gallery-deployment-nautilus
  labels:
    run: iron-gallery
  namespace: iron-namespace-nautilus
spec:
  replicas: 1
  selector:
    matchLabels:
      run: iron-gallery
  template:
    metadata:
      labels:
        run: iron-gallery
    spec:
      volumes:
      - name: config
        emptyDir: {}
      - name: images
        emptyDir: {}
      containers:
      - name: iron-gallery-container-nautilus
        image: kodekloud/irongallery:2.0
        resources:
          limits:
           memory: "100Mi"
           cpu: "50m"
        volumeMounts:
        - name: config
          mountPath: /usr/share/nginx/html/data
        - name: images
          mountPath: /usr/share/nginx/html/uploads

Every object needs namespace: in its metadata, or it lands in default. Every kubectl call afterwards needs -n. That is the cost of the isolation, and it is easy to forget halfway through a task.

Step 3: The database Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iron-db-deployment-nautilus
  labels:
    db: mariadb
  namespace: iron-namespace-nautilus
spec:
  replicas: 1
  selector:
    matchLabels:
      db: mariadb
  template:
    metadata:
      labels:
        db: mariadb
    spec:
      volumes:
      - name: db
        emptyDir: {}
      containers:
      - name: iron-db-container-nautilus
        image: kodekloud/irondb:2.0
        volumeMounts:
        - name: db
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_DATABASE
          value: database_apache
        - name: MYSQL_ROOT_PASSWORD
          value: testroot123
        - name: MYSQL_PASSWORD
          value: test123
        - name: MYSQL_USER
          value: nileena

Those are the task's credentials, in the task's plain-text form. They are also the anti-pattern, and Day 66 does the same job with Secrets instead.

Step 4: Two Services, two types

apiVersion: v1
kind: Service
metadata:
  name: iron-db-service-nautilus
  namespace: iron-namespace-nautilus
spec:
  selector:
    db: mariadb
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
---
apiVersion: v1
kind: Service
metadata:
  name: iron-gallery-service-nautilus
  namespace: iron-namespace-nautilus
spec:
  type: NodePort
  selector:
    run: iron-gallery
  ports:
    - port: 80
      targetPort: 80
      nodePort: 32678

The database Service has no type, so it is ClusterIP, which the docs describe as making the Service "only reachable from within the cluster". That is the whole security model for the database here: it has no route in from outside.

Step 5: Apply, then check the endpoints

kubectl apply -f deploy-1.yaml
kubectl apply -f deploy-2.yaml
kubectl apply -f svc-1.yaml
kubectl apply -f svc-2.yaml

kubectl get po -n iron-namespace-nautilus
kubectl describe svc -n iron-namespace-nautilus

NAME                                               READY   STATUS    RESTARTS   AGE
iron-db-deployment-nautilus-6cbdbfbb4d-tgvkz       1/1     Running   0          30s
iron-gallery-deployment-nautilus-87cf5796b-9m2l2   1/1     Running   0          3m26s

Endpoints:                10.22.0.10:3306
Endpoints:                10.22.0.9:80

The Endpoints lines are the real check. Neither Service names a Deployment anywhere; they match on db: mariadb and run: iron-gallery. Two pod IPs in the Endpoints means the labels lined up. An empty list would mean they did not, and the Service would still look perfectly healthy.

The QoS class I did not ask for

Limits:
  cpu:     50m
  memory:  100Mi
Requests:
  cpu:        50m
  memory:     100Mi
QoS Class:    Guaranteed

I wrote limits. I did not write requests. Kubernetes wrote them for me, and the docs say so: when a container specifies a limit and no request, "Kubernetes copies the limit you specified and uses it as the requested value for the resource."

Equal requests and limits on every resource in every container is the definition of Guaranteed. So the shortest path to the strongest QoS class is to specify less, not more, which is the reverse of what Day 50 led me to expect.

The database pod in the same namespace is BestEffort, because its manifest has no resources block at all. One application, two tiers, two QoS classes, and the difference is four lines of YAML.

What the docs recommend

Give the database real storage. Both deployments use emptyDir, and the docs are unambiguous about what that means: "When a Pod is removed from a node for any reason, the data in the emptyDir is deleted permanently." The schema, the user and every row go with the pod. That is acceptable for a task that only asks whether the page loads, and it is the thing to change first.

The docs also point at the workload type. StatefulSets exist for applications needing "Stable, unique network identifiers" and "Stable, persistent storage", which is a database in two bullets.

Pick the node port deliberately. 32678 falls in the dynamic band of the default range, 30086 to 32767, which is where Kubernetes allocates automatic ports from first. The static band below it is where a hand-picked number is least likely to collide with one the cluster assigned to somebody else.

Do not assume a third-party image honours the usual variables. kodekloud/irondb:2.0 has no entry in the official image documentation. The MYSQL_* variables show up in kubectl describe pod because I set them, which proves they were delivered to the container and nothing more. Even the official mariadb image only documents MARIADB_DATABASE among these and defers the rest elsewhere.

One honest note about my own transcript. kubectl get svc printed 80/TCP for the database Service while kubectl describe svc printed 3306/TCP for the port, the target port and the endpoint. The manifest says 3306 and three of the four agree, so the get line is almost certainly a copy error on my part rather than a real state. I did not re-run it, so I am flagging it rather than quietly picking the convenient one.

Task 2: Install a root SSH key without SSH

The task: put the client host's root public key into nautilus-vm's root authorized_keys, and prove password-less root login works.

Step 1: Get a shell without having a shell

The obvious route needs credentials for azureuser, which the task did not give me. Azure has a way round that:

az vm run-command invoke -g $RG -n nautilus-vm --command-id RunShellScript --scripts "..."

Microsoft: "The Run Command feature uses the virtual machine (VM) agent to run shell scripts within an Azure Linux VM", and "Scripts run by default as an elevated user on Linux." No SSH, no key, no open port. It runs over the Azure control plane through the agent already inside the VM.

That is the shape of the whole task: use a channel that does not depend on SSH to set up SSH.

Step 2: Look before writing

The script printed root's existing authorized_keys first:

BEFORE:
no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command="echo 'Please login as the user \"azureuser\"

That line is cloud-init's doing. It copies the admin user's key into root's file behind a forced command, so a root login authenticates successfully, prints the message, and disconnects.

It is identical on AWS and Azure, because it comes from cloud-init rather than from either cloud. Day 35 on the AWS track hit exactly this.

Step 3: Filter rather than append

grep -v 'Please login as' /root/.ssh/authorized_keys \
  | grep -vF '$KEYBODY' > /tmp/ak
echo '$PUBKEY' >> /tmp/ak
mv /tmp/ak /root/.ssh/authorized_keys

The reason not to simply append is that the banner line probably contains this same key already. The lab likely built the VM using /root/.ssh/id_rsa.pub from the client, and cloud-init copied it across with the forced command in front.

I had been carrying an explanation for why that blocks a clean duplicate: sshd uses the first matching line. When I went to check it, the sshd manual does not say that. Its AUTHORIZED_KEYS FILE FORMAT section describes the line format and says a forced command "is executed whenever this key is used for authentication", with no statement anywhere about ordering or precedence between two lines carrying the same key.

So the forced command certainly applies when its line is the one that authenticates. Which line wins is undocumented, and I did not test it. The script above sidesteps the question: drop every banner line, drop any existing copy of this key, append it once. That is correct either way, and it is closer to what adding a key ought to mean than overwriting the file.

-rw------- 1 root root  399 Sep 22 07:33 authorized_keys

Step 4: Fix the permissions before testing, not after

chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys
chown -R root:root /root/.ssh

StrictModes is on by default and "Specifies whether sshd(8) should check file modes and ownership of the user's files and home directory before accepting login."

When that check fails, sshd tells the client nothing. The login quietly falls through to the next authentication method, and the reason goes to the server's auth log, which you cannot read if you cannot log in. Wrong permissions look exactly like a wrong key from the outside.

Step 5: Ask sshd what it will actually do

sshd -T | grep -i permitrootlogin

permitrootlogin without-password

sshd -T is "Extended test mode. Check the validity of the configuration file, output the effective configuration to stdout and then exit." It resolves every include and default, which reading /etc/ssh/sshd_config by hand does not, and cloud images put their overrides in /etc/ssh/sshd_config.d/.

without-password is the legacy spelling. The manual: root may log in with a key "If this option is set to prohibit-password (or its deprecated alias, without-password)", with password authentication disabled for root. Had this printed no, the key would have been irrelevant.

Step 6: Prove it was the key

ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 root@$VM_IP \
  'whoami; hostname; stat -c "%a %U %n" /root/.ssh /root/.ssh/authorized_keys'

root
nautilus-vm
700 root /root/.ssh
600 root /root/.ssh/authorized_keys

BatchMode is what turns this from "login worked" into "login worked without a password". The manual: "If set to yes, user interaction such as password prompts and host key confirmation requests will be disabled."

Worth reading that precisely. It names password prompts and host key confirmation. It says nothing about passphrase prompting for a local key, so it is narrower than "disables all prompting".

What the docs recommend

Do not enable root SSH. The cloud-init banner exists to stop exactly what this task asked for, and a compromised root key is a compromised machine with no sudo trail in between. Microsoft's alternatives, in rough order: log in as azureuser and use sudo; use Microsoft Entra ID login for Linux VMs, which lets you "use Microsoft Entra ID as a core authentication platform and a certificate authority to SSH into a Linux VM" with no keys to distribute; or use Azure Bastion, after which "your virtual machines don't need a public IP address, agent, or special client software".

Know what the convenient command does besides the obvious. az vm user update --ssh-key-value is the other way to install a key, and it "appends the new public key text to the ~/.ssh/authorized_keys file for the admin user on the VM. This does not replace or remove any existing SSH keys." But the VMAccess extension behind it also "grants sudo permission to account specified in the username field", and it "changes ChallengeResponseAuthentication to no and PasswordAuthentication to yes". Turning on password authentication is an odd side effect of a command you reached for to install a key.

Check both NSG levels, and remember the rule is not symmetric. For inbound, "both NSG1 and NSG2 must include a rule that allows port 80 from the internet". For outbound, "either one of or both of the network security groups must have a rule that denies port 80". Microsoft's current advice is to sidestep it: "avoid associating network security groups with both a subnet and its network interfaces at the same time."

What I learned

Both tasks had state in them before I started. Kubernetes supplied requests to match my limits, which quietly upgraded the pod's QoS class. cloud-init had written a line into root's authorized_keys that would have beaten anything I appended after it.

The second one is where I had to change my mind about something I thought I knew. The behaviour I was relying on turns out not to be documented anywhere in the sshd manual, and the fix was to write the script so the answer does not matter.

So here is the Day 63 question. What is already configured in your environment that you have never read, because it was put there before you got the machine?

Day 63 down. Thirty-seven to go.

Part 1: Why NHS Trusts Need an HL7 v2 to FHIR Layer, and How This One Is Built

Nnamdi Felix Ibe — Wed, 14 Oct 2026 13:31:36 +0000

Hi I'm Nnamdi.

This series walks through a project I built around one question: how does a patient's data get from a hospital system that speaks HL7 v2 to services that expect FHIR? Part 1 covers the problem and the architecture. The next six parts go deeper, one area at a time.

The problem

Two standards meet here.

HL7 version 2 is the older one. HL7 International calls it "the workhorse of electronic data exchange in the clinical domain". A v2 message is plain text. Each segment is a line of fields separated by pipes, and the standard says "the segment terminator is always a carriage return". An admission is sent as an ADT^A01 message, which the standard describes as "sent as a result of a patient undergoing the admission process". Its segments include MSH for the message header, PID for patient identification and PV1 for the patient visit.

FHIR is the newer one. It models data as resources, such as Patient and Encounter, and exchanges them as JSON or XML over a REST API.

National NHS services are moving to FHIR. The NHS service manual says the PDS FHIR API "meets the latest standards". It also says that if you want to start using a PDS API, "this is the version to use". The older HL7 V3 version is under review.

So a hospital system that sends HL7 v2 needs something in between. That layer has to translate each message, check the patient against national records, and store the result somewhere that speaks FHIR. This project builds that layer.

What I built

The NHS Interop Platform is a small, production-shaped reference implementation. It isn't running in a hospital. It runs on a laptop with one command, make demo, using example data only.

Five parts do the work:

ingest is a FastAPI service that receives HL7 v2 over HTTP. It checks the MSH header, gives the message a trace ID and passes it on.
transform parses the message and maps it to a FHIR R4 transaction Bundle.
pds-client looks the patient up in the Personal Demographics Service (PDS), the national database of NHS patient details. In the local demo it runs in stub mode by default and returns a canned record, so nothing leaves the laptop. It can call NHS England's PDS FHIR sandbox instead.
A HAPI FHIR server stores the result in Postgres.
Prometheus scrapes the services and Grafana shows the pipeline. Five alert rules cover traffic, latency, errors, dead letters and the PDS circuit breaker.

The repo also holds Terraform for AWS (VPC, EKS, RDS, S3, KMS and IAM), Helm charts and Argo CD definitions. They describe where the platform would run, and I've checked them with terraform plan. Everything in this article runs locally. Part 2 covers the AWS side.

Following one admission through

This is the sample admission from the repo's test fixtures:

MSH|^~\&|CERNER|GENERAL_HOSP|INTEROP|GENERAL_HOSP|20260101120000||ADT^A01|MSG00001|P|2.5
EVN|A01|20260101120000
PID|1||9000000009^^^NHS^NH~L12345^^^GENERAL_HOSP^MR||DOE^JOHN^A||19800101|M|||1 NHS WAY^^LONDON^^SW1A 1AA^GBR
PV1|1|I|WARD1^BED1^ROOM1||||12345^SMITH^JANE|||GEN

PID carries the patient. PID-3 holds two identifiers, separated by ~: the NHS number, with NHS as its assigning authority, and the hospital's own number. PID-5 is the name, PID-7 the date of birth and PID-8 the sex.

This is the FHIR Patient the mapper builds from that segment. I ran the mapper on the fixture to get it, so it's the real output:

{
  "resourceType": "Patient",
  "identifier": [
    { "system": "https://fhir.nhs.uk/Id/nhs-number", "value": "9000000009" },
    { "system": "https://fhir.nhs.uk/Id/local-patient-identifier", "value": "L12345" }
  ],
  "name": [{ "use": "official", "family": "DOE", "given": ["JOHN"] }],
  "gender": "male",
  "birthDate": "1980-01-01"
}

The middle initial, A, isn't mapped. PID-5 has a component for it and FHIR takes more than one entry in given, so that is a real gap rather than a limitation of either format. Part 3 is about the mapping, and it's on the list there.

PV1 becomes an Encounter, and the I in PV1-2 becomes the FHIR class code IMP, an inpatient encounter.

Three things happen next:

Enrichment. transform asks pds-client for the patient by NHS number. The PDS values for name, gender, date of birth, address and telephone replace the ones from the message, and the hospital's own identifier is kept. In the local demo the stub's canned record wins, which is why the stored patient is called Paul Smith rather than John Doe.
The write. transform sends the Patient, the Encounter and a MessageHeader to HAPI FHIR as one transaction Bundle.
The failure path. If mapping or the write fails, the raw message is saved to a dead-letter folder and a counter feeds the DLQRateHigh alert. Replaying it is still a manual step.

Three design decisions

Fail open on enrichment. If PDS is slow or down, the pipeline carries on with the demographics from the message. A circuit breaker stops calling PDS after five failures in a row, and lets one call through again after 30 seconds. A slow lookup shouldn't stop an admission being recorded.

Fail closed on writes. A failed FHIR write is never quietly dropped. The raw message is saved to the dead-letter folder so it can be replayed. More on this one below, because writing this article is how I found out it wasn't true.

One transaction per message. The FHIR specification says that in a transaction, "all interactions or operations either succeed or fail together". An admission is saved whole or not at all, never as a Patient without its Encounter.

The bug I found writing this article

I wanted a screenshot of a dead-lettered message for the section above, so I pointed transform at a FHIR server that wasn't there and sent the fixture in. Nothing arrived in the dead-letter folder, and the service answered 500 rather than 502.

The write path looked fine:

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=0.2, max=2))
async def _persist_bundle(http, bundle, trace_id):
    resp = await http.post(FHIR_BASE_URL, json=bundle, ...)
    resp.raise_for_status()
    return resp.json()

and the caller catches the error the way you would expect:

try:
    result = await _persist_bundle(...)
except httpx.HTTPError as exc:
    _dlq(raw, trace_id, reason="fhir_write_error")
    raise HTTPException(status_code=502, ...)

The retry decorator is the problem. Tenacity's documentation says that when a function fails its final time, "a RetryError is raised", and that the original exception shows up "somewhere in the middle of the stack trace". RetryError is not an httpx.HTTPError, so the except never matched. Every retried write failure escaped the handler, skipped the dead-letter call and came back as an unhandled 500.

The fix is one argument. Tenacity's reraise=True raises the underlying exception instead:

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=0.2, max=2), reraise=True)

The test is the more useful half. It mocks HAPI FHIR with respx, so it needs no running stack, and it asserts the three things the rule actually promises: the retries ran, the raw message is on disk, and the counter went up.

with respx.mock(assert_all_called=True) as hapi:
    route = hapi.post(FHIR_URL).mock(return_value=httpx.Response(503))
    with TestClient(main.app) as client:
        response = client.post("/transform", content=ADT_A01,
                               headers={"X-Trace-Id": "write-fails"})

assert route.call_count == 3
assert response.status_code == 502
assert _dlq_count() == before + 1

written = (tmp_path / "write-fails.hl7").read_bytes()
assert written == ADT_A01.encode("utf-8")
assert b"\n" not in written

That last pair of lines is a second lesson. My first version of the test read the file back with read_text() and failed on a message that was written correctly, because Python's universal newline handling turns the HL7 carriage returns into line feeds on the way in. Reading the bytes is the only way to see what is really on disk.

Both changes are in the code this article is tagged against.

I'd had the "fail closed on writes" rule written down in docs/architecture.md since the first week of the project. It read like a property of the system. It was a property of the documentation.

Worth stealing

Normalise line endings before parsing. HL7 v2 ends segments with a carriage return, but files and editors add line feeds. The parser turns both into carriage returns first, and the repo's .gitattributes stops Git converting the HL7 test fixtures.
Give every message a trace ID at the front door. ingest creates one and passes it on in an X-Trace-Id header, so one admission can be followed through every service's logs.
Test the failure path, not just the happy one. A retry wrapper, a circuit breaker and a dead-letter folder all look right in a diagram. The test is what tells you whether they are wired together.
Make the demo one command. make demo builds and starts the whole stack and waits until the FHIR server answers, so anyone can check the work quickly.

Where AI helps here, and where it doesn't

I built this project with Claude Code, Anthropic's coding assistant. It drafted the scaffolding for all three services, the Helm charts, the Prometheus rules and the Grafana dashboard, and the first pass of most of the docs. I made the mapping and architecture decisions, and I reviewed everything that went in.

It was quickest where the shape was clear. The Helm charts show that best: three services, seven templates each, and the only real differences between them are the name, the port, the environment variables and the resource requests. Once the first chart is right, writing the third one by hand is just typing.

It didn't replace checking, and the failure above is the clearest case. The retry decorator, the except clause and the dead-letter helper are each individually what you would write. Nothing in the code looks wrong until something runs it. Two other passes over the repo turned up the same kind of thing: two AWS statements in the docs that no longer matched the Terraform, a description of an in-memory queue that the code doesn't have, and an alert rule, PDSCircuitOpen, that watches a metric called pybreaker_state which none of the services export. That last one can't fire at all. Part 4 is about observability, and fixing it belongs there.

The rule I'm keeping: AI drafts, and the specification, the tests and the running system decide.

What I'd do differently

I'd use SQS from the start, not a synchronous HTTP hand-off. Right now, if transform is down, ingest logs the error and still replies 200, marked forwarded: false, and nothing retries the message. That is the same failure as the one above, one service earlier, and the dead-letter folder doesn't help because the message never reached the service that owns it. A queue makes the hand-off durable instead of hopeful. It's the first item on the roadmap, and Part 6 shows the fix.

Acronyms used in this article

Acronym	Stands for	What it means here
ADT	Admission, Discharge and Transfer	The family of HL7 v2 messages a hospital sends when a patient is admitted, moved or sent home
A01	Admit/Visit Notification	The ADT event for a patient being admitted
AI	Artificial intelligence	Here, the coding assistant used to draft parts of the project
API	Application programming interface	A defined way for one piece of software to ask another for data
AWS	Amazon Web Services	Amazon's cloud platform, where the project is designed to run
CD	Continuous delivery	Shipping changes automatically once they pass checks (as in Argo CD)
DLQ	Dead-letter queue	A holding place for messages that couldn't be processed, so they aren't lost. This project uses a folder
EKS	Elastic Kubernetes Service	Amazon's managed service for running containers with Kubernetes
EVN	Event Type segment	The line of an HL7 v2 message that says what happened and when
FHIR	Fast Healthcare Interoperability Resources	The modern health data standard, built on web technology
HAPI	HL7 application programming interface	An open-source project; HAPI FHIR is the FHIR server used here
HL7	Health Level Seven	The organisation behind both standards. The "Seven" refers to the top layer of a standard networking model, where applications talk to each other
HTTP	Hypertext Transfer Protocol	The protocol web browsers and web services use to talk
IAM	Identity and Access Management	AWS's system for who and what is allowed to do what
IMP	Inpatient	The FHIR code for a patient admitted to a ward
JSON	JavaScript Object Notation	A common text format for structured data
KMS	Key Management Service	AWS's service for managing encryption keys
MSH	Message Header segment	The first line of every HL7 v2 message: who sent it, to whom, and what type it is
NHS	National Health Service	The UK's public health service
PDS	Personal Demographics Service	The national database of NHS patient details, such as name, address and NHS number
PID	Patient Identification segment	The line of an HL7 v2 message that describes the patient
PV1	Patient Visit segment	The line of an HL7 v2 message that describes the hospital stay
R4	Release 4	The version of FHIR this project uses
RDS	Relational Database Service	Amazon's managed database service
REST	Representational State Transfer	A common style for building web APIs
S3	Simple Storage Service	Amazon's file storage service
SQS	Simple Queue Service	Amazon's message queue service, which holds messages until they're processed
v2, V3	Version 2, Version 3	Versions of the HL7 standard. v2 is the older, widely used one
VPC	Virtual Private Cloud	A private, isolated network inside AWS
XML	Extensible Markup Language	An older text format for structured data

Sources

Part 2, on 21 October, goes from docker compose up to Terraform on AWS.

What does the integration engine in your organisation still speak: HL7 v2, FHIR, or both?

This is a personal project. It is not affiliated with or endorsed by NHS England or any NHS organisation. It uses the public PDS FHIR sandbox and synthetic test data only. Views are my own.

Day 62: Secrets and Tags

Nnamdi Felix Ibe — Mon, 12 Oct 2026 23:27:36 +0000

Day 62 of DevOps, Day 12 of Azure. Both tasks today turned on a name I did not choose. Kubernetes named the file inside my Secret after the file on disk, and Azure's tag commands disagree among themselves about what setting a tag means.

One Kubernetes task, one Azure task. Mount a Secret into a pod as a file, then add a single tag to a VM. The tasks come from the KodeKloud Engineer platform.

Task 1: Mount a Secret as a file

The task: create a generic Secret called official from /opt/official.txt, then run a pod, secret-xfusion, with one container, secret-container-xfusion, on debian:latest, that mounts the Secret at /opt/demo.

Step 1: Look at what is going in

cat /opt/official.txt

5ecur3

Step 2: Create the Secret from the file

kubectl create secret generic official --from-file=/opt/official.txt

secret/official created

generic is the subcommand for arbitrary data, and it produces a Secret of type Opaque, which the docs describe as "arbitrary user-defined data". The other types are for specific jobs: registry credentials, TLS material, SSH keys, service account tokens.

Step 3: Find out what the key is called

kubectl get secret
kubectl describe secret

NAME       TYPE     DATA   AGE
official   Opaque   1      4s

Data
====
official.txt:  7 bytes

This is the detail the whole task rests on. The key is official.txt, not official. The docs: "The default key name is the file name." It matters because the key is what becomes the filename inside the container, so the path you mount at is only half the address.

Seven bytes is 5ecur3 plus the trailing newline. Note what describe does not print: the value. It gives you the key and a size, and nothing else.

Step 4: Mount it in a pod

apiVersion: v1
kind: Pod
metadata:
  name: secret-xfusion
spec:
  volumes:
    - name: secret-volume
      secret:
        secretName: official
  containers:
    - name: secret-container-xfusion
      image: debian:latest
      command: ['sleep','6000']
      volumeMounts:
        - name: secret-volume
          readOnly: true
          mountPath: "/opt/demo"

kubectl apply -f po.yaml
kubectl get po

NAME             READY   STATUS    RESTARTS   AGE
secret-xfusion   1/1     Running   0          4s

sleep 6000 is there because debian:latest has nothing to run. Without it the container exits immediately and there is nothing to inspect.

Step 5: Read it from inside the container

kubectl exec -it secret-xfusion -c secret-container-xfusion -- bash
ls /opt/demo/
cat /opt/demo/official.txt

official.txt
5ecur3

One key, one file. The docs: "Each key in the Secret data map becomes a file name in this directory." A Secret with four keys would have put four files in /opt/demo.

The read-only flag that is not doing the work

kubectl describe pod shows the mount like this:

Mounts:
  /opt/demo from secret-volume (ro)

I had written readOnly: true in the manifest, so the ro looks like my doing. It is not. Kubernetes mounts Secret volumes read-only whatever the manifest says, and the volume plugin in the Kubernetes source reports ReadOnly: true unconditionally.

The documentation does not settle this for Secrets in either direction. What it does say, on the ConfigMap page, is to set the flag, which is probably where most of us picked up the habit. So the line is worth keeping as a statement of intent. It just is not the thing producing the result.

What the docs recommend

Prefer a volume to an environment variable. Both work, and they behave differently when the Secret changes. Mounted files are updated by the kubelet on their own. A Secret consumed as an environment variable "will not be seen by the container unless it is restarted". Day 66 uses the environment variable form, and that is the trade it makes.

Do not promise a one-minute update. The figure that gets repeated is the kubelet sync period, and the docs are careful to make it a sum: the total delay "can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type". The sync period defaults to one minute. The second term has no documented default, so there is no documented ceiling.

Know what the storage actually is. The caution is blunt: "Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd." Encryption at rest and tight RBAC are the two steps the docs put first.

Keep them small. "Individual Secrets are limited to 1MiB in size. This is to discourage creation of very large Secrets that could exhaust the API server and kubelet memory."

And watch out for the update that is not an update. This looks like it refreshes the Secret in place:

kubectl create secret generic official --from-file=/tmp/new-official.txt \
  --dry-run=client -o yaml | kubectl apply -f -

It does not. The key still defaults to the base name, so the manifest carries new-official.txt, and data is a map that apply merges key by key. The pod ends up with a second file while /opt/demo/official.txt keeps its old contents. The form that updates in place names the key explicitly:

kubectl create secret generic official --from-file=official.txt=/tmp/new-official.txt \
  --dry-run=client -o yaml | kubectl apply -f -

I did not run the first version, so that outcome is reasoning from the documented key naming and merge rules rather than something I watched break.

Task 2: Add one tag to a VM

The task: put Environment=dev on the VM devops-vm. One tag, one command, and a choice of command that matters more than the tag does.

Step 1: Check what is already there

az vm show -g $RG -n devops-vm --query "tags"

{}

An empty object. Not null, not an empty string. That is the fourth distinct way these labs have spelled "nothing" so far, after a missing field, an unset field and a computed empty string, and it decides how you test for it: length(tags) returns 0 for {}, while a null check matches nothing.

Step 2: Choose the command that says what it means

az tag update --resource-id $VM_ID --operation Merge --tags Environment=dev

Azure has several ways to set a tag and they do not mean the same thing. az tag update --operation Merge "allows adding tags with new names and updating the values of tags with existing names". Replace "replaces the entire set of existing tags with a new set". Delete removes the ones you name.

The one to be careful with is az resource tag, which reads like "add this tag" and whose reference describes it only as "Tag a resource."

Step 3: Verify

az vm show -g $RG -n devops-vm --query "{Tags:tags,Power:powerState}" -d

{ "Tags": { "Environment": "dev" }, "Power": "VM running" }

VM running before and after. Nothing restarted.

I went looking for a Microsoft sentence confirming that tagging never restarts a VM and could not find one, so that is an observation from this run rather than a guarantee. The closest related statement points the other way: "you can only update tags on virtual machine extensions when the virtual machine is running."

The warning the docs decline to give

I wanted to write that az resource tag replaces all tags unless you pass --is-incremental. The reference will not quite support it.

What the reference actually contains is the description of the flag, "The option to add tags incrementally without deleting the original tags", and an example captioned "It doesn't empty the existing tags." Both sentences are only meaningful if the default does delete and does empty. The behaviour is a sound inference. It is not a documented statement, and on a VM carrying Owner, CostCentre and Project the difference between those two things is three tags.

That is enough reason to use the command that names its operation. It is not enough to assert the other command's behaviour as fact without running it.

It also made no difference here, which is worth admitting. The VM had no tags, so Merge and Replace would have produced identical output. The command has to be chosen before you know whether it matters.

What the docs recommend

Expect no inheritance. "Resources don't inherit the tags you apply to a resource group or a subscription." Tag a VM and its disks, NIC and public IP stay untagged, which quietly splits a cost report: compute under dev, storage under nothing.

Azure Policy is the fix, and the policy names are close enough to misread. "Inherit a tag from the resource group" uses a modify effect and "Adds or replaces the specified tag and value from the parent resource group". "Inherit a tag from the resource group if missing" leaves an existing value alone. "Append a tag and its value from the resource group" does not touch resources created before the assignment. Picking the wrong one either overwrites deliberate values or silently skips everything that already exists.

Standardise the casing of values. "Tag names are case-insensitive for operations", but "Tag values are case-sensitive". So Environment and environment reach the same tag, while Dev and dev are two different answers in a cost report.

Know the limits before designing a tagging scheme: 50 tag name-value pairs per resource, names up to 512 characters, values up to 256. Storage accounts cap names at 128. Five resource types allow only 15 tags, Azure Automation and the DNS services among them. And tag names cannot contain <, >, %, &, \, ? or /.

Query them server-side, since that is the entire point of having them:

az resource list --tag Environment=dev --query "[].{Name:name,Type:type,RG:resourceGroup}" -o table

Note that --tag will not combine with --resource-group, and the CLI's source shows it also refuses --namespace, --resource-type, --name and --location. The error can even fire on a filter you did not type, because a configured CLI default counts.

Compare with AWS, which solves this by having one command per intent. aws ec2 create-tags "Adds or overwrites only the specified tags for the specified Amazon EC2 resource or resources", and removing tags is a separate delete-tags call. There is no flag to get wrong.

What I learned

Both tasks hid their real behaviour behind a name. The Secret's key came from the filename, so the path inside the container was decided by something I typed on the host. The tag command's behaviour came from which of four commands I reached for, and the documentation describes the safe one clearly while leaving the dangerous one's default to be inferred.

The second case is the one I will remember, because the inference is almost certainly right and I still could not write it down as a fact.

So here is the Day 62 question. When a tool's documentation describes the safe option carefully and says nothing about the default, which one do you assume you are using?

Day 62 down. Thirty-eight to go.

Day 61: Init Containers and a VM Resize

Nnamdi Felix Ibe — Sun, 11 Oct 2026 08:00:00 +0000

Day 61 of DevOps, Day 11 of Azure. Both tasks today were about something that has to finish before the real work starts. An init container must exit before the application container is allowed to run. And a VM has to restart, sometimes on different hardware, before a new size takes effect.

One Kubernetes task, one Azure task. Run an init container that prepares a file for the main container, then resize a VM from one size to the next. The tasks come from the KodeKloud Engineer platform.

Task 1: An init container that prepares a file

The task: a Deployment called ic-deploy-devops with one replica. An init container, ic-msg-devops, writes a message to a file on a shared volume. The main container, ic-main-devops, prints that file every five seconds.

Step 1: Write the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ic-deploy-devops
  labels:
    app: ic-devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ic-devops
  template:
    metadata:
      labels:
        app: ic-devops
    spec:
      initContainers:
      - name: ic-msg-devops
        image: debian:latest
        command: ['/bin/bash', '-c', 'echo Init Done - Welcome to xFusionCorp Industries > /ic/media']
        volumeMounts:
        - name: ic-volume-devops
          mountPath: /ic
      containers:
      - name: ic-main-devops
        image: debian:latest
        command: ['/bin/bash', '-c', 'while true; do cat /ic/media; sleep 5; done']
        volumeMounts:
        - name: ic-volume-devops
          mountPath: /ic
      volumes:
      - name: ic-volume-devops
        emptyDir: {}

There are two lists of containers and one volume. Both containers mount the same emptyDir at /ic, and that volume is the only thing connecting them. The init container writes /ic/media and exits. The main container reads it.

Step 2: Apply and watch the pod start

kubectl apply -f k3s-deployment.yaml
kubectl get deployments.apps
kubectl get pods -w

The pod passes through Init:0/1, then PodInitializing, then Running. The Kubernetes docs read Init:N/M as: "The Pod has M Init Containers, and N have completed so far." So Init:0/1 means the one init container has not finished yet.

Step 3: Confirm the init container finished

kubectl logs <pod-name> -c ic-msg-devops
kubectl describe pod <pod-name> | grep -A 20 "Init Containers"

The logs are empty, and that is correct. The echo is redirected into the file, so nothing goes to stdout. The evidence is in describe: State Terminated, Reason Completed, Exit Code 0.

Step 4: Confirm the main container reads the file

kubectl logs <pod-name> -c ic-main-devops

Init Done - Welcome to xFusionCorp Industries
Init Done - Welcome to xFusionCorp Industries

A new line appears every five seconds. To skip looking up the pod name, kubectl accepts the Deployment instead:

kubectl logs deployment/ic-deploy-devops -c ic-main-devops

Step 5: Delete the pod and watch it run again

kubectl delete pod -l app=ic-devops
kubectl get pods -w

The Deployment creates a replacement, and the replacement goes through Init:0/1 again. The file was on an emptyDir, so it went with the old pod, and the init container writes it afresh for the new one.

Two things my notes had wrong

My notes said the pod shows Init:Running while the init container runs. That is not a status the docs list. Their table has Init:N/M, Init:Error and Init:CrashLoopBackOff, and the count form is what kubectl prints while an init container is still going.

They also said init containers run again "even if main containers are restarted". The Pod Lifecycle page says otherwise: "init containers run only once (if successful), during Pod startup." By default, a main container crashing and restarting does not bring the init container back. A new pod does, which is what Step 5 showed. The same page describes one opt-in exception, a feature-gated restart rule called RestartAllContainers, which re-runs init containers too.

What the docs recommend

The basic contract comes first. "Init containers always run to completion", and "Each init container must complete successfully before the next one starts." If one fails, "the kubelet repeatedly restarts that init container until it succeeds", and the application container never starts in the meantime.

Write them to be safe to repeat. The docs: "Because init containers can be restarted, retried, or re-executed, init container code should be idempotent." Mine is, because > overwrites the file. An init container that appended with >>, or created something that must not already exist, would behave differently on its second run.

Do not expect probes. Regular init containers do not support livenessProbe, readinessProbe, startupProbe or lifecycle. They are meant to exit, so there is nothing to keep checking.

Count their resources. The scheduler does not add init containers to the total. It takes the highest request among them, compares it with the sum of the application containers' requests, and uses the larger. A heavy setup step can make a pod need more room than the application itself ever uses.

Know the related form from Day 55. A native sidecar is declared in the same initContainers list, with one extra line, restartPolicy: Always. That line is the whole difference: with it, the container starts first and keeps running; without it, the container must exit before anything else starts.

And debian:latest was the task's choice. Outside a lab, pin the tag.

Task 2: Resize a VM

The task: change nautilus-vm from Standard_B1s to Standard_B2s and leave it running.

Step 1: Ask what this VM can resize to

az vm list-vm-resize-options -g $RG -n nautilus-vm --query "[?name=='Standard_B2s']" -o table

Name          Cores
------------  -------
Standard_B2s  2

A VM runs on a hardware cluster that supports a particular set of sizes. Microsoft: "Deallocation may be required if the new size isn't available on the hardware cluster currently hosting the VM." This query lists the sizes this one VM can be resized to. A row means Standard_B2s is on that list. An empty result means it is not, and that is the case where deallocating first may be needed.

Step 2: Resize

The query listed Standard_B2s. I deallocated anyway:

az vm deallocate -g $RG -n nautilus-vm
az vm resize -g $RG -n nautilus-vm --size Standard_B2s
az vm start -g $RG -n nautilus-vm

The shorter route would have been the middle command by itself:

az vm resize -g $RG -n nautilus-vm --size Standard_B2s

My notes recorded the three-command version as the slower path, taken when a cheaper one was available. That is fair, with one qualification: Microsoft's own example script does the same. It checks the resize options and then "deallocates the VM, resizes it, and starts it again". The one-command version does not release the VM from its hardware first, so it should mean less downtime, although Microsoft's page gives no timings.

Neither avoids a restart. Microsoft: "Even when deallocation isn't required, changing the size of a running VM will cause it to restart."

Step 3: Verify size and state separately

az vm show -g $RG -n nautilus-vm -d --query "{Size:hardwareProfile.vmSize,Power:powerState,Prov:provisioningState}"
az vm get-instance-view -g $RG -n nautilus-vm --query "{Status:instanceView.statuses[].displayStatus,Agent:instanceView.vmAgent.statuses[].displayStatus}"

{ "Size": "Standard_B2s", "Power": "VM running", "Prov": "Succeeded" }
{ "Agent": ["Ready"], "Status": ["Provisioning succeeded", "VM running"] }

There are two requirements, so there are two checks. Size covers the resize. VM running with agent Ready covers the running state, and as on Day 58, the agent is the half that proves the guest came back up.

One caveat about the first check comes from Microsoft's resize page. If a resize fails, "the VM model will still display the requested size, but the VM will continue running on its previous size until the resize is successfully allocated". So the size field reports what was asked for. A CPU count from inside the guest would settle it, using Day 58's Run Command:

az vm run-command invoke -g $RG -n nautilus-vm --command-id RunShellScript --scripts "nproc"

On a Standard_B2s it should print 2.

The column that vanished

My first version of the Step 1 query asked for three fields:

--query "[?name=='Standard_B2s'].{Name:name,Cores:numberOfCores,MemoryMB:memoryInMb}" -o table

Two columns came back. MemoryMB was not blank. It was gone.

Two things combined. The property is spelt memoryInMB in the REST reference, with a capital B, so my memoryInMb matched nothing and returned null. Then -o table dropped the column, because every value in it was null. A wrong field name produced output that looked complete, with no error anywhere.

It is Day 58's lesson again, one step worse. Write queries against -o json, where a null is at least visible, and switch to -o table once the field names are confirmed.

What the docs recommend

Check before you deallocate. az vm list-vm-resize-options is scoped to one VM and answered on the server. It is the opposite of the az vm list-skus call that ended two lab sessions on Day 59.

Plan for the restart. Anything in memory is lost, and the temporary disk from Day 52 cannot be relied on. If you deallocate, Microsoft adds that it "also releases any dynamic IP addresses assigned to the VM. The OS and data disks are not affected."

Know what the new size changes. From Microsoft's size table:

	Standard_B1s	Standard_B2s
vCPUs	1	2
Memory	1 GiB	4 GiB
Temporary disk	4 GiB	8 GiB
Max data disks	2	4
Max NICs	2	3

The limits matter in reverse. A VM using three NICs, or more than two data disks, would not fit back into a B1s.

I removed one claim. My notes said B-series CPU credits are lost when a VM is deallocated. I could not find that in Microsoft's current B-series pages, which describe the credits a VM starts with and say nothing about stopping it. So it is not in this post as a fact.

What I learned

An init container is a rule about order: this must succeed before that may begin. A resize has the same shape, with the restart as the step you cannot skip. In both, the question worth asking early is what has to finish first, and what it costs while it does.

So here is the Day 61 question. What in your system assumes a setup step runs exactly once, and what happens the day it runs twice?

Day 61 down. Thirty-nine to go.

Day 60: Persistent Volumes and IP Configuration

Nnamdi Felix Ibe — Sat, 10 Oct 2026 08:00:00 +0000

Day 60 of DevOps, Day 10 of Azure. Both tasks today had an object in the middle that the task never names. A pod does not mount a PersistentVolume; it mounts a claim, and the claim finds the volume. A VM does not hold a public IP; an IP configuration on its network interface does.

One Kubernetes task, one Azure task. Give a web server pod storage that outlives it, then attach an existing public IP to an existing VM. The tasks come from the KodeKloud Engineer platform.

Task 1: Persistent storage for a pod

The task: a 3Gi PersistentVolume called pv-xfusion on a host path, a claim called pvc-xfusion, a pod called pod-xfusion running httpd:latest with the claim mounted, and a NodePort Service called web-xfusion on port 30008.

Step 1: Create the PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-xfusion
spec:
  capacity:
    storage: 3Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  hostPath:
    path: /mnt/security

kubectl apply -f k3s-pv.yaml
kubectl get pv

The volume shows STATUS Available. It exists, and nothing is using it yet.

Step 2: Claim it

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-xfusion
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
  storageClassName: manual

kubectl apply -f k3s-pvc.yaml
kubectl get pvc

Now both show Bound, and the claim's VOLUME column reads pv-xfusion. I never told the claim which volume to use. Kubernetes matched them on three things: the same storageClassName, a compatible access mode, and enough capacity.

Step 3: Mount the claim in a pod

apiVersion: v1
kind: Pod
metadata:
  name: pod-xfusion
  labels:
    app: xfusion
spec:
  containers:
  - name: container-xfusion
    image: httpd:latest
    volumeMounts:
    - name: xfusion-volume
      mountPath: /usr/share/nginx/html/data
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
      requests:
        memory: "64Mi"
        cpu: "250m"
  volumes:
  - name: xfusion-volume
    persistentVolumeClaim:
      claimName: pvc-xfusion

kubectl apply -f k3s-pod.yaml

The pod names the claim, not the volume. That is the whole design: the pod says what it needs, and what actually provides the storage can change without the pod's spec changing.

Step 4: Expose the pod

apiVersion: v1
kind: Service
metadata:
  name: web-xfusion
  labels:
    app: xfusion
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30008
  selector:
    app: xfusion

kubectl apply -f k3s-svc.yaml
kubectl get pods
kubectl get svc web-xfusion

The pod is Running, and the Service shows 80:30008/TCP.

Step 5: Prove the data outlives the pod

kubectl exec pod-xfusion -- sh -c 'echo "Hello from persistent storage" > /usr/share/nginx/html/data/test.html'

kubectl delete pod pod-xfusion
kubectl apply -f k3s-pod.yaml

kubectl exec pod-xfusion -- cat /usr/share/nginx/html/data/test.html

Hello from persistent storage

It is a new pod, and the file is still there, because it was never in the pod. It sits in /mnt/security on the node, held by a volume and a claim that both outlived the pod.

A path from the wrong image

Look at the mount path again: /usr/share/nginx/html/data. That sits under /usr/share/nginx/html, the nginx image's document root. This pod runs httpd, and the official httpd image serves from /usr/local/apache2/htdocs/.

So the storage worked exactly as the test shows, and Apache was never serving it. A request to the Service on port 30008 would get the image's default page, and nothing from the volume. The persistence test passed because it only tested the volume.

If the volume is meant to hold the site, the mount for this image is:

volumeMounts:
- name: xfusion-volume
  mountPath: /usr/local/apache2/htdocs

That has its own consequence, the one from Day 55: a mount hides what the image had at that path. Apache's default page would be covered, and the site would serve whatever is on the volume.

What the docs recommend

Keep hostPath for single-node testing. The Kubernetes docs list it as "for single node testing only; WILL NOT WORK in a multi-node cluster", and say of the volume type in general: "If you can avoid using a hostPath volume, you should." On a cluster with several nodes, my recreated pod could land on a different node, where /mnt/security is a different directory.

Let the cluster provision storage. In production nobody writes the PersistentVolume by hand. Dynamic provisioning "automatically provisions storage when users create PersistentVolumeClaim objects", through a StorageClass. In my manifests manual is only a matching name; no StorageClass object called manual has to exist.

Know the fourth access mode. My notes listed three. The docs have four, the newest being ReadWriteOncePod, where "the volume can be mounted as read-write by a single Pod". ReadWriteOnce is looser than it sounds: it is per node, so several pods on the same node can share the volume.

Decide what happens to the data afterwards. For a PersistentVolume created by hand, the reclaim policy defaults to Retain. Delete the claim, and the volume becomes Released, which is not the same as available: "it is not yet available for another claim because the previous claimant's data remains on the volume".

Delete in order. A claim that a pod is still using is not removed straight away; its removal "is postponed until the PVC is no longer actively used by any Pods". Pod first, then claim, then volume.

One correction to my notes: their cloud volume examples were out of date. The in-tree awsElasticBlockStore and azureDisk volume types were removed in Kubernetes v1.27, and the docs' current AWS example uses the CSI provisioner ebs.csi.aws.com.

Task 2: Attach a public IP to a VM

The task: attach the existing public IP devops-pip to the existing VM devops-vm-pip.

Step 1: Derive the NIC and IP configuration names

A public IP does not attach to a VM, or even to a NIC. Microsoft: "you associate the public IP address to an IP configuration of a network interface attached to a VM."

VM  ->  NIC  ->  IP configuration  ->  public IP

So the command needs two names the task never mentions, and I read both from the API:

NIC_ID=$(az vm show -g $RG -n devops-vm-pip --query "networkProfile.networkInterfaces[0].id" -o tsv)
NIC=$(basename $NIC_ID)
IPCFG=$(az network nic show --ids $NIC_ID --query "ipConfigurations[0].name" -o tsv)

basename works because every Azure resource ID ends in /<name>. [0] was safe because I had confirmed the VM had one NIC.

The IP configuration turned out to be ipconfigdevops-vm-pip, not the ipconfig1 that most examples use. The CLI's source explains the difference: az vm create names it ipconfig plus the VM name, while az network nic create uses ipconfig1. Microsoft's own pages are not consistent either. The page on associating a public IP uses ipconfig1 in its example, and the page on dissociating one uses ipconfigmyVM. Guessing is how this step fails.

Step 2: Attach the public IP

az network nic ip-config update -g $RG --nic-name $NIC -n $IPCFG --public-ip-address devops-pip

The VM stayed VM running before, during and after. Yesterday's NIC needed the VM deallocated. A public IP is a setting on the NIC's IP configuration, so the VM itself is not modified.

Step 3: Verify from both ends

az vm show -g $RG -n devops-vm-pip -d --query "{PubIPs:publicIps}"
az network public-ip show -g $RG -n devops-pip --query "{IP:ipAddress,AttachedTo:ipConfiguration.id}"

"PubIPs": "172.178.12.220"

"IP": "172.178.12.220"
"AttachedTo": ".../networkInterfaces/devops-vm-pipVMNic/ipConfigurations/ipconfigdevops-vm-pip"

The VM reports the address, and the public IP reports its binding, down to the IP configuration. These are two views of one relationship, and they agree.

Before the attach, that first query returned "", an empty string, not null. publicIps is not a stored property. The CLI computes it by joining the VM's public addresses with commas, and joining nothing gives an empty string. A script that tests for null there will never match.

Step 4: Test reachability separately

Assigned is not reachable. Microsoft: "Before you can connect to a public IP address from the internet, you must open the necessary ports/protocols in your network security groups." Day 57 covered why: a Standard public IP is closed to inbound traffic by default.

timeout 5 bash -c "</dev/tcp/$PIP/22" && echo PORT22_REACHABLE || echo PORT22_BLOCKED

PORT22_REACHABLE

So a rule allowing port 22 was already in place. Had it printed PORT22_BLOCKED, the attachment would still have been correct, with a VM nobody could reach.

/dev/tcp is a Bash feature, which is why this works on a host with no nc or telnet. The Bash manual: "Bash attempts to open the corresponding TCP socket." timeout keeps it from hanging on a port that silently drops packets.

What the docs recommend

Do not look for the public IP inside the VM. Microsoft: "a virtual machine's operating system is unaware of any public IP address assigned to it". ip addr in the guest shows only the private address, and Azure translates between the two.

My notes said to ask the instance metadata service instead. That needs a qualifier. The metadata service has a publicIpAddress field, but for a Standard SKU address, which is the only kind left, Microsoft's page points to the Load Balancer metadata endpoint. I have not tested what the plain field returns.

Dissociate before deleting. Microsoft only allows deleting a public IP that is not associated with any IP configuration, and its page removes the association like this:

az network nic ip-config update -g "$RG" --nic-name "$NIC" -n "$IPCFG" --public-ip-address null
az network public-ip delete -g "$RG" -n devops-pip

My notes used --remove publicIPAddress for the first line. Microsoft's current page uses --public-ip-address null. Either way, the VM loses its address the moment that line runs, so do not run it over an SSH session that arrives through the same address.

What I learned

Stated briefly, the tasks were to mount a volume in a pod and to attach a public IP to a VM. Both statements skip a layer. The claim is what lets storage change underneath a pod. The IP configuration is what lets one NIC carry several addresses. And in the first task, the part that was off was neither of those. It was a path, which no status column checks.

So here is the Day 60 question. When did a test of yours last pass while checking the wrong thing?

Day 60 down. Forty to go.

Day 59: A Broken Deployment and a NIC

Nnamdi Felix Ibe — Fri, 09 Oct 2026 17:00:00 +0000

Day 59 of DevOps, Day 9 of Azure. Both tasks today came down to what you check first. In Kubernetes, the events showed me one fault while a second sat unread in the same file. In Azure, the first check I ran was so heavy that it ended the lab session, twice.

One Kubernetes task, one Azure task. Fix a Redis Deployment whose pods will not start, then attach an existing network interface to an existing VM. The tasks come from the KodeKloud Engineer platform.

Task 1: Fix a broken Redis Deployment

The task: the pods of redis-deployment are not running. Find out why and fix it.

Step 1: Look at the pod

kubectl get pods
kubectl describe pod <redis-deployment-pod-name>

The Events section at the bottom of describe had the cause:

MountVolume.SetUp failed for volume "config" : configmap "redis-cofig" not found

Step 2: Compare with what exists

kubectl get cm

The ConfigMap in the cluster is redis-config. The Deployment asks for redis-cofig. One letter is missing.

Step 3: Fix the reference

kubectl get deployment redis-deployment -o yaml > redis-deployment.yaml

volumes:
- name: config
  configMap:
    name: redis-config    # was redis-cofig

kubectl apply -f redis-deployment.yaml
kubectl get pods

Step 4: Find the second fault

The new pod got past the volume and stopped again, this time with ImagePullBackOff. It was the same file, with a second typo:

containers:
- name: redis-container
  image: redis:alpine    # was redis:alpin

Step 5: Recreate and verify

I deleted the Deployment and created it again from the fixed file:

kubectl delete deployment redis-deployment
kubectl apply -f redis-deployment.yaml

kubectl get pods
kubectl logs <redis-pod-name>
kubectl exec <redis-pod-name> -- redis-cli ping

PONG

Running says the container started. PONG says Redis answers, which is the thing the task was really about.

What I would do differently

Read the whole spec before fixing anything. Both typos were in the Deployment from the start. The events only showed the first failure the pod hit, so the image problem stayed hidden until the volume was fixed. One careful read of kubectl get deployment redis-deployment -o yaml would have found both.

Skip the delete. A Deployment rolls out new pods whenever its pod template changes, so applying the fixed file was enough. Deleting it also deleted its ReplicaSets, because Kubernetes cascades deletion to dependents by default, and with them the revision history that kubectl rollout undo relies on, as on Day 52. The app was already down, so it cost nothing here. On a live one, it is an outage.

Use the direct tool for a one-line fix, then wait for the rollout:

kubectl set image deployment/redis-deployment redis-container=redis:alpine
kubectl rollout status deployment/redis-deployment

And preview a change before making it:

kubectl diff -f redis-deployment.yaml
kubectl apply --dry-run=server -f redis-deployment.yaml

The first shows the live object against the version you are about to apply. The second sends the request to the server without saving anything.

What the docs recommend

Start where I started. The Kubernetes debugging guide: "The first step in debugging a Pod is taking a look at it", with kubectl describe pods. It then splits the early failures in two. A pod stuck in Pending "can not be scheduled onto a node". A pod stuck in Waiting has been scheduled and cannot run, and "The most common cause of Waiting pods is a failure to pull the image." Its three checks for that: is the image name correct, has the image been pushed, and can you pull it by hand.

A ConfigMap has to exist before anything references it. The docs: "You must create the ConfigMap object before you reference it in a Pod specification", and unless the reference is marked optional, "the Pod won't start".

ImagePullBackOff means Kubernetes "could not pull a container image", and it keeps retrying with a growing delay, up to 300 seconds.

Do not read the STATUS column as the pod's state. My notes listed ContainerCreating, CrashLoopBackOff and ImagePullBackOff as pod states. The docs define five phases: Pending, Running, Succeeded, Failed and Unknown. The rest is what kubectl displays, and the docs say not to confuse the two.

Manage an object one way. I exported a live object and ran kubectl apply on it, which works, but the docs warn that an object "should be managed using only one technique", and that mixing them "results in undefined behavior". If you do export, strip the server-set fields such as resourceVersion, uid and status first.

Read events early. The API server keeps them for one hour by default.

Task 2: Attach a second NIC to a VM

The task: attach the existing NIC xfusion-nic to the existing VM xfusion-vm, and make sure the VM has finished initialising.

Yesterday's disk went onto a running VM. A NIC cannot. Microsoft: "To add a NIC to an existing VM, first deallocate the VM with az vm deallocate." That makes the order of work matter, because the VM is down from the first command to the last.

Step 1: Check what is cheap to check

Everything that could make the attach fail, I checked while the VM was still running:

az network nic show -g $RG -n xfusion-nic --query "{Subnet:ipConfigurations[0].subnet.id,AttachedTo:virtualMachine.id,Loc:location}"
az network nic show -g $RG -n xfusion-vmVMNic --query "ipConfigurations[0].subnet.id" -o tsv

xfusion-nic:        .../virtualNetworks/xfusion-vmVNET/subnets/xfusion-vmSubnet
xfusion-vmVMNic:    .../virtualNetworks/xfusion-vmVNET/subnets/xfusion-vmSubnet

Same virtual network, AttachedTo was null, and both were in eastus. The first of those is a hard rule. Microsoft: "the network interfaces must all be connected to the same virtual network", though they can sit in different subnets.

Step 2: Deallocate, add, start

az vm deallocate -g $RG -n xfusion-vm
az vm nic add    -g $RG --vm-name xfusion-vm --nics xfusion-nic
az vm start      -g $RG -n xfusion-vm

The command is deallocate, not stop. az vm stop powers the VM off but leaves it, in Microsoft's words, "allocated on a host but not running", and the CLI's own help adds: "The VM will continue to be billed." Deallocating is the state in which the VM "has released the lease on the underlying hardware", and it is the one a NIC change needs.

Step 3: Verify

az network nic show -g $RG -n xfusion-nic --query "{AttachedTo:virtualMachine.id,Primary:primary}"
az vm get-instance-view -g $RG -n xfusion-vm --query "{Status:instanceView.statuses[].displayStatus,Agent:instanceView.vmAgent.statuses[].displayStatus}"

The NIC pointed at the VM with "Primary": false, and the VM was back to VM running with agent Ready.

That false is new. Before the attach, the VM's only NIC showed "Primary": null. With one NIC the flag was unset; with two, the original kept the primary role and the new one arrived as secondary. Microsoft: "By default, the first network interface attached to a VM is the primary network interface."

The check that cost two lab sessions

There is a fourth constraint: each VM size supports a limited number of NICs. To look it up, I ran:

az vm list-skus -l southcentralus --resource-type virtualMachines --query "..."

The lab terminal disconnected, twice. Each disconnect meant a new lab with new resource names.

My notes said the command downloads the SKU catalogue for the region and filters it on the client. The CLI's source shows it is heavier than that. It sends no filter at all, so it fetches the SKU list for the whole subscription, every region, and applies --location and --resource-type afterwards. --query only shrinks what is printed.

The fix was to drop the check. Azure applies the limit itself when a NIC is added. Microsoft: "A VM can only have as many network interfaces attached to it as the VM size supports." I stayed inside the limit, so I have not seen what that failure looks like. The cheap way to know the number in advance is Microsoft's size table, which gives Standard_B1s a maximum of 2 NICs. This second NIC was the last one the VM could take.

What the docs recommend

Plan a NIC change as downtime. This is where Azure and AWS differ. AWS: "You can attach a network interface to an instance when it's running (hot attach), when it's stopped (warm attach), or when the instance is being launched (cold attach)." Azure has only the deallocated route.

Expect side effects from deallocating. Microsoft notes that it "also releases any dynamic IP addresses assigned to the VM", and the temporary disk from Day 52 cannot be relied on afterwards. Disks and networking resources are still charged while the VM is deallocated.

Finish the job in the guest. Attaching the NIC does not make it carry traffic properly. Microsoft's Linux guide: "To send to or from a secondary network interface, you have to manually add persistent routes to the operating system for each secondary network interface." My notes said current Ubuntu images sort this out by themselves. I could not find that in Microsoft's documentation, so I have left it out.

Know which NIC is primary. By default, a VM "sends all outbound traffic to the IP address that's assigned to the primary IP configuration of the primary network interface". Adding a NIC does not move that role. --primary-nic on az vm nic set does.

What I learned

In the first task I checked too little before acting, and found the second fault only by hitting it. In the second, I checked too much, and the check itself took the session down. The useful rule sits between the two: read everything that is cheap to read, and let the operation enforce what is expensive to ask about.

So here is the Day 59 question. When you troubleshoot, do you stop reading at the first error you find, or at the end of the file?

Day 59 down. Forty-one to go.

Day 58: Grafana and a Managed Disk

Nnamdi Felix Ibe — Thu, 08 Oct 2026 17:00:00 +0000

Day 58 of DevOps, Day 8 of Azure. Both tasks today had a gap between "created" and "working". A Deployment reports 1/1 before Grafana is serving a single page. A VM reports Provisioning succeeded before the agent inside it is ready, and a disk reports Attached while the operating system sees a blank device.

One Kubernetes task, one Azure task. Deploy Grafana and expose it on a node port, then attach an existing managed disk to an existing VM. The tasks come from the KodeKloud Engineer platform.

Task 1: Deploy Grafana behind a NodePort Service

The task: a Deployment named grafana-deployment-datacenter running a Grafana image, and a NodePort Service exposing it on port 32000.

Step 1: Write the Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-deployment-datacenter
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
          name: http-grafana
          protocol: TCP
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

Port 3000 is Grafana's own default. Its configuration docs say of http_port: "The port to bind to, defaults to 3000."

Step 2: Add the Service to the same file

---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  labels:
    app: grafana
spec:
  type: NodePort
  selector:
    app: grafana
  ports:
  - port: 3000
    targetPort: 3000
    nodePort: 32000
    protocol: TCP
    name: http

The --- line lets one file hold both objects. Nothing in the Service names the Deployment. As on Day 56, the only link is the label: app: grafana appears in the Deployment's selector, in the pod template and in the Service's selector, and all three have to agree.

Step 3: Apply and check both objects

kubectl apply -f k3s-deployment.yaml
kubectl get deployments.apps
kubectl get svc

The Deployment should show READY 1/1, and the Service should show TYPE NodePort with PORT(S) 3000:32000/TCP.

Step 4: Open the login page

kubectl get nodes -o wide
# then browse to http://<node-ip>:32000

# or, when the node IP is not reachable from where you sit:
kubectl port-forward deployment/grafana-deployment-datacenter 3000:3000
# then browse to http://localhost:3000

Grafana's sign-in docs give the first login as admin for both username and password, followed by a prompt to change the password.

Step 5: Look at the pod

kubectl logs deployment/grafana-deployment-datacenter
kubectl describe pod -l app=grafana
kubectl top pod -l app=grafana

Selecting by Deployment or by label saves copying the generated pod name. kubectl top only works where Metrics Server is installed; the reference says it "must be installed and running in the cluster for this command to work".

Why `1/1` is not the finish line

My manifest has no readiness probe. The Kubernetes docs say that if a container does not provide a particular probe, "the kubelet always considers the result as Success". So this pod counts as ready the moment its container starts, whether or not Grafana has finished starting inside it. 1/1 told me the container was running. Only the login page told me Grafana was.

What the docs recommend

Grafana publishes its own Kubernetes manifest, and mine falls short of it in several places:

	My manifest	Grafana's example
Memory	128Mi request, 256Mi limit	750Mi request
CPU	100m request, 200m limit	250m request
Readiness probe	None	`GET /robots.txt` on port 3000
Liveness probe	None	TCP check on port 3000
Storage	None	1Gi claim mounted at `/var/lib/grafana`
`targetPort`	`3000`	`http-grafana`, the port's name

Resources come first. Grafana's Kubernetes page lists minimums of 750 MiB of memory and 250m of CPU. My memory limit is about a third of that. The pod started in the lab, but a limit under the documented minimum is how you meet the out-of-memory kill from Day 50.

Storage matters more. Grafana's docs: "By default, Grafana uses an embedded SQLite version 3 database to store configuration, users, dashboards, and other data." In the container that lives under /var/lib/grafana. With nothing mounted there, every dashboard and user disappears when the pod is replaced.

The probes are short to add. These are the two from Grafana's example, without their timing settings:

readinessProbe:
  httpGet:
    path: /robots.txt
    port: 3000
livenessProbe:
  tcpSocket:
    port: 3000

The named port is a small thing worth copying. The Kubernetes docs: "Port definitions in Pods have names, and you can reference these names in the targetPort attribute of a Service." With targetPort: http-grafana, the number 3000 lives in one place.

Three smaller points remain. Port 32000 sits in the dynamic band of the node port range from Day 56, the band Kubernetes hands out automatically, so a hand-picked port there can already be taken. grafana/grafana:latest is what Grafana's example uses too, though Kubernetes' advice is to avoid :latest in production. And the admin password can come from a Secret through the GF_SECURITY_ADMIN_PASSWORD environment variable, a name that follows Grafana's GF_<SECTION NAME>_<KEY> rule. Grafana documents the setting behind it, admin_password, as "Set once on first-run".

Task 2: Attach a managed disk to a VM

The task: attach the existing disk datacenter-disk to the existing VM datacenter-vm, and make sure the VM has finished initialising.

Step 1: Check the two resources fit

Nothing is being created, so the risk is the two not fitting together. Two read-only calls cover it:

az vm show -g $RG -n datacenter-vm --query "{Loc:location,Zones:zones,Size:hardwareProfile.vmSize}"
az disk show -g $RG -n datacenter-disk --query "{Loc:location,Zones:zones,State:diskState,Sku:sku.name}"

Check	VM	Disk	Result
Region	eastus	eastus	Match
Zone	`null`	`null`	Match
Disk state		`Unattached`	Free to attach
Premium support	`Standard_B1s`	`Standard_LRS`	Not needed

Each row is a way the attach can fail. Microsoft's disks FAQ: "All managed disks, even shared disks, must be in the same region as the VM they're attaching to." A zonal disk attaches "only to VMs in that zone". A disk already attached elsewhere is not free. And Premium SSDs can be used "only with compatible VM series".

One state is worth knowing in advance. A disk on a deallocated VM reads Reserved, not Attached. It is still bound to that VM, and a filter on Attached will miss it.

Step 2: Confirm the VM has finished initialising

az vm wait -g $RG -n datacenter-vm --created
az vm get-instance-view -g $RG -n datacenter-vm --query "{Status:instanceView.statuses[].displayStatus,Agent:instanceView.vmAgent.statuses[].displayStatus}"

{ "Status": ["Provisioning succeeded", "VM running"], "Agent": ["Ready"] }

"Initialised" has two halves. Provisioning succeeded is Azure's side. Agent Ready is the guest's side: the Azure Linux Agent inside the VM has started and reported back. az vm wait --created only waits for the first, so the instance view is the real check.

Step 3: Attach the disk

az vm disk attach -g $RG --vm-name datacenter-vm --name datacenter-disk

The VM stayed VM running throughout. Azure picked the LUN: the REST reference says "If not specified, lun would be auto assigned", and the CLI takes the lowest free number, here 0.

Step 4: Verify from Azure, then from inside the guest

az vm show -g $RG -n datacenter-vm --query "storageProfile.dataDisks[].{Name:name,Lun:lun,Caching:caching}"
az vm run-command invoke -g $RG -n datacenter-vm --command-id RunShellScript --scripts "lsblk"

sda   30G  disk     <- OS disk
sdb    4G  disk     <- temporary disk at /mnt
sdc   30G  disk     <- datacenter-disk, raw

Run Command needs no SSH key and no open port. It uses the VM agent to run the script, so a result also proves the agent works. Microsoft documents its limits: about 20 seconds minimum, one script at a time, and only the last 4,096 bytes of output.

The disk is attached and raw, with no partition or filesystem. The task asked for attachment only, so I left it there.

The query that returned null

Checking the disk's size from both sides gave two different answers:

az disk show ... --query "{GB:diskSizeGb}"     ->  "GB": null
az vm show   ... --query "{GB:diskSizeGb}"     ->  "GB": 30

The raw JSON explained it:

az disk show -g $RG -n datacenter-disk -o json | grep -i disksize
  "diskSizeBytes": 32212254720,
  "diskSizeGB": 30,

az disk show printed diskSizeGB. az vm show printed diskSizeGb. JMESPath is case-sensitive, and a key that does not exist returns null instead of an error, so my query quietly found nothing.

My notes concluded that Azure spells the property differently on the two resource types. It does not. The REST API uses diskSizeGB in both places. The lowercase b came from how the Azure CLI formatted az vm show output, and the CLI's source code suggests that changed in version 2.84.0, which I have not tested. So the same query can work on one CLI version and return null on the next.

The fix does not depend on which version you have. When a value you know exists comes back null, suspect the query before the resource, and look at the raw object with -o json | grep -i.

What the docs recommend

Do not reference the disk as /dev/sdc. Microsoft: "Device paths in Linux aren't guaranteed to be consistent across restarts." Use the filesystem UUID, or the link the Azure Linux Agent creates for the LUN:

/dev/disk/azure/scsi1/lun0

Choose caching when you attach. The disk came up with caching None, and the REST reference explains that default: "None for Standard storage. ReadOnly for Premium storage." My notes had it as a rule about data disks; it is a rule about the storage type. Microsoft's guidance is None for write-only and write-heavy disks, and ReadOnly for read-only and read-write ones. Changing the setting later detaches and reattaches the disk, so it is better set once, with --caching on the attach command.

Before detaching, unmount the disk in the guest and remove its /etc/fstab entry. And remember that detaching does not delete. Microsoft's warning is that a detached disk "is not automatically deleted", and you keep paying for it:

az disk list --query "[?diskState=='Unattached'].{Name:name,RG:resourceGroup,Sku:sku.name}" -o table

What I learned

Every status today was true and incomplete. 1/1 meant a container had started. Provisioning succeeded meant Azure had finished its half. Attached meant a blank device had appeared. The checks that settled things were the ones that used the result: the login page, the agent's Ready, and lsblk from inside the VM.

So here is the Day 58 question. Which of your checks proves that something was created, and which proves that it works?

Day 58 down. Forty-two to go.

Day 57: Env Vars and a Public IP

Nnamdi Felix Ibe — Wed, 07 Oct 2026 17:00:00 +0000

Day 57 of DevOps, Day 7 of Azure. Both tasks today had values in them that I did not put there myself. In the pod, Kubernetes fills in $(GREETING) before the shell ever sees it. In Azure, a public IP created with nothing but a name still arrives with a SKU, an allocation method and a zone setting.

One Kubernetes task, one Azure task. Print three environment variables from a pod, then allocate a public IP address. The tasks come from the KodeKloud Engineer platform.

Task 1: Print environment variables from a pod

The task: a pod called print-envars-greeting, with one container, print-env-container, on the bash image. Three environment variables, a fixed echo command that prints them, and a restart policy of Never.

Step 1: Write the manifest

apiVersion: v1
kind: Pod
metadata:
  name: print-envars-greeting
spec:
  restartPolicy: Never
  containers:
  - name: print-env-container
    image: bash
    command: ["/bin/sh", "-c", 'echo "$(GREETING) $(COMPANY) $(GROUP)"']
    env:
    - name: GREETING
      value: "Welcome to"
    - name: COMPANY
      value: "xFusionCorp"
    - name: GROUP
      value: "Group"

Each entry under env is a name and a value. The command runs through /bin/sh -c, and restartPolicy: Never tells Kubernetes to leave the container alone once the echo has finished.

Step 2: Apply it and read the output

kubectl apply -f k3s-pod.yml
kubectl logs -f print-envars-greeting

Welcome to xFusionCorp Group

Step 3: Check how the pod ended

kubectl get pods
kubectl describe pod print-envars-greeting

get pods shows Completed. describe prints the pod's spec, so it lists the three variables under Environment, next to the command exactly as I wrote it.

Completed is what kubectl displays. The pod's phase is Succeeded, which the docs define as: "All containers in the Pod have terminated in success, and will not be restarted." The Kubernetes docs warn against mixing the two up, calling Status "a kubectl display field for user intuition".

That ending depends on restartPolicy: Never. The default is Always, which restarts a container "after any termination", a successful exit included. Without the setting, this echo would be restarted over and over, with a delay that starts at 10 seconds and is capped at 5 minutes.

Who expands `$(GREETING)`

This is the part my notes had wrong. I had written that $(VARNAME) is shell variable syntax, processed by the shell at runtime.

It is the other way round. In a shell, $(name) is command substitution, which the Bash manual says "allows the output of a command to replace the command itself". A shell would try to run a command called GREETING.

Kubernetes gets there first. The API reference says of a container's command: "Variable references $(VAR_NAME) are expanded using the container's environment." So the three references are replaced with the values from env before the container starts, and the shell is handed a finished line:

echo "Welcome to xFusionCorp Group"

The shell's own forms are $GREETING and ${GREETING}. They would have worked here too, because the variables are in the container's environment. The difference is who does the substitution, and it matters when something is misspelt. The API reference again: "If a variable cannot be resolved, the reference in the input string will be unchanged." A typo such as $(GRETING) is passed through untouched, and the shell then treats it as a command to run.

To print the literal text, double the dollar sign: $$(GREETING).

One command that cannot work here

My notes included this as a way to see the variables:

kubectl exec print-envars-greeting -- env

It cannot work on this pod. kubectl checks the pod's phase first and refuses a pod that has finished. Going by the message in kubectl's source code, the error for this pod reads:

cannot exec into a container in a completed pod; current phase is Succeeded

kubectl describe pod is the way to see the variables afterwards. To inspect the live environment, the pod has to still be running.

What the docs recommend

A pod that runs once and exits is really a Job. The Jobs page recommends "a Job rather than a bare Pod, even if your application requires only a single Pod", because a bare pod is not restarted if its node reboots or fails.

Real configuration should not be hardcoded in the pod spec. A ConfigMap holds non-confidential values, a Secret holds confidential ones, and env points at them:

env:
- name: DATABASE_HOST
  valueFrom:
    configMapKeyRef:
      name: app-config
      key: db_host
- name: DATABASE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-secret
      key: password

Two cautions from the docs go with that. ConfigMaps consumed as environment variables "are not updated automatically and require a pod restart". And Secrets are "by default, stored unencrypted in the API server's underlying data store (etcd)", so a Secret is only as protected as the cluster makes it.

Last, image: bash has no tag, and "If you don't specify a tag, Kubernetes assumes you mean the tag latest." The same page advises against :latest in production, because it is harder to track which version is running and harder to roll back.

Task 2: Allocate a public IP

The task: one public IP address named nautilus-pip. Nothing else was specified.

Step 1: Find the resource group

RG=$(az group list --query "[0].name" -o tsv)

The lab provides one resource group with a generated name, so I read it from the API instead of pasting it.

Step 2: Create the address

az network public-ip create -g "$RG" -n nautilus-pip

One thing to know before querying the response: create wraps its result in an object called publicIp, while show returns the same resource flat. A query written for one returns null for every field on the other.

Step 3: Check what was created

az network public-ip show -g "$RG" -n nautilus-pip --query "{IP:ipAddress,Sku:sku.name,State:provisioningState}"

The address was 20.84.50.139, the SKU was Standard, and the provisioning state was Succeeded. The create response had also shown the allocation method as Static. I had asked for none of it except the name.

What the defaults chose for me

The address is static, and it is already assigned. Nothing is attached to this IP, and it still has a real routable address. That follows from the SKU. Microsoft lists the allocation method for Standard as Static, and describes static assignment as: "The resource is assigned an IP address at the time it's created." Dynamic allocation only existed on the Basic SKU, and "On September 30, 2025, Basic SKU public IPs were retired."

It is closed to inbound traffic. Microsoft describes Standard as a "Secure by default model", closed to inbound traffic until a network security group allows it. Attach this address to a VM with no NSG rule, and the VM is unreachable, in a way that looks like a network fault and is really a policy.

It has a zone setting I did not choose. The CLI printed a warning on create:

[Coming breaking change] In the coming release, the default behavior will be changed as follows
when sku is Standard and zone is not provided: For zonal regions, you will get a zone-redundant IP
indicated by zones:["1","2","3"]; For non-zonal regions, you will get a non zone-redundant IP
indicated by zones:null.

My notes read that as a change still to come. Microsoft's documentation says it has already happened on the platform: "All formerly Standard non-zonal IPs are now zone-redundant in all regions that support availability zones". An address that shows no zones and one that shows zones 1, 2 and 3 "are equivalently zone-redundant". The meaning of a value I never set changed underneath an unchanged command.

It probably has a cost. Microsoft prices a Standard static IPv4 address by the hour, and its docs say "Public IPv4 addresses have a nominal charge". My notes said the charge applies whether or not the address is attached. I could not find a Microsoft page that says so in those words, so I would treat an idle address as billable until the bill says otherwise. AWS does say it outright: since 1 February 2024, it charges "for all public IPv4 addresses, whether attached to a service or not".

What the docs recommend

Pass what matters instead of inheriting it:

az network public-ip create -g "$RG" -n nautilus-pip --sku Standard --zone 1 2 3

The zone deserves the care, because Microsoft says a public IP cannot change its availability zone once created, and a zonal IP "shares fate with the health of the zone".

Find the addresses nothing is using:

az network public-ip list --query "[?ipConfiguration==null].{Name:name,IP:ipAddress,RG:resourceGroup}" -o table

And know the order for removing one. A public IP can only be deleted when it "isn't associated to any IP configuration or virtual machine network interface", so dissociate first, then delete. A static address that is deleted "can't be recovered", so anything holding the old address in DNS or a firewall allowlist breaks.

What I learned

Neither task had a hard step. The work was in noticing who supplied each value. Kubernetes substituted three variables before the shell started. Azure chose a SKU, an allocation method and a zone behaviour, and one of those has changed its meaning since the warning about it was written.

So here is the Day 57 question. Which default in your last deployment would surprise you if it changed tomorrow?

Day 57 down. Forty-three to go.