DEV Community: Nnamdi Felix Ibe

Day 53: A Shared Volume Needs a Shared Path, and Ubuntu 22.04 Has Two Names on Azure

Nnamdi Felix Ibe — Wed, 30 Sep 2026 17:00:00 +0000

Day 53 of DevOps, Day 3 of Azure. Both tasks today came down to the same thing being known by more than one name.

Two containers shared a volume and still could not see the same file, because they disagreed about where it lived. And an Ubuntu release on Azure turned out to have two valid image names, with an alias quietly choosing one of them for me.

One Kubernetes task, one Azure task. Fix a broken nginx and PHP-FPM pod, then create an Azure VM from the CLI using an image alias. The tasks come from the KodeKloud Engineer platform.

Two containers, one volume, two paths

The pod, nginx-phpfpm, runs two containers. nginx takes the requests and hands PHP files to PHP-FPM, and both mount one shared volume for the site's files. nginx's configuration lives in a ConfigMap. The site was not working.

kubectl describe pod nginx-phpfpm
kubectl get configmap nginx-config -o yaml

The ConfigMap set nginx's document root to /var/www/html, and describe showed where each container mounted the shared volume. The nginx side matched the root. The PHP-FPM container mounted the volume somewhere else.

Why does that break the site? Because nginx does not hand PHP-FPM a file. It hands it a path. Nginx passes the script's location in a FastCGI parameter, SCRIPT_FILENAME, which nginx's own documentation describes as what PHP uses to determine the script name. PHP-FPM then opens that path inside its own container. With the volume mounted elsewhere, the path pointed at a directory where the site's files were not.

The volume was shared correctly. The two containers just did not agree on where it was.

A pod you fix by replacing it

The fix was one line of YAML, and it still needed a new pod. The Kubernetes docs list what an update to an existing pod may change: container images, a couple of timing fields, tolerations and scheduling gates. Volume mounts are not on the list.

kubectl get pod nginx-phpfpm -o yaml > po.yaml
# in po.yaml, set the php-fpm-container mountPath to /var/www/html
kubectl delete pod nginx-phpfpm
kubectl apply -f po.yaml

kubectl replace --force -f po.yaml does the delete and the re-create in one step.

Then the file the task asked for:

kubectl cp /home/thor/index.php nginx-phpfpm:/var/www/html -c php-fpm-container

I copied it in through the PHP-FPM container, and that is fine, because after the fix both containers mount the same volume at the same path. A shared volume does not care which container a file arrives through. The copy has to come after the new pod exists, though. The file needs to land in the volume the new pod mounts, and if that volume is an emptyDir, its data is deleted along with the old pod.

One thing the task did not test, and worth knowing: if the ConfigMap had been the broken part, editing it would not have been enough. A container that mounts a ConfigMap with subPath never receives updates, and nginx's documentation says configuration changes are not applied until nginx is told to reload or is restarted.

An alias picks one of two names

The Azure task was another VM, this time with the image given as an alias, Ubuntu2204:

--image Ubuntu2204 --size Standard_B2s --admin-username azureuser --generate-ssh-keys

The alias resolved to Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest. Yesterday's 24.04 image was Canonical:ubuntu-24_04-lts:server:latest. Same publisher, two different naming schemes.

My notes explained that as Canonical changing its naming between 22.04 and 24.04. Canonical's own documentation says otherwise. The 0001-com-ubuntu offers are its legacy format, and "With Ubuntu 22.04 LTS, Canonical began deploying a unified offer structure". Canonical lists 22.04 under the new scheme too, as Canonical:ubuntu-22_04-lts:server:latest. So 22.04 has two valid names, and the alias happens to point at the older one.

That makes the lesson stronger rather than weaker. You cannot derive an image URN from a version number, because one release can have more than one. Use an alias, or look the URN up, and filter the search: Microsoft notes that az vm image list --all "can take several minutes to produce the entire list". The alias list itself varies by Azure CLI version and cloud, so an alias that works on one machine can be missing on another.

And check what booted, not just what was requested:

ssh -o StrictHostKeyChecking=no azureuser@$VM_IP 'hostname; grep PRETTY_NAME /etc/os-release'

devops-vm
PRETTY_NAME="Ubuntu 22.04.5 LTS"

The image SKU in the VM's configuration says what Azure was asked for. /etc/os-release says what is actually running. Only the second one is evidence.

One smaller thing carried over from yesterday. The temporary disk at /mnt was 8 GiB this time instead of 4, because it comes with the VM size, and Microsoft's size table lists exactly those figures for B1s and B2s. Some newer sizes have no temporary disk at all.

Same thing, different names

A volume that both containers mount is not a path they both agree on. A release of Ubuntu is not a single image name. In both cases the thing each side depended on existed. They were using different names for it, and nothing flagged the difference until something looked in the wrong place.

So here is the Day 53 question. Where in your setup do two components have to agree on a path or a name, with nothing checking that they do?

Day 53 down. Forty-seven to go.

Day 52: Undo Rolls Forward, and the Disk Already Mounted Is the One Not to Trust

Nnamdi Felix Ibe — Mon, 28 Sep 2026 22:03:38 +0000

Day 52 of DevOps, Day 2 of Azure. Both tasks today were about commands that do something a little different from what their names suggest.

kubectl rollout undo sounds like a rewind. It is not one; it rolls forward to an old template. And az vm create sounds like it creates a VM. It does, along with a network, a firewall rule, a public address and one more disk than I asked for, and that extra disk is the one not to keep anything on.

One Kubernetes task, one Azure task. Roll a Deployment back to its previous version, then create an Azure VM with a specific image, size and disk type. The tasks come from the KodeKloud Engineer platform.

Undo rolls forward

kubectl rollout undo deployment nginx-deployment
kubectl rollout status deployment nginx-deployment

Yesterday's rolling update kept the old ReplicaSet on purpose. Today's task is what it was kept for: a release has a bug, so go back to the previous revision.

Under the hood, kubectl's rollback code takes the pod template stored in the older ReplicaSet and patches it into the Deployment's .spec.template. A changed template is exactly what starts a rollout, as yesterday's post covered, so an undo is a new rollout with the same rolling behaviour as any other update. The Kubernetes docs put it briefly: each rollback updates the revision of the Deployment.

Two consequences are worth knowing.

Only the template goes back. The docs say "only the Deployment's Pod template part is rolled back". If someone scaled the Deployment or changed its strategy since that revision, those changes stay.

And the history has a limit. Revisions are the old ReplicaSets, and .spec.revisionHistoryLimit keeps ten of them by default. Set it to zero, and in the docs' words, "a new Deployment rollout cannot be undone".

One habit I am adding is kubectl rollout history before the undo, to see what there is to go back to. Its CHANGE-CAUSE column shows <none> unless you set the kubernetes.io/change-cause annotation, and the old --record flag that used to fill it is deprecated. A history where every revision says <none> tells you nothing at the moment you most need it.

The point I would underline, though, is this one. Undo fixes the cluster, not the source of truth. If the buggy image is still in a manifest file, the next kubectl apply of that file rolls the bug straight back out. The rollback is only finished when the file, or the commit, goes back too.

One command, a lot of resources

The Azure task was a single VM: Ubuntu 24.04, Standard_B1s, a 30 GB Standard HDD disk, SSH access.

az vm create does much more than its name says. Alongside the VM, it created a virtual network and subnet, a network security group with an SSH rule, a public IP address, a network interface, an OS disk and a data disk. Microsoft describes this as the CLI using default values to create any required supporting resources.

The catch is on the way out. Microsoft's page on deleting VMs says that by default, the disks, NICs and public IPs associated with a VM are kept when the VM is deleted. One command builds it all, and the matching delete leaves most of it behind, where a leftover disk or public IP is still billed.

Coming from AWS, the concept with no direct equivalent is the resource group. Every Azure resource lives in exactly one, and deleting the group deletes everything in it. In a lab that is the cleanest teardown Azure offers.

A naming trap while I am here: "Standard HDD" in the portal is Standard_LRS in the CLI, and Standard SSD is StandardSSD_LRS. One token apart, a different storage medium. LRS is not the disk type at all. It is the redundancy: locally redundant storage, three copies within one data centre.

Three disks, and one to leave alone

lsblk on the new VM showed three disks, not two:

sda       30G  disk
├─sda1    29G  part /
sdb        4G  disk
└─sdb1     4G  part /mnt
sdc       30G  disk

sda is the OS disk, and sdc is the data disk, attached but raw. sdb is the one I did not ask for. It is the temporary disk, and its size comes with the VM size, 4 GiB on a B1s. It arrives formatted and mounted at /mnt.

It looks like free storage, ready to use. Microsoft says data on it "might be lost during a maintenance event, when you redeploy a VM, or when you stop the VM", though it does survive a normal restart. The Linux VM FAQ puts it more bluntly: "Don't use the temporary disk, mounted under (/mnt) to store data."

The nearest AWS equivalent is instance store, and the rule is the same: nothing you need to keep goes there.

The data disk arriving raw is normal, the same as an extra EBS volume. Making it usable means partitioning, formatting, mounting, and an /etc/fstab entry by UUID with nofail. Microsoft recommends the UUID because device paths like /dev/sdc1 are not persistent and change on reboot.

The reason I had wrong

The lab host runs as root, and az vm create takes its default admin username from the local account. root is on Microsoft's list of disallowed VM usernames, along with admin, administrator, user and test. So my notes said the create would fail unless I set a username myself.

The CLI reference says otherwise. The default is the current OS username, and "If the default value is system reserved, then default value will be set to azureuser." Running as root, the CLI would have picked azureuser on its own. What does get rejected is a reserved name you pass explicitly.

I still set --admin-username azureuser, and I would again, because it makes the result independent of who runs the command. But the reason in my notes was wrong, and a wrong reason is worth correcting even when the command it produced was right.

Read what the command did

An undo that rolls forward. A create that builds a small network, and a delete that leaves most of it standing. A disk that is mounted and ready, and not safe to use. In each case the name was a fair summary and a poor specification. The rollout history, the resource list and the documentation had the real answer.

So here is the Day 52 question. The last time you rolled something back, did you also change the file that would have deployed it again?

Day 52 down. Forty-eight to go.

Day 51: A Rollout Only Starts When the Template Changes, and Azure Keeps Only Half Your Key

Nnamdi Felix Ibe — Sun, 27 Sep 2026 08:00:00 +0000

The AWS track finished yesterday, so from today the cloud half of each post comes from the Azure track instead. The series keeps its name, so everything stays in one place. Day 51 of DevOps, Day 1 of Azure.

Both tasks today turn on what gets kept. A rolling update keeps the old version around so you can go back to it. Azure keeps one half of your SSH key and hands the other half to you exactly once.

One Kubernetes task, one Azure task. Roll a Deployment to a new image without downtime, then create an SSH key pair for Azure VMs. The tasks come from the KodeKloud Engineer platform.

A rollout starts only when the template changes

kubectl describe deployment nginx-deployment
kubectl set image deployment/nginx-deployment nginx-container=nginx:1.17
kubectl rollout status deployment/nginx-deployment

The describe comes first for a reason. set image takes the container name, not the Deployment name, so nginx-container has to match the container in the pod template exactly. Get it wrong and nothing updates, because there is no such container.

Kubernetes is precise about what starts a rollout: a Deployment's rollout is triggered if and only if the pod template changes, and other updates, such as scaling, do not trigger one. Changing the image changes the template. Changing the replica count does not.

The default strategy is RollingUpdate, with two settings that both default to 25%. maxUnavailable is how far below the desired count you can drop during the update, rounded down. maxSurge is how far above it you can go, rounded up.

Those rounding directions are what make a single replica safe. With one pod, 25% rounds down to zero unavailable and up to one extra, so Kubernetes starts the new pod before removing the old one. Even a one-pod Deployment updates without a gap.

kubectl rollout status then watches until it finishes, and Kubernetes documents its exit status as 0 on success and 1 when the Deployment exceeds its progress deadline. That is what belongs in a script after an update, instead of a guessed sleep.

The old version is kept on purpose

The update does not modify the existing ReplicaSet. It makes a new one:

kubectl get replicasets

The new ReplicaSet scales up, the old one scales down to zero, and the old one stays. Kubernetes keeps ten by default, and that retained ReplicaSet is exactly what kubectl rollout undo goes back to. Tomorrow's task uses it.

There were three ways to make this change: kubectl set image, kubectl edit, or editing the manifest and running kubectl apply. All three start the same rollout. They do not leave the same thing behind. The first two change the live object, so a manifest file on disk is now out of date, and the next time someone applies that file it quietly puts the old image back. Only the apply route keeps the file as the truth.

Azure keeps only the public half

The Azure task was an SSH key pair for virtual machines.

az sshkey create --name "$KEY_NAME" --resource-group "$RG"

The resource this creates is a Microsoft.Compute/sshPublicKeys object, and its only key property is the public key. There is nowhere in it for a private key to live. Microsoft documents that each newly created key is also stored locally, and the output tells you where:

Private key is saved to "/root/.ssh/1757564000_123456".
Public key is saved to "/root/.ssh/1757564000_123456.pub".

That file is the only copy of the private key that will ever exist. No API returns it. Lose it and the key pair is useless for login, and the fix is a new pair and resetting access on every VM that used the old one.

The filename is the other trap. It is not id_rsa, so ssh will not find it by default. Every login needs -i with the full path, or the file needs renaming to something memorable straight away.

Two things in my own notes turned out to be wrong when I checked them.

I had written that Windows VMs do not accept Ed25519 keys. Microsoft's documentation lists the same two supported types for Windows as for Linux, RSA of at least 2048 bits and Ed25519. The real difference is that Azure does not provision SSH public keys to Windows machines automatically at all, of any type.

And I had uploaded a key with --public-key @~/.ssh/nautilus-key.pub. The @ tells the Azure CLI to read the value from a file, which is right. But bash only expands a tilde at the start of a word, and after @ it is not at the start, so the shell passes the tilde through untouched. Microsoft's examples use absolute paths. @$HOME/.ssh/nautilus-key.pub works because the shell expands $HOME anywhere in the word.

Coming from AWS

	AWS	Azure
Command	`aws ec2 create-key-pair`	`az sshkey create`
Private key delivery	Returned in `KeyMaterial` in the response	Written to a local file
Scope	Available only in the Region where created	Resource group plus location
Default type	RSA	RSA

The one-chance rule is the same on both. AWS keeps the public key and returns the private key inside the JSON response, so forgetting to capture KeyMaterial means it scrolls past once and is gone. Azure writing it to a file is harder to lose, and the filename is easier to misplace.

One more thing worth knowing. Deleting the Azure key resource does not lock anyone out of VMs made with it. The public key was copied into each VM when it was created, and the VM has no ongoing link back to the resource. The resource is a convenience for provisioning, not a control point for access.

What gets kept, and by whom

Kubernetes keeps the old ReplicaSet so you can go back. Azure keeps the public key so it can hand it to new VMs. The private key is kept by exactly one party, you, in a file with an unhelpful name.

So here is the Day 51 question. For the credentials your systems depend on, do you know where every copy of the private half lives right now?

Day 51 down. Forty-nine to go.

Day 50: One Number Picks a Pod's QoS Class, and a Bigger Disk Is Three Separate Jobs

Nnamdi Felix Ibe — Sat, 26 Sep 2026 08:00:00 +0000

Halfway. And today's AWS task is the last one in the 50-day AWS track, so half of the series is finished. More on that at the end.

The two tasks themselves were both about layers that look like one thing. A resource spec where a single value changes how Kubernetes treats the whole pod. A disk that has to be made bigger three separate times before the operating system notices.

One Kubernetes task, one AWS task. Set resource requests and limits on a pod, then expand an EC2 root volume without stopping the instance. The tasks come from the KodeKloud Engineer platform.

Requests, limits, and the one number

resources:
  requests:
    memory: "15Mi"
    cpu: "100m"
  limits:
    memory: "<memory limit>"
    cpu: "100m"

A request is what the scheduler reserves when it places the pod. A limit is the ceiling once it is running. And the two limits are enforced in very different ways. Kubernetes documents CPU limits as enforced by throttling, so a container over its CPU limit just runs slower. Memory limits are enforced by the kernel with out-of-memory kills, so a container over its memory limit may be terminated, though only when the kernel detects memory pressure, not necessarily at once. CPU over the limit is slow. Memory over the limit is dead.

The units deserve a second look too. 100m is a tenth of a CPU, one hundred millicpu. 15Mi is 15 mebibytes. And Kubernetes' own warning about case is worth quoting in spirit: 400m of memory is a request for 0.4 bytes, when whoever typed it almost certainly meant 400Mi.

Now the one number. I wrote the memory limit down as 15Mi, the same as the request, but I have not confirmed that was the figure the task gave, and it matters more than any other value in the file.

Kubernetes assigns each pod a QoS class from its requests and limits. A pod is Guaranteed only if every container has both CPU and memory requests and limits, all above zero, with each limit equal to its request. It is Burstable if it misses that but has at least one request or limit, and BestEffort if it has none at all.

The CPU request and limit here are both 100m. So if the memory limit is 15Mi, every limit equals its request, and the pod is Guaranteed. If the memory limit is anything higher, the same pod is Burstable. One value, two classes.

And the class has consequences. The documentation is explicit: when a node runs out of resources, Kubernetes evicts BestEffort pods first, then Burstable, and Guaranteed last. You can read the result with kubectl describe pod, on the QoS Class line.

One small thing I appreciated. My first draft of this manifest had a typo, rrequests. kubectl's --validate flag defaults to strict, which rejects unknown fields rather than dropping them, so that typo fails at apply instead of producing a pod with no requests and a very confusing QoS class.

A bigger disk is three jobs

The AWS task was to grow a root volume from 8 GiB to 12 GiB and have the instance see the space, without disrupting it.

Layer	Tool	Effect
EBS volume	`aws ec2 modify-volume`	The virtual disk is bigger
Partition	`growpart`	The partition uses the new space
Filesystem	`xfs_growfs` or `resize2fs`	The filesystem uses the bigger partition

Each one is separate, and AWS's procedure says so: before you can extend a filesystem, you must extend the partition, if the volume has one. Do only the first, and you have a 12 GiB disk holding an 8 GiB partition holding an 8 GiB filesystem, and df still says 8G.

I could see the middle state directly, which is what convinced me the layers are real. After growpart, lsblk showed the partition at 12G while df still showed the filesystem at 8G. Only xfs_growfs closed the gap.

Three details from that sequence worth keeping.

You do not have to wait for the modification to finish. AWS documents that size increases take effect once the modification reaches the optimizing state, usually within seconds, and that you can extend the partition and filesystem as soon as it does. Waiting for completed burns time for nothing.

growpart takes two arguments. The AWS procedure asks you to note the space between the device and the partition number: growpart /dev/xvda 1, not growpart /dev/xvda1.

And the two filesystem tools take different arguments. xfs_growfs wants the mount point, resize2fs wants the partition device. Check df -hT first, because Amazon Linux 2023 is xfs, Ubuntu is ext4, and using the wrong one produces an error that reads like corruption.

The rule I had wrong

My notes said a volume cannot be modified again for six hours after a change. That rule is out of date.

AWS's current documentation says you must wait for a modification to reach completed before starting another on the same volume, and that you can modify a volume at most four times in a rolling 24-hour period. The six hours seems to come from a different sentence on the same page, that a 1 TiB volume can typically take up to six hours to modify.

The practical point survives, just for a different reason. EBS volumes cannot be shrunk at all, and XFS cannot be shrunk either. And you cannot iterate freely: the next change waits for the last to finish, which on a large volume can be hours, and you only get four a day. Pick the size carefully the first time.

This is exactly why I now check every note against the current documentation before it goes out. The rule I had was plausible, specific, and wrong.

Halfway, and the end of the AWS track

Fifty tasks of AWS are done, and the cloud half of this series moves to Azure from tomorrow, under the same series name.

Looking back across the fifty, three habits came up often enough to be worth naming.

Verify the deliverable, not a status field. An ECS task can read RUNNING and be unreachable. A route can look right and read blackhole. A modification can be optimising while the filesystem is still 8G. The check that means something is the one that exercises the actual promise: the curl, the cmp, the df.

The console does things the CLI makes you do yourself. A DB subnet group, an instance profile, a listener, a resource-based permission on a Lambda. Each was invisible in the console and a separate, silent failure on the CLI.

Errors name the call that failed, not the cause. A missing zip reported three commands after a broken heredoc. AccessDenied that meant a public access block. Unable to validate the following destination configurations meant a missing Lambda permission. The fix was rarely where the message pointed.

None of that is specific to AWS, which is a good sign for the Azure half.

So here is the Day 50 question. What is a rule you are confident about that you have not checked against the documentation since you learned it?

Day 50 down. Fifty to go.

Day 49: A Deployment Keeps Pods Alive, and Routable Is Not Reachable

Nnamdi Felix Ibe — Fri, 25 Sep 2026 17:00:00 +0000

Today was layers. A Deployment that does not run pods itself, but owns a ReplicaSet that does. A peering connection that makes a subnet routable, a security group that decides whether it is reachable, and an SSH hop in between that did not carry the key I handed it.

One Kubernetes task, one AWS task. Run an application as a Deployment, then move a log file from a private VPC to S3 through a peered public one. The tasks come from the KodeKloud Engineer platform.

The layer you forget is there

kubectl create deployment httpd --image=httpd:latest
kubectl get deployments
kubectl get replicasets
kubectl get pods

Kubernetes describes a Deployment as providing declarative updates for Pods and ReplicaSets, and the ReplicaSet is the part that answers yesterday's question. A ReplicaSet replaces pods that are deleted or terminated for any reason. Delete one of this Deployment's pods and a replacement appears within seconds, with a different name.

The names give the structure away. A Deployment's pods are called something like httpd-<hash>-<suffix>, where the hash identifies the ReplicaSet that made them. The name changes every time a pod is replaced, so nothing should ever depend on it.

What the Deployment actually uses to find its pods is a label. kubectl create deployment gives them app=httpd, and kubectl get pods -l app=httpd lists exactly this Deployment's pods and nothing else.

Two facts worth having before Day 51. Scaling is not an update: Kubernetes is explicit that a rollout is triggered only when the pod template changes, so kubectl scale just changes a count. And old ReplicaSets are kept after an update, ten by default, which is what makes rolling back possible at all.

Four things have to say yes

The AWS task had an instance with no internet access writing a log file, which had to end up in S3. The route was across a VPC peering connection to an instance that did have internet access, and from there to the bucket.

The private instance had no public IP, no NAT and no internet route. The only way to reach it at all was through the public one, over the peering connection, so the networking had to be finished before a single line of configuration could reach the private side. The plumbing was not a step, it was the prerequisite.

And peering turned out to need four separate things, each documented, each capable of failing on its own:

CIDRs that do not overlap. AWS will not peer VPCs with matching or overlapping ranges, and there is no translation option. The private VPC was 10.10.0.0/16, so the new one became 10.20.0.0/16.
Acceptance. Even within one account the connection sits in pending-acceptance until someone accepts it. You can create and accept it yourself, but you do have to accept it.
Routes on both sides. AWS requires a route in the route tables for both instances' subnets. One direction only gives you packets that arrive and replies that cannot get home, and the symptom is a timeout identical to having no route at all.
Security groups. The one people miss after the routes are right. AWS's own guidance is to update the security group rules so traffic to and from the peer VPC is not restricted.

The fourth is the title. Peering made the private instance routable from the public VPC. Without a security group rule allowing 10.20.0.0/16, SSH would have been dropped at the instance with the routing working perfectly. Routes and security groups are separate layers and both have to agree.

It is also worth knowing that peering does not chain. AWS says plainly that it does not support transitive peering relationships, so a third VPC peered to the public one would still have no path to the private one.

The key that did not make the hop

This is the one that actually cost time:

ssh -i /root/.ssh/datacenter-key.pem -J ubuntu@$PUB_IP ubuntu@10.10.1.114 'hostname'

ubuntu@32.196.116.111: Permission denied (publickey).

Seconds earlier, the same key had logged into that same public instance directly. The error names the jump host, which is the only clue that the private instance was never reached.

The ssh manual has the answer, in a note under -J: configuration directives supplied on the command line generally apply to the destination host and not to any jump hosts, and you should use ~/.ssh/config to configure jump hosts. So -i went to the private instance. The public one was only offered the default key files and anything in an ssh agent, and none of those was right.

The fix is to stop passing it on the command line:

Host pubec2
    HostName 32.196.116.111
    User ubuntu
    IdentityFile /root/.ssh/datacenter-key.pem

Host privec2
    HostName 10.10.1.114
    User ubuntu
    IdentityFile /root/.ssh/datacenter-key.pem
    ProxyJump pubec2

Every hop carries its own identity, and ssh privec2 works as though the private instance were directly reachable.

One more silent failure

The log shipping ran from /etc/cron.d, and that directory has a trap:

* * * * * root scp ... /var/log/boots.log ubuntu@10.20.1.134:/home/ubuntu/boots.log

Six fields before the command, not five. The crontab manual describes five time fields followed by a username in the system crontab, and says jobs in cron.d are system jobs that need that username. A crontab -e entry has no user field because the user is whoever owns the crontab. Put a five-field line in /etc/cron.d, and cron reads scp as a username, logs a failure to syslog, and never runs anything.

Checking each stage

The file landed. I checked both halves separately, the copy on the public instance and the object in S3, because on a two-stage pipeline that one extra ls halves the search when something breaks. Nothing on the public instance points at the peering, the security group or the first cron job. A file there but not in S3 points at the role or the second.

I should also name what the task required and what I would not repeat. Both instances used the same key pair, so the private instance ended up holding a private key that also opens the public one. Compromise either and you have both. In anything real, this would be SSM Session Manager and CloudWatch Logs, and no instance would hold another's credentials.

Each layer has its own no

The Deployment works because a layer I rarely look at does the replacing. The peering works because four independent layers each agreed. And the SSH hop failed because an option I thought was global only applied to one layer of the connection.

So here is the Day 49 question. When something is unreachable, how many separate layers are you checking before you decide which one is wrong?

Day 49 down. Fifty-one to go.

Day 48: Nobody Replaces a Bare Pod, and a Stack Knows How to Take Itself Apart

Nnamdi Felix Ibe — Wed, 23 Sep 2026 17:00:00 +0000

Kubernetes starts today, and it starts with the one object that has nobody looking after it. On the other side, the same Lambda I built by hand on Day 33, rebuilt as a template that knows how to create itself and how to tear itself down.

Both are about who is responsible for a thing once it exists.

One Kubernetes task, one AWS task. Deploy a pod, then deploy a Lambda through CloudFormation. The tasks come from the KodeKloud Engineer platform.

Two routes to a pod manifest

The fast route generates the YAML:

kubectl run pod-httpd --image=httpd:latest --labels="app=httpd_app" \
  --dry-run=client -o yaml > pod.yaml

--dry-run=client -o yaml prints the object kubectl would send without sending it. The Kubernetes quick reference uses exactly this pattern to generate a spec file, and it saves typing all the boilerplate.

The careful route writes it by hand:

apiVersion: v1
kind: Pod
metadata:
  name: pod-httpd
  labels:
    app: httpd_app
spec:
  containers:
  - name: httpd-container
    image: httpd:latest

They do not produce the same file, and that caught me. kubectl run names the container after the pod, so the generated manifest has a container called pod-httpd. The task wanted httpd-container. The generated file is a starting point to edit, not a finished answer.

kubectl apply -f pod.yaml
kubectl describe pod pod-httpd

Two small things worth having. kubectl validates manifests strictly by default: --validate defaults to strict, which fails the request on unknown fields rather than silently dropping them, so a misspelled key is an error you see. And when a pod will not start, the reason is in the Events section at the bottom of describe, not in the one-word status from get.

Nobody is watching this pod

Here is the part that matters. Delete pod-httpd, or lose the node it runs on, and it is gone. Nothing brings it back.

Kubernetes documents this directly: a ReplicaSet replaces pods that are deleted or terminated for any reason, such as node failure, unlike pods a user created directly. It goes as far as recommending a ReplicaSet even when you only need one pod.

So a bare pod is useful for exactly what I did with it, which is learn what a pod is. Anything that should still be running tomorrow needs something above it whose job is to notice when it is not. That is tomorrow's task.

The same Lambda, declared instead of commanded

On Day 33 this Lambda was six CLI calls and a retry loop, the loop because create-function failed when the IAM role had been created seconds earlier and had not propagated yet. Today it is one template and one create-stack, and it deployed first time with no retry anywhere.

I want to be careful about what that proves. CloudFormation documents that in its default mode it waits for each resource to reach a fully stabilised state before reporting it complete, which fits what happened. It does not document anything specific to IAM propagation for a role a Lambda uses in the same stack. So "no retry loop needed" is what I saw, not something I can point to as a promise. That distinction matters the day it fails.

What the template genuinely removes is the ordering:

Role: !GetAtt LambdaExecutionRole.Arn

AWS's rule is that when one resource refers to another with !Ref, !GetAtt or !Sub, the referenced one is created first and deleted last. That single reference is the whole dependency graph. DependsOn is for the cases where a real dependency exists that nothing in the template points at.

And one detail that looks arbitrary until you read why:

Handler: index.lambda_handler

When code is supplied inline, CloudFormation places it in a file named index, so the handler's module part has to be index. Only the function name after the dot is yours.

Two more habits carried in from yesterday. The template uses managed policies only, because Day 47's account refused inline ones. And before deploying, one aws iam get-role to make sure no role called lambda_execution_role already existed, since the template names it explicitly and a collision would leave the stack in ROLLBACK_COMPLETE, which only a delete gets you out of.

Taking it apart

Day 33's teardown was to delete the function, detach the policy, delete the role, in that order, and get it wrong at each step.

aws cloudformation delete-stack --stack-name datacenter-lambda-app

The same references that ordered the build order the teardown, in reverse. The stack knows what it made and how to unmake it.

Who is responsible once it exists

A bare pod has nobody. A CloudFormation stack is its own record of what exists and how it fits together. Most of the difference between a lab and a system is whether something is in that role.

So here is the Day 48 question. For the thing you deployed most recently, if it disappeared tonight, what would you notice, and what would put it back?

Day 48 down. Fifty-two to go.

Day 47: Copy Requirements Before the Code, and SQS Has No Priority Feature

Nnamdi Felix Ibe — Mon, 21 Sep 2026 21:00:00 +0000

Neither of today's tasks has a setting for the thing it delivers. Docker has no switch that makes rebuilds fast, and SQS has no switch that makes a queue high priority. In both cases, the feature is the order you put things in.

One Docker task, one AWS task. Package a Python app as an image, then build priority queues with SQS and SNS in a CloudFormation template. The tasks come from the KodeKloud Engineer platform.

This is also the last Docker task. Days 35 to 47 went from installing the engine to a real application image, and Kubernetes starts tomorrow.

Two COPY lines, and which one goes first

FROM python:3.11-slim
WORKDIR /app

COPY src/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY src/ .

EXPOSE 3004
CMD ["python", "server.py"]

The obvious version copies all of src/ and then installs. It works. It also reinstalls every dependency every time you touch a line of server.py.

Docker's cache guidance explains why. A change causes a rebuild for every step that follows it, so expensive steps belong near the beginning, and installing dependencies in an earlier layer means there is no need to rebuild that layer when a project file changes. Copy the manifest on its own, install, and only then copy the code. Now editing server.py invalidates the last COPY and nothing before it.

The same logic explains --no-cache-dir, and it is a nice detail. pip's own documentation recommends leaving its cache on unless you have caching at a higher level, and it gives layered caches in container builds as the example. That is precisely this situation. The Docker layer is the cache; a pip cache sitting inside it is just weight in the image.

One thing worth knowing about that last line. CMD in exec form is a JSON array and each element is taken literally, so a stray character in "server.py" is not a syntax error. It is a different filename. The container starts, Python cannot find the file, and it exits immediately. docker logs shows it; docker ps without -a shows nothing at all.

Then the run, with the port mapping reading host first, container second:

docker run -d -p 8097:3004 --name pythonapp_nautilus nautilus/python-app
curl http://localhost:8097/

CloudFormation, and the permission that was not there

The AWS task was the first infrastructure-as-code task of the run: two SQS queues, an SNS topic routing by priority, and a Lambda that drains high before low, all in one template.

The first deploy rolled back:

User: ... is not authorized to perform: iam:PutRolePolicy on resource: role lambda_execution_role

The template gave the role permissions two ways. CloudFormation documents Policies as adding an inline policy embedded in the role, and ManagedPolicyArns as attaching standalone managed policies. The error named the IAM action the inline route used, PutRolePolicy. Attaching a managed policy is a different action, and this account allowed one and refused the other. The role had been created and the managed policy attached before the refusal, which is how you tell exactly which half was blocked.

Moving everything to managed policies fixed it, at a cost worth stating. The managed policies available are AmazonSQSFullAccess and AmazonSNSFullAccess, far broader than a function that needs to receive and delete messages on two queues. In an account that allows it, a separate AWS::IAM::ManagedPolicy resource keeps the permissions scoped while still attaching rather than embedding.

Two CloudFormation lessons came with the failure.

validate-template had passed. The documentation is blunt that it is designed to check only the syntax of your template, not whether the property values are valid, and that to check operational validity you have to create the stack. Passing validation means the YAML parses.

And the failed stack was stuck. ROLLBACK_COMPLETE only exists after a failed stack creation, and AWS says the only operation available in that state is a delete. So the first deploy of a new stack is the one that costs a full delete-and-recreate cycle when it goes wrong. A failed update is kinder, returning to the previous working state as UPDATE_ROLLBACK_COMPLETE.

The priority is in the consumer

SQS has no priority feature. The whole mechanism is two queues and a consumer that looks at one first:

response = poll_messages(os.environ['high_priority_queue'])
if response == "No more messages to poll":
    response = poll_messages(os.environ['low_priority_queue'])

What gets messages into the right queue is the SNS filter policy on each subscription:

FilterPolicy:
  priority:
    - high

AWS describes it simply: if the message attributes satisfy the filter policy, SNS sends the message to that subscriber, and otherwise it does not. Without it, SNS fan-out sends every message to every queue and the exercise means nothing.

So I checked the queue depths before invoking anything. Two and two, not four and four. That proves the routing independently of the Lambda, which matters, because four and four would mean a filter problem that no amount of debugging the function could fix.

Then four invocations:

invoke 1: "Message 'High Priority message 2' deleted"
invoke 2: "Message 'High Priority message 1' deleted"
invoke 3: "Message 'Low Priority message 2' deleted"
invoke 4: "Message 'Low Priority message 1' deleted"

High before low, exactly as required. And message 2 before message 1 in both queues, which is worth sitting with rather than dismissing.

AWS documents standard queues as delivering at least once, with messages occasionally arriving out of order because of the distributed architecture, while making a best-effort attempt to keep the send order. So this design gives strict ordering between priorities and none within one. FIFO queues would fix that, with a .fifo name and a MessageGroupId on every send, and even then the guarantee is scoped to a message group. Messages in different groups can still arrive out of order relative to each other.

The feature is the arrangement

The Dockerfile is fast to rebuild because of which line comes first. The priority system works because of which queue is polled first. Neither is configured anywhere; both are the order.

So here is the Day 47 question. In the last thing you built, which behaviour depends on the order of steps rather than on any setting, and would anyone reading it later know?

Day 47 down. Fifty-three to go.

Day 46: Compose Wires the Whole Stack, and Permission to Invoke Is Not Permission to Act

Nnamdi Felix Ibe — Sun, 20 Sep 2026 08:00:00 +0000

Two things today needed wiring on both sides, and in both cases getting one side right produces something that looks finished and does nothing.

One Docker task, one AWS task. Stand up a PHP and MariaDB stack with Compose, then build an event pipeline where an S3 upload triggers a Lambda that copies the object and logs it. The tasks come from the KodeKloud Engineer platform.

Two services, one command

services:
  web:
    image: php:7.2-apache
    container_name: php_host
    ports:
      - "8085:80"
    volumes:
      - /var/www/html:/var/www/html

  db:
    image: mariadb:latest
    container_name: mysql_host
    ports:
      - "3306:3306"
    volumes:
      - /var/lib/mysql:/var/lib/mysql
    environment:
      MYSQL_DATABASE: database_host
      MYSQL_USER: <username>
      MYSQL_PASSWORD: "<password>"
      MYSQL_RANDOM_ROOT_PASSWORD: "yes"

sudo docker compose -f /opt/sysops/docker-compose.yml up -d
curl http://localhost:8085

One up -d creates a project network, both containers and both mounts. The two containers can reach each other by service name over that network, which is the Day 42 embedded DNS point finally doing the job it exists for. Nothing here declares that relationship; it comes free with the project network.

Worth separating two names that look interchangeable and are not. The db service is addressable from web as db, the service key. container_name only fixes what docker ps and docker exec see. Pointing a connection string at mysql_host works because the container name resolves too, but db is the portable one.

Note also that this mount targets /var/www/html while yesterday's httpd file wanted /usr/local/apache2/htdocs. Same idea, different image convention, and it is worth reading the image's documentation rather than assuming a path you have seen before.

The one that catches people: those MariaDB environment variables are the image's initialisation contract, and they only apply on first start. The image initialises the database only when the data directory is empty. Mount a /var/lib/mysql that already has data and every one of those variables is ignored, in silence. A stack that "will not pick up the new password" is nearly always this.

And publishing 3306 is what the task asked for, not what you would otherwise do. The web container reaches the database over the project network with no published port at all. Putting 3306 on the host exposes the database to anything that can reach the host.

Every permission it needed, and never called

The AWS task was six resources in one path: an upload to a public bucket fires an S3 event, Lambda copies the object into a private bucket and writes an audit row to DynamoDB.

The execution role covered s3:GetObject, s3:PutObject and dynamodb:PutItem, scoped to exactly one bucket each and one table. Correct, least-privilege, and completely insufficient, because none of it lets S3 call the function.

Policy	Attached to	Answers
Identity-based	The role	What is Lambda allowed to do?
Resource-based	The function	Who is allowed to invoke Lambda?

aws lambda add-permission --function-name devops-copyfunction \
  --statement-id s3invoke --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::$PUB_BUCKET --source-account $ACCT

Get only the identity side right and nothing errors. There are no logs, because the function never ran. This is the same shape as trust policy versus permissions policy on Day 33 and execution role versus task role on Day 38, and it keeps arriving in a new costume.

The ordering is load-bearing too. add-permission has to come before put-bucket-notification-configuration, because S3 validates the destination when you set a notification. Without the permission you get:

Unable to validate the following destination configurations

Which mentions neither Lambda nor permissions nor what to change. Same category as AccessDenied on PutBucketPolicy actually meaning BlockPublicPolicy: the error names the call that failed rather than the setting that caused it. The console hides this by issuing both calls when you add a trigger through the UI.

Scope the permission, or anyone can use it

Lambda's resource-based permissions support a deliberately small set of condition keys: aws:SourceArn, aws:SourceAccount and aws:PrincipalOrgID. Leave the first two off and the statement allows any S3 bucket, in any account, to invoke your function.

That is the confused deputy problem in one line. Someone creates a bucket, points a notification at your function ARN, and your code runs on their objects with your permissions. The condition block you want to see in the response:

"Condition": {
  "StringEquals": {"AWS:SourceAccount": "245695940513"},
  "ArnLike": {"AWS:SourceArn": "arn:aws:s3:::devops-public-22976"}
}

Three details worth stealing

128 MB handles any file size here, and the reason is not generosity. s3.copy_object is server-side: S3 copies the object internally and the bytes never pass through the Lambda execution environment. Had the code done get_object then put_object instead, the file would flow through Lambda memory and 128 MB would cap the file size hard. Same apparent outcome, completely different resource profile.

put-bucket-notification-configuration takes the whole configuration document. There is no append. On a bucket already sending events to SQS or SNS, a straight write removes them silently, so the safe pattern on anything you did not create is read, merge, write.

And a hyphen is not valid in a Python module name. The provided file is lambda-function.py, the handler string is module.function, and lambda-function is not importable. Renaming it during the build step is a one-character fix that otherwise surfaces as an import error at invoke time, minutes after everything else looked fine.

The bug in the error handler

Worth recording, because it is a realistic failure mode rather than a criticism of the lab:

try:
    source_bucket = event['Records'][0]['s3']['bucket']['name']
    ...
except Exception as e:
    log_entry = {
        'SourceBucket': source_bucket,      # unbound if the first line threw

If that first line raises, on a malformed or non-S3 event, source_bucket was never assigned. The except block then throws NameError while building the error log, and the original exception is lost.

A handler written to record failures, unable to record the one class of failure that happens before its variables exist. Initialising them to None above the try fixes it. Not needed for the task to pass, and exactly how an error handler ends up hiding the error it was written to surface.

Both sides, or nothing moves

Compose wires two containers into one project, and the relationship is implicit. Lambda needs two policies pointing in opposite directions, and neither implies the other.

The tell in both cases is the same: a component that is individually correct and collectively inert. A db container running perfectly that web cannot name. A function with exactly the right permissions that nothing is allowed to call.

So here is the Day 46 question. When you grant access to something, do you check both halves, what it can do and who can reach it, or only the half that was in the ticket?

Day 46 down. Fifty-four to go.

Day 45: RUN Cannot See Your Build Context, and NAT Only Goes One Way

Nnamdi Felix Ibe — Sat, 19 Sep 2026 08:00:00 +0000

Both of today's tasks are about direction. A Dockerfile instruction that cannot reach backwards to the machine you are building on. A gateway that lets traffic out and never lets it in.

One Docker task, one AWS task. Fix a Dockerfile that will not build, then give a private-subnet instance outbound internet access. The tasks come from the KodeKloud Engineer platform.

The error is about location, not spelling

cd /opt/docker/
docker build -t nautilus:latest_img .

cp: cannot stat 'certs/server.crt': No such file or directory

The file is right there. ls shows certs/server.crt sitting next to the Dockerfile. The line that failed:

RUN cp certs/server.crt /usr/local/apache2/conf/server.crt

cp is not wrong and the path is not wrong. The instruction is. RUN executes a command inside the image filesystem, where certs/ has never existed. The file is in the build context, on the host, and RUN cannot see it.

COPY certs/server.crt /usr/local/apache2/conf/server.crt
COPY certs/server.key /usr/local/apache2/conf/server.key
COPY html/index.html /usr/local/apache2/htdocs/

COPY takes files from the build context and puts them into the image. That is the one instruction that spans the boundary, and it is the only way anything on your machine gets into the build.

What makes this a good puzzle is that the same file also has four RUN sed lines that are completely fine:

RUN sed -i "s/Listen 80/Listen 8080/g" /usr/local/apache2/conf/httpd.conf

Those edit httpd.conf, which is already in the image because the base image put it there. Editing a file that exists inside the image is exactly what RUN is for. Reaching for a file on the host is not. Four correct RUN lines and three wrong ones, in the same file, differing only by which side of the boundary the file lives on.

One detail to check before blaming the Dockerfile. The trailing . on docker build is the build context, and COPY paths resolve against it. Build from the wrong directory and COPY certs/server.crt fails identically, for the opposite reason: the instruction is right and the context does not contain the file.

The NAT gateway goes in the subnet it is not serving

This one reads backwards until it clicks. A NAT gateway serving a private subnet is placed in the public subnet, and AWS states it as a requirement: you create a public NAT gateway in a public subnet and must associate an Elastic IP with it at creation.

The reason is that a NAT gateway is itself a client of the internet. It takes traffic from private instances, rewrites the source address to its own Elastic IP, and forwards it out. To forward anything it needs its own route to an internet gateway, and that route lives in the public subnet's table.

Put it in the private subnet and it creates without complaint, reaches available, and drops every packet it forwards. Nothing in describe-nat-gateways suggests a problem.

The finished path is two tables and two hops:

EC2 (private subnet)
  -> priv-rt: 0.0.0.0/0 -> nat-gateway
    -> NAT gateway (public subnet, holds the Elastic IP)
      -> pub-rt: 0.0.0.0/0 -> internet gateway
        -> internet

The private subnet never references the IGW directly.

One way only

"Internet access" is doing a lot of work in the task description, so it is worth being precise. AWS says instances behind a NAT gateway cannot receive unsolicited inbound connections from the internet.

	Internet gateway	NAT gateway
Outbound from instance	Yes	Yes
Inbound to instance	Yes	No
Instance needs a public IP	Yes	No
Costs money at rest	No	Yes

The instance still has no public IP and never will. It can install packages, call AWS APIs and upload to S3, and nothing on the internet can open a connection to it. That asymmetry is the entire reason to use a NAT gateway rather than just moving the instance to a public subnet.

The Elastic IP belongs to the gateway, not the instance, so every private instance behind it shares one source address. Which is also what makes it useful when a third party wants an IP to allowlist.

Two things the task was right to insist on

It said explicitly not to touch the VPC's Main route table, and that is not fussiness. The Main table is the fallback for any subnet with no explicit association, so a 0.0.0.0/0 route there grants internet access to every unassociated subnet in the VPC, including ones created months later by someone who assumed a new subnet would be isolated. A dedicated table with an explicit association makes the intent readable and keeps everything else isolated by default.

And AZ placement is not cosmetic. AWS documents each NAT gateway as created in a specific availability zone and made redundant within that zone. Which means if resources across several AZs share one gateway and its zone goes down, the resources in the healthy zones lose internet access too. The recommendation is one per AZ with routing to match. Here one private subnet in one AZ meant one gateway in the same AZ, which is both correct and cheapest.

While we are on cost: a NAT gateway bills hourly whether anything flows through it or not, plus a per-gigabyte processing charge. For this workload, an instance in a private subnet talking only to S3, a Gateway VPC Endpoint would have done the job for nothing:

aws ec2 create-vpc-endpoint --vpc-id $VPC --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids $PRIV_RT

Free, keeps the traffic on the AWS network, and covers S3 and DynamoDB only. The task asked for NAT, so NAT is what got built, but the cheaper answer exists.

Checks that come before the wait

Two habits carried over from Day 40, both of which turn an ambiguous outcome into a clear one.

Check the route state before waiting on the result. active rather than blackhole proves the target resolves, so an empty bucket three minutes later means the cron job has not fired yet rather than that the path is broken.

And establish the negative first. The bucket was confirmed empty before anything was built, which is what turns the later listing into evidence. A leftover file from a previous attempt would have looked exactly like success.

Which side of the line

RUN cannot reach the host. A NAT gateway cannot be reached from the internet. In both cases the thing that feels like a limitation is the boundary doing its job, and in both cases the failure mode is a command that looks reasonable and fails somewhere you were not looking.

So here is the Day 45 question. In the last thing you built, do you know which direction each connection can be opened from, or only that it works?

Day 45 down. Fifty-five to go.

Day 44: Compose Stopped Needing a Version, and the CLI Gives You No Grace Period

Nnamdi Felix Ibe — Fri, 18 Sep 2026 17:00:00 +0000

Two defaults today, both of them different from what a reasonable person would assume. Compose no longer wants the version key that every tutorial still opens with. And an Auto Scaling group created from the CLI has a health check grace period of zero, where the console gives you 300 seconds.

One is a warning. The other is an infinite loop.

One Docker task, one AWS task. Write a Compose file serving httpd from a host directory, then rebuild Day 36's load-balanced instance as an Auto Scaling group. The tasks come from the KodeKloud Engineer platform.

The version key is obsolete

services:
  web:
    image: httpd:latest
    container_name: httpd
    ports:
      - "5000:80"
    volumes:
      - /opt/sysops:/usr/local/apache2/htdocs

No version: "3.9" at the top, and that is deliberate. Compose's documentation says the top-level version property exists only for backward compatibility, that it is informative, and that using it produces a warning telling you it is obsolete. Compose validates against the most recent schema regardless of what you write there.

So the line every older tutorial opens with now buys nothing and costs you a warning on every run.

Three other things in that short file are worth more than they look.

ports and volumes both read host first, container second, matching -p and -v on the command line. Quote the port mapping, because YAML will read an unquoted 5000:80 as a sexagesimal number in some parsers and hand you a value you did not write.

The document root is /usr/local/apache2/htdocs, not /var/www/html. That second path belongs to the Debian and Ubuntu Apache packages; the official httpd image lays itself out differently. Mount over the wrong one and you get a running container cheerfully serving the image's default page.

And container_name fixes the name rather than letting Compose generate <project>-<service>-1. It makes docker exec predictable and makes scaling that service past one replica impossible, since two containers cannot share a name.

The default that would have cost me the task

The AWS half was Day 36 again with the hand-launched instance replaced by an Auto Scaling group, a launch template and a target tracking policy. Two things in it are genuinely worth carrying.

First, launch templates do not encode user data for you.

# run-instances: CLI v2 base64-encodes this automatically
--user-data file:///root/ud.sh

# create-launch-template: you do it yourself
UD=$(base64 -w 0 /root/ud.sh)

run-instances has a dedicated parameter with special handling. create-launch-template takes one opaque blob where UserData is a string field the API expects already encoded. Pass raw text and the call succeeds. The template is created, the ASG launches an instance, cloud-init cannot make sense of the payload and moves on. You get a running instance with no nginx, no error in any API response, and a target group that never goes healthy. The only trace is in a log file on an instance you have not set up SSH for.

Second, and this is the one I would not have found without reading the documentation:

--health-check-grace-period 300

The grace period is the minimum time a new instance stays in service before being terminated for failing a health check. AWS documents the default as 300 seconds in the console and 0 seconds via the CLI or SDK, where 0 turns it off entirely.

That matters because the task also required --health-check-type ELB, which makes the target group's HTTP check drive instance replacement rather than the default EC2 status check. The default only asks whether the VM is running, so nginx could crash completely and the ASG would see a perfectly healthy instance.

Put those together with no grace period and the sequence is:

Instance launches
Health check fails, because nginx is still installing
ASG terminates it as unhealthy
ASG launches a replacement to satisfy desired capacity
Back to step 2, forever

An infinite launch-and-terminate loop that burns money and never converges, and it shows up in the ASG activity history rather than anywhere you would think to look. Preventing exactly that is what AWS says the grace period is for.

Worth noting the warning runs the other way too. Set it too high and a genuinely broken instance stays in service longer, blunting the health checks you turned on.

Three smaller things

The ASG registers targets, you do not. Naming --target-group-arns on the group means it registers every instance it launches and deregisters every one it terminates. Running register-targets by hand against an ASG-managed group is actively wrong, because the next scaling action silently undoes it.

Early unhealthy is normal and the reason field tells you which kind it is. Elb.RegistrationInProgress and Elb.InitialHealthChecking mean wait. Target.Timeout means no TCP connection, so a security group or nothing listening. Target.ResponseCodeMismatch means it answered with the wrong status. Only the last two are unambiguously broken.

And escape \$Latest. It is a literal AWS keyword, and unescaped inside double quotes bash expands it as a shell variable, finds nothing, and sends Version= to an API that rejects it.

Reading the default rather than assuming it

The Compose version key is a default that used to be required and now warns. The grace period is a default that differs depending on whether you used the console or the CLI, in a direction that turns a working configuration into a loop.

Neither is discoverable by doing the obvious thing and watching it work. The Compose file runs fine with the version key, just noisily. The ASG would have failed in a way that looks like an application problem.

So here is the Day 44 question. For the last resource you created from a script rather than a console, do you know which of its defaults differ between the two?

Day 44 down. Fifty-six to go.

Day 43: A Port Is Closed Until You Publish It, and an AZ Name Is Not an AZ

Nnamdi Felix Ibe — Wed, 16 Sep 2026 17:00:00 +0000

Today both tasks turned on reading a label correctly. A port number on the host is not the port inside the container, and they do not have to match. An availability zone name is not the availability zone, and in someone else's account it points somewhere else entirely.

One Docker task, one AWS task. Publish a container port on a different host port, then provision an EKS control plane. The tasks come from the KodeKloud Engineer platform.

Publishing, and the direction of the colon

docker pull nginx:stable
docker run -d --name demo -p 6000:80 nginx:stable
curl http://localhost:6000

Host first, container second. -p 6000:80 listens on 6000 on the host and forwards to 80 inside. Reverse it and you get a host listening on 80, forwarding to a container port nothing is bound to, and a connection that opens and returns nothing.

The two numbers are independent, which is the part worth internalising. nginx is still on 80 inside the container and has no idea the host calls it 6000. Nothing in the image changes to publish it elsewhere.

And without -p the container is not broken, it is just not reachable from outside. Docker puts it plainly: ports on bridge networks are accessible from the Docker host and from other containers on the same network, and are not accessible from outside the host or, by default, from containers on other networks.

That splits cleanly against yesterday's task. The network decides which containers can see each other. Publishing decides whether the host's port space does. Two different questions, and EXPOSE in a Dockerfile answers neither, since it only documents which port the application listens on.

One habit worth forming early: -p 127.0.0.1:6000:80 binds to loopback only. Plain -p 6000:80 binds on every interface, which on a public host means the internet.

The error names a zone that does not generalise

The EKS task was a control plane to a specific configuration. Named cluster, latest stable Kubernetes, default VPC, three availability zones, Auto Mode off, private endpoint.

Passing all six subnets in the default VPC produced this:

Cannot create cluster because EKS does not support creating control plane instances in us-east-1e

Clear enough, and I nearly wrote it down as the rule. It is not the rule.

AWS documents the restriction against availability zone IDs, not zone names. Subnets cannot sit in use1-az3 in us-east-1, usw1-az2 in us-west-1, or cac1-az3 in ca-central-1. Zone names are mapped to zone IDs independently per account, so us-east-1e is this account's name for use1-az3, and in your account use1-az3 may well be called something else.

Which means "avoid us-east-1e" is a note that works in exactly one account and silently avoids the wrong zone everywhere else. The portable version:

aws ec2 describe-availability-zones --query 'AvailabilityZones[].{Name:ZoneName,Id:ZoneId}'

This is the kind of thing that only shows up when you go and read the documentation for something you already appeared to understand. The error told me the truth about my account and nothing about the rule.

Shorthand cannot hold a list inside a struct

--resources-vpc-config subnetIds=subnet-a,subnet-b,subnet-c,endpointPublicAccess=false

AWS CLI shorthand separates struct fields with commas. It also separates list items with commas. So that string is genuinely ambiguous, and the parser cannot tell where subnetIds ends and the next field begins.

JSON removes the question:

--resources-vpc-config "{\"subnetIds\":[$SUBNET_JSON],\"endpointPublicAccess\":false,\"endpointPrivateAccess\":true}"

That is now the third list convention in this challenge. elbv2 create-load-balancer --subnets wants spaces. ecs create-service wants commas inside brackets. eks create-cluster wants a JSON array. There is no rule, only the habit of checking, and the working heuristic is to reach for JSON the moment a struct contains a list.

Two smaller things, and one honest caveat

Resolve the version rather than hardcoding it. aws eks describe-cluster-versions returns what is currently valid, and Kubernetes ships three minor releases a year, so a hardcoded version is wrong within months. Same habit as SSM parameters for AMIs and orderable instance options for RDS engine versions: ask AWS what is valid now.

The service principal for an EKS cluster role is eks.amazonaws.com. That is the fourth different principal in this run, after lambda.amazonaws.com, ec2.amazonaws.com and ecs-tasks.amazonaws.com. Mostly it is <service>.amazonaws.com, with ECS as the exception. A wrong one gives you a role that looks right in the console and can never be assumed.

And the caveat, because the task says "ready for workloads" and that sentence is doing some work. The cluster is ACTIVE and has no compute attached. No managed node group, no Fargate profile, nothing. Schedule a pod today and it sits Pending forever with no nodes available. That is the correct scope, since the task specified nothing about nodes and inventing a node group would have been inventing requirements. But ACTIVE is not schedulable, and the node role that comes next needs three managed policies and is a different role from the cluster role.

Labels are not the things they label

6000 on the host and 80 in the container are the same service under two names. us-east-1e and use1-az3 are one zone under two names, one of which travels and one of which does not.

The port case is harmless because you chose both numbers. The zone case is not, because the name was chosen for you, per account, and the error message hands you the local one as though it were universal.

So here is the Day 43 question. The last error message you turned into a rule, was it telling you about the system, or about your particular instance of it?

Day 43 down. Fifty-seven to go.

Day 42: A Custom Network Gives You DNS, and DynamoDB Only Declares Its Key

Nnamdi Felix Ibe — Mon, 14 Sep 2026 21:00:00 +0000

Both of today's tasks declare far less than you would expect, and both have a rule waiting in the part you did not declare. A Docker network is three flags and changes how containers find each other. A DynamoDB table declares one attribute out of three, and the one word the task chose for another is reserved.

One Docker task, one AWS task. Create a user-defined network, then build a DynamoDB table and prove two items have the right status. The tasks come from the KodeKloud Engineer platform.

The reason to create a network is not isolation

docker network ls
docker network create <name> --driver <driver> --subnet <CIDR> --ip-range <CIDR inside it>
docker network inspect <name>

Six drivers ship with Docker, and they are not variations on a theme. bridge is the default, host removes network isolation from the host entirely, none isolates completely, overlay joins multiple daemons for Swarm, and ipvlan and macvlan put containers onto the physical network, the latter making them appear as devices on the host's own network.

But the thing that makes a user-defined network worth creating is smaller and more useful than any of that. Containers on the default bridge cannot refer to each other by name. Containers on a network you created use Docker's embedded DNS server and can reach each other by container name.

That is the whole feature. Every multi-container stack that connects to db rather than 172.18.0.3 is relying on it, including the Compose files coming up later this week, which get a project network for free and never mention DNS anywhere.

Two smaller things worth knowing. --subnet is the CIDR the network occupies, and --ip-range allocates container addresses from a sub-range inside it, which is how you keep the rest of the block for addresses you assign yourself. And macvlan and ipvlan are not drop-in swaps for a bridge: they attach to a physical interface and need the host's network to cooperate.

Finish with inspect, not ls. ls proves a name exists. inspect shows the driver, the addressing and what is attached, which is what the task actually specified.

You declare the keys and nothing else

aws dynamodb create-table --table-name datacenter-tasks \
  --attribute-definitions AttributeName=taskId,AttributeType=S \
  --key-schema AttributeName=taskId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

The items carry taskId, description and status. Only one of those appears in the command, and that is correct. AWS defines AttributeDefinitions as an array of attributes that describe the key schema for the table and indexes, and nothing beyond that. Everything else comes into existence when an item carries it.

This reliably catches people arriving from SQL, who read --attribute-definitions as a column list and try to declare all three. Adding description there without using it in a key is an error, because DynamoDB has no use for the definition.

One habit before the first write:

aws dynamodb wait table-exists --table-name datacenter-tasks

create-table returns CREATING, and writing to a table in that state fails with ResourceNotFoundException, which reads exactly like a typo in the table name.

Every value carries its type, and numbers are strings

{"taskId":{"S":"1"},"description":{"S":"Learn DynamoDB"},"status":{"S":"completed"}}

S string, N number, B binary, plus BOOL, L, M, NULL and the set types. The one that surprises people is that N values are written as JSON strings: {"N": "42"}, never {"N": 42}. AWS gives the reason directly, which is that numbers are sent across the network as strings to maximise compatibility across languages and libraries. They are still treated as numbers once they arrive.

Note taskId is {"S":"1"}, the character, not the integer. The key schema declared it S, so {"N":"1"} would be rejected outright as a key type mismatch.

The word the task chose is reserved

aws dynamodb scan --table-name datacenter-tasks \
  --filter-expression "status = :v" \
  --expression-attribute-values '{":v":{"S":"completed"}}'

Invalid FilterExpression: Attribute name is a reserved keyword; reserved keyword: status

DynamoDB reserves several hundred words in its expression language and they are ordinary English. status, name, size, count, timestamp, year, data, value, source, region, owner, hash and key are all on the published list. Almost any attribute name you would reach for first is on it somewhere.

The fix is an expression attribute name, which is a placeholder:

aws dynamodb scan --table-name datacenter-tasks \
  --filter-expression "#s = :v" \
  --expression-attribute-names '{"#s":"status"}' \
  --expression-attribute-values '{":v":{"S":"completed"}}' \
  --query 'Items[].taskId.S' --output text

Two substitution mechanisms, and conflating them is the usual next mistake. #name stands in for an attribute name and its value is a bare string. :value stands in for a value and its value is a typed descriptor. # is what to look at, : is what to compare against.

Worth noticing that put-item needed none of this. Reserved words only matter inside expressions, so an attribute literally called status stores perfectly well and only becomes awkward the moment you query on it. Which points to the real lesson, upstream of all of it: do not name an attribute status. taskStatus costs nothing and deletes the problem from every future query.

One last thing about that verification, because it is doing something the get-item before it did not. get-item shows the item and lets me read the status off the screen. The filtered scan makes DynamoDB assert the status matches and hand back the ID. Same answer, different authority. That said, FilterExpression is applied after the read, so a scan reads the whole table and pays for all of it before discarding what does not match. Fine for two items, a bad habit at a million.

The undeclared part still has rules

The Docker network and the DynamoDB table are both mostly implicit. You do not wire containers to each other, you put them on a network and naming resolves. You do not declare a schema, you write items, and the attributes appear.

In both cases, the implicit part is not lawless. Name resolution only works on a network you created. Undeclared attributes are fine until one of them collides with a reserved word you had no reason to know about.

So here is the Day 42 question. In the system you work on, what behaviour are you relying on that nothing in your configuration actually states?

Day 42 down. Fifty-eight to go.