Day 71: One Command, Two Machines

Day 71 of DevOps, Day 21 of Azure. Both tasks today turned on the same mistake: treating one line of a command as one unit of work.

A Jenkins build step I wrote as a single remote command was actually two commands on two different machines, and only one of them reached the server I meant. An az vm create I thought of as one command turned out to be a deployment of six resources, so when it failed, five of them stayed behind.

One Jenkins task, one Azure task. Build a parameterised job that installs a package on a remote host, then gives a VM a static public IP. The tasks come from the KodeKloud Engineer platform.

Task 1: A Jenkins job that installs a package

The task: a freestyle job named install-package, with a string parameter PACKAGE, which installs the named package on the storage server ststor01.

Step 1: Find out what the target runs

cat /etc/os-release

NAME="CentOS Stream"
VERSION="9"
ID="centos"
PLATFORM_ID="platform:el9"

CentOS Stream 9, so dnf. The Jenkins node, from Day 68, is Ubuntu 24.04, so apt. Two package managers in one job, which is the detail that makes the next step go wrong.

Step 2: The parameter, and how it reaches the shell

A String parameter named PACKAGE, then a shell build step using $PACKAGE.

That works because of something Jenkins documents directly: "parameterised freestyle projects pass parameter values to Shell and Batch build steps as environment variables." The parameter is not substituted into the script by Jenkins. It is exported, and the shell expands it.

Worth noting what is not documented. Jenkins' own site has no page describing the freestyle "Execute shell" build step, its shell, or its exit-status behaviour. The documented equivalent is Pipeline's sh step, which "Runs a Bourne shell script", uses "the system default shell ... using the -xe flags", and fails the build when "a script which exits with a nonzero status code" runs. Freestyle behaves the same way in practice. I could not cite that from Jenkins.

Step 3: A key for the storage server

Jenkins has no credentials for ststor01, so:

ssh jenkins@jenkins
ssh-keygen -t rsa -b 4096
ssh-copy-id natasha@ststor01

The key has to belong to the jenkins user, because that is who the build runs as. Logging in as root and generating a key in /root/.ssh produces a key the build cannot see. ssh-keygen offering /var/lib/jenkins/.ssh/id_rsa is how you know you are the right user, since JENKINS_HOME is /var/lib/jenkins.

-t rsa -b 4096 also overrides two defaults, and not the ones usually assumed. The ssh-keygen manual gives the default key type as ed25519, and for RSA, "the minimum size is 1024 bits and the default is 3072 bits". So 4096 is an upgrade from 3072, not from 2048.

sudo then works over that connection without a password because natasha has a passwordless rule, and it has to, because there is nowhere to type one. The ssh manual: ssh "by default will only request a pseudo-terminal (pty) for interactive sessions when the client has one". Supplying a command is not an interactive session, so there is no terminal for a prompt to appear on.

I had attributed that to sudo's requiretty. It is a different thing. Requiretty "will only run when the user is logged in to a real tty", it "is off by default", and it governs whether sudo runs at all, not whether it can prompt.

Step 4: The build step was wrong

This is what went into the job:

ssh natasha@ststor01 sudo dnf makecache && sudo dnf install -y $PACKAGE

Read as one remote command, it makes sense. It is not one remote command.

&& belongs to the local shell. Bash's manual: "A list is a sequence of one or more pipelines separated by one of the operators ';', '&', '&&', or '||'", and the shell "Breaks the input into words and operators" before it runs anything. So ssh never receives && as an argument at all. By the time ssh starts, the line has already been split.

ssh builds its remote command only from the arguments it does receive. Its synopsis is destination [command [argument ...]], and "If a command is specified, it will be executed on the remote host instead of a login shell."

Which means the line runs on two machines:

sudo dnf makecache              -> on ststor01
sudo dnf install -y $PACKAGE    -> on the Jenkins node

And the Jenkins node is Ubuntu. It has no dnf. The install half never ran anywhere it could work.

The fix is to quote the whole remote command:

ssh natasha@ststor01 "sudo dnf makecache && sudo dnf install -y $PACKAGE"

Double quotes, deliberately, not single. The local shell expands $PACKAGE before sending, which is what is wanted here, because the Jenkins parameter only exists on the Jenkins side. Single quotes would send the literal $PACKAGE, and the remote shell would expand it to nothing.

One thing I cannot reconcile, and would rather say than quietly tidy up. My own notes record the build succeeding, and vim-enhanced.x86_64 2:8.2.2637-27.el9 present on ststor01 afterwards. The command as I wrote it down cannot produce that. The likeliest explanation is that the build step in Jenkins was not identical to the line I copied into my notes, which is the gap a screenshot-driven write-up leaves. The corrected form above is the one to use either way.

What the docs recommend

The risk in this job is not the package manager. It is that $PACKAGE is free text that ends up inside a command line on a remote host, with passwordless sudo at the other end.

Jenkins warns about exactly this shape. Using a Windows example: "A variable value that contains valid Batch commands may result in the execution of these commands by Jenkins", and "special care may need to be taken to filter or sanitise environment variable values". Then the general case: "Any user who can add environment variables with a name they choose may be able to modify the behaviour of builds this way."

So anyone who can trigger this build can run an arbitrary command on the storage server as a user with passwordless sudo. A validated choice parameter instead of a free string closes most of it. And as on Day 69 and Day 70, the job's definition is worth living in a file where that constraint is reviewable, rather than in a form where it is one edit away from gone.

Task 2: A static public IP for a VM

The task: datacenter-vm, Ubuntu, Standard_B1s, with an SSH key and a static public IP named datacenter-pip, reachable over SSH.

It took four attempts, and the sequence is the interesting part.

Step 1: The default OS disk is banned

RequestDisallowedByPolicy: Resource 'datacenter-vm_OsDisk_...' was disallowed by policy.
Reasons: 'Global: This storage configuration is not allowed. Ensure the disk size is
128 GB or less and the SKU is not Premium for Compute disks.'

The fix was to stop leaving the disk at the default:

--storage-sku Standard_LRS --os-disk-size-gb 30

Day 2 and Day 3 both passed --storage-sku Standard_LRS and both worked, not because I knew about the policy but because those tasks specified a disk type. This one said nothing about disks, so the default applied, and the default is the one thing this subscription refuses.

My explanation of which default was wrong, and it was specific enough to be worth correcting. I had written that az vm create picks Premium_LRS whenever the VM size supports premium storage, which Standard_B1s does.

The CLI does not do that. --storage-sku has no documented default, and the only hardcoded Premium_LRS in the CLI's validation applies to the old unmanaged disk profiles rather than to managed disks. On the managed-disk path, which is the default, the CLI sends no storage account type at all and leaves the Compute service to choose. What it chooses is not stated in the CLI reference.

Which makes the lesson stronger, not weaker: something downstream picked a Premium disk, and I cannot read the rule it used. In a policy-constrained subscription, the values you do not specify are the ones most likely to be rejected, and the mechanism behind them may not be documented anywhere you can check.

Step 2: az vm create does not roll back

The failed attempt left this:

datacenter-pip      Microsoft.Network/publicIPAddresses
datacenter-vmVNET   Microsoft.Network/virtualNetworks
datacenter-vmNSG    Microsoft.Network/networkSecurityGroups
datacenter-vmVMNic  Microsoft.Network/networkInterfaces
datacenter-vm       Microsoft.Compute/virtualMachines    <- provisioningState: Failed

az vm create is one command and six resources. When the VM fails, the five that already exist stay, and the VM itself remains as a resource in Failed state.

That there is no rollback is not documented for the command. The surrounding behaviour is the same, and it points the same way: ARM's undo is opt-in. --rollback-on-error is "The name of a deployment to roll back to on error, or use as a flag to roll back to the last successful deployment", with a warning that it "rollbacks a failed deployment by redeploying an earlier deployment" and the result "may be different than what you would expect from undoing the failed deployment". ARM replays. It does not reverse. And it does nothing unless asked.

This is Day 70's finding showing its consequences. An ARM deployment is a record of an operation, not an owner of resources, so nothing owns the half-built output and nothing cleans it up.

CloudFormation is the other way round. Day 47 failed on an IAM permission and rolled itself back, because OnFailure defaults to ROLLBACK. The price there was that the stack had to be deleted before retrying. The price here is cleaning up by hand.

Step 3: My own cleanup caused the next two failures

OperationNotAllowed: Operation 'Update VM' is not allowed on VM 'datacenter-vm'
since the VM is marked for deletion.

Caused by this:

az vm delete -g $RG -n datacenter-vm --yes 2>/dev/null

Two mistakes on one line. 2>/dev/null hid whether it worked at all. And Azure's control-plane operations are asynchronous, so success means started, not finished: "Some Azure REST operations run asynchronously because the operation can't be completed quickly", returning a status code and a header to poll. I then ran az vm create against a VM that was still being deleted.

The obvious waiter did not help:

az vm wait -g $RG -n datacenter-vm --deleted
The operation failed

It reported failure while the VM was still listed. What --deleted actually polls is not documented; the help says "Wait until deleted." and gives an interval and a timeout. What I can say is what it did here, which is give up on a resource that was already in a failed state.

What worked was testing the only thing that matters:

az vm delete -g $RG -n datacenter-vm --yes; echo "delete exit=$?"
until ! az vm show -g $RG -n datacenter-vm >/dev/null 2>&1; do sleep 10; done

until ! az vm show asks whether the resource still exists, rather than asking a waiter for its opinion about state.

Step 4: The IP, which was the point

Created as its own resource first, then referenced by name:

az network public-ip create -g $RG -n datacenter-pip --sku Standard --allocation-method Static
az vm create ... --public-ip-address datacenter-pip

The address did not change across four attempts:

Attempt 1 (policy failure):  20.119.41.127
Attempt 4 (success):         20.119.41.127

Because deleting a VM does not delete a public IP. Microsoft: "By default, disks, NICs, and Public IPs associated with a VM are persisted when the VM is deleted." And the CLI documents the dual behaviour of the flag: --public-ip-address is the "Name of the public IP address when creating one (default) or referencing an existing one".

Had I let az vm create generate the IP, each failed attempt would have produced a different address, and every retry would have invalidated whatever DNS record or firewall allowlist pointed at the last one. Which is the problem the task exists to solve.

--allocation-method Static is redundant with --sku Standard, since Standard's allocation method is Static with no alternative. I passed it anyway, so the requirement is visible in the command rather than implied by a SKU.

What the docs recommend

Three resources survived my cleanup on purpose:

datacenter-pip      the whole point, keep it
datacenter-vmVNET   az vm create reuses a VNet of this name
datacenter-vmNSG    az vm create reuses an NSG of this name

az vm create generates names from the VM name, {vm}VNET, {vm}Subnet, {vm}NSG, {vm}PublicIP and {vm}VMNic, and for most of them it will reuse an existing resource of that name instead of failing. The reference says so for the NSG: "The name to use when creating a new Network Security Group (default) or referencing an existing one."

The NIC is the exception, in the CLI's source and in practice. It holds the private IP reservation and the public IP association, so a new VM cannot adopt one already sitting there under the generated name. That is why only the VM, its NIC and its disk had to be removed between attempts.

Two more small things that only show up on a retry. --generate-ssh-keys printed its message on the first attempt, the one that failed, and the later attempts reused the keys, because it is documented to "Generate SSH public and private key files if missing" and stores them in ~/.ssh. And azureuser rather than the local username, because the local user here is root, and root is on Azure's list of disallowed Linux admin usernames. The CLI anticipates that: "Default value is current username of OS. If the default value is system reserved, then default value will be set to azureuser."

Verification, as on Day 13:

ssh -o BatchMode=yes azureuser@20.119.41.127 'whoami; hostname'

azureuser
datacenter-vm

BatchMode=yes makes a password prompt impossible, so that login could only have come from the key.

What I learned

One line is not one unit of work, and both halves of today punished that assumption in different ways.

The ssh command looked like one instruction to one machine and was two instructions to two machines, because && is handled by the shell in front of ssh, not by ssh. Nothing errored loudly. Half the work simply happened on the wrong host, where the tool it needed does not exist.

az vm create looked like one instruction and was a deployment of six resources, so failure is not a single event. Five resources existed, one sat in Failed state, and the command meant to clean up started an operation it did not wait for, with its error output discarded.

The static IP is the counterweight. By creating the address as its own resource and referring to it by name, the one thing that had to survive four rebuilds did, while everything around it was destroyed and recreated repeatedly.

So here is the Day 71 question. In your scripts, which single line is actually doing its work on more than one machine, or on more than one resource, than you would say if asked?

Day 71 down. Twenty-nine to go.