Day 75 of DevOps, Day 25 of Azure. Both tasks today punished something I had written down as a fact when it was really a snapshot.
A Jenkins agent would not start because the Java version on it had been fine until the controller moved past it. And an Azure disk that was sda before the task was sdb after it, with a different disk holding the old name.
One Jenkins task, one Azure task. Attach the app servers to Jenkins as build agents, then expand one disk and add another. The tasks come from the KodeKloud Engineer platform.
Task 1: Three servers as build agents
The task: attach the three app servers to Jenkins as agents. For the first one, name App_server_1, label the same, remote root directory /home/tony/jenkins.
Step 1: The plugin, and what it is called now
The plugin is SSH Build Agents. It "is formerly known as 'SSH Slaves Plugin'" and was renamed in version 1.31.0, and its plugin ID is still ssh-slaves.
So three names for one thing, which is why searching for any of them lands in the same place. The KodeKloud task uses the old vocabulary, and so does the identifier you would put in a configuration file.
Its documented scope is almost nothing: it "Allows to launch agents over SSH", using "a Java implementation of the SSH protocol", and it "Requires Jenkins 2.504.1". That is the whole of what its page says about behaviour.
Step 2: Create the node, and find the fields are undocumented
Name, label, remote root directory, host, credentials, host key verification strategy. Jenkins' Managing Nodes page defines none of them. It does not describe the Remote root directory, the Labels field, the Usage field, or the launch methods, and does not list SSH among them.
That is the third day running. Day 71 was the Execute shell build step, with no page. Day 72 was String and Choice parameters, with no page. Today it is the entire node configuration form.
What Jenkins does document is the requirement and the reason for the whole exercise. "An agent can use any operating system that supports Java", and you should "Ensure that you have a java version available on your agent machine". And then, plainly: "running tasks on the built-in node is discouraged for security, performance, and scalability reasons."
That last line is why this task exists. Days 71 through 74 ran everything on the controller, directly or over ssh from it. This is the day the work moves off the machine Jenkins itself lives on.
Step 3: The agent refuses to start
java.lang.UnsupportedClassVersionError: hudson/remoting/Launcher has been compiled by a
more recent version of the Java Runtime (class file version 61.0), this version of the
Java Runtime only recognizes class file versions up to 55.0
Agent JVM has terminated. Exit code=1
[tony@stapp01 ~]$ java -version
openjdk version "11.0.20.1" 2023-08-24 LTS
Java 11 on the agent. Installing java-17-openjdk and relaunching brought it online.
The fix was right. My explanation of it was wrong in three ways, and the correction is more useful than the fix.
I had written that Jenkins requires Java 17 "because it is the current Long-Term Support (LTS) version, and its components are compiled to run on it, ensuring better stability".
Java 17 is not the current LTS release and was not then. Java 21 is a later one, and Jenkins has moved past both: "Beginning with Jenkins 2.555.1, Jenkins requires Java 21 or Java 25 on both the controller JVM and agent JVMs." The same page adds that "With the release of Jenkins 2.555.1, the Java 17 images have been retired."
The reason Java 17 was required at all is a project decision with a date on it. Jenkins raised its baseline at the 2.463 weekly release in June 2024, announced as "Jenkins requires Java 17 or newer", with 2.462 being "the last weekly release to support Java 11". The requirement covers "agent JVMs (i.e., JVMs running remoting.jar)" explicitly, and the announcement names this exact failure: "Running a Java 11 agent with a Java 17 controller produces an UnsupportedClassVersionError."
And it is not about stability. The agent's JVM cannot load a class file built for a newer JVM than itself, so the agent does not start. Saying Java 17 gives "better stability" implies Java 11 would be worse. Java 11 would be nothing.
The fix was still correct for this Jenkins: Day 68 established the lab runs 2.541.3, the last LTS release supporting Java 17. Written as a general rule for today, "install Java 17 for Jenkins agents" is already out of date. The general rule is to install whatever the controller's version requires, and to upgrade controller and agents together, agents first.
What the docs recommend
Two choices in this node configuration are worth more attention than the Java version.
The host key verification strategy was set to Non verifying, which I justified as skipping manual trust setup in a lab. Practical, and the strategy names turn out to be undocumented: the plugin's page lists no host key verification options at all.
What its page does carry is a security advisory, and it is precisely this one: "Man-in-the-middle vulnerability due to missing host key verification", affecting version 1.14 and earlier.
So the non-verifying strategy is not a missing feature. It is an option to reinstate, per node, the exposure that advisory was issued to fix. Choosing it accepts that anyone who can answer on the agent's address and port can impersonate the agent, and Jenkins will hand over the connection and the credentials used to open it. Which in this task are tony's password.
That is the second choice. The credential here is a username and password, where Days 71 and 74 both set up key-based trust for the same class of connection. The plugin supports SSH keys. A password is a reusable secret sitting in the credential store for no benefit.
To be fair to the credential store, it is a boundary. Jenkins gives credentials their own permission group, with Credentials/View and Credentials/Update separate from job permissions, so credential access is deliberately not the same as job access. A password in the store is at least behind that line.
A password typed into a build step is not. That is tomorrow's problem.
Task 2: Expand one disk, add another
The task: grow the VM's disk from 32 to 64 GiB, then create a 64 GiB standard HDD named devops-disk and mount it at /mnt/devops-disk.
Step 1: The outage belongs to the OS disk, not the task
az vm deallocate ...
az disk update -g $RG -n $OSDISK --size-gb 64
az disk create -g $RG -n devops-disk --size-gb 64 --sku Standard_LRS
az vm disk attach -g $RG --vm-name devops-vm --name devops-disk
az vm start ...
I had written that expanding a managed OS disk requires the VM deallocated. Right conclusion, and Microsoft never says it in those words, so it is worth showing how it follows.
Expanding without downtime is documented, and its scope is explicit: it "Is supported only for data disks." For the path that does not qualify, "the VM must be deallocated." An OS disk is outside the downtime-free feature, so it lands on the deallocation path by elimination rather than by a rule written about OS disks.
Two things follow that I had not drawn out. The data disk alone could have been expanded with the VM running, so the outage was the OS disk's requirement, not the task's. And deallocate is not a synonym for stop: "Stopping the VM with az vm stop doesn't release the compute resources. To release compute resources, use az vm deallocate."
Since the VM had to stop anyway, creating and attaching the data disk in the same window meant one outage covering both halves of the task instead of two.
Step 2: The reboot did the other two layers, and I overstated how
Day 50's lesson was that a disk has three layers and each needs its own action. After the VM started:
sdb1 63.9G part /
/dev/root ext4 62G 1.7G 61G 3% /
Already grown, with nothing run inside the guest.
I credited that to cloud-init running growpart on every boot. The documented claim is narrower and says nothing about every boot: "On Ubuntu 16.x and newer, the root partition of the OS disk and filesystems are automatically expanded to use all free contiguous space on the root disk by cloud-init. A small amount of free space must be available for the resize operation."
So cloud-init does it, on Ubuntu 16.x and newer, to the root partition of the OS disk, and it needs a little free space. Whether it checks on every boot or only when something changed is not stated.
The mirror-image reading survives the correction, and it is the part worth keeping:
- AWS allowed an online resize, so no reboot happened, so cloud-init never ran, so the partition and filesystem were manual.
- Azure required a deallocation, and the restart that followed is what expanded them.
The platform constraint supplied the convenience. Which matters, because a data disk expanded with the VM running gets no reboot, and so still needs the filesystem grown by hand.
Step 3: The letters moved while I was working
Before this task the OS disk was sda. After attaching one data disk:
sda 64G <- devops-disk, the NEW disk
sdb 64G sdb1 / <- the OS disk
sdc 4G sdc1 /mnt <- the temporary disk
The OS disk became sdb, and the new data disk took the name the OS disk used to have.
Day 8 and Day 50 both warned that device letters are assigned in enumeration order and are not stable. This is that warning arriving. A script written as "format /dev/sda1" against yesterday's layout would have reformatted the root volume today.
So the script resolved the device instead of naming it:
DEV=$(readlink -f /dev/disk/azure/scsi1/lun0)
The LUN is the slot on the VM and does not change. The letter is an artefact of enumeration order. And for the persistent mount, the UUID does the same job, because an fstab entry naming /dev/sda1 would, after the next disk change, mount the wrong filesystem or fail the boot.
Which is the Jenkins half of today in different clothing. sda meant one disk and now means another. remoting.jar ran on Java 11 and now will not. Both were true when written down and neither was a stable fact about the system.
What the docs recommend
The mount point the task asked for is the thing I would change in real infrastructure.
/mnt/devops-disk sits inside /mnt, and on an Azure Linux VM /mnt is where the ephemeral temporary disk is mounted. So this is a persistent disk mounted at a path inside a filesystem that gets wiped on deallocation. It works now. After the next stop and start, the directory will not exist on the fresh temporary disk.
nofail keeps that from blocking the boot, and that is exactly the problem: the data disk may simply not be mounted, and nothing complains. Azure's own fstab line for the temporary disk shows the pattern for this, ordering itself after cloud-init, and the equivalent here would be x-systemd.requires-mounts-for=/mnt.
I have not applied it, because the task asked for the mount and the mount is there. But a persistent disk underneath an ephemeral mount point is the wrong shape, and /data or /srv/devops-disk at the root avoids the whole question.
One more number worth knowing before someone asks. The 64 GiB disk reports 63G to the filesystem and 60G available, and both are correct: ext4's metadata takes the first slice and its root reserve takes the second. Three numbers, one disk, nothing wrong.
What I learned
Everything that broke today was something I had recorded as a fact and that was really a reading at a moment in time.
sda was the OS disk. Java 11 was sufficient for the agent. Neither statement was wrong when written. Both became wrong without anything on those machines changing: one because a disk was attached, the other because the Jenkins project moved its baseline, twice, and my note about it was already stale when I wrote it down.
The defence in both cases turned out to be the same, which is what makes it worth noticing. Do not name the thing, resolve it. The disk script asks for LUN 0 and follows the symlink to whatever letter the kernel chose this boot. The agent should take whatever Java the controller's version requires, checked against the controller, rather than the version that worked in a tutorial.
And the two configuration choices nobody made me justify were the ones carrying actual risk: a host key strategy that reinstates a patched man-in-the-middle vulnerability, and a password where a key would have done.
So here is the Day 75 question. Which version number or device name is hardcoded in your scripts because it was true the day you wrote them?
Day 75 down. Twenty-five to go.
Top comments (1)
I really felt the part about the SSH Build Agents plugin having three names for one thing, because I have spent too much time tracking down which plugin actually provides a capability in Jenkins. That is the lesson I keep coming back to: with Jenkins, the plugin-per-capability model means maintenance is part of the price, not an afterthought, especially when a rename or an undocumented field changes how a node is configured. Your point about not hardcoding a version that was true when you wrote it down also landed for me, since the Java 11 to Java 17 agent failure is exactly the kind of silent assumption that breaks a build. Do you now pin the controller and agent Java versions together in your lab notes, or check them as part of the agent setup itself?