Day 76 of DevOps, Day 26 of Azure. Both tasks today went green, and both were quietly resting on something that will not hold.
A database backup job ran on schedule, produced a file, copied it across two machines, and backed up no tables at all. And an Azure network behaviour I had written in my notes as how Azure works turned out to be a platform default that Microsoft is in the middle of retiring.
One Jenkins task, one Azure task. Schedule a database backup across three servers, then put a VM in a purpose-built virtual network. The tasks come from the KodeKloud Engineer platform.
Task 1: A database backup on a ten-minute schedule
The task: a job named database-backup that dumps a database on stapp01 every ten minutes and copies the dump to the storage server.
Step 1: Two hops, two keys
Three machines are involved, and the Jenkins controller never touches the storage server directly. So each hop needs its own key, belonging to the account that opens it.
jenkins@jenkins:~$ ssh-keygen -t rsa -b 4096
Created directory '/var/lib/jenkins/.ssh'.
jenkins@jenkins:~$ ssh-copy-id tony@stapp01
jenkins@jenkins:~$ ssh tony@stapp01
[tony@stapp01 ~]$ ssh-keygen -t rsa -b 4096
[tony@stapp01 ~]$ ssh-copy-id natasha@ststor01
The second key had to be generated as tony, because tony@stapp01 is where the first hop lands and therefore the account that makes the second call. One key cannot do both hops.
This is the chain Day 74 could not account for: there, the transcript showed the second hop keyed and never showed the first, and the build still went green. Here both are on the record.
One detail on the second hop worth noticing:
The authenticity of host 'ststor01 (10.244.240.154)' can't be established.
ED25519 key fingerprint is SHA256:yEyN8qvzhNxfcKVE+H05zwQPmQMKCXj4JyGWuOP1HIg.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
That prompt is exactly what Day 75's agents switched off with their non-verifying host key strategy. Answered once interactively, recorded, done.
Step 2: The build step, and one thing it gets right
ssh tony@stapp01 "
mysqldump -u kodekloud_roy -pasdfgdsd kodekloud_db01 > /tmp/db_\$(date +%F).sql &&
scp /tmp/db_\$(date +%F).sql natasha@ststor01:/home/natasha/db_backups/
"
The && is inside the quotes, which is Day 71's bug written correctly. There, ssh host cmd1 && cmd2 split in the local shell and only the first half ever reached the remote machine. Here the whole conditional is one argument to ssh, so the scp runs on stapp01 and only if the dump succeeded.
The escaping is right too, and for a reason my notes got slightly wrong. I had written that the backslash makes the filename come out in the right format. What it actually does is choose whose clock names the file: \$ stops the local shell expanding the substitution, so the literal text is sent and date runs on stapp01.
Which is the exact inverse of Day 71, where "$PACKAGE" had to expand locally because the Jenkins parameter only exists on the Jenkins side. Same mechanism, opposite intent. Escape it to evaluate remotely, leave it bare to evaluate locally, and know which one you mean.
Step 3: The job works. The backup does not.
The file arrived on both machines and the build went green. Then I read it:
-- MariaDB dump 10.19 Distrib 10.5.29-MariaDB, for Linux (x86_64)
--
-- Host: localhost Database: kodekloud_db01
-- ------------------------------------------------------
-- Server version 10.5.29-MariaDB
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
...
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
...
-- Dump completed on 2026-04-14 7:02:52
A header, session settings, the restore of those session settings, and a completion line. No CREATE TABLE. No INSERT. Nothing whatsoever between the opening SET block and the closing one.
So kodekloud_db01 either has no tables, or kodekloud_roy cannot see the ones it has. Either way this job backs up zero rows, successfully, every ten minutes, and produces a plausible-looking file while doing it.
The task is satisfied. It asked for a backup job and there is one. The backup is worthless, and nothing anywhere in the pipeline says so: valid SQL, reasonable size, green build, file present at the destination.
This is Day 74's lesson with the stakes raised. There, a scheduled job copying no logs succeeds on time forever, and the fix was to confirm the source files existed first. Here the equivalent check is one line:
mysql -u kodekloud_roy -p... -e "SHOW TABLES IN kodekloud_db01"
And the verification that belongs in the job itself asserts on content rather than exit status:
grep -qi 'CREATE TABLE' "$DUMP" || { echo "dump has no tables"; exit 1; }
An empty lab database and a privilege problem look identical from outside, and mysqldump exits zero for both. Only the content tells you which you have.
What the docs recommend
Three things, and the first is the one that would matter in a real environment.
The password is in the job. MySQL does not hedge: "Specifying a password on the command line should be considered insecure. To avoid giving the password on the command line, use an option file." The documentation also explains the odd-looking flag, since "there must be no space between --password= or -p and the password following it."
In a Jenkins job this is worse than at a terminal, because the command is stored rather than typed. The build step lives in the job's config.xml, and anyone with Job/Configure on the job can read it in a browser.
Day 73 granted Job/Configure on a job to two people. The same grant here hands over a database credential, and nothing in the permission grid hints at it. The column says Configure. What it conveys is the contents of every build step.
Jenkins points at the fix. Credentials have their own permission group, with Credentials/View and Credentials/Update kept separate from job permissions, which is Jenkins separating credential access from job access on purpose. And its Pipeline documentation gives the pattern directly: never interpolate a secret into the command text, send a literal command string, and let "the shell read secrets from its environment", which the credentials binding has already populated. Two things improve at once, because the value leaves config.xml and Jenkins masks it in the console output.
Second, there is a latent bug in the date. date is called twice, and the two substitutions are evaluated at different moments. If midnight falls between the dump finishing and the scp starting, mysqldump writes one filename and scp looks for another and fails. On a ten-minute schedule that is one build a day exposed to it, presenting as an occasional failure with no obvious cause.
Computing it once removes the whole class of problem, and lets the quoting get simpler because nothing needs to expand locally any more:
ssh tony@stapp01 '
D=$(date +%F)
mysqldump --defaults-file=~/.my.cnf kodekloud_db01 > "/tmp/db_$D.sql" &&
scp "/tmp/db_$D.sql" natasha@ststor01:/home/natasha/db_backups/
'
Third, the schedule. H/10 * * * * rather than */10 * * * *, for Day 74's reasons: H "should be used wherever possible" and is "a hash of the job name, not a random function". With a log-copy job already on a six-minute cycle, round intervals guarantee the two collide eventually, and H separates them without anyone coordinating. And mysqldump without --single-transaction locks tables while it reads, which costs nothing on an empty database and blocks writes every ten minutes on a real one.
Task 2: A VM in a purpose-built network
The task: a VNet, a subnet, and a VM inside it with a public IP and SSH reachable from the internet.
Step 1: Name things rather than letting the VM name them
--vnet-name devops-pub-vnet --subnet devops-pub-subnet
--public-ip-address devops-pub-vm-pip --public-ip-sku Standard
--nsg devops-pub-vm-nsg
Every earlier VM in this track let az vm create build its own network, which produces names derived from the VM, like <vm>VNET and <vm>VMNic. Here the network came first and the VM was told where to go, so the names come from the design. That starts mattering the moment a second VM shares the network.
Naming the public IP also carries the Day 21 benefit: it survives the VM being deleted and recreated, so the address stays put.
Step 2: What makes the network public, and why my answer has expired
My notes said a VNet has no public or private attribute, and that reachability comes entirely from a public IP on the NIC plus an NSG rule. The first half still holds. The second is now incomplete, and what has changed is outbound rather than inbound.
Subnets do now carry a relevant property, and Microsoft's own word for the result is private: "Azure is changing the default subnet configuration for new virtual networks to require explicit outbound connectivity." With a date attached that has already passed: "For the API version released after March 31, 2026, subnets in new virtual networks have defaultOutboundAccess set to false by default."
So a VNet created today with a current API version gets subnets whose VMs "can't rely on default outbound access". My note's claim that Azure handles the routing and the NSG is the only gate describes a platform behaviour being retired, not a property of Azure networking.
Three things keep that from affecting this task, and the second is the interesting one.
Existing networks are untouched: "Existing virtual networks aren't changed", and deployments on earlier API versions "retain the earlier default behavior", which the lab's CLI may well have done.
And this VM already has an explicit outbound method, without my having chosen one for that purpose. Microsoft lists "Associate a Standard public IP to any of the virtual machine's network interfaces" among the four explicit methods, and --public-ip-sku Standard provided exactly that. A VM built to be reachable from the internet happens to have the thing that also makes its egress explicit.
Of those four methods, Microsoft marks one as preferred: "Associate a NAT gateway to the subnet of your virtual machine. Note this is the recommended method for most scenarios."
Which means the AWS comparison in my notes needs qualifying rather than deleting. Azure still has no internet gateway or route table to configure for basic reachability, which is why Day 40 and Day 45 have no counterpart on this track. What it no longer has is outbound connectivity you can assume. A private subnet needs a NAT gateway, and that is the closest Azure analogue to the work Day 45 did on AWS.
What the docs recommend
Two smaller things, both about not doing the same work twice.
The port 22 rule was created twice. --nsg-rule was not passed to az vm create, and its accepted values are {NONE, RDP, SSH} with Linux defaulting to SSH, which creates a default-allow-ssh rule at priority 1000. The explicit az vm open-port --port 22 --priority 1001 then added a second rule for the same port. Harmless, because both are Allow and the one at 1000 matches first, and still redundant configuration in two places. Either --nsg-rule NONE plus the explicit call, or --nsg-rule SSH and no call, leaves one rule and one place to change it.
And the disk flags, which I have now carried forward three times and explained wrongly twice:
--storage-sku Standard_LRS --os-disk-size-gb 30
Day 21 failed on this, and Days 22, 23 and 26 all pass the flags without being asked to. My explanation, that az vm create defaults the OS disk to Premium on premium-capable sizes, is wrong, as Day 72 covered: --storage-sku has no documented default at all, and the managed-disk path sends no storage account type whatsoever. Something below the CLI chooses, and the rule is not published. The flag is still mandatory in this subscription.
What I learned
Both halves of today were green and both were built on something that does not hold up.
The backup job is the one I would want to find before someone needed it. Every signal said it worked: exit zero, file created, file copied, file identical on both machines, build green, schedule firing. The only thing that said otherwise was reading the file, and what it said was that the job had been faithfully backing up nothing. No monitoring built on exit codes would ever have caught it, because nothing failed.
The Azure half is the same error made against documentation rather than against a file. I wrote "Azure handles the routing" in my notes as a fact about the platform. It was a default, and Microsoft had already announced its retirement with a date that has since passed. The VM survives on a technicality that happens to be sound: the Standard public IP that makes it reachable is also, separately, one of the four documented ways to make its egress explicit.
The pattern underneath both is the same one from Day 75, arriving by a different route. A thing that is true today, recorded as a thing that is true.
So here is the Day 76 question. Which of your backups has anyone actually opened?
Day 76 down. Twenty-four to go.
Top comments (2)
And here we are, already at Day 76! π
Thank you for being so consistent and for sharing such wonderful content with us. I genuinely admire the effort and dedication you put into every single post. It's really lovely to see someone show up day after day with so much passion.π
Your consistency deserves so much appreciation. Keep doing what you do, and keep inspiring us along the way. π₯
Sending you lots of support and appreciation for this beautiful journey! π·
@technogamerz Divya, thank you, thank you for such kind words. This is genuinely so kind of you, and I appreciate the support. π₯°
Iβll be honest with you, though. The consistency gets credited to me, and it isnβt all mine. On the days when the lab is dull and I canβt find anything worth saying, itβs just knowing a handful of people will actually read it that keeps me going. Youβre one of the reasons thereβs a Day 76 at all.
I finished the lab tasks for the 100 Days of DevOps today. Over the next few weeks, Iβll be going back through my notes and deep-diving into the documentation to write up the remaining posts. Itβs been a journey, and kudos to the people who built the KodeKloud platform, without which this series wouldnβt have been possible.
Twenty-four to go. Thank you for being here for so many of them. π