Day 81: Two Stages and a Regional Quota

Day 81 of DevOps, Day 31 of Azure. Both tasks today involved reading something that was not there.

A pipeline Test stage checks that an application is accessible, and would report success whether or not the Deploy stage that ran before it accomplished anything. And an Azure quota error left one field blank, and I read the blank as evidence about what the quota applied to.

One Jenkins task, one Azure task. Build a two-stage pipeline that deploys and then tests, then deploy a Python web app. The tasks come from the KodeKloud Engineer platform.

This is the last Jenkins day. Days 82 to 93 are Ansible, which is the gap I have been calling the clearest one on my CV since August.

Task 1: Two stages, deploy and test

The task: a Pipeline job named deploy-job with a Deploy stage that pulls from master and a Test stage that confirms the application is accessible.

Step 1: Stages, not jobs

Day 80 chained two jobs. This one uses two stages in one job, and both are right for what they are.

Deploy and restart on Day 80 were separate operations with separate reasons to exist, and manage-services is worth triggering on its own. Deploy and test here are one unit of work with one result. A stage boundary gives separate pass and fail reporting without separate configuration, which is exactly what you want when the only sensible response to a failed test is to look at the deployment that preceded it.

Step 2: The quoting, which goes the other way from Day 78

environment {
    APP_URL     = 'https://8091-port-ceyx5ty225wyfbc2.labs.kodekloud.com/'
    DEPLOY_DIR  = '/var/www/html'
}

stages {
    stage ('Deploy') {
        steps {
            sh """
                cd ${DEPLOY_DIR}
                git pull origin master
            """
        }
    }
}

Day 78's pipeline used sh '''...''', triple single quotes, which do not interpolate in Groovy, so the shell expanded the variable from its own environment. This one uses triple double quotes, which do.

So Groovy substitutes ${DEPLOY_DIR} into the command text before the agent receives it, and that is the pattern Jenkins documents as the thing to avoid. Groovy string interpolation "can inject rogue commands into command interpreters via special characters" and "can result in problems analogous to SQL injection", and the documented fix is to "use single quotes to avoid Groovy interpolation".

The exposure here is small, because these values come from the environment block rather than from a user. Nobody can set APP_URL at build time.

What makes it worth fixing anyway is that the double quotes buy nothing at all. The environment directive has already "defined as environment variables for all steps" every one of those names. They are in the shell's environment either way. Single quotes give an identical result, with the interpolation layer gone, so the day somebody adds a parameter to this pipeline the hazard is not already sitting there waiting for it.

Two pipelines in one batch, three days apart, on opposite sides of the one rule Jenkins documents about this. And the safer one is also the simpler one.

Step 3: The Test stage would pass without the Deploy stage

This is the finding worth keeping.

curl --fail --silent --show-error ${APP_URL} || (echo 'APPLICATION IS DOWN' && exit 1)

That checks that the application answers. It checks nothing about what was deployed.

A stale index.html from last month returns 200. A page from a different branch returns 200. The exact content the Deploy stage failed to install returns 200, as long as Apache is running.

So the stage proves the web server is up. The task's wording, "test if the application is accessible", is satisfied by that. Accessibility is not what a deployment pipeline needs to know immediately after deploying.

The walkthrough's own validation shows the gap without naming it. It changes index.html in the repository, runs the pipeline, and then opens a browser to check the new content appeared. That manual check is the real test. The pipeline's Test stage did not do it.

A stage that tested the deployment would assert on content:

curl --fail --silent --show-error "$APP_URL" | grep -q 'Welcome to xFusionCorp Industries'

Better still, compare against what the repository says it should be, so the assertion survives the next content change:

EXPECTED=$(git -C "$DEPLOY_DIR" show origin/master:index.html)
curl --fail --silent --show-error "$APP_URL" | diff -q - <(printf '%s' "$EXPECTED")

This is Day 76's lesson arriving in a new place. There, a backup job succeeded every ten minutes while its dump contained no tables at all, because nothing checked the content. Here, a Test stage passes while the Deploy stage could have done nothing. Exit codes tell you a command ran, not that it achieved anything.

What the docs recommend

Three smaller things, and the first is the kind of bug that makes a check worse than no check.

... || (echo 'APPLICATION IS DOWN' && exit 1)

That exit 1 runs inside a subshell, because of the parentheses, so it exits the subshell rather than the script. The build still fails, but only because the || list is the last command in the script and so its status becomes the script's. Move the curl anywhere, but the end and a failed check goes green. Braces fix it regardless of position:

curl --fail --silent --show-error "$APP_URL" || { echo 'APPLICATION IS DOWN'; exit 1; }

A Test stage whose failure path works by accident of line ordering is worse than no Test stage, because it reports success instead of nothing.

Second, --fail --silent --show-error is genuinely the right combination and worth knowing as a set: --fail turns an HTTP error status into a non-zero exit, --silent kills the progress meter, and --show-error puts the error message back so a failure says something.

Third, the sleep 5. It is honest about a real problem and solves it the way Days 22 and 23 argued against. A fixed sleep tuned to one run wastes time on a fast one and fails on a slow one. A retry loop around the curl waits on the condition instead.

And the URLs do not belong in the file. That hostname is session-specific, so a new lab brings a new one, and this pipeline then fails in the Test stage against a host that no longer resolves, which reads like an application outage. REPO_URL is declared and never used, too, since the Deploy stage pulls from whatever remote the existing working copy has. Same shape as Day 80's unused plugin: configuration that looks load-bearing and is not.

Task 2: A Python web app

The task: a Python web app on a B1 Linux App Service Plan, with two tags and without Application Insights.

Step 1: The error with the blank field

The first two labs both failed:

Operation cannot be completed without additional quota.
Additional details - Location:
Current Limit (B1 VMs): 0
Current Usage: 0

East US appeared in az appservice list-locations --sku B1, so the SKU was offered there. The limit was zero anyway.

I read that blank Location: field as evidence the limit was subscription-wide. I said trying another region would "most likely give the same message", and recommended reporting the lab as broken rather than running the one diagnostic I had just finished writing out.

B1 was created immediately in South Central US. The quota was regional. East US was exhausted.

The mistake was reasoning from an empty field. A blank Location means the message template did not populate it, not that the limit has no regional scope. When an error omits the field that would tell you its scope, test the scope rather than inferring it. The test was one command and I talked straight past it.

Which is today's other half, in the opposite direction. The Test stage reads a 200 as evidence of a successful deployment, which it is not. I read an absence as evidence of scope, which it was not. Both are conclusions drawn from something that was never saying what it looked like it was saying.

The related judgment was wrong in effect too. I listed every workaround as breaking a requirement, which was true of SKU changes and would have been true of a region change if East US were mandatory. Presenting that as "stop and report" closed off the option that worked.

Step 2: Plan and app are different things

az appservice plan create -n devops-learn-python --sku B1 --is-linux
az webapp create -n devops-webapp-347137090 -p devops-learn-python --runtime "PYTHON:3.10"

The plan is the compute: VM size, OS, region, and the thing that gets billed. The app is a workload running on it, and one plan can host several apps sharing its CPU and memory.

Two properties are fixed when you create the plan. The OS, where --is-linux produces reserved: true, and a Windows plan cannot host a Linux Python app. And the region, which the plan and every app on it inherit.

The SKU can change later with az appservice plan update --sku. Worth setting beside Day 79's finding that an ACR tier moves freely in either direction with no downtime: here one dimension scales and two are decided once.

reserved: true meaning "Linux" is a genuinely unhelpful field name. The app's kind: app,linux says it readably.

Step 3: Two syntaxes for one runtime value

Command	Format
az webapp create --runtime	PYTHON:3.10
az webapp config set --linux-fx-version	`PYTHON\
Stored value	{% raw %}`PYTHON\

The reference confirms the colon form for create: {% raw %}--runtime is "Canonicalised web runtime in the format of Framework:Version". The pipe form is what the API stores and what every read returns, so passing a colon to --linux-fx-version sets a literal string that matches no runtime.

az webapp list-runtimes prints the colon form, because it exists to feed --runtime. One thing worth knowing before trusting it: its --support parameter accepts {active, all, eol, near, supported} and defaults to supported, so end-of-life runtimes are hidden. A version that worked last year may simply not be listed, and --support all is how you tell a retired runtime from a typo.

One correction to my notes while I was in there. I had used --deployment-container-image-name as the container-mode flag. The CLI now marks it deprecated: "Option '--deployment-container-image-name' has been deprecated and will be removed in a future release." The point survives, since there is still no publish-mode flag and the mode is still inferred from which argument you pass. Only the flag name was stale.

What the docs recommend

Read configuration from the command whose job it is. The az webapp create output said:

"linuxFxVersion": "",
"pythonVersion": null,

Which looks like the runtime did not apply. It did:

az webapp config show -g $rg -n devops-webapp-347137090 --query linuxFxVersion
PYTHON|3.10

The create response returns a sparse siteConfig, with most fields unpopulated regardless of their real values. Third time this has caught me in the Azure track, after Day 18's missing contentMd5 and Day 29's null Tags. The response shape belongs to the command, not the resource.

pythonVersion being empty is correct rather than contradictory: it is the Windows stack field, and Linux apps express their whole runtime through linuxFxVersion. A check written against it would conclude no Python runtime was configured.

And verifying an absence takes two lookups, not one. App Insights leaves two traces, an instrumentation key in app settings and a microsoft.insights/components resource, so confirming it is off means enumerating both. Same reason as Day 27's NSG rule listing: when the requirement is that something is not present, you have to look at what exists rather than confirm what you intended.

What I learned

Two conclusions drawn from things that were not evidence.

The pipeline's Test stage is the one I would want to find before trusting it. It passes, it is named Test, it sits after a Deploy stage, and it would report success if the deployment had silently done nothing, because all it establishes is that Apache answers on a port. The walkthrough then checks the real thing by hand, in a browser, and does not notice that this is the actual test and the pipeline is not doing it.

My quota error was the same mistake with the parts swapped. A field was blank, and I treated the blankness as a fact about the quota rather than a fact about the message template. One command would have settled it, I had already written the command out, and I argued for giving up instead.

Both are the same habit seen twice: deciding what something means before checking what it says.

So here is the Day 81 question. Which of your tests would still pass if the thing it runs after had quietly done nothing at all?

Day 81 down. Nineteen to go.