DEV Community

Cover image for Day 64: A Bad Image and a Small Disk
Nnamdi Felix Ibe
Nnamdi Felix Ibe

Posted on AI-assisted

Day 64: A Bad Image and a Small Disk

#devops #kubernetes #azure #containers

Day 64 of DevOps, Day 14 of Azure. Today I had to withdraw an explanation. The Kubernetes half went fine, but the Azure half rested on a tidy theory I had built from one observation on Day 8, and when I checked it against the source, it was wrong.

One Kubernetes task, one Azure task. Fix a Flask app whose pod will not start and whose Service points at the wrong port, then create a managed disk. The tasks come from the KodeKloud Engineer platform.

Task 1: Two faults, one application

The task: python-deployment-devops is not running, and python-service-devops does not reach it. Find out why and fix both.

Step 1: Ask the Deployment what it thinks it is running 

kubectl get deploy
kubectl describe deployments.apps
Enter fullscreen mode Exit fullscreen mode 
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-devops   0/1     1            0           4m35s
Enter fullscreen mode Exit fullscreen mode 
Containers:
 python-container-devops:
  Image:         poroko/flask-app-demo
  Port:          5000/TCP
Enter fullscreen mode Exit fullscreen mode

Two facts worth holding on to: the image name, and that the container declares port 5000.

Step 2: Ask the pod what went wrong 

kubectl get po
kubectl describe po
Enter fullscreen mode Exit fullscreen mode 
NAME                                        READY   STATUS         RESTARTS   AGE
python-deployment-devops-7dd8f6ddf8-4z6qd   0/1     ErrImagePull   0          5m52s
Enter fullscreen mode Exit fullscreen mode 
State:          Waiting
  Reason:       ImagePullBackOff

Events:
  Warning  Failed   Failed to pull image "poroko/flask-app-demo": failed to resolve reference
           "docker.io/poroko/flask-app-demo:latest": pull access denied, repository does not
           exist or may require authorisation: server message: insufficient_scope
  Warning  Failed   Error: ErrImagePull
  Warning  Failed   Error: ImagePullBackOff
  Normal   BackOff  Back-off pulling image "poroko/flask-app-demo"
Enter fullscreen mode Exit fullscreen mode

The existing image is poroko/flask-demo-app. The Deployment asks for poroko/flask-app-demo. Two words transposed.

The registry error offers three explanations and only one applies: "pull access denied, repository does not exist or may require authorisation". A registry will not tell an anonymous client whether a private repository exists, so a typo and a missing credential produce the same message. That is why the docs' first check is the dull one: "Make sure that you have the name of the image correct."

Step 3: Fix the image in place 

kubectl edit deployment
Enter fullscreen mode Exit fullscreen mode 
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
python-deployment-devops   1/1     1            1           12m
Enter fullscreen mode Exit fullscreen mode

kubectl edit is enough here, and for a specific documented reason: "A Deployment's rollout is triggered if and only if the Deployment's Pod template (that is, .spec.template) is changed, for example if the labels or container images of the template are updated." The image is in the template, so editing it rolls out new pods by itself.

The alternative some walkthroughs take, deleting the Deployment and recreating it from an exported file, also deletes its ReplicaSets, and with them the revision history kubectl rollout undo needs. On an app that is already down, it costs nothing. On a live one, it is an outage.

Step 4: Now the second fault becomes visible 

kubectl get svc
kubectl describe svc python-service-devops
Enter fullscreen mode Exit fullscreen mode 
NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
python-service-devops   NodePort   10.43.140.241   <none>        8080:32345/TCP   13m

Port:        <unset>  8080/TCP
TargetPort:  8080/TCP
NodePort:    <unset>  32345/TCP
Endpoints:   10.22.0.10:8080
Enter fullscreen mode Exit fullscreen mode

The node port is 32345, which the task required. The target port is 8080, and the application listens on 5000. The Endpoints line is pointing at a port nothing is bound to.

This fault was there the whole time and could not be seen, because a Service with no running pods and a Service aimed at the wrong port look the same from outside. Fixing the image is what made it observable.

Step 5: Fix the target port 

kubectl edit svc python-service-devops
Enter fullscreen mode Exit fullscreen mode

Changing targetPort from 8080 to 5000 took effect without touching the pod. Whether that is guaranteed is not something the docs say; the Service page ties EndpointSlice updates to the set of pods changing, not to a port change. What I can report is that it worked, which is consistent with a Service being a routing rule rather than part of the pod.

The status kubectl shows you that nobody documented

kubectl get po said ErrImagePull. kubectl describe po said Reason: ImagePullBackOff. Same container, seconds apart.

They are two stages of one failure: the pull is attempted and fails, then the kubelet waits before trying again. ImagePullBackOff is documented, including the fact that "Kubernetes raises the delay between each attempt until it reaches a compiled-in limit, which is 300 seconds (5 minutes)".

ErrImagePull is not. I checked the Images page, the Pod Lifecycle page and the Debug Pods page, and the string appears on none of them. It exists in the kubelet source as ErrImagePull = errors.New("ErrImagePull") and nowhere in the documentation. So one of the two statuses you will see most often during an image failure cannot be looked up.

What the docs recommend

Read the port the container declares rather than guessing the framework default. The Deployment said Port: 5000/TCP, which is the containerPort field, and that field is informational: "Not specifying a port here DOES NOT prevent that port from being exposed." Declaring it opens nothing and omitting it closes nothing. It is documentation that happens to be machine-readable, and on this task it was the fastest route to the answer.

5000 is also Flask's default, and the default is conditional. Flask's reference gives the port as "the port of the webserver. Defaults to 5000 or the port defined in the SERVER_NAME config variable if present", with the host defaulting to 127.0.0.1 on the same condition. Two conditions are one more than most people remember, which is an argument for reading the manifest.

Work through the Waiting checklist in order. The debugging guide: a pod stuck in Waiting "has been scheduled to a worker node, but it can't run on that machine", and "The most common cause of Waiting pods is a failure to pull the image." Its three checks are the image name, whether the image was pushed, and whether you can pull it by hand.

Task 2: Create a managed disk

The task: create xfusion-disk, Standard_LRS, 2 GiB. The title says "Create and Attach Managed Disks", and the numbered requirements mention no VM.

Step 1: Check whether there is anything to attach to 

az vm list -g $RG --query "[].{Name:name}" -o table
Enter fullscreen mode Exit fullscreen mode 
(empty)
Enter fullscreen mode Exit fullscreen mode

No VM exists in the resource group, so attaching was not merely unrequested, it was impossible. The requirements are the contract and the title describes a topic area. Day 11 said "Using Console" and was done with the CLI for the same reason.

Step 2: Create it 

az disk create -g $RG -n xfusion-disk --size-gb 2 --sku Standard_LRS
Enter fullscreen mode Exit fullscreen mode 
"diskSizeGB": 2, "diskSizeBytes": 2147483648, "diskState": "Unattached"
Enter fullscreen mode Exit fullscreen mode

2147483648 is 2 × 1024³, which confirms the Day 8 finding that Azure labels disk sizes GB and provisions GiB.

The explanation I had to withdraw

Querying the size gave two different answers:

az disk create ... --query "{GB:diskSizeGB}"   ->  "GB": null
az disk show   ... --query "{GB:diskSizeGB}"   ->  "GB": 2
Enter fullscreen mode Exit fullscreen mode

On Day 8 I had found that az disk show prints diskSizeGB while az vm show printed diskSizeGb, and I generalised: the response shape belongs to the command, not the resource type, and create and show disagree even within az disk.

It is a neat rule, and it is not true. Both commands are generated from the same schema, and in both generated files the line is identical:

properties.disk_size_gb = AAZIntType(serialized_name="diskSizeGB",)
Enter fullscreen mode Exit fullscreen mode

So the null was not caused by the two commands naming the field differently. The likeliest remaining explanations are the shape of the object my query ran against, since create returns the result of a long-running operation, or a nesting difference between the top level and properties. I did not work out which, and I am not going to publish a second tidy theory to replace the first one.

What survives is the habit, which never depended on the theory being right:

az disk show -g $RG -n xfusion-disk -o json | grep -i disksize
Enter fullscreen mode Exit fullscreen mode

grep -i matches whatever the casing and whatever the nesting. When a value you know exists comes back null, look at the raw object rather than reasoning about why.

2 GiB costs the same as 32

This is the part that actually matters for a 2 GiB disk, and Microsoft states it plainly: "managed disks are billed according to their provisioned size. Azure maps the provisioned size (rounded up) to the nearest offered disk size." The disks FAQ works an example: "A standard managed disk created from an 80-GiB VHD is treated as the next available standard disk size, which is an S10 disk."

The Standard HDD tiers start at S4:

Tier S4 S6 S10 S15 S20 S30
Size in GiB 32 64 128 256 512 1,024

A 2 GiB disk lands on S4 and is billed as 32 GiB, with 30 GiB of unused space you have already paid for. Asking for less than the smallest tier saves nothing.

It is specific to Standard HDD. Premium SSD starts at P1 with 4 GiB and Standard SSD at E1 with 4 GiB, so the floor is much lower there.

What the docs recommend

Find the orphans. The disk exists, is fully provisioned, is fully billed and is attached to nothing:

az disk list --query "[?diskState=='Unattached'].{Name:name,RG:resourceGroup,GB:diskSizeGB,Sku:sku.name}" -o table
Enter fullscreen mode Exit fullscreen mode

Microsoft's warning is about what survives a VM: "After a VM is deleted, you will continue to pay for unattached disks." It is the storage equivalent of the unattached public IP from Day 7.

Pass the region explicitly when it matters. Leaving --location off worked because the task named no region, and the CLI documents the fallback per command: "If location is not specified and no default location specified, location will be automatically set as same as the resource group." That is a statement about az disk create, not a general Resource Manager rule.

Snapshot before deleting. "Deleting an unattached managed disk is permanent. You can't recover its data after deletion." Azure Advisor's version is the one to act on: "Deleting a disk is irreversible. Create a snapshot before deletion and confirm the data is no longer needed."

And know that Standard_LRS means different things in different places. On a disk, it is "Standard HDD locally redundant storage", with LRS replicating "your data three times within a single data centre in the selected region". On a storage account, which is Day 66, the same string describes a performance tier and a redundancy option with no HDD involved.

What I learned

The Kubernetes half was ordinary: two typos in two objects, one hiding the other, found by reading what the system already said about itself. The Azure half was the useful one, because it made me take a rule out of my notes rather than add one.

The rule was plausible, it explained something I had actually seen, and it was wrong. What saved the note was that the practical advice sitting on top of it, look at the raw object when a query returns null, works regardless of the cause.

So here is the Day 64 question. Which rule of thumb are you carrying that explains something you really observed, and has never been checked against the source?

Day 64 down. Thirty-six to go.

Top comments (0)