DEV Community

Cover image for Day 89: A Dead Service and a Private Container That Serves
Nnamdi Felix Ibe
Nnamdi Felix Ibe

Posted on AI-assisted

Day 89: A Dead Service and a Private Container That Serves

Day 89 of DevOps, Day 39 of Azure. Both tasks today produced something that reads as broken and is documented as correct.

A service came back inactive (dead), and that was the right result, one day after an identical command had to say active (running). And an Azure storage container holding a live website is private, returns 404 to anonymous requests, and the website serves anyway.

One Ansible task, one Azure task. Install logrotate on three servers, then host a static site on a storage account. The tasks come from the KodeKloud Engineer platform.

Task 1: Install logrotate

The task: a playbook that installs logrotate on all three app servers. One package, no service requirement.

---
- hosts: app_servers
  become: yes

  tasks:
    - name: Install logrotate package.
      yum:
        name: logrotate
        state: present
Enter fullscreen mode Exit fullscreen mode
stapp01 : ok=2  changed=1   stapp02 : ok=2  changed=1   stapp03 : ok=2  changed=1
Enter fullscreen mode Exit fullscreen mode

Step 1: Then the verification that looks like a failure

[tony@stapp01 ~]$ sudo systemctl status logrotate
○ logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
     Active: inactive (dead)
TriggeredBy: ○ logrotate.timer
       Docs: man:logrotate(8)
Enter fullscreen mode Exit fullscreen mode

inactive (dead). Yesterday the same command on vsftpd had to say active (running) for the task to be done, and here the opposite is correct.

Three details in that output explain why.

TriggeredBy: logrotate.timer is the mechanism. logrotate is not a daemon. It is a batch job that systemd runs on a schedule, so being inactive almost all the time is exactly what it does. The question "will this run" is answered by the timer, not the service.

static in the Loaded line means the unit has no [Install] section, so it cannot be enabled or disabled at all. Which is also why the playbook correctly has no service task. There is nothing to start and nothing to enable, and had the task asked for state: started, the right move would have been to question the task rather than write it.

And the unit exists, which is what the install produced. Before the playbook, the same command said Unit logrotate.service could not be found.

So two consecutive days, one command, opposite right answers:

Day Package Correct state
87 vsftpd enabled, active (running)
88 logrotate static, inactive (dead)

The habit worth having is working out which kind of unit you are looking at before deciding what success looks like. And the check that actually answers the question for a timer is systemctl list-timers logrotate.timer, which is the one command neither the task nor I ran.

What the docs recommend

Installing logrotate gives you the binary, the unit, the timer, and /etc/logrotate.conf with its include /etc/logrotate.d line. It does not give you a rotation policy for anything.

So the task is satisfied, and nothing is being rotated that was not already. A playbook doing useful work would drop a file into /etc/logrotate.d/ saying which logs to rotate and how often, which is one more task using copy or template.

Worth saying because "logrotate is installed" and "logs are being rotated" are different claims. It is Day 76's backup-with-no-tables pattern in a milder form: the thing that was asked for is in place, and the outcome anyone actually wants is a step further on.

One small thing to fix in my own notes. I wrote the playbook path as /home/thor/ansible/playbook.yml and then ran it from ~/playbook. The transcript shows the run working, so ~/playbook is the real location. Anyone following the stated path writes the file where the command will not find it.

Task 2: A static site on a storage account

The task: a storage account with static website hosting, serving an uploaded index.html publicly.

Step 1: The container is private, and the site works

web_endpoint:  http=200    (https://...z13.web.core.windows.net/)
blob_endpoint: http=404    (https://...blob.core.windows.net/%24web/index.html)
Enter fullscreen mode Exit fullscreen mode

Same account, same blob, same anonymous caller. One endpoint serves it, the other denies it exists.

$web was never given a public access level. It was created implicitly by enabling static website hosting, and its publicAccess is unset, which is why the blob endpoint 404s. Enabling hosting is the entire grant, and it has its own anonymous read path that does not go through container ACLs at all.

Which is the morning's finding from the other side. There, the state that makes a task idempotent is two comment lines in the file being edited, not anything in the playbook. Here, the permission that makes a site public is a service property on the account, not an ACL on the container holding the content. Both times, the place you would reach for first is not where the answer is kept.

Step 2: The thing I left open is documented

My notes ended that section honestly and incompletely:

Not tested: whether the site still serves with allowBlobPublicAccess: false. Microsoft documents the static website endpoint as unaffected by that switch... that is currently an inference, not a result.

It is documented, in one sentence on the page about disallowing anonymous access:

"Disallowing anonymous access for a storage account doesn't affect any static websites hosted in that storage account. The $web container is always publicly accessible."

So the inference was right, and the documented claim is broader than the one I was reaching for. Not "unaffected by that switch" but "always publicly accessible", which is a statement about $web itself rather than about two settings interacting.

The useful consequence is the configuration I had guessed might exist. Static website hosting is the way to serve public content from a storage account with anonymous blob access completely disabled. Account-wide anonymous access off, so no container can ever be made public by accident, and the site still served. That is what a real deployment would want, and I had it as a maybe.

Worth being precise that this still was not observed. The account was created with the switch on, because the task says "allow public access" and a grader reads that field. What changed is that the claim is documented rather than inferred, which is a different kind of support than a test.

What the docs recommend

Three things, and the first costs the most time when you get it wrong.

https://xfusionwebst360856247.z13.web.core.windows.net/
                              ^^^^
Enter fullscreen mode Exit fullscreen mode

That z13 is a DNS zone partition assigned to the account. It is not derivable from the account name, the region, or anything else visible, and it varies between accounts. Every other storage endpoint is predictable from the account name. This one is not.

WEB=$(az storage account show -g "$RG" -n "$SA" --query "primaryEndpoints.web" -o tsv)
Enter fullscreen mode Exit fullscreen mode

Guessing the hostname gives a DNS failure, not an HTTP error, so the error points nowhere near the real cause, and you end up debugging a storage account that was correct the whole time. Same trap as the s3-website-us-east-1 hyphen on AWS Day 39.

Second, set the content type explicitly even when the CLI would guess right:

--content-type text/html
Enter fullscreen mode Exit fullscreen mode

The failure mode is nasty. An index.html stored as application/octet-stream returns HTTP 200 and the browser downloads it instead of rendering it. The URL works, the bytes are correct, curl is perfectly happy, and only a real browser shows the problem. A 200 with the wrong content type is harder to find than a 404. Read it back off the response header rather than the blob's metadata, so the check is end-to-end.

Third, $web needs single quotes in bash:

-c '$web'      # correct
-c "$web"      # expands to empty string
Enter fullscreen mode Exit fullscreen mode

It is a legal container name and a variable reference at the same time, and double quotes make bash substitute an undefined variable and pass an empty container name. It also appears as %24web in anything URL-shaped, because $ is reserved in a URI path.

Worth the comparison with S3 while it is fresh, because the two platforms are exactly inverted here. On S3, enabling website hosting turns on index document resolution and grants nothing, so public read needs a separate bucket policy with the access block off, and the objects then become readable through the REST endpoint as well. Azure keeps the blob endpoint shut and opens only the web one, which is a smaller exposed surface for the same outcome. And Azure's web endpoint does HTTPS, whereas an S3 website endpoint cannot do TLS at all, which is why CloudFront sits in front of essentially every real S3 static site.

What I learned

Both halves produced something that reads as broken and is documented as correct.

inactive (dead) is the better of the two findings, because it is cheap and permanent. Some systemd units are daemons, and some are batch jobs with timers, and the same systemctl status command is the right check for both with opposite correct answers. The tell is three words in the output: static and TriggeredBy.

The Azure half is the one I got a response to. I wrote down an open question and said plainly that my reasoning was suggestive and the evidence was absent. Going and looking turned it into a documented behaviour, and a stronger one than I was expecting: $web is always publicly accessible, full stop, which makes a static website the way to serve public content from an account where anonymous blob access is switched off entirely.

So here is the Day 89 question. How many of the alerts you would act on are firing on a state that something's own documentation calls correct?

Day 89 down. Eleven to go.

Top comments (0)