I wrote a monitoring stack (Prometheus, Grafana, Alertmanager) and a README that says "it just works". It had always worked on my own server. But I had set that one up by hand, over time, from memory. I had never once followed my own README from the first line to the last.
So I took a clean VM and did exactly that, the way a newcomer would. First on Ubuntu 22.04, then on 24.04 and 26.04. It took about a day.
I found 24 problems. Three of them stuck with me, because I'm the one who got them wrong.
The alert that fired on a healthy machine
On Ubuntu 22.04 a fresh VM with nothing running went to pending on a critical alert: FilesystemReadonly. My first thought was that I'd broken something.
It was my rule. systemd 249 mounts a read-only ramfs for its own needs, and my alert counted it as a failing disk. Ubuntu 24.04 and 26.04 don't do this, so on newer systems I'd never have noticed.
The fix is one word: add ramfs to the exclusion list. The more interesting part is that my own checks had missed it. I was verifying that targets are up and dashboards load. I wasn't verifying that a healthy system is quiet. Now /api/v1/alerts must be empty on every fresh stand.
The restore that restores nothing
My backup doc explained how to make a backup. Restore was one sentence without commands. So I tried it.
The Grafana archive has a top-level grafana/ directory, the Prometheus one doesn't. Unpack Grafana to the wrong path and nothing fails. You keep the old data and believe you restored.
I wrote the steps down and ran them. To be sure it really restored something, I created a folder with a marker in Grafana before the backup. After the restore it was there, and all 7 targets were up.
A backup you haven't restored is a hope, not a backup.
"Ansible 2.9+"
The README said Ansible 2.9 or newer. I had never run it on 2.9, I just wrote the number.
I tried. On 2.9 the syntax check passes, then the run fails on the target with ModuleNotFoundError, a known clash with Python 3.12. From 2.12 to the latest 2.21 everything worked, and the second run gave changed=0. The README now says 2.12+.
Everything else
The other 21 are smaller: a compose block in the wrong place, a stale link, flags in an example config that no code reads, Alertmanager needing restart instead of up -d. All fixed in 15 pull requests, #26 to #40.
What I did not check: a real x86_64 machine (I used emulation), Debian and RHEL, load.
A check I ran by hand once is a snapshot. The next version bump will break something again, so the next step is a CI smoke test: compose up, targets up, Ansible twice with changed=0. Docs should run like code.
How do you check that your install docs still work: CI, a manual run before a release, or you find out from an issue?
Repo: github.com/Airat71/monitoring-stack
Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.
Top comments (3)
"The newest version hid it" is the line I keep relearning.
I ran the same exercise on my own tool and found exactly two problems — because I ran it once, on one machine, in one state. You found 24 because you varied something. That gap isn't diligence, it's sample size.
Same shape showed up in a measurement I published: 10,000 real text samples through the actual tokenizer. Prose came in at 4.58 characters per token and I was ready to write "about 4.5" in the README. Structured data came in at 3.78. One content type I rarely feed it moved the number 18% — meaning the sentence I wanted to write was true for exactly the input I happened to test with.
Worth one more pass to your method: after "follow the README from line one on a clean VM", do it once more on a machine that already has stuff on it. That third run finds a class neither of the first two can.
Thanks, fair point. I varied the conditions and you ran it once, so the gap is sample size, not diligence. A run on a machine that already has leftovers (busy ports, old volumes, an earlier install) is something I didn't cover. Worth adding to the method.
Agreed - and that dirty-machine gap maps to exactly what I keep hitting with token estimates: run against clean benchmark prompts and you undercount errors, the same way a fresh machine undercounts install failures. The leftovers are the real test subjects; the docs just have to survive them. If the method gets a part two, that's the run worth publishing.