This morning I logged into seven servers one by one and started counting. I was not chasing a fault; what I wanted to know was duller than that. After all these years, how many pieces of automation on these machines did I actually write, and what do they really do?
The answer surprised me. The overwhelming majority of my automation does nothing. It only talks.
Most writing about production ends with "you should automate that too." I am curious about the opposite. When one person carries seven machines alone, what do they deliberately keep doing by hand? Because the real skill is not handing everything to the machine — it is deciding which work has to stay with a human.
The count first: seven machines, fifty-three timers
I ignored the units that ship with the distribution — logrotate, apt-daily, fstrim and friends. I only looked at the .timer files I put under /etc/systemd/system/, plus the cron entries I wrote myself.
| Server | Distribution | My timers | My cron jobs | What it carries |
|---|---|---|---|---|
| vps1 | Ubuntu 25.04 | 12 | 0 | two blog pipelines |
| vps2 | Ubuntu 25.04 | 8 | 0 | mailcow (mail) |
| vps3 | Ubuntu 24.04.4 | 20 | 6 | blog, köprü, gitea, runners |
| vps4 | Ubuntu 24.04.4 | 2 | 0 | log collection |
| vps5 | Ubuntu 26.04 | 4 | 0 | second mail server |
| vps6 | Ubuntu 26.04 | 4 | 1 | web + database |
| vps7 | Ubuntu 26.04.1 | 3 | 0 | hosting panel |
I got the cron column wrong on the first pass: I had written down all 24 files under /etc/cron.d/ as they were. But asking dpkg -S showed most of them had a clear owner — certbot, e2scrub_all, sysstat, awstats, php. Once I separated out the ones no package owns, seven entries remained, and six of those sit on a single machine. Almost all of my own automation is a systemd timer; I have drifted away from cron over the years and only noticed while counting.
Fifty-three timers in total. On all seven machines systemctl --failed returned zero lines, so at the moment I measured, nothing was broken anywhere. A pleasant morning.
Here is the interesting part. Among those 53 units, only two names appear on all seven machines. alert-monitor.timer and security-report.timer. No shared backup scheme, no shared deployment path, no shared observability stack. The only common denominator across my fleet is two scripts that send me email.
I had never seen quite this clearly that the floor of my automation is notification.
The word "watchdog" is lying to me
alert-monitor.sh on vps3 is 372 lines and runs every minute. Exactly one line in it contains systemctl restart, and that one is for köprü's extra-IP unit. The other 371 lines measure, compare, and produce mail when needed.
health-watchdog.sh is the more interesting case. The name says guard dog; the file is 136 lines. Inside it there are seven alert calls — site down, origin down, white-page risk, blog has not produced content, Actions queue stuck, disk full, RAM low. Remediation actions: zero. That is not a watchdog, it is a smoke detector.
runner-watchdog.sh, by contrast, is 137 lines — almost exactly the same size — and it genuinely intervenes: stops the unit, kills the parent process first, collects orphaned builds with kill -9, and even rescues the build cache before killing anything.
Two scripts, similar length, completely different authority. I drew that line on purpose, and the reason fits in one sentence: I know exactly how the runner breaks, and I do not know how the site breaks.
The runner's failure pattern is narrow and repetitive: duplicate listeners, a hung worker, an orphaned build process. For a fault like that, "stop, clean up, start" is almost always the right answer. But the sentence "the site returns 502" can hide a hundred different stories, and in some of them restarting the service destroys the evidence. Restarting is sometimes not a repair but the destruction of the evidence.
Google's SRE book tells this trap through its "Diskerase" story: automation interprets an empty result set as "everything" and wipes the disks of an entire CDN fleet within minutes. The chapter opens with a sharp warning: automation done thoughtlessly creates as many problems as it solves. The line I draw is the practical form of that sentence: I grant remediation authority only when the failure pattern is narrow and familiar.
A reminder that talks to a log file
This line sits in vps3's cron directory:
0 6 * * 0 root echo "$(date -Iseconds) HATIRLATMA: haftalik Contabo snapshot al!" >> /var/log/snapshot-reminder.log
A cron job that runs every Sunday at six in the morning and whose only job is to append a line to a file. I opened the file: 19 lines. The first from 17 May 2026, the last from 20 September 2026. Nineteen weeks, nineteen reminders.
There are two separate things going on here, and it matters not to confuse them.
The first is a deliberate choice. I did not automate taking snapshots from the provider panel, because that operation grows the storage bill and the answer to "which moment is worth freezing" depends on what I was doing that week. I kept the decision and gave the reminder to the machine. That part is right.
The second is plainly a design flaw: the reminder is written to a log file. It reaches nobody. On the same machine there is a 372-line script running every minute that is perfectly capable of sending mail, and yet this reminder talks into a quiet corner. My records also cannot tell me whether those snapshots were ever taken — the reminder only records that it reminded, not that anything happened.
So not every "I did not automate this" is a mature decision. Some of them are just unfinished work. The only way to tell the two apart is to sit down and count.
What I did automate: rebooting
Then I found the opposite where I did not expect it. On all seven servers unattended-upgrades is enabled and Unattended-Upgrade::Automatic-Reboot is true on every one of them. These machines restart themselves, without asking me, when a security update touches the kernel.
That is exactly what happened on the evening of 23 September. Five machines rebooted on their own, and their boot times match my configured values precisely: vps6 at 21:30:07, vps3 at 21:31:20, vps4 at 21:31:47 (all three configured for 21:30), vps5 at 18:30:26 and vps7 at 18:30:09 (both configured for 18:30). vps1 and vps2 stayed put; they have no pending kernel update and have been running on the same boot since 9 July.
Nobody called me, and I did nothing. I have enabled, on seven machines at once, the thing we are taught for years never to do in production — and I am glad I did, because the alternative was the fate of a nineteen-week-old snapshot reminder.
There is a fine distinction here. I automated installing the update and rebooting, but I kept noticing that the kernel has gone stale separate. On vps1 there is another weekly unit, and its own description in systemctl cat ends the argument:
Description=Weekly kernel update availability check (state-change email only)
The note in parentheses is a boundary I wrote for myself: if the state changed, send mail, do nothing else. Automatic reboot only covers the case where the package manager says "this update requires a restart"; a deferred kernel, a held package or a manually installed module all fall outside that gate. What closes the gap is a one-line email once a week.
But that setting could have drifted silently
While digging I found something small. In vps3's apt configuration directory there is a backup file I left behind on 8 September: 52unattended-upgrades-local.bak-20260908, containing the old reboot time 04:00. The live file next to it says 21:30.
The machine's reboot history confirms the change exactly: 20 August at 04:01, 7 September at 04:01 — then 10 September at 21:30 and 23 September at 21:31. I changed the setting on 8 September and the server obediently followed.
So which file won? While answering that, apt-config dump also printed its own warning:
N: Ignoring file '52unattended-upgrades-local.bak-20260908' in directory '/etc/apt/apt.conf.d/' as it has an invalid filename extension
Unattended-Upgrade::Automatic-Reboot-Time "21:30";
APT never read that backup. According to the man page, files in fragment directories are read in ascending alphanumeric order, but only those with no extension or with conf as the extension are considered; on top of that, names ending in .bak, .disabled or ~ are silently ignored.
I was lucky here, not careful. My luck was that the extension counted as invalid — because .bak-20260908 sorts after the live file alphabetically, and had it been read it would have overridden the live setting. Curious about what would have happened had it been read, I tested it in a temporary directory. When the same key is assigned twice, the second assignment wins:
$ printf 'Lab::Deger "ONCE";\nLab::Deger "SONRA";\n' > main.conf
$ APT_CONFIG=main.conf apt-config dump | grep '^Lab::'
Lab::Deger "SONRA";
That experiment lives in a single file, but since the man page says fragment files are read in order and fed to the same parser, the behaviour is identical across files. In other words, had I separated that backup suffix with a hyphen instead of a dot — 52unattended-upgrades-local-bak-20260908 — APT would have treated it as a valid fragment, read it later, and quietly pulled the reboot time back to 04:00. Half my fleet would have restarted at the wrong hour for three months, and I would have had no way to notice: no alarm, no log line.
That N: warning has been printed on every apt invocation since 8 September. I read it for the first time today, for this article.
The bill for the thing I automated most
The most automated thing in my fleet is this blog. The repository holds 18 GitHub Actions workflows and more than 17,000 lines under scripts/. Content generation, translation, cover images, deployment, social posting, health checks — all of it is written down.
Then I asked the git history. In the last six months, 199 commits touched the workflow files, and 110 of them start with fix or hotfix. I have repaired this automation far more than I have extended it.
The commit subjects tell their own story: a date calculation turned out to be fragile, URL encoding broke something, a duplicate key produced an empty failing run on every push, a guard silently turned a whole workflow into a no-op. None of them say "the automation made a wrong decision"; all of them say "the automation is itself software that needs maintenance."
The SRE book's definition of toil helps here: work that is manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as the service grows. The last item is the critical one. By automating the blog pipeline I genuinely zeroed out the toil of producing content — but I replaced it with a new linear cost: 18 workflows that need looking after.
Automation is not an asset, it is a tenant. It pays rent in maintenance. Whether that rent is worth paying is, for me, a clear yes — those 110 repairs bought hundreds of articles — but you should say yes knowing the answer is not "free."
Silence costs more than noise
On 25 June the blog produced no articles at all for roughly a day. The machine was perfectly healthy: 73–87% of the CPU idle, more than eighty gigabytes of free RAM, zero swap. It was not a capacity problem. A build had timed out, repeated failures had left orphaned processes behind, and the runner had locked up looking "busy."
What bothered me was not the outage. It was that no warning came. The automation was running; it simply told nobody.
What I added afterwards was not a repair script but an observation: a check that says "the blog has not produced an article in N hours." Today it has two separate carriers — one of the seven alerts inside health-watchdog.sh, and a separate pipeline-health unit that runs every four hours. I built them at different times, out of the same fear. The duplication I was trying to weed out at the start of this article accumulates exactly like that. I had written before about why excessive alerting systems fall short; this incident was the other side of that coin — a blindness caused not by too many alerts but by none at all.
Black boxes spend their worst nights without telling anyone.
So does the voice of those two scripts reach me?
While writing this I realised my whole arrangement leans on a single assumption: that the automation which notifies can actually notify. With the snapshot reminder, that assumption did not hold. So I had to measure the main channel too.
I looked at this morning's run of the security-report unit on all seven machines. Every one of them: exit=0, result=success; on vps3 there is not a single failed run in the last thirty days. So handing the mail to SMTP works on all seven.
But that measurement proves less than I assumed. exit=0 means "the letter entered the post office"; it says nothing about it landing in my inbox, let alone about me reading it. I carry the mail myself, so I am in a position to look past the handoff — and I never have. I have not wired the reminder into mail either; that is the first item on the to-do list this article handed me.
A decision frame: five questions
When I decide whether to automate something, I now ask these five questions in order. The answers push me into one of three boxes: notify, remediate narrowly, or keep it in my hands.
- Is the failure pattern narrow? "The runner spawned duplicate listeners" is narrow. "The site returns 502" is not. Where the pattern is not narrow, automation's job is to notify, not to intervene.
- Is the cost of a wrong intervention reversible? Restarting a service is reversible. Erasing a disk is not. Every irreversible action stays with a human.
- Does the decision depend on context? Taking a snapshot is technically a one-liner, but the answer to "which moment this week is worth freezing" depends on what that week held. Decisions that need context do not get automated — only reminded.
- Who will maintain this automation? If the answer is "me," how many hours is the rent? A hundred and ten repair commits taught me that question is not rhetorical.
- When will I come back to this decision? I added this step to the list today, because half the evidence in this article grew out of its absence: a broken reminder that went nineteen weeks unnoticed, and a warning line printed on every apt invocation for seventeen days that nobody read. A decision without a date is not a decision.
There is also a lesson that did not make the list but fell out of the counting itself. vps4 is the least automated machine in my fleet — only two timers I wrote, zero cron entries. But it is not unprotected: a backup agent runs on it. On that machine I bought the automation instead of writing it. Reading the table as "fewest timers means most risk" would have been wrong.
Conclusion
Seven servers, 53 timers, and as the common denominator two scripts that send me mail. The sentence that came out of the count is this: in a one-person team, automation's real job is not to do the work but to steer my attention.
Because the bottleneck of a one-person team is not manual labour. I am not maintaining seven machines by hand as it is; that part was delegated long ago. The bottleneck is how many things I can watch at once. When I build automation as a guide that tells me where to look rather than as a worker that does the job, seven machines fit into one person.
And when that guide falls silent, the system does not crash — something far worse happens: nothing happens, and I find out a day later.
The cheapest question you can ask your own setup is this: of the automation you have written, how much of it does something and how much tells you something — and do the ones that tell you actually reach you, or do they fall into a log file?
Top comments (0)