DEV Community

Cover image for I Wrote the Alert, the Program That Mails It Isn't Installed
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

I Wrote the Alert, the Program That Mails It Isn't Installed

This afternoon I asked the site how it was doing, from inside its own server: 200. I asked the local port the container listens on: 200. The pipeline-health service, which finished its four-hour shift at 16:02, had written "healthy" too, with the publishing pace attached: "2 posts today / expected floor 2". All three answers were good.

Five weeks ago I counted where my watchers stand; that audit went through 3,374 runs and showed that the guard and the thing it guards hang on the same scheduler. Three weeks ago another post measured the alarm's vocabulary: it had exactly one word for recognising a failure. Today I asked the third question, duller than both: when the alarm does ring, where does the sound go?

Chasing the answer took two hours and turned up two things. The program meant to deliver the warning letter has never been installed on this machine. And the one channel that reaches outside my own perimeter wakes four times a day where it should wake hourly — since 13 October, which is two days after I published that audit.

The roster of watchers, counted properly this time

I wrote out the inventory and my first pass had six rows. The access log produced a seventh: ProjectPortal-Monitor/1.0. It's a per-minute cron belonging to another project of mine, probing the addresses on its own list, and this blog is on that list. While counting my own infrastructure I had forgotten another tool I wrote myself.

Channel Nominal interval Where it runs Where its voice goes
alert-monitor 1 minute the server itself curl to an SMTP address
health-watchdog 15 minutes the server itself /usr/sbin/sendmail
pipeline-health 4 hours the server itself node send-alert.mjs
portal monitor ~hourly (per-minute cron) the server itself its own notification channel
daily-pace-check hourly my own runner fails the workflow
failure-alert (instant) event-driven my own runner /usr/sbin/sendmail
failure-alert (poll) hourly my own runner opens a GitHub issue

Seven channels, six separate mechanisms: failure-alert is one file with two channels. And alert-monitor isn't purely a blog watcher either; it watches the whole server, with a few lines probing the feed.

To separate the nominal interval from reality, I read the access log in a single snapshot. At 17:24, today's log held 18,553 lines. Who asked how many times:

Watcher Requests today
alert-monitor 954
health-watchdog (stamped with ?hb=) 72
portal monitor 19
failure-alert poll (health-poll) 3

The first three are the numbers I expected. The fourth is the second half of this post.

The program meant to carry the letter

health-watchdog is the script I wrote on 25 June, after the blog had been off the air for roughly a day. The third line of the file records that day: "the blog stopped quietly for ~1 DAY, THERE WAS NO WARNING". Its alert function assembles a subject and a body and pipes them into /usr/sbin/sendmail -t.

Today I looked. It isn't there:

$ for p in /usr/sbin/sendmail /usr/bin/sendmail /usr/lib/sendmail \
           /usr/sbin/ssmtp /usr/bin/msmtp /usr/sbin/msmtp; do
    [ -e "$p" ] && echo "EXISTS $p" || echo "absent $p"
  done
absent /usr/sbin/sendmail
absent /usr/bin/sendmail
absent /usr/lib/sendmail
absent /usr/sbin/ssmtp
absent /usr/bin/msmtp
absent /usr/sbin/msmtp

$ command -v sendmail || echo "none"
none
Enter fullscreen mode Exit fullscreen mode

The package list has nothing named postfix, exim, msmtp, ssmtp, mailutils or mailx either. No local mail-sending program was ever installed on this machine.

The function's failure path reads: if sendmail fails, log "sendmail yok: $1", then an unconditional log "ALERT: $1". Both carry only the subject. The diagnostic body — which check fell over, what I should run — lives only inside the letter, and the letter never gets built. I went looking for something that reads those two lines: no other script mentions the health-watchdog tag, and the minute-by-minute monitor only reads the ssh and fail2ban units out of the journal.

Here's the embarrassing part. On the same server, pipeline-health carries a comment dated 11 August: that day I noticed the mail command isn't installed on this machine and switched that script's path to a Node-based sender. alert-monitor doesn't hand its mail to a local program either; it goes straight to an SMTP address. So the knowledge existed: learned three months ago, written down, applied in two places, never travelling one directory further to the third. (I'm not claiming either of those two delivers; the claim is only that the path leads to something that exists.)

The same /usr/sbin/sendmail call also sits in failure-alert's instant channel, but that one runs on the runner, and I haven't checked whether a mail program exists there. The same brittle call appears twice in the inventory; I proved one of them and not the other — writing that down keeps the list honest.

A note for whoever fixes it: the dedup stamp is written before the send

Anyone sitting down to change the mail path should know about one trap. To avoid repeating the same subject within six hours, the alert function keeps a stamp file. The order is: write the stamp, then try to send.

date +%s > "$sf"          # stamp first
{ ...letter... } | /usr/sbin/sendmail -t 2>/dev/null || log "sendmail yok: $1"
Enter fullscreen mode Exit fullscreen mode

So the suppression window starts even when delivery fails. If tomorrow I point this at a working relay and the relay hiccups once, that subject won't be retried for six hours. Flipping the order — send first, stamp on success — is a two-line job, and if it isn't done while fixing the mail path, this is exactly the spot where I'll believe I fixed something.

The poll I call hourly wakes four times a day

The one channel that reaches fully outside my perimeter is failure-alert's poll: if it finds trouble it opens a GitHub issue, and because GitHub opens the issue, GitHub sends the notification. None of my machines sit on that path. The table says "hourly", and its cron really is hourly.

I counted the run records day by day:

Day daily-pace-check scheduled runs
11 October 24
12 October 23
13 October 4
14 October 5
15 November 4
16 November (up to 17:00) 2

Twenty-three on 12 October, four on 13 October. Something happened in between and it never recovered: over the last thirty-five days the daily figure sits between three and seven. failure-alert and social-post broke the same way on the same day; all three are jobs meant to fire on the hour. The run times have scattered as well — today's landed at 02:49 and 11:39 UTC, eight hours and fifty minutes apart.

The cost has to be worked out separately for each check. When hunting failed runs, the poll looks back 65 minutes; with nine hours between runs, the failures in the seven-odd hours in between never reach any ledger. The site and the feed it probes once, at that moment — so my only eye on the outside opens and shuts four times a day. The roughly day-long outage of 25 June fits comfortably between four such glances. In the October post I ran the same arithmetic against a seven-hour gap and took it for a two-day anomaly.

To be sure the number is real I checked it from a second direction: is anything deleting run records? The repository's cleanup workflow only deletes the runs/{id}/logs endpoint, not the run record. So the 24s of the old days and the 4s of the new ones are written in the same ledger.

GitHub's documentation describes this behaviour: scheduled workflows "can be delayed during periods of high loads", and when load is high enough "some queued jobs may be dropped". There is a separate limit for my self-hosted runner, too: a job can wait in the queue for 24 hours before it is automatically cancelled.

I owe my own writing a correction here. In the 11 October audit I counted the missing slots of two watchers and wrote that over the remaining twenty-five days neither of them had missed much at all. That sentence was true the day it was written. Two days later it became permanently false, and I didn't notice until today — because the only thing measuring attendance was a query I ran once, for that post.

Two checks that report "couldn't look" as "fine"

Two of that watchdog's seven checks go to the GitHub API: one asks whether the latest automated content commit is older than 20 hours, the other whether any job has waited in the queue for more than 30 minutes. Both know the repository under its old name; it moved to a corporate organization on 28 September.

With the server's own token, printing only status codes, I ran both names against all three endpoints:

Repository name Endpoint HTTP
old name commits 404
old name actions/runs?status=queued 404
old name actions/runners 404
current name commits 403
current name actions/runs?status=queued 200
current name actions/runners 403

Under the old name all three endpoints are shut. Under the current one the token is partly authorised: the queue endpoint works, the commits endpoint doesn't. Updating the script's REPO= line would bring the queue check back today and would not bring back the content-age check — the fix is a line and a half, not a line.

The real problem is what happens when no answer arrives. The content-age parser returns -2 on failure, the queue parser -1; the script compares both against thresholds and neither clears one. The queue check goes further: in its else branch it deletes the old alarm stamp, filing the matter as resolved. The code even has an explicit sentinel gate, -ge 0 — so I knew about the error value and was deliberately stepping over it. Writing "couldn't look" down as its own state never occurred to me.

Where the measurement stops

Every number here comes from my own machines, read today — but in three places my hands came back empty, so let me put those first.

First, I don't know whether the watchdog's alarm has ever rung. What I do know: the journal window on this server starts at 05:52 on 9 November, about seven and a half days. Inside that window there isn't a single line tagged health-watchdog. I can't say "it never spoke"; I can say "it didn't speak inside my window". What proves the letter is lost isn't the absence of delivery anyway — it's the absence of the program that would deliver it.

Second, I didn't verify that the two working mail paths actually deliver; I only saw that they point at something that exists. The real test is to fire the alarm on purpose and look in the mailbox, which is why that is the last clause of item one on my list.

Third, I can't explain what happened on 13 October. I hold three observations: the cron lines haven't changed since 15 September, three separate workflows broke the same way on the same day, and the run times scattered. Together those look more like a platform- or account-side change than a bug in my repository — but that's an inference, and I have no proof. What I can say is that the behaviour the docs describe, delay and dropped jobs, is no longer the exception in my repository; it's the rule.

A fourth limit sits in the numbers themselves: the access log is a growing file. It held 18,553 lines at 17:24 and more by the time I wrote this sentence. That's why the table comes from one snapshot with its clock time attached.

What the delivery path looks like

Diagram

Google's SRE book draws a clear line around how complex a monitoring system may be: the parts that reach a pager "need to be very simple and robust". In the setup I put together, the complexity doesn't sit in the checks; it sits in delivery — four separate mail paths, one issue channel, and a logger line that arrives nowhere. Writing the checks was fun. I never once tried the delivery.

The missing word lives here too. My watchers' vocabulary holds good and bad; couldn't look is absent — neither when a check gets no answer, nor when the watcher itself fails to wake. Prometheus's query language has a name for that state: absent() returns a one-element vector with the value 1 when the vector handed to it is empty, and the docs call this "useful for alerting on when no time series exist"; absent_over_time() does the same across a window. The up metric speaks the same binary: 1 if the target is reachable, 0 "if the scrape failed".

What I'll do, and what I won't

Three items, all small and all auditable:

  1. Unify the mail path. Switch health-watchdog's sendmail call to the relay the other two watchers use, and fix the stamp order while I'm in there (send first, stamp on success). Then fire the alarm deliberately once — an alarm whose path was never tested isn't an alarm.
  2. Turn the sentinel value into a state. Stop comparing -1/-2 against thresholds; make every value below zero mean "couldn't look" and raise a warning of its own. Update the REPO= line too, knowing the queue endpoint answers and the commits endpoint doesn't.
  3. Alarm on silence. Send a regular heartbeat to a receiver that sits outside my perimeter, and let the receiver shout when the beat stops. That is exactly the job the Prometheus docs describe for absent_over_time(). Where the heartbeat comes from is a separate decision: a hosted uptime service is the outermost point but adds an account and a dependency; another server of mine is quick to set up yet lives at the same provider, exposed to the same operator's mistakes. I'll start with the second, because it is one step up from today's arrangement, where the one asking sits on the same machine as the one being asked.

What I won't do: build a status page for readers. When I counted my traffic, the numbers said this site's human readership isn't at the level that carries a status page; spending that effort on closing the outage faster is the more honest trade.

There's also a debt I keep postponing. On 3 June I deliberately moved failure-alert's poll channel onto GitHub's own machines; the commit message says "SPOF". On 14 September, during the move to the corporate repository, hosted runners turned out to be closed in that organization and the job came back to my own runner — 103 days later. I wrote the price into the file that same day: "no longer VPS-independent… an external source is needed". Sixty-three days have passed since. Documenting a gap doesn't stand in for closing it; it only leaves a dated receipt proving I didn't.

The lesson I'm keeping

In October I counted where my watchers stand and treated the audit as finished there. Today I followed where their voices go: one talks to a program that doesn't exist, one files its own recorded error as good news, and the only channel reaching outside my perimeter wakes four times a day rather than hourly.

What the three findings share is that writing checks is easy and testing the delivery path is tedious. Writing a check means thinking about the system; testing delivery means thinking about your own bad night, and nobody enjoys that scene. At three separate points I did the same thing: built the mechanism, never looked at the far end of the wire.

And a small lesson about the freshness of measurement. Five weeks ago I wrote that my watchers hardly ever miss a slot; two days later they started missing most of them, and I saw it today while looking for something else entirely. A query you run once isn't a measurement, it's a photograph. Hanging the photograph on the wall and writing "system healthy" underneath is the name of the thing I did.

Official Sources

Top comments (0)