Mustafa ERBAY

Posted on Oct 11 Originally published at mustafaerbay.com.tr

I Audited My Watchdogs: 1,973 Runs, 15 Lines in the Logbook

#yazilimkariyeri #otomasyon #olcum #process

Over six months I added one watchdog after another to the automation behind this
blog. Each came from a reasonable fear: what if a workflow breaks overnight and I
never hear about it, what if publishing stalls and I don't notice, what if the
pace slips. Adding them always felt free, because the bill never shows up
anywhere visible.

Today I got curious. How much do these watchdogs run, and what do they give me
back? I pulled the repository's GitHub Actions records one by one. Between 14
September 19:05 UTC and 11 October 10:45 UTC — 26 days and 16 hours — there were
3,374 runs. Of those, 1,973 — roughly three out of every five — belong to two
workflows that produce nothing at all and only ask "is everything fine?" In the
same window, the lasting output of both watchdogs together is 15 lines on a
single GitHub issue.

Eight days ago I wrote about fourteen straight days of articles coming out of one
category after the
topic queue ran dry, and how none of my indicators turned red. That piece was
hunting for a missing measurement: nobody was watching the composition of the
output. The question here is different, and more uncomfortable for me: what do the
watchdogs I already have deliver, what do they cost, and are they in any shape to
tell me when it matters? The first was a coverage question; this one is about
delivery.

The run census

I counted workflow by workflow first, then downloaded every run record and counted
again from scratch. The two methods differed by a single run, and that one run
taught me something I'll come back to near the end.

Workflow	Runs	Job
Workflow Failure Alert	1,360	watchdog
Social Media Post	719	posting
Daily publish pace check	613	watchdog
Deploy Only	139	publishing
Veo Prompt	130	production
CodeQL (two records)	237	scanning
Other 10 records	176	production/maintenance

I split them into three groups. The ones carrying publishing and content: 1,119
runs (33.2%). Inspection work — code scanning, repo maintenance, runner
self-tests: 282 runs (8.4%). The two health watchdogs: 1,973 runs (58.5%). Most of
this blog's CI time belongs to the thing watching, not the thing working.

In the same window, 19 runs failed — 5.6 per thousand. I keep the watchdogs
standing for those 19 events, and on its own that isn't a bad trade; insurance
works like this. The question is whether the insurance pays out.

The logbook: one issue, open for twenty-six days

When the hourly watchdog finds a problem, it opens a GitHub issue. If an open
issue with the same title already exists, it adds a comment instead of opening a
new one — a nicety I wrote myself to avoid duplicate notifications. Here's how
that played out: the issue opened on 14 September at 20:42 UTC and never closed.
Twenty-six days later it is still open. It carries 14 comments, every one of them
signed by a bot. Not a single comment from me.

Look at what the first line says. The logbook's first alarm is the watchdog
itself:

🚨 Saglik kontrolu (2026-09-14 20:42 UTC) sorun tespit etti:

**Fail eden workflow(lar)** (son 65dk):
  - Runner Selftest (container): .../runs/34890375653
  - Workflow Failure Alert: .../runs/34888488140

The second name is the watchdog's own. The event that opened the logbook was the
watchdog falling over, and next to it sits a runner self-test started by hand. A
guard whose first report is about itself would annoy me if it weren't slightly
funny.

The 15 lines break down like this: 7 are "this workflow broke" notices, 7 are
one single publishing stall, and 1 is a transient feed.json read failure.
All seven stall lines are from 16 September and all say the same thing, with the
hourly watchdog bumping a counter each time it woke: no new article for 37 hours,
then 38, 39, 40, 41, 42, 43. One incident, seven lines. I had written about the
cost of repetition in alert design
before, looking at other people's systems; in my own logbook it sits there
sevenfold.

I went through the failure notices one by one. The Deploy Only failures on 15, 16
and 26 September all came in through workflow_dispatch — runs I triggered by
hand, which means someone was in front of the screen while they broke. The CodeQL
failure on 1 October was on a push, sitting red in the checks list.

That leaves one. The seven lines on 16 September told me something I couldn't have
seen elsewhere: publishing had stalled. I dug the rest of that story out of the
records too — articles kept being written (three commits each on 15 and 16
September), but between 15 September 14:21 and 16 September 20:00 there is no
Deploy Only run at all. For some thirty hours production flowed and publishing
didn't. The logbook's seventh line landed at 19:40; the first manual deploy
attempt came at 20:00. I can't prove what the twenty minutes in between mean, but
that is the order they happened in.

The failure I didn't expect was never reported

The watchdog has two channels. The hourly poll looks at everything; there's also
an event-driven channel that fires the moment a workflow finishes and sends mail.
That channel doesn't listen to every workflow — it listens to four names written
out by hand in the file. I opened the list today. One of the four is Generate Content, with zero runs in the window, because I retired it in September. A
quarter of the instant channel's watch list is a ghost.

Worse is what's not on the list: Deploy Only. It produces more failures than
anything else, 7 of the 19 — and the instant mail channel never listens to it.
Google's SRE book wrote my sentence for me: "You can only alert on the failures
that you expected." I didn't expect Deploy Only to break, so I never put it on the
list.

The channel's cost is interesting too. GitHub's docs are explicit about
workflow_run: "A workflow run is triggered regardless of the conclusion of the
previous workflow." It fires even when the workflow it listens to succeeds; the
"was it a failure?" filter is applied by my own step. There were 747 triggers, 744
skipped immediately, and 3 where the mail step actually ran.

Seven hundred forty-seven runs, three emails. At this point in the draft I was
going to write the channel off as waste. Then I looked at which three they were,
and changed my mind.

One of the three emails caught what the poll structurally cannot

All three went out on 26 September: two Lighthouse failures at 13:47 and 13:57,
and a Social Media Post failure at 15:43. The last one matters, because that same
failure never entered the hourly watchdog's logbook. I reconstructed why, step by
step:

The failing run's created_at is 15:32:22; the run itself finished around 15:43.
When the 15:42:35 poll ran, the run was not yet counted as failed, so it saw nothing.
The 16:42:10 poll looked back 65 minutes, that is to 15:37:10. The failure's created_at sat 4 minutes and 48 seconds outside that window.
Issue #2 has no comment between 26 September 14:42 and 1 October 14:48. It never went in.

The fault here isn't "the watchdog was asleep." The poll filters on when a run
started, not when it failed; a long-running job that doesn't fail until its
start time has scrolled out of the window becomes invisible. However wide I make
that window, this class of failure keeps escaping as long as I measure on the
wrong axis.

And the only thing that caught it was the instant channel I was about to write off
as waste. That Social Media Post happens to be on its list is luck; that Deploy
Only isn't is the same luck running the other way. The two channels don't
complement each other — they're just blind in different places.

The watchdog's own absence

I read the number backwards as well: how many times should an hourly job run in
this window? 639 hours, 613 runs. Twenty-seven slots missing. The pace guard: 635
hours, 608 runs, 28 missing.

The gaps aren't spread across the month. Every gap longer than two hours falls
on 3–4 October, and together they swallow 27.3 slots — the entire shortfall. In
the other twenty-five days neither watchdog missed anything worth counting. The
watchdog's largest gap is 7.0 hours (3 October 19:45 → 4 October 02:44); the pace
guard's is 8.4 hours. Their gaps land in the same two-hour windows, with the
minute marks drifting a few minutes apart. I can't prove the cause — on those two
days every scheduled workflow in the repository went hours without running.
GitHub's docs describe the behaviour: "The schedule event can be delayed during
periods of high loads of GitHub Actions workflow runs," and "If the load is
sufficiently high enough, some queued jobs may be dropped." The docs explain the
mechanism; I have no incident record for that day.

What matters is that both eyes closed at the same time. Because I hung the guard
on the same scheduler as the thing it guards, a shared outage silences both.

Now put two numbers side by side. The poll's lookback window is 65 minutes.
Its own absence runs up to 7 hours. After seven hours of silence the poll
wakes, looks at the last 65 minutes, and never asks about the remaining five-plus.

That this isn't theoretical is visible on the repository's first day. On the
evening of 14 September three runs broke back to back: the old Cloudflare Pages
deploy at 19:05, Deploy Only at 19:16, social-post at 19:18. In between, the 19:42
watchdog run failed too (that's the second name in the logbook's first line). The
next poll woke at 20:42 and looked back 65 minutes — to 19:37. All three failures
sat outside that window, and none was ever written down.

To total it up: of 19 failures, 15 made the logbook and 4 did not. Three of those
four are from the evening of 14 September, one from 26 September. Three reached no
channel at all; the fourth was caught by the mail channel.

The gate checked 608 times and never once rang

I described the pace guard in that piece eight days
ago: it counts today's
feat: yeni makale commits on main and fails if the count exceeds four. The
new information to add there is its cost: 608 scheduled runs, not one failure. It
has two failures, both from runs started by hand.

Curious how close it came to the threshold, I counted with the gate's own
definition — the GitHub API's since filter, which goes by commit date. In this
window, three days produced exactly four articles: 17 and 23 September, and 2
October. And here I owe my own writing a correction: eight days ago I gave "10 and
17 September" for this same metric, and running the gate's own query puts 10
September at three. Author date and commit date are not the same thing, and
rebases pull them apart; my first count read the author date.

None of this means "delete this guard." The threshold's whole job is to never
ring; the day it rings I learn that something broke structurally. But knowing its
price is separate: 608 runs, twenty-six days. The only reason I thought it was
free is that the bill never arrived as its own line item.

The evidence evaporates in three days

Mid-audit I also ran into my own housekeeping. I wanted to read the run logs
behind the lines in the logbook; anything older than three days has none. The
cause is my own cleanup job in the repo, whose stated target is to stay under the
free tier's 0.5 GB of Actions storage. The evidence for my alarms was going out
with it. I could still answer the 16 September question above from run history —
but I did that despite the missing logs, not with them.

The lifetime of the records changed this month, too. GitHub's retention doc gives
the default as "retained for 90 days before they are automatically deleted," and
adds: "Starting October 1, 2026, these policies will apply to checks, workflow
runs, and commit statuses in addition to artifacts and logs." The previous state
was "retained for 400+ days." If I'm on the default — and I haven't changed it at
the org level — I won't be able to repeat today's count for this window a year
from now.

One measurement trap deserves a note, because I nearly started with the wrong
number. When I asked the repository-wide endpoint "how many runs?" with a date
filter, the API gave the same answer for 1 August, 1 September, 14 September
and 20 September: 2,500. Four different windows, one number — a ceiling. Here's
the interesting part: the docs describe that ceiling as 1,000 — "This endpoint
will return up to 1,000 results for each search when using the following
parameters," and created is on the list. The mechanism is documented, the figure
doesn't match; the ceiling that comes back is 2,500. I trusted the measurement over the
document and changed method.

Changing method also explained that one-run difference from the beginning. Adding
up workflow by workflow gave 3,373; downloading every record gave 3,374. The
missing run belonged to an old workflow deleted on 14 September (deploy.yml, the
Cloudflare Pages deploy) — and that run is the repository's oldest failure. A
deleted workflow disappears from the listings; its runs stay. What I took for a
race condition was a structural hole.

Three questions to ask a watchdog

After this count, what I ask before adding monitoring has changed. There used to
be one question: "would I want to know about this?" The answer was always yes, so
I always added another guard. Today's three are:

Is its list alive? Do the names it listens to still exist, and do they still run? A quarter of my list was the name of a retired workflow, and counting runs by hand was what it took to notice.
What does it fall with? If one scheduler wakes both the guard and the thing it guards, a shared outage silences both. Every gap in both of my guards lands on the same two days; that's the proof they aren't two independent eyes.
Who closes its logbook? If the place it writes findings has no owner, the log just accumulates. Mine has been open twenty-six days with 14 comments and I've touched none of them. The SRE book's line fits exactly here: "Every page should be actionable." An issue nobody closes is an archive by definition.

Four changes for my own pipeline came out of these. First, add Deploy Only to the
instant channel's list and drop the dead name. Second — and I only realised while
writing that my first instinct was wrong — don't widen the poll's window, keep
state: store the last processed run id and report only new failures. Widening to
eight hours would have reported the same failure eight times, turning the "one
incident, seven lines" behaviour I complained about above into a rule. Third, make
closing the alert issue a task for the next watch. Fourth, drive log cleanup from
the logbook rather than the calendar: exclude the run ids that produced alarms.

There's a fifth I haven't solved: as long as the same scheduler wakes both guards,
I need an eye from outside. The answer's name is already written in
failure-alert.yml's own comment — an external source is required for the
server-down signal. You can't hear your own repository's silence from inside it.

If you want to count your own

This audit isn't expensive: half an hour of API queries and a table. Three things
will likely trip you; I learned all three by tripping.

Don't ask the repository-wide endpoint with a date filter — the filtered query hits
a ceiling and hands you a fixed number. Second, count with two different methods
and compare; even when the difference isn't an error it teaches you something, and
my one-run gap surfaced a deleted workflow. Third, you can't answer "did it catch
anything?" by looking at run conclusions: my poll returns success whether or not
it found a problem, because it exits zero after writing the issue. The catch count
can only be read from the logbook itself. Run counts show cost, the logbook shows
yield, and the two don't live in the same place.

Then write these two numbers next to each other: your watchdog's lookback window
and your watchdog's longest outage. Mine is 65 minutes against 7.0 hours. If the
first is smaller than the second, you have a blind window that appears on no
dashboard. After that, check which field your poll filters on; mine looked at when
a run started, and that's why it missed 26 September.

The real split isn't watcher versus worker

While building the groups, something caught my eye. Social Media Post, which I put
in the "posting" group, is second on the list with 719 runs. Its job is real: it
sends a published article to the platforms and marks what it sent with a commit on
main. In the same window there are 91 of those marker commits, against 80 new
articles. So at least 628 of those 719 runs woke up, looked at the queue, found
nothing to do, and went back to sleep.

My binary split was drawn in the wrong place. The real division in this pipeline
runs between something that does a thing and something that checks whether a thing
happened. Add up the second kind and you get 2,601 runs rather than 1,973: 77% of
the total. Polling is an architectural choice, and I've repeated it reflexively for
every new need — "I'll put an hourly cron on it, it'll look."

In its defence: these runs don't burn GitHub's paid minutes, they run on my own
runners. The cost isn't money. Slots stay busy, and, the harder part to notice, I
see constant movement on the dashboard. A list flowing green feels like a working
system; today was the first time I counted how much of that movement is actual
work.

Conclusion

This audit taught me why adding watchdogs feels so easy. The cost accumulates in
run counts, the benefit in what goes uncaught. The run count is quiet, even
flattering: a green column on a dashboard. The uncaught failure is written
nowhere, because the thing keeping the log never saw it.

Out of 3,374 runs, the number that stays with me isn't 1,973. Four of the 19
failures never entered any logbook, and three of them landed on the repository's
first evening: the guard fell once that night, its next waking looked twenty-one
minutes short, and three failures stayed outside the record. The fourth was caught
by the very channel I was about to call waste. Twenty-six days later, I'd know
neither fact if I hadn't counted the records one by one. Setting up monitoring is
reassuring; auditing what you set up is not — but it's the only way I know.

DEV Community