Silent failures
Your daily report stopped arriving two days ago. Nobody noticed, because a message that does not arrive looks exactly like a quiet day.
A reader told me my fix was only a promise
Count the scheduled jobs in your repository that end by sending something. A report, a digest, a backup confirmation, an alert. Now say, without opening anything, which of them actually sent something yesterday. Most teams cannot answer that, and the cost stays at zero until the one day it does not.
I wrote about this gap two days ago and shipped a guard for it: any script whose own header says it runs daily must appear in a workflow file that has a schedule. A reader named Mads Hansen replied that this does not go far enough. A guard that reads workflow files proves only that a declaration exists in source control. He suggested a deployed canary instead: run the thing on the real path, and check it uses the same identity, secrets and result sink as production.
He was right within a day, and it cost me two days of silence.
The change was green on day one and broke on day two
I built a second daily probe and, on his advice, ran it once by hand against production instead of waiting for its first scheduled run. It went red immediately. Then it went red again for a different reason. The second failure was not in the new probe at all. It was in the older one that had been running for weeks.
Here is the mechanism, and it is worth borrowing. The probe writes its history to a separate branch, so the main branch does not collect one commit per data point. The step checks that branch out, appends a line, pushes. That branch contains exactly one file. Checking it out therefore removes every other file from the working tree, including the notifier that the next step calls.
On the day I wrote that step, the branch did not exist yet. So the code took the other path, the one that creates an orphan branch, and that path leaves the working tree alone. The run was green. The next morning the branch existed, the first path ran, and the notifier was gone. The daily message stopped arriving, and the only trace was a red run nobody read.
A change that passes on the day it ships and fails the day after is the worst kind, because the review, the test and your memory of it are all from day one.
Ten minutes, three checks, no tooling required
Steal this. Two greps and one command, and the first two work on any repository with GitHub Actions in it.
# 1. Scheduled workflows that also push to a branch.
grep -l 'schedule:' .github/workflows/*.yml | xargs grep -l 'git push'
# 2. In each hit: does any step AFTER that push run a script from the
# repo? The branch checkout may have removed it. This is a handful
# of files - read them, do not automate it.
# 3. The check no file in your repo can do for you:
gh workflow run <name>.yml && gh run watch
If step two finds anything, the fix is to move the notification before the push. The message matters more than the bookkeeping, and a failed push must never be allowed to silence it.
Then do the one that has nothing to do with branches. Trigger one scheduled job by hand today, against production, and watch it. Not the test suite. The job. Everything you learn in that minute is something no file in your repository could have told you.
Make sure the alarm can still say no
A canary that always passes is just another thing to maintain. So break it on purpose once. Remove the token, point the notifier at a wrong channel, and confirm that the run goes red and that you hear about it.
The new probe aborts when its API key is missing, before it touches the network, and it exits non-zero. It does not print a tidy empty report. That distinction is the whole point: found nothing and never asked must not look the same, and if your job prints the same thing in both cases, it is lying to you politely.
The same shape, three more places
Any step that switches branches or checks something out mid-job. Everything after it is running against a working tree you did not think about.
Any create-if-missing path. The first run takes one branch of the code and every later run takes the other, so the version that matters was never the version you reviewed.
Any notifier called through plain curl. curl exits 0 as soon as the request was made, including a 401 where nothing was delivered. Green step, no message, no trace.
What changed, and what is still a promise
The probe now runs on the real path, and I watched it fail twice before it worked: wrong permissions on the token, then the missing notifier. Both took under a minute to find, because they happened when I asked instead of at 07:00 on some morning in two weeks. The older probe sends its daily message again, and I only know that because I made it send one while watching.
Mads suggested something else that I have not built: a durable ledger, a row written before each run is due, so that ran and found nothing is mechanically different from never ran. My canary proves the path works today. A ledger would prove it ran yesterday. That part is still a promise in source control, which is exactly what he warned me about, and I would rather write that down than pretend otherwise.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Free tier, hosted in the EU: cachly.dev
Top comments (4)
Competitor saying hello, since I run a memory server over MCP too and your last paragraph is my product's whole failure mode.
Your three shapes have a fourth, and it's the one that got me. Every failure you listed is a step that ran and did the wrong thing. Mine was a step that never ran at all.
I moved an email send off the request path into an asyncio background task, which is the obvious thing to do. Python garbage collected the task, because nothing held a strong reference to it. No exception. No red run. The delivery row sat there as pending and the worker had simply never been scheduled. create_task returns successfully the moment I call it, which is the same lie curl tells when it exits 0 on a 401. Started is not finished, and both of them report started.
My fix was three lines, a module level set that holds the task and a done callback that discards it. What I still don't have is a test that catches it, because there's no failure to inject.
Which is why I'd push you toward Mads' ledger over the canary, if you only build one. A canary proves the path works when I ask it to. A ledger is the only thing that separates ran and found nothing from never ran, and never ran is the case with no red run for anyone to read.
The rule I landed on is to alert on the absence of a success instead of the presence of a failure. A job that never started has an error rate of zero.
Competitor hellos are the best kind — welcome, and thanks for a comment that's sharper than most postmortems.
Your fourth shape deserves the top spot on the list. We ended up encoding it as a rule: a verdict has three states, not two — green, red, and NOT MEASURED — and "not measured" must be a first-class, visible state, never a silence that renders as green. Same energy as your error-rate-zero line. Our gate literally refuses with "not started is not green" when a required CI workflow never got scheduled — we learned that one from GitHub spending limits: a workflow that never starts fails zero steps and just looks slow.
Two things that helped us where you say there's no failure to inject:
Steps prove their output, not their exit code. Every check carries an expected proof string; "no tests found" plus exit 0 trips a wire instead of passing.
For the guard itself we demand a Kontrolle: the test first runs a copy without the guard and must actually see the failure — a watchdog you can't prove sighted is worthless. For your GC'd task that means testing the absence-detector, not the task: advance the clock, assert the "no success inside the window" alarm fires. There's your injectable failure — it lives one layer up.
And fully agreed on ledger over canary — ours is append-only JSONL with those three states in it, and the "never ran" rows are the ones that earn their keep.
Good luck with Humaux. The memory space is big enough for everyone who writes down what broke. — Heinrich
You just answered the thing I said I didn't have. Test the absence detector, not the task. I read that three times.
Your spending limit case has a worse cousin on my side. I dispatch my content gate through workflow_dispatch, and twice now I've fired it with a required input missing. The API hands back a 422 and no run gets created at all. There's no red run to read, and there isn't a grey one either. The row simply isn't in the list, and an empty list looks exactly like a quiet day. Your ledger catches that. A run status column never will, because the thing that failed didn't get far enough to have a status.
I hit the softer version too. My chain has a style gate sitting ahead of the analysis jobs, and when that gate failed the jobs behind it came back as skipped. Skipped renders grey. I'd been reading grey as nothing to see here, for longer than I'd like to admit, when what it actually meant was never measured.
On the Kontrolle, here's the trap I think I'm walking into, and you've clearly been down this road already. My window is wall clock. To run the guardless copy I have to advance the clock, and if I advance it inside the test process while the detector reads now() from Postgres, all I've proven is that the guard fires under my clock. Production runs on the other one. The Kontrolle goes green and the watchdog still isn't sighted, which is your own point folded back one layer down.
I haven't built it yet. I'll take the ledger over the canary, and I'll come back and tell you what the guardless run actually showed.
That distinction between grey-as-nothing and grey-as-never-measured is the one I keep having to re-learn, and your 422 case is the cleaner example of it. A run that fails at least leaves a corpse. A dispatch that never becomes a run leaves an absence, and an absence has no colour at all — it just looks like Tuesday.
The skipped-jobs version is the crueller one, because it does render. Grey feels like information. It took me embarrassingly long to internalise that a skipped job and a passing job are equally silent about the thing they were supposed to measure.
Your clock trap is the part I'd worry about most, and I think you've already named the exit: don't move the clock, move the data. If the detector reads now() from Postgres, leave that alone and backdate the row instead. Then the guard runs on production's clock against a state that is genuinely old, and a green Kontrolle means something. The moment the test owns the time source, you've built a second system and proven that one works.
One small thing that helped me on the dispatch side: have the dispatcher write its ledger row before it fires, not after. A 422 then leaves behind a row that says "dispatch attempted, no run id" — the absence gets a record, which is the only way absence ever becomes visible. It also gives you the pairing you need later: attempts without runs is a number you can watch.
Genuinely curious what the guardless run shows. That's the measurement I'd trust over any canary too.