DEV Community: Jennifer Smith

A Dead-Man's Switch for Scheduled Jobs (Because Silent Failures Rot Data)

Jennifer Smith — Thu, 30 Jul 2026 19:40:21 +0000

A scheduled job has two failure modes. The loud one throws an error and your alerting catches it. The quiet one just stops running, and nothing catches it, because your alerting is wired to the job itself. No job, no error, no alert. The data it maintains gets staler by the day and every downstream system keeps serving it like nothing's wrong.

I run a handful of production systems by myself. One of them is a niche content site with a curated data feed behind it, plus a release-tracking pipeline on Cloudflare Workers and D1, plus some geolocation alerting. There's no ops team. The quiet failure mode is the one that actually bit me.

How I found out

The curation job that refreshes the feed died without a sound. No exception. Nothing in the logs that would make you stop scrolling. The site kept serving the last good copy of the feed, so every health check I had stayed green — the checks were all downstream of the dead job, and downstream was fine.

I found out days later, reading my own site as a user, thinking huh, this hasn't changed in a while.

My first instinct was to go add error handling, which was the wrong instinct, because the job didn't error. What I'd actually built was alerting for when things fail. I had nothing for when things stop happening. Took me an embarrassingly long time to see that those are different problems.

The pattern

A dead-man's switch flips the alerting relationship around. The job doesn't report failure; it has to keep proving success. Silence is what fires the alarm.

Two parts, both small.

The job stamps every successful run. Last thing before exit, write a timestamp somewhere cheap — a KV key, a D1 row, or in my case a field right in the feed file:

{
  "items": [ ... ],
  "generated_at": "2026-07-30T14:00:11Z"
}

Only on success. After the output is actually verified. Not in a finally block — a stamp that gets written on failure is worse than no stamp, and yes, I initially had it in the wrong place.

A separate watcher checks the stamp's age. Separate is the load-bearing word. Different runtime, different schedule, different failure domain. Mine is a tiny scheduled Worker on its own cron that reads the stamp and does one comparison:

const ageHours = (Date.now() - Date.parse(stamp)) / 3600_000;
if (ageHours > MAX_AGE_HOURS) {
  await notify(`Feed is stale: last success ${ageHours.toFixed(1)}h ago`);
}

Job dies, watcher fires. Job hangs, watcher fires. I fat-finger the cron expression in a deploy, watcher fires. It doesn't know why the stamp is old and doesn't need to.

The threshold is the hard part

Not the code. The number.

My instinct was to set the threshold just above the job's schedule — job runs daily, alert at 26 hours, right? That version paged me constantly, because my feed is curated: the job runs daily but the content only genuinely changes when there's something worth adding, which some weeks is barely at all. So I'd get a stale alert, check it, find nothing wrong, dismiss it. After the third or fourth round of that I'd trained myself to ignore the watcher, at which point it protected nothing.

The threshold I actually run on that feed now is 720 hours. Thirty days. That number looks absurd next to a daily cron until you accept that the thing being monitored is the content's rhythm, not the job's. The rule I eventually landed on: set it long enough that an alert is always worth investigating, short enough that the damage window is one you can live with. For a different system of mine — event scraping, where a dead day matters — the threshold is tight. Same pattern, wildly different number.

Make the alert say what broke

The notification goes to a Discord channel I actually read. It carries three things: which system, how stale, where to look first.

⚠️ feed stale: last success 39.2h ago (threshold 24h).
Check: worker cron logs → feed generator → upstream source.

At 7 a.m., "something is wrong somewhere" and "this specific thing is wrong, start here" are very different messages to receive. When you're the whole on-call rotation, the second one is a two-minute fix. The first one is a knot in your stomach until the weekend.

Cost

Basically zero. The watcher is a scheduled Worker on the free tier and the stamp is a few bytes. An hour to build the first one, minutes for each system after.

Since the stamps and watchers went in, no silent failure has made it past one threshold window. The loud failures were never the problem. If your alerting only fires when jobs fail, it's worth checking what happens when one simply stops.

My Auto-Publish Pipeline Shipped a Two-Year-Old News Story. Here's the Fix — All Three Layers of It.

Jennifer Smith — Thu, 30 Jul 2026 19:39:23 +0000

I run a content pipeline that finds industry news, has an LLM judge score it, and publishes what clears the bar. Fully automated, Cloudflare Workers and D1, one operator: me. It runs daily and the whole point is that I don't babysit it.

In July it published a news story from 2024 as if it had just happened.

Nobody was harmed. One item, niche site, caught on a routine quality check within a few days, unpublished. But "the robot posted two-year-old news and nothing stopped it" is the kind of failure that quietly costs a site its credibility, and it deserved a real postmortem instead of a shrug. What went wrong turned out to be more interesting than I expected, and the fix ended up being three layers instead of the one I planned to write.

What actually went wrong

The pipeline already had two mechanisms that should have caught this.

There was a recency check. But when I built it, the stale-content problem I was seeing was old product releases resurfacing on aggregator sites. So the year-sanity check ran against product items. News items took a different path through the gate and never met it. The check had no bug. It was standing at the wrong door.

There was an LLM judge. It scores every item before publication: relevant? interesting? worth the reader's time? It read the 2024 story and scored it well — because the story was relevant and interesting. I had never asked the judge whether the story was current. That question wasn't in the rubric, so it wasn't asked. A human editor has a reflexive "hang on, when is this from?" An LLM judge has exactly the reflexes you wrote down and none you didn't.

Upstream, an aggregator had resurfaced the old story with a fresh feed timestamp. Everything downstream took the feed's word for it. Two guards, both working as designed, both blind to this one.

The fix

My first draft of the fix was one line: extend the year check to news items. Done, ship it. I've shipped enough one-line fixes to know that the incident you just had is rarely the exact incident you'll have next, so it became three thinner layers instead. Each one covers a different way the other two fail.

Layer 1: widen the deterministic check. The year-sanity check now runs against every content type that passes the gate, not just the type that misbehaved historically. When a deterministic guard misses, check its scope before its logic — in my experience the case almost always walked around the guard, not through it.

Layer 2: put recency in the judge's rubric. The judge now explicitly evaluates currency: when did this event happen, does its age undermine its value, penalize items whose event date sits far behind the feed date. This is the layer that catches what rules can't express. It only works for dimensions you remember to name. Rubrics are like tests that way — they encode the failures you've already imagined, and nothing else.

Layer 3: extract the event date from the content itself. This one goes at the root cause: trusting upstream metadata. A resurfaced story arrives wearing a fresh timestamp; the packaging lies even when the content doesn't. The pipeline now tries to pull the actual event date out of the item's text and compares it to the feed's claimed freshness. Disagreement gets flagged for me instead of published. It's also the flakiest of the three layers — date extraction from prose fails in fun ways — which is exactly why the two blunter layers stand in front of it.

Any one of these would have caught July's incident. I've watched each kind of layer fail before, though: deterministic checks miss on scope, judges miss on rubric gaps, extraction misses on weird input. So it's all three.

What I'd tell anyone running auto-publish

Your failure modes are public. A bad batch job embarrasses you in a log file. A bad publish embarrasses you in front of readers. Budget your quality-gate effort by audience size, not code size.

LLM judges have no instincts. Every implicit check a human editor does for free has to be written into the rubric explicitly. The judge will hand out perfect scores along every axis you specified while the item fails one you didn't.

Don't trust a timestamp you didn't compute. Feed metadata describes the feed's behavior, not the content's age. If freshness matters, derive it from the content.

The item came down, the three layers shipped the same week, and the pipeline went back to running unattended — which is the whole point. One person can run automated publishing. The tax is that every incident has to buy a structural fix, not a patch for the instance. Patch the instance and you've just scheduled the next one.