Three approaches I use to catch silent failures in a cron-heavy GitHub Actions pipeline

The most common failure mode in my cron-heavy GitHub Actions setup isn't a red workflow — it's a green one that did nothing useful. A content refresh completes but stores stale data because an API rate limit fired early. A queue-drainer runs and logs "0 items processed" for a week because the input file format changed silently. (My nightly ETL is not one of these, to be clear — its entry point exits 1 on any uncaught error, so when it crashed in July the run went properly red.)

Red failures are easy. You get a notification, you look at it, you fix it. Green failures that produce wrong output take days to notice and are usually discovered by the symptom — the site doesn't have new entries, the YouTube queue isn't draining — rather than by the monitoring.

Here are three lightweight approaches I've wired up for this, in increasing complexity order.

GitHub Actions Insights: good for trend audits, useless for real-time alerting

GitHub's built-in Actions Insights (Settings → Actions → Insights in the repository) shows billable minutes by workflow, run frequency, and failure rates over a rolling 90-day window. This is useful for two things: spotting which workflows are costing the most quota, and reviewing whether a cron is actually running as scheduled.

It's not useful for real-time alerting. The data updates with a delay, there's no way to trigger a notification from it, and the failure rate chart shows hard failures only — not the green-exit-wrong-output case.

I check it roughly once a week as a billing audit. The GitHub Actions quota reclaiming work I did in June came directly from noticing in Insights that one workflow was consuming 40% of my monthly free tier. That's the right use for it: backward-looking audit, not forward-facing alert.

A daily pipeline-health watchdog that opens and closes a GitHub Issue

The more useful piece is a separate pipeline-health workflow that runs every night at 23:30 UTC, after all the content cron jobs have had time to complete. It calls a Python script that inspects recent workflow run statuses via the GitHub API and checks whether the content publishing pipelines actually produced output (not just exited 0).

The key distinction from simple failure alerting: the watchdog checks for no progress as a distinct condition from hard failure. A workflow can succeed and produce nothing — an empty queue drained is a successful no-op. So the watchdog keys off actual output: the posted_at timestamps inside the Bluesky queue file and the uploaded_at timestamps on uploaded YouTube queue files, flagging when either hasn't moved in 36 hours. It also flags stale queue buildup and a stale market-trends data file.

When it finds a problem, it opens a single GitHub Issue with a structured report. If that issue already exists from a previous day, it updates it in place rather than opening a new one. When the pipeline recovers, it closes the issue automatically. This means my notifications (GitHub Issue → mobile alert) are edge-triggered: I get one notification when something breaks and one when it recovers. Not a daily status email regardless of health.

The biggest limitation: it's a lagging indicator. By the time the watchdog runs at 23:30 UTC, a failure that happened at 03:00 UTC has been sitting for 20 hours. For content pipelines where daily freshness is the goal, this is acceptable. For something where every-hour data matters, I'd need a faster polling loop.

Proactive API balance check before the expensive work runs

The third piece is a check-anthropic-balance workflow — currently disabled — that was designed to run at 01:00 UTC, one hour before the content-generation cron. It makes a minimal API call to the Anthropic endpoint and checks the HTTP status code:

HTTP_CODE=$(curl -sS -o /tmp/resp.json -w "%{http_code}" \
  https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"."}]}' || true)

A 402 response means billing has lapsed. A 403 means the key is invalid or revoked. A 200 means the API is reachable and billing is active. The workflow exits 0 regardless — the point is to send a notification, not to fail loudly in the Actions UI.

I disabled it in May when I removed the Anthropic API dependency from the content pipeline entirely — with no API consumer left in the crons, there was nothing to pre-flight. (Discord itself is still my notification sink for other workflows.) The pattern is still correct — the one thing that used to silently kill an entire night's content generation was an expired Anthropic API key, and a pre-flight check 60 minutes before the main cron gives me time to rotate the key without losing a full day's run.

The pattern generalizes: for any paid external API your pipeline depends on, a cheap proactive check the hour before matters more than a failure notification the hour after. The notification after the failure is too late — the work didn't happen, and you're not getting that cron slot back.

---

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.