The most common failure mode in my cron-heavy GitHub Actions setup isn't a red workflow — it's a green one that did nothing useful. A nightly ETL exits 0 after crashing midway through a for-loop. A content refresh completes but stores stale data because an API rate limit fired early. A queue-drainer runs and logs "0 items processed" for a week because the input file format changed silently.
Red failures are easy. You get a notification, you look at it, you fix it. Green failures that produce wrong output take days to notice and are usually discovered by the symptom — the site doesn't have new entries, the YouTube queue isn't draining — rather than by the monitoring.
Here are three lightweight approaches I've wired up for this, in increasing complexity order.
GitHub Actions Insights: good for trend audits, useless for real-time alerting
GitHub's built-in Actions Insights (Settings → Actions → Insights in the repository) shows billable minutes by workflow, run frequency, and failure rates over a rolling 90-day window. This is useful for two things: spotting which workflows are costing the most quota, and reviewing whether a cron is actually running as scheduled.
It's not useful for real-time alerting. The data updates with a delay, there's no way to trigger a notification from it, and the failure rate chart shows hard failures only — not the green-exit-wrong-output case.
I check it roughly once a week as a billing audit. The GitHub Actions quota reclaiming work I did in June came directly from noticing in Insights that one workflow was consuming 40% of my monthly free tier. That's the right use for it: backward-looking audit, not forward-facing alert.
A daily pipeline-health watchdog that opens and closes a GitHub Issue
The more useful piece is a separate pipeline-health workflow that runs every night at 23:30 UTC, after all the content cron jobs have had time to complete. It calls a Python script that inspects recent workflow run statuses via the GitHub API and checks whether the ETL and content-refresh pipelines actually produced output (not just exited 0).
The key distinction from simple failure alerting: the watchdog checks for no progress as a distinct condition from hard failure. A workflow can succeed and produce nothing — no new rows inserted, no new files committed. The watchdog tracks the last commit timestamp for the content directories and the last-updated timestamp for database rows and flags when those haven't moved in 24 hours.
When it finds a problem, it opens a single GitHub Issue with a structured report. If that issue already exists from a previous day, it updates it in place rather than opening a new one. When the pipeline recovers, it closes the issue automatically. This means my notifications (GitHub Issue → mobile alert) are edge-triggered: I get one notification when something breaks and one when it recovers. Not a daily status email regardless of health.
The biggest limitation: it's a lagging indicator. By the time the watchdog runs at 23:30 UTC, a failure that happened at 03:00 UTC has been sitting for 20 hours. For content pipelines where daily freshness is the goal, this is acceptable. For something where every-hour data matters, I'd need a faster polling loop.
Proactive API balance check before the expensive work runs
The third piece is a check-anthropic-balance workflow — currently disabled — that was designed to run at 01:00 UTC, one hour before the content-generation cron. It makes a minimal API call to the Anthropic endpoint and checks the HTTP status code:
HTTP_CODE=$(curl -sS -o /tmp/resp.json -w "%{http_code}" \
https://api.anthropic.com/v1/messages \
-H "x-api-key: $ANTHROPIC_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{"model":"claude-haiku-4-5","max_tokens":1,"messages":[{"role":"user","content":"."}]}' || true)
A 402 response means billing has lapsed. A 403 means the key is invalid or revoked. A 200 means the API is reachable and billing is active. The workflow exits 0 regardless — the point is to send a notification, not to fail loudly in the Actions UI.
I disabled it when I stopped using Discord as a notification sink; the workflow was posting balance alerts to a Discord webhook that nobody was watching. The pattern is still correct — the one thing that used to silently kill an entire night's content generation was an expired Anthropic API key, and a pre-flight check 60 minutes before the main cron gives me time to rotate the key without losing a full day's run.
The pattern generalizes: for any paid external API your pipeline depends on, a cheap proactive check the hour before matters more than a failure notification the hour after. The notification after the failure is too late — the work didn't happen, and you're not getting that cron slot back.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)