The trigger was a stretch of days where yt-publish and Publish articles sat red and nobody noticed — nothing was watching the Actions tab, so a red run stayed on screen exactly as long as a green one. The failure that sharpened the lesson — the one I root-caused the day after shipping the watchdog — never went red at all: the Bluesky queue had stalled because bluesky-queue.yml had timeout-minutes: 5 while the job opens with a random 0–5 minute start delay to dodge the cron spike. A high roll cancelled run 27902025841 at 5m4s, and the queue backed up from 7 to 12 unposted items. The job status did say cancelled — but check_failed_runs() matches only conclusion == "failure", so the watchdog's first signal skipped straight past a status that was visible the whole time, and the Bluesky QC gate had nothing to say because the 5-minute cap expired before the qc and post steps ever ran. The no-progress check was the fallback that could catch the stalled output.
The conclusion I took from that: job status is not the same as output progress. I'd built the three-site content pipeline assuming those were equivalent. The assumption was wrong.
The fix is scripts/pipeline-health.py, a Python script that runs every night in GitHub Actions at 23:30 UTC — after the day's content routines complete — and checks three separate signals. It uses the GitHub Actions REST API and the GitHub Issues API via urllib.request, so there are no dependencies beyond the Python standard library and the GITHUB_TOKEN that Actions injects automatically. When any signal fires, it opens a GitHub Issue with a structured diagnostic body. When everything clears, it closes the Issue automatically. The rest of the time, it's silent.
The three failure modes and why job status misses two of them
The script watches four content workflows by display name: yt-publish, Publish articles, yt-publish-longform, and Bluesky queue post. For each, it queries the GitHub Actions API for runs in the last 26 hours and reports any run whose conclusion is failure. That catches the obvious failure: a job that crashed. It does not match cancelled, which is exactly why the timed-out Bluesky run above never showed up in this signal — the no-progress check is the one that would catch that.
The obvious failure is not the dangerous one.
Signal 2 is no-progress detection. The YouTube publish workflow reads from a queue directory and exits 0 whether it published a video or found an empty queue — both cases look identical from CI. The script reads the uploaded_at field out of every JSON in content/yt-queue/uploaded/ and takes the newest one. If nothing in that directory is newer than 36 hours, video production has stalled even though no workflow failed. The same threshold applies to Bluesky: 36 hours without a logged post triggers the alert.
Signal 3 is queue staleness, and it only covers the YouTube side: stale_queue() is hardcoded to content/yt-queue and checks whether the oldest file there is more than 14 days old. A video sitting in the queue for 14 days almost certainly missed its window — either a gate blocked it permanently, or the queue processor is stuck on it. Neither is visible from job status. The Bluesky queue has no equivalent check: it's a single JSONL file (content/bluesky-queue.jsonl), and the script only reads posted_at timestamps out of it for the no-progress signal.
The three signals catch distinct failure classes:
| Signal | What it catches | What it misses |
|---|---|---|
| Workflow failures | Crashed jobs, config errors | Successful no-ops, and cancelled runs such as timeouts |
| No-progress (36h) | Stalled pipelines with clean CI | Bad-but-published content |
| Stale yt-queue (14d) | Video queue backed up past its useful lifetime | Short-term stalls, and the Bluesky queue entirely |
The gaps in coverage matter as much as the coverage itself. The health monitor doesn't read article content, doesn't verify publish quality, and doesn't know whether a video is good. The content quality gate handles article-level problems; the ETL health queries catch data-level stalls. The pipeline monitor only asks whether something shipped at all.
Why a single deduplicated Issue beats individual notifications
Deduplication went into the first version, before the watchdog ever ran on a schedule — one open pipeline-health Issue at a time, updated in place. That was a deliberate choice rather than a lesson learned the hard way: a daily watchdog that opens an Issue per detection would have produced a new Issue every night for as long as a stall lasted, and I know I'd have stopped reading them.
The script embeds <!-- pipeline-health-bot --> as a marker in the Issue body, searches for that marker on each run, and updates the existing Issue in place if found. No new Issue is opened while an alert is active; the body is replaced with the current diagnostic state and the timestamp of detection. Even so, one root cause can crowd a single Issue: #18 ended up listing "Publish articles failed" 11 times, all from one expired Hashnode token.
When all three signals clear — no workflow failures, progress markers are recent, queue isn't stale — the script calls the GitHub Issues API to close the open alert Issue. The next failure opens a new one. The effect: the Issue tracker contains at most one pipeline-health issue at any moment, and its open/closed state maps directly to pipeline health.
The closed-issue history becomes a free audit trail. Filtering closed Issues by the pipeline-health label shows exactly when the pipeline stalled and for how long, without any additional storage or logging infrastructure.
I considered Slack or email notifications instead. The problem with both: they create a parallel notification stream I have to check separately from my normal work. A GitHub Issue surfaces in the same place I track everything else. It auto-assigns priority by being open. When it closes, I don't have to manually mark anything resolved.
Implementation: stdlib only, no npm install, no external packages
The script uses only Python's standard library: urllib.request, json, os, datetime, sys. No requests, no PyGithub, no install step.
def _req(method, path, body=None):
url = path if path.startswith("http") else f"{API}{path}"
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Authorization", f"Bearer {TOKEN}")
req.add_header("Accept", "application/vnd.github+json")
req.add_header("X-GitHub-Api-Version", "2022-11-28")
if data:
req.add_header("Content-Type", "application/json")
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read().decode()
return json.loads(raw) if raw else {}
The GitHub Actions step needs two permissions: actions: read to query workflow run history, and issues: write to open and close the alert Issue. GITHUB_TOKEN and GITHUB_REPOSITORY are provided automatically in the Actions context — no additional secrets setup.
The workflow runs once daily at 23:30 UTC, not on every push or every workflow completion. Running on a fixed schedule avoids double-alerting when a single stall causes multiple overlapping failures. The 30-minute offset after the top of the hour puts it after the nightly content jobs finish, which avoids flagging in-progress work.
on:
schedule:
- cron: "30 23 * * *"
workflow_dispatch: {}
The workflow_dispatch trigger lets me run it manually to check state without waiting for the nightly run. This was useful while debugging the no-progress thresholds.
How I picked the thresholds
The 36-hour no-progress threshold came from failure data. The shortest real stall that caused content gaps was about 30 hours. The GitHub Actions cron timing bugs I'd hit previously mean the watchdog itself might fire off schedule. 36 hours gives one missed daily run plus a six-hour margin before it trips; 24 hours produced false positives in testing.
The 14-day queue staleness threshold mirrors QUEUE_MAX_AGE_DAYS in yt-publish.yml. The YouTube publisher drops files older than 14 days rather than publish them out of context. A file that old in the queue is one the publisher won't touch — it signals a stall that's gone past the point of self-repair.
The 26-hour window for checking workflow runs is slightly longer than a calendar day. This prevents a gap when the nightly run schedule drifts by a few minutes and a failure from the previous night's run falls just outside a 24-hour window.
I haven't changed any of these since the initial deployment — both the YouTube and Bluesky no-progress limits have been 36 hours since the first commit of the script. 36 hours also gives me room to pause posting for a day without triggering an alert.
What changed after I had this running
The monitoring changed how I think about the GitHub Actions free quota. When I was cutting workflow runs to stay under the free tier ceiling, I evaluated each workflow by its compute cost. The pipeline health monitor runs for about 20 seconds and queries the API a handful of times — nearly free in compute terms. But its value is asymmetric: 20 seconds of Actions minutes catches failures that would otherwise take me days to notice. I kept it without hesitation.
It also surfaced a pattern I hadn't noticed: what stalls the Bluesky JSONL queue is infrastructure, not content. The stall I root-caused was the workflow timeout above, and the noise that flooded the alert Issue came from an expired Hashnode token turning Publish articles red run after run. Neither was a bad post. That told me where the error handling was actually missing — a broken token is now a deferred, warned-about condition instead of a red job.
The monitor doesn't eliminate production failures. It shrinks the detection window from "whenever I notice the feed went quiet" to "the morning after the failure occurred."
What this doesn't catch
Bad-but-published content. If a YouTube video uploads with corrupted audio, or a Bluesky post goes out with a broken image, the pipeline-health monitor doesn't know. It only asks whether something was published, not whether what was published was correct. That's a separate concern that belongs in the QC gates before publish, not in a post-hoc health check.
Cascade failures where the pipeline restarts fast enough that no 36-hour window opens. A job that fails and restarts within an hour — GitHub's built-in retry behavior — looks healthy from the no-progress signal even if it failed and retried five times.
Intentional pauses. If I decide to take a break from publishing, the monitor trips after 36 hours. I handle this by closing the alert Issue manually and ignoring it for the pause duration, then re-opening if things haven't resumed. There's a cleaner solution — a PAUSED_UNTIL environment variable the script could check — but I've hit this scenario infrequently enough that I haven't implemented it.
FAQ
Does this replace checking the GitHub Actions UI directly?
No. The UI shows job-level status and step logs, which the health monitor doesn't replicate. This adds a layer on top: watching outputs rather than jobs, and creating a persistent artifact (the Issue) that survives past the run.
Why Python instead of a Node.js script like the other monitoring tools in this project?
No strong reason. I wrote it during a session already in Python and urllib.request handles everything without an npm install step. A Node.js version would work equally well.
What's the QUEUE_MAX_AGE_DAYS threshold based on?
It mirrors the hard cap in the publisher: the queue processor drops files older than 14 days rather than publish stale content. A file that's been in the queue past that threshold is one the publisher won't handle automatically — the stall is permanent until a human intervenes.
Could GitHub's built-in failure notifications replace this?
GitHub can notify on workflow job failures. It can't detect no-progress signals or queue staleness. Those two signals are specific to my pipeline's output semantics and require code that understands what "progress" means in this context.
Why auto-close the Issue instead of requiring manual review?
Open issues accumulate and lose signal value. A closed issue with the self-resolution timestamp is more useful for retrospective analysis than an issue that requires someone to manually close it after confirming everything's working. If the pipeline is healthy, there's no issue to read; if it's not, there's exactly one issue that says what's wrong.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)