The trigger was a four-day gap I noticed in retrospect. The Bluesky queue had silently stalled because an API token expired mid-run, every daily workflow returned exit code 0, and nothing in GitHub Actions went red. The Bluesky QC gate was working correctly — it blocked a bad post, then the token expired before the next run, and after that the gate started rejecting everything because authentication was broken. Four days later I noticed the account had gone quiet.
The conclusion I took from that: job status is not the same as output progress. I'd built the three-site content pipeline assuming those were equivalent. The assumption was wrong.
The fix is scripts/pipeline-health.py, a Python script that runs every night in GitHub Actions at 23:30 UTC — after the day's content routines complete — and checks three separate signals. It uses the GitHub Actions REST API and the GitHub Issues API via urllib.request, so there are no dependencies beyond the Python standard library and the GITHUB_TOKEN that Actions injects automatically. When any signal fires, it opens a GitHub Issue with a structured diagnostic body. When everything clears, it closes the Issue automatically. The rest of the time, it's silent.
The three failure modes and why job status misses two of them
The script watches four content workflows by display name: yt-publish, Publish articles, yt-publish-longform, and Bluesky queue post. For each, it queries the GitHub Actions API for runs in the last 26 hours and reports any failure or cancelled conclusion. That catches the obvious failure: a job that crashed or was abandoned.
The obvious failure is not the dangerous one.
Signal 2 is no-progress detection. The YouTube publish workflow reads from a queue directory and exits 0 whether it published a video or found an empty queue — both cases look identical from CI. The script reads content/yt-queue/uploaded/ and checks the timestamp of the most recent file. If nothing in that directory is newer than 36 hours, video production has stalled even though no workflow failed. The same threshold applies to Bluesky: 36 hours without a logged post triggers the alert.
Signal 3 is queue staleness. The Bluesky queue is a directory of pending posts, and the script checks whether the oldest file in that queue is more than 14 days old. A post sitting in the queue for 14 days almost certainly missed its window — either a QC gate blocked it permanently, or the queue processor is stuck on it. Neither is visible from job status.
The three signals catch distinct failure classes:
| Signal | What it catches | What it misses |
|---|---|---|
| Workflow failures | Crashed jobs, timeouts, config errors | Successful no-ops |
| No-progress (36h) | Stalled pipelines with clean CI | Bad-but-published content |
| Stale queue (14d) | Content backed up past its useful lifetime | Short-term stalls under the threshold |
The gaps in coverage matter as much as the coverage itself. The health monitor doesn't read article content, doesn't verify publish quality, and doesn't know whether a video is good. The content quality gate handles article-level problems; the ETL health queries catch data-level stalls. The pipeline monitor only asks whether something shipped at all.
Why a single deduplicated Issue beats individual notifications
The first version opened a new Issue every time the watchdog detected a problem. Within two weeks there were 47 open Issues about variations of the same stalled pipeline. I stopped reading them.
The second version writes to one Issue. The script embeds <!-- pipeline-health-bot --> as a marker in the Issue body, searches for that marker on each run, and updates the existing Issue in place if found. No new Issue is opened while an alert is active; the body is replaced with the current diagnostic state and the timestamp of detection.
When all three signals clear — no workflow failures, progress markers are recent, queue isn't stale — the script calls the GitHub Issues API to close the open alert Issue. The next failure opens a new one. The effect: the Issue tracker contains at most one pipeline-health issue at any moment, and its open/closed state maps directly to pipeline health.
The closed-issue history becomes a free audit trail. Filtering closed Issues by the pipeline-health label shows exactly when the pipeline stalled and for how long, without any additional storage or logging infrastructure.
I considered Slack or email notifications instead. The problem with both: they create a parallel notification stream I have to check separately from my normal work. A GitHub Issue surfaces in the same place I track everything else. It auto-assigns priority by being open. When it closes, I don't have to manually mark anything resolved.
Implementation: stdlib only, no npm install, no external packages
The script uses only Python's standard library: urllib.request, json, os, datetime, sys. No requests, no PyGithub, no install step.
def _req(method, path, body=None):
url = path if path.startswith("http") else f"{API}{path}"
data = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(url, data=data, method=method)
req.add_header("Authorization", f"Bearer {TOKEN}")
req.add_header("Accept", "application/vnd.github+json")
req.add_header("X-GitHub-Api-Version", "2022-11-28")
if data:
req.add_header("Content-Type", "application/json")
with urllib.request.urlopen(req, timeout=30) as resp:
raw = resp.read().decode()
return json.loads(raw) if raw else {}
The GitHub Actions step needs two permissions: actions: read to query workflow run history, and issues: write to open and close the alert Issue. GITHUB_TOKEN and GITHUB_REPOSITORY are provided automatically in the Actions context — no additional secrets setup.
The workflow runs once daily at 23:30 UTC, not on every push or every workflow completion. Running on a fixed schedule avoids double-alerting when a single stall causes multiple overlapping failures. The 30-minute offset after the top of the hour puts it after the nightly content jobs finish, which avoids flagging in-progress work.
on:
schedule:
- cron: "30 23 * * *"
workflow_dispatch: {}
The workflow_dispatch trigger lets me run it manually to check state without waiting for the nightly run. This was useful while debugging the no-progress thresholds.
How I picked the thresholds
The 36-hour no-progress threshold came from failure data. The shortest real stall that caused content gaps was about 30 hours. The GitHub Actions cron timing bugs I'd hit previously mean the watchdog itself might fire off schedule. 36 hours gives one missed daily run plus a six-hour margin before it trips; 24 hours produced false positives in testing.
The 14-day queue staleness threshold mirrors QUEUE_MAX_AGE_DAYS in yt-publish.yml. The YouTube publisher drops files older than 14 days rather than publish them out of context. A file that old in the queue is one the publisher won't touch — it signals a stall that's gone past the point of self-repair.
The 26-hour window for checking workflow runs is slightly longer than a calendar day. This prevents a gap when the nightly run schedule drifts by a few minutes and a failure from the previous night's run falls just outside a 24-hour window.
I changed these thresholds once after the initial deployment: the Bluesky no-progress threshold started at 24 hours and produced a false alert after I intentionally paused posting for a day without updating the configuration. 36 hours now gives me room to pause for a day without triggering an alert.
What changed after I had this running
The monitoring changed how I think about the GitHub Actions free quota. When I was cutting workflow runs to stay under the free tier ceiling, I evaluated each workflow by its compute cost. The pipeline health monitor runs for about 20 seconds and queries the API a handful of times — nearly free in compute terms. But its value is asymmetric: 20 seconds of Actions minutes catches failures that would otherwise take me days to notice. I kept it without hesitation.
It also surfaced a pattern I hadn't noticed: the Bluesky JSONL queue stalls more often than the YouTube queue, and almost always from auth issues rather than content issues. The no-progress signal has fired three times total. Two of those three were Bluesky auth failures; one was an unrelated disk quota issue in the upload step. That distribution told me where to add better error handling, which the monitoring made visible.
The monitor doesn't eliminate production failures. It shrinks the detection window from "whenever I notice the feed went quiet" to "the morning after the failure occurred."
What this doesn't catch
Bad-but-published content. If a YouTube video uploads with corrupted audio, or a Bluesky post goes out with a broken image, the pipeline-health monitor doesn't know. It only asks whether something was published, not whether what was published was correct. That's a separate concern that belongs in the QC gates before publish, not in a post-hoc health check.
Cascade failures where the pipeline restarts fast enough that no 36-hour window opens. A job that fails and restarts within an hour — GitHub's built-in retry behavior — looks healthy from the no-progress signal even if it failed and retried five times.
Intentional pauses. If I decide to take a break from publishing, the monitor trips after 36 hours. I handle this by closing the alert Issue manually and ignoring it for the pause duration, then re-opening if things haven't resumed. There's a cleaner solution — a PAUSED_UNTIL environment variable the script could check — but I've hit this scenario infrequently enough that I haven't implemented it.
FAQ
Does this replace checking the GitHub Actions UI directly?
No. The UI shows job-level status and step logs, which the health monitor doesn't replicate. This adds a layer on top: watching outputs rather than jobs, and creating a persistent artifact (the Issue) that survives past the run.
Why Python instead of a Node.js script like the other monitoring tools in this project?
No strong reason. I wrote it during a session already in Python and urllib.request handles everything without an npm install step. A Node.js version would work equally well.
What's the QUEUE_MAX_AGE_DAYS threshold based on?
It mirrors the hard cap in the publisher: the queue processor drops files older than 14 days rather than publish stale content. A file that's been in the queue past that threshold is one the publisher won't handle automatically — the stall is permanent until a human intervenes.
Could GitHub's built-in failure notifications replace this?
GitHub can notify on workflow job failures. It can't detect no-progress signals or queue staleness. Those two signals are specific to my pipeline's output semantics and require code that understands what "progress" means in this context.
Why auto-close the Issue instead of requiring manual review?
Open issues accumulate and lose signal value. A closed issue with the self-resolution timestamp is more useful for retrospective analysis than an issue that requires someone to manually close it after confirming everything's working. If the pipeline is healthy, there's no issue to read; if it's not, there's exactly one issue that says what's wrong.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)