I run a small Python pipeline on GitHub Actions that generates and uploads Shorts for a few YouTube channels. Five scheduled workflows, including one I proudly named "Self-Healing Health Check".
Last weekend I finally opened the Actions tab. Every recent run was red. Upload runs, growth jobs, and the self-healing job itself. Meanwhile the bot had pushed 1,376 automated commits since May, so from a distance the repo looked alive. Last successful upload before the fix: May 25. That is 175 days of nothing.
I sat down with an AI coding agent and had it run every stage and classify each failure instead of fixing anything. Result: 14 separate problems.
- 4 were expired or revoked credentials
- 5 were upstream changes (a TTS endpoint returning 403 until I upgraded the library, a deprecated image model, a free tier that added rate limits, a removed vision model, a trends endpoint returning 404)
- 5 were my own code or config
So 9 of 14 had nothing to do with my code. The world moved and my pipeline stayed still.
The actual root cause was not in the repo at all. My Google OAuth app was still in "Testing" status, which means refresh tokens die after 7 days. Every re-auth I had ever done was a 7-day fix. The error in the logs was just invalid_grant.
Fixing that had its own trap: the "Publish app" button was greyed out with a vague message about completing the Branding page, even though every required field was filled. It only unlocked after I added a home page URL, a privacy policy URL and an authorized domain. For a personal uploader script. I used a tiny public GitHub repo with a README and PRIVACY.md.
Other things I learned the hard way that night:
- My self-healing workflow shared the same dead token as the jobs it was supposed to heal. It failed every time, quietly. A watchdog in the same failure domain is not a watchdog.
- My local repo had six months of uncommitted changes. Production was running code from May. I had been "fixing" a version that was never deployed.
- The agent printed my git remote URL, which had a personal access token embedded in it. Rotated it to a fine-grained token, forgot the Secrets permission, got a 403 on gh secret set, fixed that too.
- I asked the agent to delete temp files matching probe. The match was case-insensitive and it deleted 41 files of a finished video about the Parker Solar Probe. That one is on me for the sloppy instruction.
What fixed production in the end was almost no code. It was: find the real cause, change one Google setting, re-auth once, update one secret. First green scheduled run in a long time followed a few minutes later.
What I am changing:
- Check outcomes, not exit codes. "Did a video actually appear on the channel today" is the only health check that matters.
- Keep an expiry ledger: every token, key, free tier and model name, with the date it dies.
- Alerts go somewhere I actually look, and the alerting path must not depend on the thing it monitors.
Question for people running unattended automations: how do you catch the "everything is technically running but nothing is being produced" case? Heartbeats? Output checks? Something smarter?
Note: I wrote this post with AI assistance. The incident, numbers and fixes are from my own pipeline.
Top comments (5)
Silent failures like that 7-day Google OAuth token expiry in testing mode are the classic trap of unattended pipelines reporting false green statuses. The fix is moving your monitoring layer entirely outside the execution path by pinging an external heartbeat (like Healthchecks.io) only after a confirmed YouTube API upload response. Let me know what orchestration or notification webhooks you use for alerts and I can outline a resilient heartbeat architecture right here
Thanks, and agreed on both points. Pinging only after a confirmed upload response is the right signal, and my "self-healing" job living inside the same execution path was exactly why it stayed quiet.
What I keep running into is the step after the alert. Knowing it stopped took one check. Working out that the cause was a consent-screen setting, and which human steps would fix it, took hours. Do you use anything that helps with that part, or is it always manual triage once the heartbeat goes missing?
That is the exact gap most teams hit: detecting the failure takes 10 seconds; diagnosing the root cause takes 3 hours.
Once a heartbeat goes missing, manual triage is brutal because you're starting from scratch. To eliminate that scramble, here are the three patterns I use in production to make remediation practically instant:
For Google OAuth, an expired 7-day token returns:
{"error": "invalid_grant", "error_description": "Token has been expired or revoked."}Instead of just catching the exception, we map known failure signatures to human-actionable runbooks in our error-handling wrapper:
invalid_grantis caught, the alert webhook (Slack/Telegram) doesn't just say "Upload failed".[reauth-url].Having the exact diagnostic hypothesis and URL in the alert drops triage time from 2 hours to 2 minutes.
Run a lightweight, dedicated canary script (e.g., every 6 hours) that only performs an OAuth token refresh and hits a read-only endpoint (like
youtube.channels.list(mine=true)).If the token expires or the consent screen status drops, your alert fires hours before the actual upload pipeline is scheduled to run, giving you a buffer to re-auth without missing a production run.
Prompt it strictly: "Analyze this API error payload. Output: (1) Root Cause in plain English, (2) The 2 exact steps the engineer must take in the vendor console to fix it."
Injecting those 2 lines into your alert eliminates 90% of the Googling and doc-diving.
Are your pipelines running as custom Python/Node scripts or inside an orchestrator like n8n/Airflow? That usually dictates how easy it is to drop in an error interceptor.
Custom Python on GitHub Actions, no orchestrator. That's part of why it stayed quiet: there was no error hook to put any of this in, and the "successful" runs were swallowing failures with || true.
I ended up building roughly your three patterns as a separate watchdog job rather than inside the pipeline, so it works without touching the pipeline code: outcome check from outside, a ledger of expiring tokens/models/quotas, and LLM triage on failed and suspicious "green" run logs. Your point 1 is a good push though. Right now every diagnosis goes through the model, and known signatures like invalid_grant should map straight to a runbook. Adding that next. I'll open-source it once it has run on my own pipelines for a few days.
1,376 commits and zero uploads is a perfect picture of a pipeline that looks alive from a distance. The detail that stood out to me was the self-healing job sharing the same dead token as the jobs it was meant to heal, and the root cause being an OAuth app left in Testing status, so every re-auth was only ever a 7-day fix. Your reply about the "successful" runs swallowing failures with || true explains why nothing turned red where you would have looked.
At Ascendra Ventures we worked on a similar problem for a seed-stage talent platform, where the candidate pipeline did not fail loudly, it just stopped. A slow step could leave a person in an unclear state with no error anywhere. The fix was to rebuild the profile processor as a durable workflow: urgent profiles run straight away, the rest are batched, it falls back to polling when callbacks go quiet, and every candidate either finishes or visibly fails. Your watchdog that checks outcomes from outside the pipeline is the same idea: the job is not done because it ran, it is done when the upload exists.
Did finding that 9 of the 14 problems came from upstream changes make you rethink how many outside services the pipeline should depend on?