The analytics looked wrong. I'd been running a YouTube Shorts pipeline for about a month, and the performance data for my best-guess archetype — game-vs-game comparisons for indie titles — showed n≈1. One published video. The archetype I'd built the pipeline around had barely any measured data, even though the generation routine had been running daily.
I checked the analytics directive file expecting to find a reason to pivot. What I found instead was a measurement artifact: the directive was working correctly, but it was measuring the wrong universe. Eight Short scripts had been written, committed, and pushed — and never published.
The root cause: the publisher reads main, sessions commit to branches
My YouTube Auto-Script routine runs inside a Claude Code session that creates a new branch — something like claude/wizardly-ptolemy-abc123 — for each run. That's the default session behavior, and it makes sense for code changes (isolation, review, merge). For a content queue, it's a silent trap.
The publisher, yt-publish.yml, reads content/yt-queue/ exclusively on main. A JSON file sitting on claude/wizardly-ptolemy-* might as well not exist. The workflow doesn't scan remote branches. No error is thrown — the file just sits there, accumulating age.
Eight scripts sat there for approximately ten days: Papers Please, The Stanley Parable, Stardew Valley, Don't Starve, Valheim, A Short Hike, Lethal Company, and Hollow Knight. All written, all committed, all pushed to origin — none reaching the publisher. The only data points in the analytics directive came from a handful of videos that happened to land on main in the earliest runs before I set up session branching.
Why the silence was complete
Three factors combined to make this invisible:
1. The publisher exits cleanly when there's nothing new. yt-publish.yml picks the oldest unpublished file from content/yt-queue/ on main. If that directory has nothing new, it exits with success. There's no concept of "I expected new files and found none today."
2. Session branches look productive. Running git branch -r showed plenty of origin/claude/* refs with recent timestamps. From a distance, the generation routine appeared healthy. New commits, new pushes — just to the wrong destination.
3. Analytics reinforced the wrong picture. The directive file is built from published video data, so it reflects reality accurately — but that reality was an incomplete sample. "Honest data from a thin sample" is indistinguishable from "honest data from a working system" until you notice the sample size. n=1 after a month of daily generation was the tell, but I wasn't watching that number.
This failure mode is worth naming: branch drift. The generator is productive; the publisher is starved; nothing errors. The artifact storage problems I hit earlier were a different kind of invisible failure, but both share the same character — the pipeline looks fine and produces wrong results silently.
Fix 1: A routing contract document
I created docs/yt-queue-routing-contract.md with a single hard rule stated plainly:
Generated Short scripts MUST land on
main'scontent/yt-queue/. A script committed to a session branch is never published and never measured — it just rots.
The document explains why this is the rule (yt-publish only reads main), what "landing on main" means (direct commit or a PR merged the same day, never a persistent session branch), and how to handle the dedup constraint (match by Steam appid, not title, because titles are comparison strings that mislead).
A routing contract is different from a comment inside a YAML file. Comments are per-file; this document is findable by any tool, any future session reading the repo, any grep. The goal is that the constraint is visible before someone generates the first file in a new session — not discoverable only after tracing why nothing is publishing.
Fix 2: Remove the push trigger from the publisher
The original yt-publish.yml had both triggers:
on:
push:
branches: [main]
schedule:
- cron: '0 21 * * *' # 06:00 JST
This created a second problem beyond branch drift: when the generation routine commits to main and yt-publish fires on that push, there's a narrow window where the queue file exists but the commit isn't fully visible to the publisher's checkout. That race is low-probability but not zero.
Removing the push trigger entirely — keeping only cron — resolves this and makes the architecture cleaner. The publisher runs once per day at 06:00 JST, regardless of when scripts landed on main. A backlog of several scripts drains one per day on its own. There's no double-publish risk.
This is the correct relationship between a producer and a consumer queue: they run on independent schedules, communicate only through the queue state, and fail independently. The scheduling patterns article goes into why this matters for the broader multi-workflow monorepo setup, but for this specific problem the answer was simple: pick one trigger and let the queue do its job. GitHub's workflow trigger documentation covers the difference between push, schedule, and workflow_dispatch in detail — worth reading once before designing a queue like this.
Fix 3: A staleness watchdog
The structural fix is a daily workflow that checks whether the queue has received anything new in the last two days:
newest=$(find content/yt-queue -maxdepth 2 -type f -name '*.json' 2>/dev/null \
| sed -E 's#.*/([0-9]{4}-[0-9]{2}-[0-9]{2}).*#\1#' \
| grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}$' | sort | tail -1 || true)
age_days=$(( ( $(date -u +%s) - $(date -u -d "$newest" +%s) ) / 86400 ))
if [ "$age_days" -ge 2 ]; then
gh issue create \
--title "yt-queue stale: no new Short on main for ${age_days} days" \
--body "Newest file date: ${newest}. Check the Auto-Script routine is committing to main."
fi
Three decisions worth calling out:
Filename dates, not git log timestamps. A file named 2026-06-24T0004Z-*.json always extracts to 2026-06-24 regardless of how it arrived (direct push, merge commit, squash merge). Git log timestamps vary by mechanism, which would produce false alarms on the PR-merge path.
GitHub issues, not webhook notifications. Notifications disappear. A GitHub issue stays open until explicitly closed, appears in the repo's issue list, and is searchable. The operational signal should be durable.
Two-day threshold, not one. A single missed day might be intentional (a rest day, a holiday, a session that ran late). Two consecutive days means the production rate has genuinely dropped to zero. I close the issue manually if I pause deliberately and note why — that overhead is acceptable once a month.
What I recovered
I moved five of the eight stranded scripts to main's queue: Papers Please, Stanley Parable, Stardew Valley, Don't Starve, and Valheim. The other three I excluded:
- A Short Hike and Lethal Company had already published elsewhere under different titles, so moving them would create appid duplicates (Steam appid is the dedup key, not title).
- Hollow Knight was held because its title conflicted with a same-day upload already in the queue.
Those five scripts will publish one per day through the normal cron. The analytics directive now has a real sample for the game-comparison archetype. The n≈1 reading is gone.
The pattern generalizes
Every automated queue that feeds something visible needs three observable components:
| Component | What it catches |
|---|---|
| Routing contract | Producers committing to wrong destinations |
| Decoupled consumer | Publisher not depending on producer's push event |
| Staleness watchdog | Silent starvation when either side stops |
I now apply this to every queue I operate. The Bluesky post queue has a similar contract: entries land in content/bluesky-queue.jsonl on main; the publisher reads only main; a backed-up queue (12 unposted at one point, from a different set of pipeline failures) is now caught by explicit count checks rather than discovered after the channel goes quiet.
The post-deploy verification checks I run after each site build catch build-time failures. Branch drift happens earlier in the chain — at commit time, before deployment — so the two layers are complementary rather than overlapping.
FAQ
Why not make session branches auto-merge into main?
That adds merge-conflict handling and review logic without solving the root issue. The routing contract is simpler: state where files must land, enforce with a watchdog, audit failures manually. Auto-merge would hide the contract from future readers.
What if the watchdog fires during an intentional pause?
Close the issue manually with a note. That's acceptable overhead — once per pause period, not daily. The cost of a false alarm (one click to close) is lower than the cost of a silent stall going unnoticed for ten days.
Could branch protection rules prevent this?
Branch protection rules govern merging, not pushing. You can require review for merges, but you can't prevent a session from pushing to claude/* branches. The contract-plus-watchdog approach is more direct for queue scenarios.
Should every CI queue have a staleness guard?
Any queue that feeds something user-visible (a channel, a website, a social account) should. Queues that feed only internal state are lower priority, but if stakeholders notice when output stops, a guard should exist.
How granular should the watchdog threshold be?
Match your production cadence. Daily production → two-day threshold. Weekly production → five-day threshold. The goal is catching a complete stoppage, not a delay within normal variance.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)