The analytics looked wrong. I'd been running a YouTube Shorts pipeline for about a month, and the performance data for my best-guess archetype — game-vs-game comparisons for indie titles — showed n=2. Two published videos. The archetype I'd built the pipeline around had barely any measured data, even though the generation routine had been running daily.
I checked the analytics directive file expecting to find a reason to pivot. What I found instead was a measurement artifact: the directive was working correctly, but it was measuring the wrong universe. Eight Short scripts had been written, committed, and pushed — and never published.
The root cause: the publisher reads main, sessions commit to branches
My YouTube Auto-Script routine runs inside a Claude Code session that creates a new branch — something like claude/wizardly-ptolemy-abc123 — for each run. That's the default session behavior, and it makes sense for code changes (isolation, review, merge). For a content queue, it's a silent trap.
The publisher, yt-publish.yml, reads content/yt-queue/ exclusively on main. A JSON file sitting on claude/wizardly-ptolemy-* might as well not exist. The workflow doesn't scan remote branches. No error is thrown — the file just sits there, accumulating age.
Eight scripts sat there for approximately ten days: Papers Please, The Stanley Parable, Stardew Valley, Don't Starve, Valheim, A Short Hike, Lethal Company, and Hollow Knight. All written, all committed, all pushed to origin — none reaching the publisher. The only data points in the analytics directive came from a handful of videos that happened to land on main in the earliest runs before I set up session branching.
Why the silence was complete
Three factors combined to make this invisible:
1. The publisher exits cleanly when there's nothing new. yt-publish.yml picks one unpublished file from content/yt-queue/ on main — the oldest inside the highest-ranked archetype tier present, not the globally oldest. If that directory has nothing new, it exits with success. There's no concept of "I expected new files and found none today."
2. Session branches look productive. Running git branch -r showed plenty of origin/claude/* refs with recent timestamps. From a distance, the generation routine appeared healthy. New commits, new pushes — just to the wrong destination.
3. Analytics reinforced the wrong picture. The directive file is built from published video data, so it reflects reality accurately — but that reality was an incomplete sample. "Honest data from a thin sample" is indistinguishable from "honest data from a working system" until you notice the sample size. n=2 after a month of daily generation was the tell, but I wasn't watching that number.
This failure mode is worth naming: branch drift. The generator is productive; the publisher is starved; nothing errors. The artifact storage problems I hit earlier were a different kind of invisible failure, but both share the same character — the pipeline looks fine and produces wrong results silently.
Fix 1: A routing contract document
I created docs/yt-queue-routing-contract.md with a single hard rule stated plainly:
Generated Short scripts MUST land on
main'scontent/yt-queue/. A script committed to a session branch is never published and never measured — it just rots.
The document explains why this is the rule (yt-publish only reads main), what "landing on main" means (direct commit or a PR merged the same day, never a persistent session branch), and how to handle the dedup constraint (match by Steam appid, not title, because titles are comparison strings that mislead).
A routing contract is different from a comment inside a YAML file. Comments are per-file; this document is findable by any tool, any future session reading the repo, any grep. The goal is that the constraint is visible before someone generates the first file in a new session — not discoverable only after tracing why nothing is publishing.
Fix 2: Remove the push trigger from the publisher
The original yt-publish.yml had no cron at all — publishing fired on pushes to main:
on:
push:
branches: [main]
paths:
- 'content/yt-queue/*.json'
workflow_dispatch:
That tied publishing directly to the generation routine pushing to main, which is exactly the dependency that broke. It also created a second problem beyond branch drift: when the generation routine commits to main and yt-publish fires on that push, there's a narrow window where the queue file exists but the commit isn't fully visible to the publisher's checkout. That race is low-probability but not zero.
Replacing the push trigger with a daily cron — schedule: - cron: '0 21 * * *', 06:00 JST — resolves this and makes the architecture cleaner. The publisher runs once per day at 06:00 JST, regardless of when scripts landed on main. A backlog of several scripts drains one per day on its own. There's no double-publish risk.
This is the correct relationship between a producer and a consumer queue: they run on independent schedules, communicate only through the queue state, and fail independently. The scheduling patterns article goes into why this matters for the broader multi-workflow monorepo setup, but for this specific problem the answer was simple: pick one trigger and let the queue do its job. GitHub's workflow trigger documentation covers the difference between push, schedule, and workflow_dispatch in detail — worth reading once before designing a queue like this.
Fix 3: A staleness watchdog
The structural fix is a daily workflow that checks whether the queue has received anything new in the last two days:
newest=$(find content/yt-queue -maxdepth 2 -type f -name '*.json' 2>/dev/null \
| sed -E 's#.*/([0-9]{4}-[0-9]{2}-[0-9]{2}).*#\1#' \
| grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}$' | sort | tail -1 || true)
age_days=$(( ( $(date -u +%s) - $(date -u -d "$newest" +%s) ) / 86400 ))
if [ "$age_days" -ge 2 ]; then
gh issue create \
--title "yt-queue stale: no new Short on main for ${age_days} days" \
--body "Newest file date: ${newest}. Check the Auto-Script routine is committing to main."
fi
Three decisions worth calling out:
Filename dates, not git log timestamps. A file named 2026-06-24T0004Z-*.json always extracts to 2026-06-24 regardless of how it arrived (direct push, merge commit, squash merge). Git log timestamps vary by mechanism, which would produce false alarms on the PR-merge path.
GitHub issues, not webhook notifications. Notifications disappear. A GitHub issue stays open until explicitly closed, appears in the repo's issue list, and is searchable. The operational signal should be durable.
Two-day threshold, not one. A single missed day might be intentional (a rest day, a holiday, a session that ran late). Two consecutive days means the production rate has genuinely dropped to zero. I close the issue manually if I pause deliberately and note why — that overhead is acceptable once a month.
What I recovered
I moved five of the eight stranded scripts to main's queue: Papers Please, Stanley Parable, Stardew Valley, Don't Starve, and Valheim. The other three I excluded:
- A Short Hike and Lethal Company had already published elsewhere under different titles, so moving them would create appid duplicates (Steam appid is the dedup key, not title).
- Hollow Knight was held because its title conflicted with a same-day upload already in the queue.
Those five scripts will publish one per day through the normal cron. The analytics directive now has a real sample for the game-comparison archetype: by 06-24 it read n=4 instead of n=2.
The pattern generalizes
Every automated queue that feeds something visible needs three observable components:
| Component | What it catches |
|---|---|
| Routing contract | Producers committing to wrong destinations |
| Decoupled consumer | Publisher not depending on producer's push event |
| Staleness watchdog | Silent starvation when either side stops |
I now apply this to every queue I operate. The Bluesky post queue has a similar contract: entries land in content/bluesky-queue.jsonl on main; the publisher reads only main; a backed-up queue (12 unposted at one point, from a different set of pipeline failures) is now caught by explicit count checks rather than discovered after the channel goes quiet.
The post-deploy verification checks I run after each site build catch build-time failures. Branch drift happens earlier in the chain — at commit time, before deployment — so the two layers are complementary rather than overlapping.
FAQ
Why not make session branches auto-merge into main?
That adds merge-conflict handling and review logic without solving the root issue. The routing contract is simpler: state where files must land, enforce with a watchdog, audit failures manually. Auto-merge would hide the contract from future readers.
What if the watchdog fires during an intentional pause?
Close the issue manually with a note. That's acceptable overhead — once per pause period, not daily. The cost of a false alarm (one click to close) is lower than the cost of a silent stall going unnoticed for ten days.
Could branch protection rules prevent this?
Branch protection rules govern merging, not pushing. You can require review for merges, but you can't prevent a session from pushing to claude/* branches. The contract-plus-watchdog approach is more direct for queue scenarios.
Should every CI queue have a staleness guard?
Any queue that feeds something user-visible (a channel, a website, a social account) should. Queues that feed only internal state are lower priority, but if stakeholders notice when output stops, a guard should exist.
How granular should the watchdog threshold be?
Match your production cadence. Daily production → two-day threshold. Weekly production → five-day threshold. The goal is catching a complete stoppage, not a delay within normal variance.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)