Running five GitHub Actions workflows daily across a monorepo sounds simple until the third one fires at 3:30am instead of midnight, the fourth publishes the same video twice, and the fifth silently cancels before it finishes. I've hit all four of these. Here's what each one looked like and how I fixed it.
Bug 1: Top-of-hour queue contention causes 3+ hour drift
The GitHub Actions job queue is global and shared. Scheduling a cron at 0 0 * * * (midnight UTC) puts you in the same batch as thousands of other repos. GitHub's documentation on scheduled events notes that scheduled workflows can be delayed during high loads — they don't commit to exact delivery time. During busy hours, GitHub's scheduler queues those jobs and delivers them when a runner is available. The result: I scheduled a Bluesky post for midnight UTC and watched it run at 03:27 UTC on a Wednesday.
For most CI tasks this is fine — a build running an hour late is rarely critical. For social media posting, a "midnight UTC" post landing at 3:30am matters.
The fix is off-minute scheduling with an earlier offset. Instead of 0 0 * * *, I use 37 7 * * * (07:37 UTC → ~16:37 UTC, landing near my 17:00 JST target). The offset moves the job out of the peak batch; the randomized start delay inside the workflow (sleep $(( RANDOM % 300 ))) handles the remaining variance without creating a bot-posting pattern.
Bug 2: Push+cron double-publish
I had this in yt-publish.yml:
on:
push:
branches: [main]
schedule:
- cron: '0 21 * * *'
The intent was to publish immediately when a new Short script landed on main, and also publish daily at 06:00 JST in case the routine pushed outside the push window. In practice, this meant that when the generation routine pushed a new queue file to main at 21:00 UTC, two jobs spawned simultaneously — one from the push trigger, one from the matching cron. Both picked the same file as the oldest unpublished entry. The second upload succeeded too; YouTube accepted it as a distinct video and issued a new video ID.
The fix: remove the push trigger entirely. A content queue with a daily consumer doesn't benefit from an immediate-publish trigger — it benefits from a predictable publish cadence. The cron handles that. A backlog of files drains one per day; workflow_dispatch handles on-demand publishes when I need to jump the queue.
If your workflow genuinely needs to trigger on push and schedule, use concurrency with cancel-in-progress: true to ensure only one run proceeds. But for a daily content queue, picking one trigger is cleaner.
Bug 3: Default job timeout cancels the job before it finishes
GitHub Actions jobs have a default timeout of 6 hours, but individual steps don't. If you specify timeout-minutes at the job level, that's the ceiling for the entire job. I set timeout-minutes: 5 on my Bluesky posting workflow, thinking: "post one tweet, can't take more than 5 minutes."
It can, when you add a random start delay.
The Bluesky workflow opens with a sleep $(( RANDOM % 300 )) — up to five minutes — to avoid posting at an obvious cron-exact timestamp. On a high-roll (say, 295 seconds), the delay alone consumed almost the entire budget. The QC check and actual post then bumped the total past the limit. The job was cancelled at 5 minutes 4 seconds, logged as successful (timeout cancels the step, not the job), and left the queue entry in a half-processed state.
The symptom was a Bluesky queue that appeared to be processing (commit timestamps updated daily) but hadn't actually posted anything for about a week. Twelve unposted entries had backed up.
The fix: set timeout-minutes to at least (max_delay / 60) + (actual_work_minutes) + (margin). For Bluesky with a 5-minute random delay and ~2 minutes of actual work, timeout-minutes: 12 gives comfortable headroom. If the timeout is genuinely the constraint (cost, billing minutes), reduce the random delay range instead.
Bug 4: Concurrent workflow pushes cause rejected commits
Five workflows that push to main daily — analytics updates, content refreshes, queue state, Bluesky queue, article generator — each commit and push at roughly the same time window. When two of them try to push simultaneously, one gets:
error: failed to push some refs to 'github.com/...'
hint: Updates were rejected because the remote contains work that you do not have locally.
The naive fix is a force push. Don't do that — you'll silently drop the other workflow's commit. The correct fix is a rebase retry loop:
PUSH_OK=0
for attempt in 1 2 3; do
if git push; then
PUSH_OK=1; break
fi
echo "push attempt $attempt failed, rebasing..."
git pull --rebase || true
sleep $((attempt * 2))
done
[ "$PUSH_OK" = "1" ] || { echo "push failed after 3 attempts"; exit 1; }
Exponential backoff (2s, 4s, 6s) gives the competing workflow time to finish. --rebase preserves the current commit on top of the new main instead of creating a merge commit. The || true on the rebase handles the case where the rebase itself conflicts — that should be rare for workflows writing to non-overlapping files, but it prevents the loop from aborting on a transient conflict.
I now include this retry block in every workflow that modifies files and pushes to main. The three attempts are usually enough; I've never seen a fourth attempt be necessary with well-separated file writes.
None of these bugs produced visible errors on first glance. The push contention showed in workflow logs as a failed step, but the job still exited 0 because the retry succeeded. The double-publish was a duplicate YouTube video with a new ID. The timeout cancellation looked like a completed run. Top-of-hour drift just looked like "the post was a bit late."
The pattern: GitHub Actions is optimistic about success and tends to swallow soft failures. Anything time-sensitive or quota-sensitive needs explicit guards — randomized delays, concurrency locks, retry loops, timeout math — rather than assuming the scheduler delivers on the label.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)