Running eight GitHub Actions workflows daily across a monorepo sounds simple until the third one fires at 3:30am instead of midnight, the fourth quietly stops publishing videos at all, and the fifth silently cancels before it finishes. I've hit all four of these. Here's what each one looked like and how I fixed it.
Bug 1: Top-of-hour queue contention causes 3+ hour drift
The GitHub Actions job queue is global and shared. Scheduling a cron at 0 0 * * * (midnight UTC) puts you in the same batch as thousands of other repos. GitHub's documentation on scheduled events notes that scheduled workflows can be delayed during high loads — they don't commit to exact delivery time. During busy hours, GitHub's scheduler queues those jobs and delivers them when a runner is available. The result: I scheduled a Bluesky post for midnight UTC and watched it run at 03:45 UTC on a Monday — and at 03:29 the Saturday before it, and 03:48 the Sunday. A consistent ~3.5-hour drift, not a one-off.
For most CI tasks this is fine — a build running an hour late is rarely critical. For social media posting, a "midnight UTC" post landing at 3:30am matters.
The fix is off-minute scheduling with an earlier offset. Instead of 0 0 * * *, I use 37 7 * * * — 07:37 UTC, 16:37 JST nominal, offset early so that a moderate queue delay lands the post near my 17:00 JST target. The offset moves the job out of the peak batch, and the randomized start delay inside the workflow (sleep $(( RANDOM % 300 ))) keeps the exact minute from looking bot-generated.
Honesty about the outcome: it hasn't hit the target yet. The queue file's own posted_at timestamps for the week before writing cluster around 10:20–12:10 UTC — 19:20–21:10 JST, roughly three hours after the nominal fire time. Off-minute scheduling reduced the top-of-hour pile-up; it did not buy me a predictable delivery time. If the exact minute matters, cron is the wrong instrument.
Bug 2: A push trigger as the cadence engine
I had this in yt-publish.yml:
on:
push:
branches: [main]
paths:
- 'content/yt-queue/*.json'
workflow_dispatch:
The intent was to publish immediately when a new Short script landed on main. That makes publishing depend on the generation routine pushing to main every day — and that dependency is exactly what broke. Eight generated scripts drifted onto session branches (origin/claude/*) instead of main, so the push trigger never fired, the main queue went stale, and the backlog sat unpublished for about ten days while I assumed the channel was running.
The fix: replace the push trigger with a daily cron (0 21 * * *, 06:00 JST). A content queue with a daily consumer doesn't benefit from an immediate-publish trigger — it benefits from a predictable publish cadence that holds regardless of when scripts land. A backlog of files drains one per day; workflow_dispatch handles on-demand publishes when I need to jump the queue. I deliberately did not keep both triggers: on days the routine does push, push + cron would publish twice and break the one-Short-per-day policy.
If your workflow genuinely needs to trigger on push and schedule, use concurrency with cancel-in-progress: true to ensure only one run proceeds. But for a daily content queue, picking one trigger is cleaner. I also added a watchdog workflow that opens an issue if no new dated Short reaches main for two days, because the silent-stall failure mode is the one I actually hit.
Bug 3: Default job timeout cancels the job before it finishes
GitHub Actions jobs have a default timeout of 6 hours, but individual steps don't. If you specify timeout-minutes at the job level, that's the ceiling for the entire job. I set timeout-minutes: 5 on my Bluesky posting workflow, thinking: "post one tweet, can't take more than 5 minutes."
It can, when you add a random start delay.
The Bluesky workflow opens with a sleep $(( RANDOM % 300 )) — up to five minutes — to avoid posting at an obvious cron-exact timestamp. On a high-roll (say, 295 seconds), the delay alone consumed almost the entire budget. The QC check and actual post then bumped the total past the limit. One run was cancelled at 5 minutes 4 seconds, before it ever posted.
The symptom was a Bluesky queue that quietly stopped draining. A timed-out job shows up as cancelled, not failed, which is easy to miss when you're scanning a run list for red X marks — and unposted entries backed up from 7 to 12 while the channel looked fine from the outside.
The fix: set timeout-minutes to at least (max_delay / 60) + (actual_work_minutes) + (margin). For Bluesky with a 5-minute random delay and ~2 minutes of actual work, timeout-minutes: 12 gives comfortable headroom. If the timeout is genuinely the constraint (cost, billing minutes), reduce the random delay range instead.
Bug 4: Concurrent workflow pushes cause rejected commits
Six workflows that push to main daily — analytics updates, content refreshes, trends fetch, YouTube queue state, Bluesky queue, article publisher — each commit and push at roughly the same time window. When two of them try to push simultaneously, one gets:
error: failed to push some refs to 'github.com/...'
hint: Updates were rejected because the remote contains work that you do not have locally.
The naive fix is a force push. Don't do that — you'll silently drop the other workflow's commit. The correct fix is a rebase retry loop:
PUSH_OK=0
for attempt in 1 2 3; do
if git push; then
PUSH_OK=1; break
fi
echo "push attempt $attempt failed, rebasing..."
git pull --rebase || true
sleep $((attempt * 2))
done
[ "$PUSH_OK" = "1" ] || { echo "push failed after 3 attempts"; exit 1; }
Linear backoff (2s, 4s, 6s — attempt * 2, not a doubling) gives the competing workflow time to finish. --rebase preserves the current commit on top of the new main instead of creating a merge commit. The || true on the rebase handles the case where the rebase itself conflicts — that should be rare for workflows writing to non-overlapping files, but it prevents the loop from aborting on a transient conflict.
I include this retry block in every daily workflow that modifies files and pushes to main, with one straggler I haven't backfilled yet — yt-publish-longform.yml still does a single-attempt git pull --rebase origin main && git push origin main. The three attempts are usually enough; I've never seen a fourth attempt be necessary with well-separated file writes.
None of these bugs produced visible errors on first glance. The push contention showed in workflow logs as a failed step, but the job still exited 0 because the retry succeeded. The dead push trigger just meant no runs at all — nothing to look at. The timeout cancellation looked like a cancelled run rather than a failed one. Top-of-hour drift just looked like "the post was a bit late."
The pattern: GitHub Actions is optimistic about success and tends to swallow soft failures. Anything time-sensitive or quota-sensitive needs explicit guards — randomized delays, concurrency locks, retry loops, timeout math — rather than assuming the scheduler delivers on the label.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)