Running three sites from one monorepo means three data refreshes a day. The refresh matrix commits and pushes per app, so a full run can land three separate commits on main — and because every Cloudflare Pages project rebuilds on every push (more on that below), those three commits can fan out to as many as nine builds and deploys. All of it has to happen without stepping on anything else. What surprised me is how little scheduling machinery that actually needs. After six weeks of iteration starting from launch on April 23, the whole thing runs on one cron, one matrix, and two conventions. Here are the four patterns that stabilized it.
One cron and a serialized matrix
The naive approach is three separate workflows on 0 2 * * *. They fire simultaneously, race for the same API rate-limit windows, and one of them fails when HuggingFace or the GitHub API returns a 429 mid-run. The obvious fix is staggered offsets — but then you own the offset arithmetic forever, and the offsets are wrong the moment one source gets slower.
I collapsed all three into a single workflow instead. refresh-content.yml has exactly one schedule entry and runs the three apps as a matrix that never runs two at once:
on:
schedule:
- cron: "23 1 * * *" # ~10:23 JST nominal
workflow_dispatch: {}
concurrency:
group: refresh-content
cancel-in-progress: false
jobs:
refresh:
strategy:
fail-fast: false
max-parallel: 1
matrix:
app:
- ai-tools
- indie-games
- oss-alternatives
max-parallel: 1 is the whole trick. The second app starts when the first finishes, however long the first takes, so there is no offset to tune and no way for two jobs to compete for the same rate-limit window. fail-fast: false stops one site's ETL failure from cancelling the other two, and the concurrency group keeps a manual dispatch from overlapping the scheduled run.
The one cron detail worth stealing is the off-minute. GitHub Actions schedules use the standard five-field POSIX format, and they are heavily delayed at the top of the hour — I saw roughly 3.5 hours of drift pushing 02:00 UTC runs past 05:30 UTC. Scheduling at :23 sidesteps the global queue spike and the run lands close to when I meant it to.
The Steam ETL behind the indie games site also has its own throttle inside the script rather than in the schedule — await sleep(250) between requests, annotated in the source as aggressive but usually fine against Steam's roughly 200-requests-per-5-minutes ceiling. Nothing about that depends on start time. (itch.io entries don't come from this ETL at all; those are added by hand with apps/indie-games/tools/add-itch-game.mjs.)
Skip markers in commit messages
Several workflows here commit back to main. publish-articles.yml regenerates OG images and then records the published URLs into frontmatter; the Bluesky queue workflow marks a post as sent. Both of those pushes match the same path filter that triggered the workflow in the first place, which is an infinite loop.
The guard is a marker in the commit subject plus a job-level condition:
jobs:
publish:
if: "!contains(github.event.head_commit.message, '[skip publish-articles]')"
with the bot's own commits carrying it:
chore(og): regenerate OG + summary images [skip publish-articles]
chore(articles): record published URLs [skip publish-articles]
Each workflow owns its own marker — [skip publish-articles], [skip bluesky-queue] — so one workflow's bot commit doesn't accidentally mute another's trigger.
It's worth being precise about what this is: a self-trigger guard, not a general isolation layer between pipelines. The ETL commits carry no marker at all. They land as plain chore(ai-tools): refresh content 2026-06-03, because refresh-content.yml is cron-only and has nothing to guard against, and because the files it writes (apps/*/src/data/*.json) fall outside the article workflow's path filter anyway. I originally assumed I'd need the marker on every bot commit; I didn't.
Path filters on push-triggered workflows
Path filters are the second convention, and the thing I had to unlearn is that they don't gate deploys. Since the 2026-05-07 move from Vercel to Cloudflare Pages, deployment is entirely outside GitHub Actions: each of the three Cloudflare Pages projects watches main through the git integration and builds itself on every push. There is no per-site deploy workflow in .github/workflows/ to filter, and no way to stop a Cloudflare build from the Actions side. Verifying a deploy is therefore a separate concern — that's what the post-deploy checks are for.
What path filters gate is which workflows wake up. publish-articles.yml only runs when article sources change:
on:
push:
branches: [main]
paths:
- "content/articles/**/*.md"
- "packages/publish/**"
- ".github/workflows/publish-articles.yml"
The three YouTube publishers watch only their own queue directories in the same way — content/yt-queue/*.json, content/yt-longform-queue/*.json, and content/yt-queue-samurai-princess/*.json — so a queue file landing for one never wakes the others.
The filter that actually saved me money was the inverse one, on CI:
on:
push:
branches: [main]
paths-ignore:
- "content/**"
pull_request:
branches: [main]
CI builds four apps in parallel: the three directory sites plus the dashboard. The bots commit under content/ several times a day — articles, YouTube queue moves, Bluesky queue refills — and every one of those was firing a full four-app build that could not tell me anything new. paths-ignore: content/** on push removed the single largest consumer of my Actions minutes. Pull requests still get full CI, because paths-ignore is declared only on the push trigger. Deploy health is unaffected either way: Cloudflare builds every push regardless of what Actions decides to skip.
Manual dispatch — and what mine still doesn't do
Every scheduled workflow here also carries workflow_dispatch, which is what I reach for when an ETL run fails or I want a refresh right after adding a data source.
What refresh-content.yml doesn't have is a selector:
workflow_dispatch: {}
No inputs. A manual dispatch reruns the whole three-app matrix, and there's no dry-run flag that would fetch and parse without writing anything. When only one app matters I re-run that single matrix job from the run page instead, which works but is a worse interface than a dropdown.
The pattern I want is already in this repo — yt-publish.yml takes typed dispatch inputs, a file path and a boolean disable_pexels, and the Actions UI renders them properly:
workflow_dispatch:
inputs:
file:
description: 'queue file path. Empty = oldest unprocessed'
required: false
default: ''
disable_pexels:
description: 'Skip Pexels stock bg (A/B test for shadowban hypothesis)'
required: false
default: 'false'
type: boolean
A choice input for the target site and a boolean dry_run on the refresh workflow would be maybe fifteen lines. I just haven't ported it.
When patterns conflict
These four work together, but they fail quietly when they disagree, and the failure always has the same shape: something you expected to run didn't.
The interaction most likely to confuse me is that a path filter and a skip marker can both suppress the same workflow. If a bot commit touches only paths outside a workflow's filter and carries that workflow's marker, you get a double block — usually correct, occasionally baffling when you're staring at an empty Actions tab.
I don't have a doc for this, which is the honest gap in the setup. What I have is a habit: before editing any workflow file, read its on: block and its job-level if:, because all of the routing lives in the first thirty lines. That scales because there are only about a dozen workflow files. A .github/WORKFLOWS.md mapping trigger, marker, and filter per workflow is the obvious thing to write, and I keep not writing it.
Six weeks in, the scheduling is stable enough that I don't think about it most days. The patterns aren't clever. They're just explicit enough that I can read any workflow file and reconstruct exactly when and why it will run.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)