The GitHub Actions free-tier billing block hit in May. I cleared it, tightened up the pipeline, and it hit again in late June. The software optimization pass I did after the first block cut roughly 280 minutes per month by adding pip requirements caching, removing npm global reinstalls, and narrowing trigger scopes so content commits didn't fire a full CI build. It worked for a few weeks.
Then the pipeline kept growing. I added a daily YouTube analytics job, then a Bluesky queue refill job, then a weekly long-form video pipeline. Each one was individually cheap — 2–4 minutes per run — but daily crons multiply fast. A third exhaustion was coming, and I knew the software-only approach had a ceiling.
The permanent fix was moving the two highest-volume jobs off GitHub-hosted runners entirely.
Why self-hosted instead of paying GitHub
The calculation came down to cost trajectory and failure mode.
On cost: GitHub's private-repo free tier is 2,000 Linux minutes per month. The original CI design runs a 4-app matrix build on every code push, plus three content-refresh jobs daily. That's the dominant volume. GitHub charges $0.008/minute for Linux and $0.016/minute for macOS once you clear the free tier. At 2,000 minutes of overflow per month, that's $16–32/month in compute. The rest of the stack costs $2.25/month — three domains at Cloudflare Registrar, and $0 for hosting since the sites moved to Cloudflare Pages in May and Vercel Pro was cancelled. So Actions overflow would cost several times more than everything else in the project combined, and the project has no revenue yet.
On failure mode: a self-hosted runner introduces a new failure mode — if the machine is offline when a scheduled job fires, the job doesn't run. That's acceptable for CI builds (a code build skipping one day is a minor inconvenience) and for content-refresh (missing one day's data update doesn't affect readers). It's not acceptable for the publish pipeline — if an article or video that was supposed to publish doesn't because the runner was asleep, that's externally visible. So the scope matters.
The decision: move CI builds and the daily content-refresh to self-hosted. Leave article publishing, Bluesky queue posts, and all failure-notify jobs on GitHub-hosted Ubuntu, where GitHub guarantees execution.
The migration: two lines changed, several surprises followed
The core yaml change was small:
# Before
runs-on: ubuntu-latest
# After
runs-on: [self-hosted, macos]
GitHub routes jobs to runners that match all labels in the array. The Mac has self-hosted and macos registered in the repository Actions settings. Adding a third descriptive label (mac-coo-runner) makes the config self-documenting — when I look at the workflow a month later, I know which machine this is.
The first thing that needed adjusting was the timeout. Default GitHub jobs have a 6-hour hard limit, but I had explicit timeout-minutes: 10 in most jobs. On a cold self-hosted runner — first run after a restart — pnpm needs to download and populate its store, Node.js binaries need to be fetched by actions/setup-node, and pnpm install --frozen-lockfile for a 4-app Astro monorepo takes 8–10 minutes. A 10-minute timeout killed the cold run. Setting it to 20 minutes absorbed the cold start. Subsequent warm runs complete in 2–3 minutes.
The content-refresh workflow also needed a max-parallel: 1 addition — I'll come to that in the push-race section. But the most time-consuming problem was the pnpm cache hang, which showed up as jobs silently running for 30+ minutes without finishing.
The pnpm cache hang: 30+ minutes for a post-step no one needed
The setup-node step I was using had cache: 'pnpm' enabled:
- uses: actions/setup-node@v4
with:
node-version: 22
cache: 'pnpm'
On a GitHub-hosted runner, this is beneficial: restore the pnpm store from GitHub's cache before install, upload it after the job completes. It shaves 30–60 seconds off installs by reusing previously downloaded packages.
On a self-hosted runner with a persistent filesystem, it becomes a problem.
The runner already has a persistent pnpm store on disk — on macOS it lives at /Users/a/Library/pnpm/store. The store doesn't disappear between runs. When I install packages on run 1, they're cached on disk for run 2 automatically. The cache: 'pnpm' restore step does nothing useful (the store is already there and up to date), but the post-step — which runs after the job completes — tries to upload the entire local pnpm store to GitHub's Actions cache API.
That upload was the hang. The store for a project with 4 Astro apps, shared TypeScript packages, Python tooling, and dev dependencies accumulated to several hundred megabytes. The post-step was hitting timeouts or rate limits uploading to GitHub's cache API, silently stalling the job for 20–40 minutes before eventually failing or timing out. The job logs showed the build completing cleanly, then the progress bar on the cache upload step hanging with no error message.
Removing cache: 'pnpm' entirely fixed it:
- uses: actions/setup-node@v4
with:
node-version: 22
# No cache: 'pnpm' — self-hosted runner persists pnpm store at
# /Users/a/Library/pnpm/store between runs. actions/cache post-step
# tries to upload the full store to GitHub's cache API and hangs.
After the fix, jobs that were running 30+ minutes finished in 3–4 minutes. The local store gives the full caching benefit without the upload overhead.
This applies to any actions/cache wrapper for a package manager on a persistent self-hosted runner. The pattern that helps on ephemeral hosted runners (restore from cache → use → upload back) adds noise on persistent runners. The cache is already on disk. The comment in the yaml is important here: future me will wonder why cache: 'pnpm' isn't there and be tempted to add it back.
Managing git push races in the content-refresh matrix
The content-refresh workflow runs three apps in a matrix — ai-tools, indie-games, oss-alternatives — each one fetching API data, generating content JSON, committing, and pushing to main.
On GitHub-hosted runners, each matrix job starts from a fresh checkout and can run in parallel. On a single self-hosted Mac, they queue behind each other by default (only one job can run at a time on one machine). But GitHub's job scheduler doesn't guarantee the checkout for job N happens after job N-1's push. If job 2 checks out main before job 1 finishes pushing, job 2's commit will diverge from job 1's commit, and the push will fail as non-fast-forward.
The fix has two parts.
First, max-parallel: 1 in the matrix strategy ensures jobs start sequentially — job 2 doesn't start until job 1 completes, including its push:
strategy:
fail-fast: false
max-parallel: 1
matrix:
app: [ai-tools, indie-games, oss-alternatives]
Second, a pull step before each commit handles the case where another workflow (like the daily yt-analytics job) pushed to main between the start of this job and the commit step:
- name: Pull latest
run: git pull --ff-only
- name: Commit and push
run: |
git add apps/${{ matrix.app }}/src/data/*.json
if git diff --cached --quiet; then exit 0; fi
git commit -m "chore(${{ matrix.app }}): refresh content $(date -u +'%Y-%m-%d')"
for attempt in 1 2 3; do
if git push; then break; fi
git pull --rebase || true
sleep $((attempt * 2))
done
The retry loop on push is worth having even with max-parallel: 1. Other workflows — yt-analytics, bluesky-queue, trends-fetch — all push to main on their own schedules. The cron timing stagger I use means these rarely overlap, but rarely isn't never. The retry loop handles the overlap case without manual intervention. The same pattern appears in several other ETL workflows across the project.
Keeping downstream jobs resilient on hosted runners
After moving the main refresh job to self-hosted, I kept two downstream jobs on ubuntu-latest:
jobs:
refresh:
runs-on: [self-hosted, macos]
indexnow:
needs: refresh
if: always() && needs.refresh.result != 'cancelled'
runs-on: ubuntu-latest
notify-failure:
needs: refresh
if: failure()
runs-on: ubuntu-latest
The reason is failure-mode composition. If the Mac is offline when the cron fires, refresh enters a failed state (runner not found, job queued and timed out). I want notify-failure to send a Discord alert in that case. But if notify-failure is also on the Mac, it never runs — the machine that would have run it is the machine that's offline.
By keeping both downstream jobs on hosted Ubuntu, GitHub guarantees they run even when the self-hosted runner is unavailable. The if: failure() condition on notify-failure catches both "the refresh step actually failed" and "the runner was offline and the job errored." The pipeline health monitor I run separately also watches for failed workflow runs via GitHub API, so this is a belt-and-suspenders setup.
The indexnow job stays on hosted for similar reasons — it pings IndexNow with updated content URLs, and I'd rather have that ping fail gracefully (hosted job errors) than silently skip (no runner available). The four-monitoring-tools setup for the three sites watches traffic and uptime, not search-engine submissions — nothing in it would notice a skipped IndexNow ping — so the hosted guarantee is doing the work here.
What worked, what didn't, what I'd do differently
What worked well: the pnpm store persistence on the self-hosted runner is genuinely better than the actions/cache restore cycle on hosted. After the first cold run, every subsequent run starts from a warm store and installs in under 30 seconds. I also appreciate having a machine I own running the jobs — the failure mode (Mac asleep) is predictable and doesn't depend on GitHub's runner availability.
What didn't work initially: I underestimated how much the cache: 'pnpm' post-step would affect things. The option is documented as a straightforward optimization, not a potential hang source. Spending several hours debugging a silently-stalling job before realizing the post-step was the cause was frustrating. The clue was the post-step progress bar hanging — it's easy to miss if you're not looking at the right section of the job log.
What I'd do differently: audit for cache: options in any step immediately when migrating to self-hosted. The pattern is consistent: actions/cache and cache: options in setup-* steps are designed for ephemeral hosted runners. On persistent self-hosted runners, they add upload latency without restore benefit. I'd also explicitly set max-parallel: 1 for any matrix that writes to the repository from the start, not as a fix after seeing push failures.
FAQ
Which jobs should stay on GitHub-hosted runners?
Publish pipelines, external notification jobs, and anything where a miss is externally visible. For this project: article publishing to Dev.to and Hashnode, the Bluesky queue drain, and all failure-notify jobs. These need guaranteed execution independent of whether a specific machine is online.
How do you register a macOS machine as a self-hosted runner?
Repository → Settings → Actions → Runners → New self-hosted runner → macOS. GitHub provides a shell script that downloads the runner agent, configures it with a registration token, and registers it with the repository. You run ./run.sh in the foreground or install it as a macOS launch agent to start automatically on login. Label it with at least self-hosted and the OS name; any additional labels are optional but useful for documentation. The GitHub self-hosted runners documentation covers the full registration process and security considerations (note that self-hosted runners on public repositories can be a security risk — only use them on private repos).
Does this approach work for parallel matrix jobs?
A single self-hosted runner serializes matrix jobs regardless of max-parallel setting — there's only one machine. For jobs where order and push order matter (like content-refresh), serialization is fine and max-parallel: 1 makes it explicit. If you needed genuinely parallel matrix builds, you'd need multiple registered runners or a cloud-hosted runner in your organization.
What about secrets access on a self-hosted runner?
GitHub Actions secrets work identically on self-hosted runners — the runner agent fetches them from GitHub on each job run, the same way hosted runners do. I don't store secrets directly on the Mac; they come through the Actions secrets mechanism at job time. The one difference is that GITHUB_TOKEN on self-hosted has the same permissions as on hosted (controlled by the permissions: block in the workflow yaml), so no special configuration is needed.
How do you handle the runner being offline?
For jobs that must run, I don't use self-hosted. For jobs where a miss is acceptable, I accept the risk. The notify-failure job on hosted Ubuntu catches most cases. The larger pipeline health monitor runs daily, asks the GitHub Actions API for content-workflow runs that concluded in failure, and opens or updates a single deduplicated GitHub Issue — closing it again once things are healthy.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)