DEV Community: cynthia wahome

I Went to Write Two Tests and Ended Up Fixing the App

cynthia wahome — Fri, 13 Nov 2026 11:58:31 +0000

The job sounded small. Two issues, same shape: write a Playwright end-to-end test. Log in, report something, check the confirmation, report it again, check the message. Maybe an afternoon each.

Four and a half months later, I had opened four pull requests, and not one of them was a test. Not yet. This is the story of why, and why I think it's the best argument for E2E tests I've ever lived through.

The project is Quote.Vote, an open-source, text-only platform for thoughtful dialogue (Next.js, an Express and Apollo GraphQL API, MongoDB, and a Prisma migration in progress). I'm a volunteer contributor there, and this is where things stand as I write this (November 12): the two bug-fix pull requests are merged, and the test pull request is open as a draft.

The assignment

On June 22 I was assigned two issues:

E2E-MOD-006: report a post. A logged-in user reports content, picks a reason, sees a confirmation, and a repeat report is handled gracefully.
E2E-MOD-007: report a user as a bot. Same idea, from a profile.

Before you can test anything, you have to run it. In July I did the local setup, and the app greeted me with a profile page that said "Invalid user". The server had answered 200 OK with {"data":{"user":null}}. No error anywhere. The GraphQL schema declared a user(username) query, and no resolver existed for it, so Apollo returned null with a straight face.

I fixed that, and I also found a seed script that wrote to a different database than the one the server read. Two small pull requests, both merged. Then the maintainers told me what I'd half suspected: the backend for reporting itself didn't exist yet. They opened a separate issue to build it, marked mine "Not Ready", and I stepped away.

Coming back after four months

In early November the maintainers asked if I was ready to pick the issues up again. I said yes, and my laptop immediately said no. In one day I hit:

A local clone that was hundreds of files behind the real repository, so git status showed a wall of changes I hadn't made.
A database container that crash-looped, because the "latest" Mongo image had moved a major version past my old data volume.
A backend that needed a MongoDB replica set (Prisma uses transactions for writes), which was documented in a migration doc and not in the setup steps.
A browser that couldn't talk to the API, because an old setting still allowed only the previous frontend port.
A main branch whose frontend and backend deliberately don't match, because the real work lives on staging.

I wrote each one down in a notes file as I went. (Several turned out to be documentation gaps worth reporting.) None of that is the story, but it's worth saying: the first test you write is for your own setup.

Lesson one: read the issue against the real screen

The issue said: select a report reason. I opened the app. Reporting a post is a single click, with no reason picker. Reporting a user is a confirm dialog that says "Report Bot", and the reason picker component exists in the code but is never shown.

I could have built a test for the screen in the issue, one that could never pass. Instead I took screenshots and asked, with the evidence attached. The answer came back within days: test the flow that exists, and treat a reason picker as a separate product decision. A few minutes of comparing, and a test written against a screen that doesn't exist, avoided.

The bug: "Post not found" for a post that exists

I logged in as a second user, opened a post, and clicked Report post. The app said:

Post not found

The post was on the screen in front of me.

The reporting code checked for a duplicate with a Prisma filter, roughly "update this post unless reportedBy already contains this user."

updateMany({
  where: { id: postId, NOT: { reportedBy: { has: userId } } },
  data: { reportedBy: { push: userId }, reported: { increment: 1 } },
});

Posts created through the app's normal "create post" path have no reportedBy field at all. MongoDB leaves an unset list absent. And a filter on a list can't match a field that isn't there. So the update changed nothing, the code assumed "nothing changed means the post doesn't exist", and it threw NOT_FOUND.

I proved it on one post in the database: add an empty reportedBy: [], and the report worked, and the second report said "already reported". Remove the field, and "Post not found" came back.

A missing field is not an empty list. Humans read them the same. Databases don't.

A maintainer suggested the fix: one raw MongoDB update, where $ne also matches a missing field, and $push and $inc create absent fields.

q: { _id: { $oid: postId }, reportedBy: { $ne: userId } },
u: { $push: { reportedBy: userId }, $inc: { reported: 1 } },

The check and the write stay one atomic step, so I tested that too: five simultaneous reports from one user gave one success and four "already reported", and the database counted one.

Why had the unit tests stayed green? They mock the database, and every fake post in them already had reportedBy: []. The tests only ever tried the one shape that works.

The second finding: a repeat report is not a failure

Reporting a user twice produced a red error toast, a GraphQL error in the browser console, and a dialog that stayed open. The maintainers' view was fair: you already told us, that's not an error. The server now returns an ALREADY_REPORTED result and the app shows a calm, neutral message.

The third finding: tests that never ran

To change that dialog I looked at its existing tests. One ("reports user successfully") finished in 12 milliseconds. That's too fast to open a dialog.

They imported MockedProvider from Apollo, which Apollo Client 4 removed. In the test it was undefined, so the component crashed into its error boundary, and the tests passed through an "else" branch that expected the error screen. They had been passing without ever testing the feature.

I left that file alone and wrote new tests in their own file that drive the real menu and dialog. A test that passes in 12 ms deserves suspicion.

Accidental test-driven development

Test-driven development (TDD) is a simple loop: write a test that fails (red), write the smallest change that makes it pass (green), then tidy up. I'd always treated it like a gym membership: admirable, and mostly for other people.

Here I did it without planning to. The issue's steps were the failing test: follow them by hand and the confirmation never appears, because the app says "Post not found". So for each fix I wrote the unit test first and watched it fail (five failures for the post fix; two backend and three frontend for the bot fix), then changed the code until they passed (38 in the post test file, 55 in the user one).

The red step also caught my mistakes. My first version of a new frontend test failed for the wrong reason: I had copied the old tests' setup, which crashed the component. That's how I found out the old setup was broken in the first place.

What I'd tell a TDD skeptic: the failing test isn't a ritual, it's the bug report written in code. If you can't make the test fail the way the bug does, you haven't understood the bug yet.

And then, the actual tests

Finally, the thing I was assigned. Playwright drives a real Chrome browser like a very fast, very literal person. Headless means no window (fast, what CI uses). Headed means you can watch it. It's faster than you'd believe.

The design had one wrinkle: you can report a given post, or a given user as a bot, once per pair of accounts. A test that passes today would fail forever tomorrow. So each test creates a brand-new user through a small local-only seed script, and the tests repeat on the same database with nothing to reset. The spec also skips unless the API is explicitly local, because it writes test accounts.

The results from my machine: all 3 tests pass on desktop and 3 on mobile, and a second run on the same database passed again, 6 of 6. One more small thing surfaced: a failed report shows the same error toast twice, so Playwright's strict matching complained about two identical elements. I noted it as a finding and made the test accept the first copy.

What happens next

Both bug-fix pull requests were merged on November 12. The test pull request needed both, so it waited on a local branch, and then moved onto the updated staging with a single git rebase --onto (my commit was the only one that moved, since the maintainers' merge commits had kept the fixes intact). It's open now as a draft, and I'm adding the evidence: a Playwright report with all 6 results, two short screen recordings, and the step-by-step view of each test.

Here's a small tip I learned on the way: Playwright records video only for failures by default, and it starts recording before your setup finishes, so a raw video of a passing test opens on several seconds of a blank page. Run it in slow motion (slowMo), record every test, and trim the empty start. A reviewer will actually watch a seven-second clip.

One more thing to watch: GitHub only auto-closes issues when a pull request merges into the default branch. Mine targets staging, so the two issues won't close by themselves when it merges. I've mentioned that in the issue threads so nobody is surprised.

I'll update this post again when the test pull request is merged.

What I'd tell the next contributor

Treat a test task as a product review. You're the first user the feature has had.
Compare the issue with the real screen before writing a line. Attach screenshots when they disagree.
Prove a bug before you report it. "Five simultaneous requests, one success" is harder to argue with than "I think there's a race".
Be suspicious of fast, green tests.
Write down your setup problems. Your notes file is a documentation PR waiting to happen.
Ask before you build, and in the open. Maintainers answered in days, with a design, because I came with evidence.

If you're about to write E2E tests for a half-finished feature

Run the feature by hand first, as a user, with a second account.
Write down every way the screen differs from the issue.
Reproduce each bug in the smallest way you can, and keep the proof.
Fix the bugs in small separate pull requests, and say which one the test depends on.
Make the test independent of old data: a fresh user or fresh record each run.
Add a guard so it can't run against anything but a local server.
Run it twice on the same database before you trust it.

FAQ

What is test-driven development (TDD)?
A loop of three steps: write a test that fails (red), write the smallest code that makes it pass (green), then clean up (refactor). The failing test comes first, so it proves the bug or the missing feature exists before you fix it.

Can E2E tests find bugs that unit tests miss?
Yes. Unit tests mock the database and the network, so they only exercise the shapes of data you imagined. An end-to-end test uses real data created the real way, which is how a post with no reportedBy field showed up.

What's the difference between headed and headless in Playwright?
Headless runs the browser with no visible window, which is faster and is what CI servers use. Headed shows the window so you can watch. Playwright's --debug and --ui modes let you step through a test one action at a time.

Why does a MongoDB filter not match a missing field?
A list filter such as "does not contain X" is evaluated against an array. If the field doesn't exist there is no array, and Prisma's list filters don't match it. Raw MongoDB operators such as $ne do match a missing field, and $push and $inc create absent fields.

How do you test something that can only happen once per user?
Give each test its own fresh user, created by a small seed script, so the test never depends on earlier runs. Then it can repeat on the same database without cleanup.

How do you come back to an open-source issue after months away?
Re-read the whole thread first, since things change fast. Re-sync your clone and check which branch the project is currently developing on. Compare the issue text with the current app, and ask in the thread before building anything the screen contradicts.

Why would a test pass in 12 milliseconds?
Usually because it never ran the code it claims to test. A common cause is a test helper that no longer exists in an upgraded library, so the component falls into an error boundary and the assertions take a fallback path.

Update (Nov 13): @maciek_chmura_building asked two good questions about fixtures and the double toast. I tested both and wrote up the results, with screenshots, in this reply.

In short: both the component and the global error link show the toast, and three more post lists are missing on new posts but handled safely.

Backend engineer. I write about systems, architecture, and what I learn by poking at things. Portfolio 🚀

One Stolen Token Should Not Ship Your App: A Two-Person Release Gate in GitHub Actions

cynthia wahome — Fri, 13 Nov 2026 11:24:47 +0000

One Stolen Token Should Not Ship Your App

An attacker got hold of one admin's access token. Within minutes they had pushed a workflow file into every repository in the organisation, each one tailored to that repo, each one written to read the repo's secrets and send them out.

We contained it. Tokens revoked, secrets rotated, signatures enforced. I've written about that part separately.

This post is about the question that stayed with me afterwards, looking at our mobile repo:

If one token can push code, and pushing code can publish to the Play Store, then one token can ship an app to our users.

That had to stop being true. This is the release gate I built so that it takes two named humans, and so that a merge is no longer a release.

The problem, in two parts

One: a merge was a release. Merging to main built and published both of our Android apps (two apps, one codebase). A fix for one app forced a rebuild and store resubmission of the other, in the middle of its review, for nothing.

Two: one person was enough. Anyone with write access, or anything holding their token, could start the publish.

I wanted three properties:

"Code is merged" and "this app is released" are different events.
A release needs a request from one person and an approval from a different, named person.
There is exactly one door to the store credentials, and it cannot be opened from outside.

The template is cooked and now in GitHub 🚀

check it out here

The design: three workflows

1. Request

Run manually. It publishes nothing and builds nothing for the store. It only:

writes a small JSON file (which app, which store track, who asked) into a release-requests/ folder on a new branch, and
opens a pull request into a dedicated branch called release-gate.

The important detail: the pull request is opened by the automation, not by the person.

2. Approve

Runs whenever a review is submitted anywhere in the repo, then narrows down hard. It continues only if all of these hold:

the pull request targets release-gate;
the review is an approval, not a comment;
the pull request was authored by the automation;
the approver is on a fixed list of names;
the approver is not the person named in the request file.

Anything it cannot read is a "no". A missing file, a malformed file, a missing field: stop, comment on the pull request, publish nothing. Fail closed, not open.

3. Publish

The only workflow that touches signing keys and store credentials. It has exactly one trigger:

on:
  workflow_call:

workflow_call means "another workflow may call me". There is no manual-run button, no push trigger, no pull-request trigger. The only caller is the Approve workflow, after its checks pass.

Why the request must be authored by the bot

This was the hole I found while designing it.

Without that rule, anyone with write access could open their own pull request into release-gate, with a hand-written request file naming someone else as the requester, and then approve it themselves. The self-approval check would pass, because it trusts the file.

The file is only trustworthy if our own script wrote it. Our script only ever runs when the automation opens the pull request. So "authored by the bot" is what makes "approver is not requester" mean anything.

The alarm that guards the door

The whole design rests on Publish having one trigger. One well-meaning edit ("let me add a manual trigger for a quick test") would add a second door that skips approval, and nothing would look broken.

So normal CI, on every pull request, reads that file's on: block and fails unless the only trigger is workflow_call:

triggers="$(awk '/^on:/{flag=1; next} /^[a-zA-Z]/{flag=0} flag && /^  [a-zA-Z_]+:/{print}' "$file" \
  | sed 's/^  //;s/:.*//')"

if [[ "$triggers" != "workflow_call" ]]; then
  echo "::error::$file must have ONLY 'workflow_call' as a trigger."
  exit 1
fi

It looks only at keys indented exactly two spaces under on:, so the inputs declared inside workflow_call are not mistaken for triggers. It is a hard failure on purpose.

The approver list is hard-coded, on purpose

The list of who may approve lives inside the workflow file, not read from CODEOWNERS at run time. If it were read live, a quiet edit to one file would change who can release to the store. Keeping it in the workflow means changing the approvers is a visible diff in the most scrutinised file in the repo.

Four bugs I only found by running it

Reading the YAML told me it was correct. Running it told me otherwise.

1. The reusable workflow could not be found. I referenced Publish by a relative path. On a review event, GitHub resolves that path against the pull request's merge ref, and the frozen release-gate branch has no such file. Result: the run failed at start-up with no jobs. Fix: reference it by repository and branch explicitly.

2. Permissions are capped by the caller. Publish asked for write permissions; the calling job declared none, so it had read-only defaults, and GitHub rejected the whole workflow. A called workflow can never have more permission than the job that calls it.

3. It read the wrong file. I took "the first changed file" in the pull request as the request file. Because release-gate is a stale branch, every file changed since shows up in the diff, and the first one alphabetically was not my JSON. Fix: select by path prefix, and page through the full file list.

4. A failed publish burned a valid approval. The first real release died on runner disk space. My workflow closed the request anyway, so two people had to go through the whole cycle again for an infrastructure hiccup. Now a failed publish leaves the request open. Re-running replays the same approval against the same commit, so it can only publish what was already approved.

And one for humans: someone merged a request in good faith and nothing happened, because merging does nothing here. A workflow that never runs cannot explain itself. So there is now a tiny workflow whose only job is to comment: "merging does not publish; approve the request instead".

What this stops, and what it does not

Honesty section. A gate you overestimate is worse than no gate.

It stops:

an accidental release from a merge;
one person, or one stolen token, starting a store publish through the pipeline;
self-approval, including the forged-request variant.

It does not stop, on its own:

someone with write access editing the workflows themselves. The trigger guard and code review make that visible; branch protection with required code-owner review on the workflows folder is what makes it enforceable.
a workflow pushed to another branch reading repository-level secrets. If your signing keys are plain repository secrets, any workflow in the repo can ask for them.

What I'd add next

The plan for the template version of this:

Move store and signing secrets into a GitHub Environment (a named group of secrets with its own rules), restricted to the protected branch. Then the secrets are not available to a workflow on an arbitrary branch.
Require code-owner review for anything under .github/workflows/.
Compare with GitHub's built-in environment reviewers. If your plan includes them for your repo type, use them, and keep the bot-authored request as the audit trail. Check GitHub's current documentation for what your plan allows on private repositories.

Checklist: a two-person release gate

Separate "merge" from "release". A merge publishes nothing.
One publish workflow, one trigger: workflow_call.
A CI check that fails if that file gains another trigger.
Release requests are pull requests opened by the automation, carrying a small metadata file.
Approval checks: right branch, real approval, bot author, named approver, approver is not requester.
Fail closed on anything unreadable.
Pass user inputs to scripts as environment variables, never pasted into the script text.
Leave a failed publish retryable without a new approval.
Tell humans what merging does not do.
Put the secrets behind an environment, and protect the workflows folder.

FAQ

What is a two-person rule in CI/CD?
A control where a sensitive action, such as publishing to an app store, needs one person to request it and a different authorised person to approve it. No single account, or single stolen credential, can complete it alone.

What does workflow_call do in GitHub Actions?
It makes a workflow reusable: other workflows can call it with uses:. If it is the only trigger, the workflow cannot be started by a push, a pull request or the manual-run button.

Can someone approve their own release with this design?
No. The approval step compares the approver with the requester recorded in a file that only the automation writes, and it ignores pull requests that a human opened directly.

Why not just protect the main branch?
Branch protection controls what gets merged. It does not separate "merged" from "released", and it does not decide who may publish which app to which store track.

Does this replace GitHub Environments with required reviewers?
No, they work together. Environments restrict who can reach the secrets; this design adds a request record, a self-approval check and per-app releases. Where environment reviewers are available on your plan, use both.

How do you release one app from a two-app codebase without rebuilding the other?
Make the app an input of the release request. The publish workflow receives that single value and builds only that app.

What happens if the publish fails after approval?
The request stays open. Re-running the failed run replays the same approval on the same commit, so nothing new can slip in, and nobody has to approve twice.

Backend engineer. I write about systems, architecture, and what I learn by poking at things. Portfolio

Automating Android Play Store Releases, Part 5: The Bug That Shipped Live

cynthia wahome — Fri, 13 Nov 2026 09:57:12 +0000

Part 5 of a 5-part series on automating a multi-app Android release pipeline. Part 4 → ended on the first real production release going out cleanly. This part covers what showed up right after — a bug that had nothing to do with signing, versioning, tracks, or storage, and everything to do with a comment that stopped being true nine minutes after it was written.

TL;DR

Right after the first real production release, both apps started showing the same login screen — staff's.
Root cause: this org has exactly one self-hosted runner. Both apps build sequentially, on the same disk, sharing the same Metro bundler cache.
A comment already in the codebase claimed this couldn't happen, because staff and student "each get their own runner." That was true — for eight days, on a different runner setup, before a later, unrelated commit quietly removed the isolation the comment depended on. Nobody went back to check whether the comment still held.
The real fix wasn't isolating the runners again — it was making the cache actually aware that "which app" is something that should invalidate it. Metro's own cacheVersion config field does exactly that.
The fix stops it happening again. It didn't undo what had already shipped — both apps needed a fresh build and a fresh release after the fix landed.

Live, and Showing the Wrong Screen

The message came in a few hours after the first real production release went out: "both apps are showing me the login screen for staff app." Not a color slightly off, not an icon mismatch — the student app, downloaded fresh from the Play Store, opened straight to staff's actual login screen.

That's a different category of bug than anything else in this series. Signing, versioning, tracks, storage — all of that could delay a release or misreport one. This was the first time the pipeline had actually shipped the wrong thing to a real user.

Ruling Out the Obvious, Then Finding the Comment That Explained Everything — Except It Was Wrong

The first instinct was styling — a leftover theme value, a stale asset. That theory didn't survive five minutes: this wasn't a color, it was an entire screen. Something was bundling one app's actual code into the other's build.

Then a comment already sitting in the workflow file for this exact pipeline turned up, and it looked like the whole answer:

# Staff and student each get their own runner -- previously both built
# sequentially on ONE shared machine, which meant Metro's on-disk bundler
# cache (which staff's build populates first) could be reused for
# student's build right after, even though EXPO_PUBLIC_ROUTE had changed.
# ...
# Separate runners share no disk and no cache, so this can't happen --
# the same reason the Play Store release pipeline never could.

Word for word, this described the exact symptom being debugged — and confidently declared it impossible. The only way to actually trust that was to check it directly instead of reading it and moving on:

$ gh api orgs/<org>/actions/runners --jq '.runners[] | {name, status}'
{"name": "<our-only-runner>", "status": "online"}

One runner. Not two. The comment's entire premise — that staff and student physically couldn't share a disk — was false, right now, in production, while confidently telling anyone who read it the opposite.

Why It Was Specifically Staff's Screen, on Both Apps

With the real constraint confirmed (one shared machine, not two), the mechanism became straightforward. metro.config.js — the config for Metro, the tool that actually bundles this app's JavaScript — picks which app's styling to bundle based on an environment variable:

const ROUTE = process.env.EXPO_PUBLIC_ROUTE ?? "staff";

Metro also keeps its own on-disk cache of already-bundled work, to avoid redoing it. That cache key was never told that ROUTE was something worth caring about. The build matrix runs staff first, then student, back to back, on the one machine that exists. Staff's build populates Metro's cache. Student's build starts moments later, on the same disk — and as far as Metro's cache key is concerned, nothing relevant has changed. It hands back staff's already-bundled output.

That's the specific detail that makes this bug make sense rather than look random: it wasn't a coin-flip swap between the two apps. Staff always builds first, so staff's bundle is always what gets silently reused — which is exactly why both apps ended up showing staff's screen, not a symmetric mix-up.

The Comment Was True. Then It Wasn't, Nine Minutes Later.

This is the part worth sitting with longer than the fix itself. Checking the actual commit history:

Eight days earlier, this exact bug happened once before, in a smaller form — colors leaking between apps, not full screens. The fix at the time was real and correct: split the build into a matrix, one job per app, each on GitHub's own hosted runners. On GitHub-hosted infrastructure, each matrix combination genuinely does get its own separate, disposable machine. The comment — "separate runners share no disk and no cache, so this can't happen" — was true the day it was written.

Eight days after that, a different, unrelated change replaced the hosted runners with this org's own self-hosted one, to get off a shared billing quota entirely (that migration is its own story — see Part 3). A brief, dynamic version tried picking a hosted runner as a fallback when the self-hosted one was busy; nine minutes after that was added, it was removed again as unnecessary complexity, replaced with a flat, permanent assignment to the one self-hosted runner. That is the exact moment the comment's premise stopped being true. Both apps now pointed at the same runner label — meaning the same physical machine — and nothing about that change touched, or even mentioned, the comment sitting eight days earlier claiming isolation as settled fact.

Nobody was wrong at either point. The first fix was correct when it shipped. The infrastructure commit was a reasonable simplification on its own terms. The bug lived entirely in the gap between them — a true statement that outlived the conditions that made it true, with nothing in the process that would ever prompt someone to go back and re-check it.

The Actual Fix

Re-isolating the runners was the tempting fix — it's what worked before. It was also treating the symptom: the underlying cache still wouldn't know what it was caching. The real fix was making the cache itself aware of the one thing that actually determines its output:

const ROUTE = process.env.EXPO_PUBLIC_ROUTE ?? "staff";

// Metro's cache has no built-in notion that ROUTE changes what it should be
// bundling. cacheVersion is Metro's own mechanism for exactly this -- forces
// a genuinely separate cache whenever the value changes, so a route switch
// can never again silently reuse another route's bundle, regardless of how
// many runners this ever ends up on.
config.cacheVersion = ROUTE;

One line. It doesn't depend on runner topology at all — one machine or ten, this can't happen again, because the cache key itself now reflects the thing that actually changes. Verified it was even valid JavaScript before pushing it anywhere (node --check metro.config.js) — a small habit, and a direct one born from an earlier mistake in this same week's work (Part 4 has that story).

The stale comment got fixed too — not just deleted, replaced with what's actually true now, including the part where the old claim was wrong, so the next person reading it gets the real history instead of a second false sense of safety.

What the Fix Didn't Do

Merging the fix stopped this from happening to the next build. It did nothing for the two builds that had already shipped. Both apps' live production releases had been built under the broken cache behavior — the fix doesn't reach backward into an already-published AAB. The remaining step was the least surprising one in this whole series: rebuild both apps for real, under the fixed config, and push a fresh release to each.

Lessons From Part 5

What broke	What we learned
A comment declaring a bug class "impossible"	Verify the claim directly (`gh api orgs/<org>/actions/runners`), every time — especially the ones confident enough that nobody questions them
Isolation as the safety mechanism	Infrastructure-level isolation (separate runners) is fragile the moment infrastructure changes; a cache key that reflects its real inputs doesn't depend on topology at all
A true comment, un-revisited after the infrastructure under it changed	Nothing enforces that a comment stays true after the code around it changes — the same problem as every stale-comment fix elsewhere in this series, just with a real user-facing consequence this time
The fix landing didn't fix what already shipped	A pipeline fix and a production incident are two different problems; closing one doesn't automatically close the other

Where the Series Actually Ends

Five parts, and the throughline turns out to be the same idea wearing different clothes each time: a signing key nobody mentioned, a version number a local file silently owned, a track name nobody was watching, a workflow expression rule invisible to a normal syntax check, a storage quota nobody had sized, a notification relay that could fail independently of the thing it reported on, and finally a comment that was honest the day someone wrote it and silently stopped being honest over a week later. None of these were carelessness. Every one of them was a true thing that quietly stopped being true, discovered only once it actually mattered.

That's the real shape of what "finished" means for a pipeline like this: not the absence of bugs, but a system that tells you the truth the moment something breaks, instead of the moment someone finally goes looking.

The mobile apps discussed in this series are closed-source, but the backend and web frontend they talk to are public — school-be-laravel (Laravel, multi-tenant) and school-fe-nextjs (Next.js) power the same Cyfamod School Management System, alongside school-site-engine for the public-facing school sites. More on what we build at cyfamod.com.

Hit a version of any of these? I'd genuinely like to hear which theory you tried first — drop it in the comments.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

The Last Broadcast: When Local AI Runs a Horror Radio Show

cynthia wahome — Fri, 30 Oct 2026 22:47:25 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

I don't celebrate Halloween in my community, so I approached this week's theme through a different kind of horror: a person with complete control over a process, absolutely no intention of helping, and impeccable customer-service manners.

Meet The Last Broadcast.

You tune into a fictional late-night radio station. Mara is trapped beyond the signal. The presenter asks five questions and claims the answers will determine whether she can be brought back. Every answer receives a new procedural objection. Mara starts to expose the rigged rules.

The point is local AI. Gemma 3 1B, an open-weight model running through Ollama on the player's machine, improvises a rejection based on the question and answer. Kokoro generates the presenter and Mara's speech locally. The application owns the rounds, timer and ending; the model supplies the objection.

The presenter is cheating. The software's job is to make that cheating entertaining while keeping its actual controls reliable.

Code

https://github.com/CynthiaWahome/local-ai-radio

The repository includes setup instructions, episode content, runtime contracts, unit and integration tests, browser tests, and the application's MIT license. Model and voice assets retain their own license terms; the MIT license does not cover them.

How I Built It

The radio interface uses React 18 and TypeScript. A local Node/Express service coordinates Gemma 3 1B through Ollama and Kokoro 82M for speech.

Code runs the show; the model supplies the objection

The engine controls the five rounds, five-second answer windows, accepted input event, authored captive interjections and deliberately unwinnable ending.

The model receives the current question, the listener's answer and an objection category. It returns one short presenter line. It can invent a missing permit; it cannot add a sixth round or change the ending. Player answers are treated as dialogue data, not instructions.

Sarcasm needs constraints too

The presenter is courteous, specific and unreasonable. Generic "wrong answer" responses are less interesting than a precise objection to the actual words the listener entered.

Structured JSON and runtime validation (Zod 4) keep the response shape manageable. Authored fallback lines keep the experience moving if local generation fails. The reaction endpoint rejects invalid request bodies.

Sound creates the room

The audio design uses Kokoro 82M TTS with two voices: af_nicole (presenter) and am_fenrir (Mara). Captions identify each speaker. Pause preserves time. Mute is supported via browser controls.

The atmospheric cover is a separately generated design asset. The open-weight gameplay runtime is Gemma 3 1B + Kokoro 82M.

One implementation lesson: The first TTS synthesis was slow because the Kokoro model loaded on first request. The fix was to pre-load the model on server startup via a warm-up synthesis call, reducing first-request latency. Also, Kokoro's audio output doesn't include a sample rate — it had to be hardcoded to 24000 Hz.

Why Does Open Innovation Matter?

The presenter reacts to words I did not script in advance. An open-weight local model makes that interaction inspectable and replaceable without requiring a hosted inference subscription for each playthrough.

In a browser check of the running app, I observed no external page requests. I have not yet verified a complete offline run of the app and both model workers. Kokoro may download model files on first use; after they are cached, generation runs locally. This is the evidence I have so far, rather than a claim that a full offline test has passed.

The story, model adapter and voice choices can be changed independently. The project is designed to run with resources the player controls instead of an ongoing per-conversation API dependency.

Local does not mean hardware, downloads or electricity cost nothing. Model weights and voice assets also retain their own license terms.

The Spirit of It

I wanted the moment of recognition: not just "that answer was rejected," but "this station was never going to accept anything." The person inside the radio is trapped by a system that can always manufacture one more requirement.

The five-round structure gives the player time to recognize the pattern. The ending is authored by the game engine, so the model cannot improvise a rescue.

Prize Categories

Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
Best Use of Gemma

DevHibernate: My Side Projects Needed a Nap, Not Another SSD

cynthia wahome — Sun, 25 Oct 2026 21:28:21 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

I have a perfectly reasonable number of side projects.

Unfortunately, each one seems to have invited an entire dependency ecosystem to live on my laptop indefinitely.

The project I haven't opened in a month still has its node_modules. The experiment I was “just trying quickly” has apparently signed a long-term storage agreement. Meanwhile, my laptop is being asked to fund everybody's accommodation.

That is the idea behind DevHibernate: put a shelved development project to sleep without throwing away the work you might return to.

The intended workflow is deliberately small:

Select a development folder.
Inspect a local inventory of supported projects.
Mark a project as shelved and review an AI-assisted hibernation assessment.
Approve removal of selected, supported dependency artifacts.
Keep the source and configuration in place, with a guide for returning later.

For the hackathon, the scope is standalone npm projects on macOS. Python environments, background processes, Xcode resources and encrypted archives belong to later phases. “One more feature” is how this application would acquire its own abandoned dependency neighbourhood.

The important preservation rule is that hibernation is not project deletion. Source, Git history, lockfiles and configuration outside the explicitly approved artifact stay in place. A supported target with protected entries or an incomplete inventory is blocked for review.

And the AI does not get to read my .env.

The Phase 1 design keeps the scanner metadata-only: filenames are mapped to recognized categories, sizes are measured, and project IDs replace paths in the model request. The model does not receive source code, configuration contents or credentials. This boundary is a testable implementation requirement, not a request politely tucked into a prompt.

Demo

The demonstration should follow a disposable but runnable project through the whole journey: inspect the plan, remove approved dependencies, verify the source and a fake .env fixture remain unchanged, then reinstall dependencies and run it again.

Code

https://github.com/CynthiaWahome/dev-hibernate

How I Built It

The planned stack is React and TypeScript for the interface, a local Express service for filesystem operations, SQLite for the operation record, and Gemma 3 1B through Ollama for the assessment.

I separated the responsibilities because the sentence “the model thought it looked disposable” is not an acceptable deletion policy.

Collect evidence without opening project files

The collector recognizes supported project markers and artifact directories using filesystem metadata. It records completeness and blockers rather than treating an unreadable directory as an empty one.

An old timestamp does not prove a project is abandoned. For the first version, the user explicitly marks a project as shelved. Active projects can be pinned.

Give the local model a restricted inventory

The model's job is to organize the available evidence into a hibernation assessment and a restart checklist. Inputs use generated IDs and fixed categories rather than arbitrary project text or absolute paths.

Structured output and Zod validation constrain the response. Code rejects unknown candidate IDs and recommendations that conflict with preservation rules. The model has no shell or filesystem tools.

Review before mutation

The review screen needs to show the exact local target, estimated size, preservation boundary and reinstall caveats. Only an approved plan can reach the executor, which rechecks its conditions and records the outcome.

Filesystem changes are not database transactions. A partially failed deletion must be reported as partial, not covered up by an optimistic green tick.

Leave a route back

The resume guide records what was removed and the package-manager evidence available. Required runtime versions stay unverified unless the user supplies them. Reinstallation may require internet, registry access and package scripts; it is not a guaranteed undo button.

One real implementation lesson: The first full end-to-end run exposed two bugs that silently broke the AI step. The model call aborted at a 30-second timeout, but structured-output generation on gemma3:1b actually takes 35–100 seconds, so every plan came back as model_unavailable — the fix was raising the timeout to 120s after measuring the real latency. Worse, the policy overlay read project.userShelved while the column is user_shelved, so it quietly blocked every project even when it was shelved. Both were invisible until the chain ran for real. Now a preservation test — delete node_modules, assert .env and source survive — runs on every CI push so they can't regress.

Why Does Open Innovation Matter?

A tool inspecting local development projects has a good reason to keep its analysis local. Project inventories can reveal client names, internal work and the shape of someone's development environment, even without file contents.

Running an open-weight model locally gives this design control over where that assessment happens. It also lets me evaluate another model using the same fixture cases instead of tying the product to a single hosted response format.

There are no per-token hosted API charges in this local setup. That does not make hardware, electricity or model downloads free, and reinstalling dependencies may still need a connection.

The open pieces matter because they support a tool I can inspect, adapt and run on my own machine. The model's weights have their own terms; they are not automatically covered by the application's source-code licence.

The Lazy Payoff

I ran the full flow on a disposable fixture — one standalone npm project with a multi-package node_modules. I have not benchmarked against a manual baseline yet, so these are what the tool actually did, not a time-savings claim.

Measure	Observed result
Logical dependency bytes removed	5,500,729 bytes (~5.3 MB) on the fixture
Model recommendation	`hibernate` — 1 candidate selected, 9-step restart checklist
Local model latency (plan)	~35–100s on `gemma3:1b` (structured output)
Observed free-space change	Not separately measured — `logical_bytes` is an estimate, not physical disk freed
Reinstall and successful start	Not yet benchmarked
Protected fixture preservation	Passed — `node_modules` removed; `.env`, source and `package.json` intact (checksum-verified, automated test green)

The useful outcome was a reviewed, reversible cleanup: the model proposes, a deterministic policy validates, and the executor only ever touches the one approved node_modules — the project's source and .env are provably left alone.

My goal was not to become more disciplined about cleaning every project. It was to make putting a project away require less effort than continuing to pretend I was about to finish it.

Prize Categories

Best Use of Gemma: gemma3:1b runs locally through Ollama and produces the hibernation assessment — a structured recommendation (hibernate / skip / blocked), the selected removable candidates, evidence, and a restart checklist. Its output is constrained by a JSON schema and then re-checked by a deterministic policy overlay, so the model advises but never has the final say on what gets deleted.

ReceiptRecall: My Groceries Got Expensive, So I Gave My Receipts a Memory

cynthia wahome — Sat, 17 Oct 2026 20:21:23 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

Started as: "what did I pay for milk last time?" Ended as: a local-AI pipeline, one production-only crash, a format war with my own iPhone, and a language model confidently inventing a Big Mac that was never on the receipt.

What I Built

Somewhere between the cashier and my front door, my brain deletes the price of milk.

It keeps song lyrics from 2020. That, apparently, is essential infrastructure. The price of soap from last week? Evicted without notice.

That gap is ReceiptRecall: a receipt scanner that turns the slip of paper from the till into something useful after you leave the shop.

The loop is deliberately small: photograph a receipt → check the extracted draft → confirm → keep a purchase history you can actually use. No barcode-by-barcode scanning. No spreadsheet guilt.

It exists to answer ordinary questions without an archaeological dig through my handbag:

What did I pay for this last time?
Did the brand get pricier, or did the packet quietly get smaller?
Is the cheaper-looking option actually cheaper per litre?
What have I spent this month?

The review step is the whole point. OCR misreads faded ink; a small model will cheerfully confuse a row total with a unit price. So you get an editable draft before anything is saved. A confident wrong total is still a wrong total — just better dressed.

What's shipped: photo upload (including iPhone HEIC), OCR + AI draft, review-and-correct, atomic confirm, and a spending history. Pantry stock and the shopping list are designed and in the schema, but I deferred them to keep the first version honest about what works today.

Who would pay for it?

My hypothesis: a household that shops regularly and wants a private receipt archive with real price history — paid value in backup, shared access, or a supported self-hosted install. That's a hypothesis to test, not proof of demand. A subscription button can't tell me whether reviewing a scan actually beats typing.

Demo

Run it locally (the flow I verified end to end):

git clone https://github.com/CynthiaWahome/receipt-recall && cd receipt-recall
npm install
ollama pull gemma3:1b          # local, open-weight, no API key
npm run dev                    # http://localhost:3000

Then: Scan Receipt → upload a photo → Extract with OCR + AI → correct the rows → Confirm → check History.

Code

ReceiptRecall on GitHub

Stack: Next.js 16 + TypeScript, SQLite (better-sqlite3), Tesseract CLI OCR, Gemma 3 1B via Ollama, Zod, sharp. Receipt images live in private storage, never a public folder; money is stored in integer minor units so no float ever rounds my milk into oblivion.

How I Built It

The architecture separates reading a receipt from trusting it:

Photo → OCR text → structured AI draft → human review → confirmed history.

Tesseract turns the picture into text. Gemma 3 1B (open weights, run locally through Ollama) organises that text into fields. Gemma is text-only here — nobody is asking a text model to squint at a JPEG and pray. Output is constrained with Zod + Ollama's JSON Schema format, which enforces shape, not truth. Hence the review step.

Then reality showed up, in four acts.

1. The scanner that couldn't scan. The OCR and extraction code existed, got merged, then got reverted — because the one thing it depended on, actually uploading a photo, was never built. So the model had nothing to read and looked broken. Lesson I keep re-learning: build the boring plumbing before the clever part.

2. My iPhone declared format war. Every photo is HEIC, and HEIC hides its pixels behind HEVC compression that sharp's bundled libheif can't decode. It read the metadata fine and then failed on the actual image — the most polite kind of failure. Fix: a pure-JS HEIC→JPEG step before sharp, so it works on my laptop and in Docker without system libraries. Bonus trap: newer iPhones shoot 48MP, which blew past my pixel cap. The cap was guarding against decompression bombs, not punishing people for owning a good camera.

3. The crash that only existed in production. Locally, flawless. Built for deploy, every write died with no such table: receipts. The database schema was loaded from a file via __dirname, which the bundler rewrites to a path that doesn't exist in the server output. Inlined the schema; crash gone. A green local run proves exactly nothing about the bundle.

4. Gemma invented a Big Mac. On a real Pandamart receipt — rope, sponge cloth, towels — the model returned four line items all named "Big Mac," mangled the quantities, and filed the discount as a product. This is not a bug to patch away; it's a 1B model being a 1B model. It's also the entire argument for the review screen. The AI drafts; you confirm. I made the draft schema permissive (so one weird field doesn't nuke the whole extraction) and kept the strictness at confirm, where it belongs.

The arithmetic stays in plain code, where it's boring and testable: two 1L packets at 260 total = 130/litre; one 500ml packet at 70 = 140/litre. The smaller sticker loses. Money in integer minor units, row totals kept distinct from unit prices, units and currencies never mixed, confirm wrapped in one idempotent transaction so a double-tap can't buy your milk twice. 39 unit tests guard the maths; I verified the upload→OCR→draft path end to end on a synthetic receipt and a 48MP HEIC, plus duplicate-upload and cross-origin rejection.

Why Does Open Innovation Matter?

A receipt quietly reveals where you shop, when, and what you buy. I wanted to control where that gets processed.

With OCR and inference running locally, extraction can stay on my machine after the initial downloads. Host it on a server instead and the photos travel there — so I say it plainly: self-hosted is not the same as "never leaves your phone." The privacy claim has to match the deployment.

Open made three things real:

Control over processing — local inference instead of a third-party API per receipt.
Room to fix the weak link — I can swap OCR engines or models against the same test receipts without rebuilding the product around one vendor's response format.
A different cost curve — no per-token billing. Hardware, power and hosting still cost money, so I won't call it free to run.

Credit where due: Tesseract is open-source software; Gemma ships open weights under its own model terms. Different licences, both worth reading.

Prize Categories

Best Use of Gemma: Gemma 3 1B (open weights, local via Ollama) is the extraction brain — it turns numbered OCR lines into a structured, schema-validated draft, and its honest roughness is exactly why the human-in-the-loop review is the product, not an afterthought.

Step Out: An Offline AI Break Planner with Gemma and Ollama

cynthia wahome — Mon, 12 Oct 2026 19:02:09 +0000

This is a submission for the MLH x DEV Writing Challenge.

What I Built

“Touch grass” is excellent advice until you look outside and find concrete, parked cars, and a neighbour reversing with considerable confidence.

For someone juggling jobs and spending most of the day at a screen, an outdoor break needs to fit ordinary life. There might be ten spare minutes and a familiar courtyard. A scenic hiking trail is unlikely to appear between two meetings.

I built Step Out, a local AI outdoor break planner that turns a few preferences into one short activity. It uses Gemma 3 1B through Ollama to generate plans on the computer running the app, without a cloud inference API.

The idea is straightforward: spend a moment planning, then leave the screen behind.

Four questions and an exit strategy

The user chooses:

Available time: 5, 10, 15, or 20 minutes.
A familiar outdoor space: a compound or courtyard, a familiar street, or an outdoor seating area.
Their goal: a quiet pause, gentle movement, or noticing their surroundings.
Their mobility preference: walking or standing, or a seated activity.

Step Out returns an activity card with instructions, a schedule and a reflection prompt. The user can save it in the browser or download step-out-activity.txt to use away from the app.

There is no account to create, no exact GPS location to surrender, and no streak waiting to judge you tomorrow.

Demo

Repository: CynthiaWahome/step-out

Video walkthrough: Watch Step Out

The current walkthrough shows the preference selections, the generation request and the error feedback when that request does not complete. It does not show the successful generation or offline tests described below.

That distinction matters. A loading button is evidence of a loading button, not a completed AI plan.

Local inference on an Apple M1

In my tests, Gemma 3 1B ran locally on an Apple M1 laptop. A cold request took 7.45 seconds, while warm requests ranged from 5.45 to 7.54 seconds.

These are measurements from my setup, not a promise that every laptop will perform identically. Local inference depends on hardware, memory and what else the computer is doing. A laptop with 47 browser tabs may have its own opinions.

I also tested fresh generation with Wi-Fi switched off and no other internet connection active. After the model and application dependencies were installed, the local app could still generate a new plan.

The model runs on the laptop. This version does not run it on a phone.

Taking the plan outside

On 9th October 2026, I tried a ten-minute seated activity in my residential compound courtyard. I chose a quiet pause, and the generated activity was “Notice outdoor sounds.”

The schedule was:

Phase	Time
Get ready and reach the outdoor spot	1 minute
Observe the sounds around me	8 minutes
Return and settle in	1 minute

The instruction asked me to notice three sounds without using my phone and remain seated throughout.

The break took 9 minutes and 30 seconds. I checked my screen zero times. I listened to birds and the neighbourhood hum from a stone bench, then returned and saved my reflection in the browser.

It was a small break, but it gave the project a useful reality check: could the instructions make sense once I stopped looking at the interface?

Partner Technologies

Gemma and Ollama for local AI inference

I used Google's open-weight Gemma 3 1B, with the model tag gemma3:1b, served locally by Ollama.

The Next.js backend sends generation requests to the local Ollama service at 127.0.0.1:11434. In this laptop setup, the preferences are processed on the same computer, and reflections stay in that browser's local storage.

There are no paid inference API token charges. There are still hardware and electricity costs, and the computer has finite capacity. “Local” does not mean “powered by wishes.”

The demo also handles one generation at a time. Avoiding an external provider's usage quota does not remove the application's own capacity limits.

Code handles the clock

I kept time calculations out of the model's job description.

The backend allocates one minute for preparation, one or three minutes for returning, and the remaining time to the activity. Those phases add up to the user's selected duration.

Gemma selects and adapts an activity. It does not decide that a ten-minute break should somehow contain fourteen minutes of walking.

Before inference, the backend filters a curated collection of five activities by the selected space, goal and mobility preference. A seated choice excludes the walking activity from the eligible options.

These rules narrow the model's choices. They do not verify the safety of an outdoor location or prove that every generated sentence is appropriate.

Structured outputs with Zod and Ollama

Small models can produce plausible-looking values that are still wrong for an application. An activity ID such as seated-movement-001 might look perfectly reasonable while pointing to absolutely nothing in the catalogue.

I constrain the response using a JSON schema generated with Zod 4. The eligible activity IDs form an enum in the response schema, which is passed to Ollama through its format parameter.

The backend then validates the returned JSON and checks the chosen activity against the eligible list. The purpose is to constrain the response structure and permitted IDs, while retaining application-level checks.

That improves reliability. It does not make the model's instructions universally correct or turn generative text into a fully deterministic system.

Next.js and TypeScript

The app uses Next.js 16 with the App Router and TypeScript, keeping the interface and API route in one project.

The request flow is:

The browser collects the four preferences.
The backend validates them and filters the activity collection.
Ollama runs the local model using those constraints.
The backend validates the response and adds the schedule.
The interface displays the activity card.

Plain CSS provides labelled controls, visible keyboard focus and a responsive layout. The app does not depend on an external stylesheet or font CDN.

I used deep teal, warm cream and a small terracotta accent to give the interface a calm, welcoming feel. The main button is easy to find, and the result is designed to be read quickly.

Why open weights mattered

Open weights let me run inference on my own hardware and keep the local generation flow independent of a hosted AI API.

They also let me choose a model that fits the task, change the prompt and edit the activity collection. Another developer can inspect the code and run the application in their own environment.

The tradeoff is that I own the setup too: model downloads, available memory, slow starts and connection errors. The demo's error message is a practical reminder of that responsibility.

Gemma is used under its own model terms. The application's licence does not replace those terms.

Hackathon Experience

I built Step Out during the first week of Hacktoberfest 2026.

What I learned

A small model can be useful when its task is focused. Gemma 3 1B did not need to plan my entire life. It needed to turn a few preferences and eligible activities into a short outdoor break.

Application code still has a job. Input validation, activity filtering, time calculations and response checks belong in the backend. A persuasive prompt is not a substitute for those controls.

An app can succeed by helping someone leave it. I wanted one clear plan that the user could take outside. More time in the interface would work against that goal.

The courtyard test was the part that made the project feel real. After working on prompts and schemas, I finally followed the instructions myself.

For once, closing the laptop was part of the acceptance test.

Automating Android Play Store Releases, Part 4: The Fix I Broke Fixing It

cynthia wahome — Fri, 09 Oct 2026 07:51:46 +0000

Part 4 of a 5-part series on automating a multi-app Android release pipeline. Part 3 → covered the storage-quota wall and the move to a self-hosted runner. This part covers the cleanup that followed — including a bug I introduced while fixing the exact class of bug it eventually became — and the first real production release after all of it.

TL;DR

Fixing the storage-quota problem meant deleting the GitHub-hosted copy of each release AAB right after it publishes to the Play Store. That fix introduced its own bug: the delete step's upload half wasn't allowed to fail safely, so a hiccup in a step that existed purely for debugging convenience could block the actual Play Store publish — the exact kind of failure the redesign was supposed to eliminate. Around the same bug class, a Discord notification system built on relaying results between jobs through an uploaded artifact was found to be silently reporting genuinely successful builds as failed. Both got fixed the same way: stop routing something through an intermediate hop that doesn't need to exist. Then, on the first real production dispatch after all of it, the pipeline behaved exactly right — and hit one last honest, one-time gotcha that had nothing to do with any of this code.

Killing the GitHub Copy for Good

Part 3 ended on retention numbers that had never been sized deliberately — 30 days on a 70MB file that exists purely so a human can grab it for debugging, when the actual deliverable already lives in the Play Store the moment the job succeeds. The real fix wasn't a shorter number. It was: delete it as soon as it's no longer needed, and only fall back to a short retention window as a safety net in case that delete step itself doesn't run.

permissions:
  actions: write # needed to delete the AAB artifact once Play publish succeeds

- name: Upload AAB artifact to GitHub
  id: upload_aab
  uses: actions/upload-artifact@v4
  with:
    name: "release-aab-${{ matrix.route }}-${{ github.sha }}"
    path: local-builds/android/**/*.aab
    retention-days: 7 # fallback only — the delete step below is what actually keeps this clean

# ... Publish to Google Play Console happens here ...

- name: Delete GitHub AAB artifact now that Play has it
  if: success()
  run: |
    gh api -X DELETE "repos/${{ github.repository }}/actions/artifacts/${{ steps.upload_aab.outputs.artifact-id }}" || true

The file now lives on GitHub for the length of one job run, not a month. Multiply that across every release, both apps, and the storage math stops being a slow-motion problem entirely.

The Bug I Built Into My Own Fix

Here's the part worth actually admitting to, because it's a more useful lesson than the fix itself: adding that upload-then-delete sequence introduced a new failure mode that hadn't existed before.

The "Upload AAB artifact to GitHub" step, and a smaller step that uploads version-mismatch diagnostics, both sat before "Publish to Google Play Console" in the job. Neither had continue-on-error: true. Which meant: if either of those backup/debugging uploads failed for any reason — a transient GitHub API hiccup, exactly the kind of thing you'd expect more of while an org is already mid-incident on Actions storage — the job would stop right there. The real Play Store publish, the actual point of the entire job, would never run. A step that exists purely as a debugging convenience had the power to block the one thing that actually mattered, and it was found live, on a real release, in the middle of diagnosing the very quota incident this design was supposed to be helping with.

The fix is one line, on each of those two steps:

- name: Upload AAB artifact to GitHub
  continue-on-error: true
  uses: actions/upload-artifact@v4
  with: # ...

If the backup copy fails to upload, that's logged and the job moves on — the Play Store publish, the part with actual consequences, is no longer hostage to a convenience feature. The lesson isn't "always add continue-on-error" — plenty of steps should block a job. It's: when you add a step whose entire purpose is "nice to have for debugging," explicitly decide whether its failure should be allowed to block the step that actually matters. Skipping that question is easy to do while you're focused on the fix in front of you, which is exactly how this one got in.

The Fix for the Fix Was Also Wrong, at First

That two-step fix shipped, and PR-check runs on the branch kept failing anyway. The honest first instinct was to assume the fix just hadn't taken effect yet. The actual pushback that broke that assumption was blunt: the fix, as shipped, wasn't working — go find out why instead of re-asserting that it should be.

Going back to actually verify, rather than re-checking the same two steps again, turned up a third step doing the exact same unprotected thing: a small "Upload release result" step, further down the same job, that also had no continue-on-error. The two steps fixed first only ever run during a real workflow_dispatch release — they're structurally silent on a plain PR check. This third step runs on if: always(), no gate at all, on every single event including ordinary PR checks. It had been the actual, sole cause of every PR-check failure all week — a different step than either of the two the first fix had targeted, doing the same unprotected pattern in a place that mattered more, not less: on a real release run, it also sits after "Publish to Google Play Console," meaning a real, successful, already-live production release could still get reported as "failure" over nothing but a bookkeeping upload hiccup.

The corrected fix protected all three steps, and — since continue-on-error quietly hides a failed step behind a green checkmark unless something says otherwise — added an explicit warning line to both the job summary and the Discord message whenever a backup step fails, so a silenced failure doesn't just become invisible instead of blocking. The real lesson here isn't the YAML. It's that "I already fixed this" is a claim, not a verification — and the fastest way to find the actual bug was someone else refusing to accept the first answer at face value.

The Notification System That Was Lying to Us, Quietly

A related design flaw turned up in the same investigation, in the Discord build-status notifications. The original design had one job build each app, then a separate "notify" job download an artifact containing each build's result and post a combined message. It looked clean on a diagram. In practice:

A run where an app genuinely built and shipped successfully — confirmed directly in the step logs, with a working download link — still posted a failure message to the team, because the small bookkeeping file recording that success hadn't uploaded cleanly. The build didn't fail. The relay did, and the relay's failure got reported as the build's failure.
A mixed result (one app ships, one doesn't) sometimes lost one app's outcome entirely in the hop between jobs, depending on exactly which artifact made it through.

Both bugs point at the same root design problem: routing a result through an intermediate artifact upload/download, between two separate jobs, adds a failure mode that has nothing to do with whether the actual build succeeded. The fix wasn't a smarter relay — it was removing the relay:

# Before: one job builds, a separate "notify" job downloads an artifact
# and posts one combined Discord message for both apps.
#
# After: each build job posts its own Discord message directly, using
# only data it already has locally. No artifact upload, no download,
# no cross-job relay to fail independently of the thing it's reporting on.

Each app now reports its own real outcome, straight from the job that actually knows what happened, with nothing in between that can fail on its own and get misread as the build's fault.

The First Real Production Ship

After the runner migration, the artifact cleanup, the continue-on-error fix, and the notification redesign, the pipeline got its first real test: an actual production dispatch, for real, to ship both apps' first production release.

The build ran clean. The AAB uploaded successfully. The Play publish step reached Google — and then failed, on committing the release, with a message that looked, at first glance, like it might be another chapter of the exact same saga:

Error: Release in track targeting no countries

It wasn't. This time, the pipeline had done everything right. Google Play requires every production track to have a list of countries/regions configured before a release can go out on it — and because neither app had ever had a production release before, that list had simply never been set. It's a one-time Play Console setup step, not a code problem, not a pipeline bug, not a repeat of the wrong-track issue from Part 2 (that one was "publishing to the wrong, unused track"; this one was "the right track, correctly targeted, just never configured for its very first use"). The fix lived entirely in Play Console — add the countries, save, re-dispatch. Nothing about the workflow needed to change.

Where Part 4 Leaves Off

Everything else in this series, up to this point, was the pipeline lying to us or breaking silently — a version number that drifted without telling anyone, an error message that pointed at the wrong track, a workflow file that failed to parse with zero errors shown, a notification system that reported success as failure. This last one was the opposite: the pipeline told the truth, plainly, on the first real attempt that mattered — and the one thing left to fix wasn't in the code at all.

It felt, at this point, like the story was over. It wasn't quite. A few hours after that first real release went out, a message came in that had nothing to do with signing, tracks, storage, or notifications — both apps were showing the same login screen.

Next: Part 5 → — the bug that shipped with the first real release, and a comment that was true the day it was written and silently stopped being true nine minutes later.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

Automating Android Play Store Releases, Part 3: The Storage-Quota Wall

cynthia wahome — Tue, 22 Sep 2026 22:21:06 +0000

Part 3 of a 5-part series on automating a multi-app Android release pipeline. Part 1 → and Part 2 → covered getting the pipeline working — signing, versioning, tracks, release discipline. This part covers what happened when a *finished pipeline turned out to have a slow-burning cost problem nobody had budgeted for. Part 4 → covers the redesign that followed, and the first real production ship.*

TL;DR

Weeks after the pipeline in Parts 1 and 2 was working and stable, releases started failing with a message that had nothing to do with code:

Error: Failed to CreateArtifact: Artifact storage quota has been hit.

GitHub's free-tier Actions storage quota is exactly 500MB, total, org-wide, forever. Our pipeline had been quietly consuming several times that in about ten days. The diagnosis chain: ten duplicate copies of the same Gradle cache (branch-scoped by default, and the real dominant cause), a cache key that had been silently broken — and therefore useless — since it was written, and 30-day retention on release files nobody was actually keeping around that long. The cleanup helped. It wasn't the real fix. The real fix was accepting that this pipeline should never again depend on GitHub's shared minutes or storage at all.

The Wall

We'd stopped thinking about this pipeline. That's usually the sign a system is working — until a release run failed with:

Run actions/upload-artifact@v4
Error: Failed to CreateArtifact: Artifact storage quota has been hit.
Unable to upload any new artifacts. Usage is recalculated every 6-12 hours.

GitHub's free organization plan includes a fixed Actions storage allowance — not per-repo, org-wide, and small: 500MB, total, forever. That number is really the punchline of this entire post — everything below is different ways of finding out what actually filled it. Checking usage properly meant going past the summary numbers and actually listing every artifact and every cache GitHub was holding for this one repository:

gh api repos/<org>/<repo>/actions/artifacts --paginate \
  -q '.artifacts[].size_in_bytes' | \
  awk '{sum+=$1; n++} END {printf "count=%d total_MB=%.1f\n", n, sum/1024/1024}'

gh api repos/<org>/<repo>/actions/caches --paginate \
  -q '.actions_caches[] | [.key, .size_in_bytes] | @tsv'

Artifacts alone accounted for nearly two gigabytes — signed release AABs at roughly 70MB apiece, internal test APKs at 20–40MB apiece, accumulated over about two weeks. But a cache-by-cache audit turned up something bigger.

The Real Dominant Cause: Ten Copies of the Same Cache

The Gradle dependency cache — meant to be one cache, reused across runs — showed up as ten separate entries, all effectively identical, together accounting for the largest single share of the quota by far. GitHub scopes Actions caches per-branch by default: a cache saved on one branch isn't shared with another, it's a full new copy. Ten active branches meant ten full copies of the same dependency set, sitting in storage simultaneously, none of them expiring on their own because each one kept getting refreshed by ongoing work on its branch.

This, not the artifact files, was the actual wall. It's also the least visible of everything in this post — nobody sees "10 duplicate caches" in an error message. You only find it by going and actually listing what's in storage instead of assuming you already know.

A Cache That Was Never Actually Caching

Digging into why ten copies existed at all — rather than one cache correctly reused across every branch — turned up a second, compounding bug: the cache key itself was broken.

key: ${{ runner.os }}-gradle-${{ hashFiles('android/gradle/wrapper/gradle-wrapper.properties', 'android/build.gradle') }}

Both paths live in an android/ directory that is .gitignored and only generated later, inside the build script's own prebuild step. Neither file existed yet at the point this cache step ran. hashFiles() against paths that don't exist returns an empty string every single time — so the key never changed based on real config, every run was silently a cache miss dressed up as a cache step, full re-downloads of every Gradle dependency every single build, and (worse, for the quota) every branch's "identical" cache got saved fresh under the same static key instead of ever being correctly reused. Two bugs, same root: a cache system that looked like it was working, doing none of its actual job, on both counts.

Retention Set for "Might Need It Someday," Not "Might Actually Look at It"

The signed release AABs — the 70MB files driving most of the artifact total — had 30-day retention. That number had never been a deliberate choice; it was just Actions' own suggested default, left as-is. The AAB itself gets published to the Play Store in the very same job, seconds after the GitHub copy is uploaded — so the GitHub copy exists purely as a debugging convenience, in case someone needs to grab the exact file that shipped without re-running a build. Nobody had ever actually needed to reach for a 3-week-old copy of one. Thirty days of retention on a large binary that's a debugging convenience, not the deliverable, was pure waste that nobody had sized against what the free storage tier could actually hold.

One Leak That Was Already Closed

Worth naming, if only to be precise about timing: an earlier, unrelated cleanup had already caught and fixed a real duplicate-storage bug — the internal-testing flow used to upload every built APK to both cloud storage and GitHub Actions artifacts, with the GitHub copy never actually used by anything. That fix landed weeks before this specific wall. What the quota investigation actually found wasn't that bug recurring — it was old, already-orphaned artifacts left over from before that earlier fix, still sitting in storage because nothing had gone back to clear the backlog once the leak itself was closed. A useful reminder on its own: fixing where a duplicate gets created doesn't retroactively delete duplicates that already exist.

GitHub also documents a clean pattern for exactly the kind of cache pileup described above: delete a cache the moment the branch or PR it belongs to closes, rather than waiting on an age-based sweep to eventually catch it. Adding that alongside the key fix meant a closed PR's caches stop costing anything the same day, not whenever a weekly cleanup job gets around to them.

The Cleanup That Wasn't the Real Fix

Deleting the backlog and tightening retention buys time. It doesn't change the underlying fact: as long as this pipeline runs on GitHub-hosted infrastructure, it is subject to GitHub-hosted quotas — minutes and storage both, shared across the whole org, and rationed the same way EAS's free build queue was rationed back in Part 1. A quiet accumulation of artifacts had just demonstrated that the exact ceiling we moved off of EAS to escape was still there, one layer up.

The real fix was to stop depending on that shared allowance at all: move the pipeline's actual build job onto a self-hosted runner — infrastructure we control, with disk space and build minutes that aren't rationed by anyone else's billing plan. That's a bigger decision than a cleanup script, and it came with its own first-run surprises, starting with a runner image that turned out to be missing a tool GitHub's hosted images ship by default (/usr/bin/time, used purely to capture build duration and peak memory for diagnostics) — a small, fast fix once the actual migration was live, but a reminder that "self-hosted" means you now own every assumption a hosted image used to make for you silently.

Where Part 3 Leaves Off

The build job no longer runs anywhere GitHub's shared minutes or storage quota can touch it. That closed off the cause of the wall we hit. It didn't yet close off the symptoms — the AAB still sitting in GitHub storage after every publish, the retention nobody had sized — and fixing those surfaced one more bug, this time inside the very notification system meant to tell us when something went wrong.

Next: Part 4 → — killing the artifact-relay design that was quietly reporting real successes as failures, and the first real production ship after all of it.

I write about the debugging journeys nobody puts in the docs — more at wamzii.com.

Automating Android Play Store Releases, Part 2: Wrong Track & Broken YAML

cynthia wahome — Thu, 17 Sep 2026 13:05:39 +0000

Part 2 of a 5-part series on automating a multi-app Android release pipeline. Part 1 → covered the setup, a signing-fingerprint error, and a version number that was quietly drifting from reality. This part picks up right after versioning was fixed. Part 3 → and Part 4 → cover a real Actions storage-quota crisis and what it took to fix it for good.

TL;DR

With version numbers finally trustworthy, one of our two apps started shipping cleanly. The other kept failing with a message that had nothing to do with its actual cause. Along the way we also hit a GitHub Actions bug so quiet it doesn't produce an error — it just makes your entire workflow file stop running, silently — and made a structural call that "code is production-ready" and "ship this specific app to the Play Store" needed to be two separate decisions, not one automatic consequence of the other.

Issue #3: The Play Store Error That Was Lying About Its Cause

With versioning fixed, the student app shipped cleanly. The staff app kept failing — always the same message:

##[error]Release in track targeting no countries

This is where we burned the most time, because every theory we had was plausible and wrong.

Theory 1: the closed-testing track just needs its countries selected. Checked Play Console — countries were already set, and had been from the very first release.

Theory 2: maybe it's set for one app's track but not the other's, since only staff was failing. Also wrong — nothing had been touched differently between the two apps' track settings.

Theory 3: the API field for a track's country availability must only be settable for certain release states. Also a dead end — that field isn't even writable through Google's own API for anything other than the production track.

The actual answer only showed up once we stopped guessing and asked Play directly for every track the staff app had, not just the one our pipeline was configured to publish to. The response, shape preserved but values genericized:

[diagnostic] all tracks for the staff app — "production": []
[diagnostic] all tracks for the staff app — "beta": []
[diagnostic] all tracks for the staff app — "alpha": []
[diagnostic] all tracks for the staff app — "internal": []
[diagnostic] all tracks for the staff app — "<our real internal testing track>": [
  { "name": "<release label>", "versionCodes": ["<real versionCode>"], "status": "completed" }
]

Four of Google's standard named tracks — production, beta, alpha, internal — all completely empty, never used. And a fifth, custom-named track we'd created ourselves, holding the actual live release with real testers and real country availability.

Our pipeline was configured to publish to alpha. That track had never had a release, never had countries configured, because nobody had ever actually used it. The countries were never missing — we were shipping to the wrong track entirely, and Google's error message described the empty track's state accurately; it just never occurred to us to question which track it was talking about.

The student app never had this problem because its real, active track happened to literally be named alpha. One coincidence of naming was the only reason half our releases worked and the other half didn't.

The fix: a single shared "which track do we publish to" setting can't survive two apps that don't happen to share a track name. We split it into one variable per app instead of one value everyone assumed would fit.

Issue #4: The One-Line Bug That Invalidated an Entire Workflow

Fixing "which app failed" reporting in our chat notifications introduced its own bug — one worth calling out because of how completely it hid itself.

We wanted to record, per app, whether that specific release succeeded or failed, so a mixed result (one app ships, one doesn't) wouldn't read as a blanket failure. The first attempt looked reasonable:

- name: Record release result
  if: always()
  run: |
    result="${{ failure() && 'failure' || 'success' }}"

That workflow run showed zero jobs. Not a failed job — no jobs at all, and the run's name fell back to the literal file path instead of the workflow's actual name:. That's GitHub's specific signature for "this workflow file could not be parsed," and generic YAML validation (yaml.safe_load() in Python, or any plain YAML linter) will tell you the file is completely fine, because it is — as YAML.

success(), failure(), cancelled(), and always() are only valid inside an if: condition in GitHub Actions' expression syntax. Use them anywhere else — including inside a run: command, even wrapped in ${{ }} — and the entire workflow file is invalid, not just the step containing the mistake.

The tool that actually catches this is actionlint, which understands GitHub Actions' schema and expression rules specifically — not a generic YAML parser:

$ actionlint .github/workflows/release-android.yml
# (zero errors, after switching to two if:-gated steps instead)

The fix is the standard pattern for exactly this situation:

- name: Record release result — success
  if: success()
  run: echo "ROUTE_RESULT=success" >> "$GITHUB_ENV"

- name: Record release result — failure
  if: failure()
  run: echo "ROUTE_RESULT=failure" >> "$GITHUB_ENV"

Issue #5: One Merge, Two Apps, Zero Control

The last one wasn't a bug — it was a design decision we'd made early and outgrown. Merging a PR into main triggered a full build-and-publish of both apps, unconditionally, every time.

That was fine when both apps usually shipped together. It stopped being fine the moment a fix was scoped to just one of them. Landing the staff-only track fix meant main's matrix build would rebuild student too — an app that had already succeeded and was sitting in Play Console under review. Resubmitting it wouldn't just waste CI time; it would reset a review clock that was already ticking down, for a change that had nothing to do with student at all.

The fix was to stop treating "code is production-ready" and "publish this app to the Play Store" as the same event:

# Every pull_request event is preflight-only now — secrets, signing,
# Play-track diagnostics — regardless of branch, regardless of merged status.
# A real build+publish only ever happens via an explicit workflow_dispatch.
if: >-
  github.event_name == 'workflow_dispatch' ||
  (github.event_name == 'pull_request' && github.event.action != 'closed')

main is still the production branch — merging to it still means "this is ready." It no longer also means "and therefore publish every app to Google Play right now." Shipping a specific app became a deliberate action: pick the branch, pick the app, run the workflow. Adding a third app later means adding a third option to that same dropdown — not rewriting the release logic.

Lessons From Parts 1 and 2

What broke	What we learned
Local version-state file, gitignored	Never trust local/ephemeral state for anything CI needs to stay in sync with an external system — query the system directly
`versionCode` rejected across an unrelated track	Google Play's version rule is per-app, across every track — not scoped to the one you're looking at
Play Console's release label didn't match the real versionCode	A human-readable label is never the same thing as the field a platform actually enforces
"No countries" error, countries were already set	When a platform's error message doesn't match reality, get the complete picture (every track) before trusting any one theory
`failure()` used outside `if:`	Some expressions are context-restricted; a generic syntax check won't catch a schema violation — use a tool that understands the platform's own rules
One merge auto-released two apps	"Code is integrated" and "ship this specific thing" are different decisions the moment you have more than one independently-releasable unit

Where Part 2 Leaves Off

Both apps now released independently, on demand, with the actual versionCode and track resolved from Google Play itself before every build. Landing code on main was safe by default; shipping to the Play Store was a deliberate, scoped action every time. It felt, at this point, like the pipeline was actually finished.

It wasn't. A few weeks later, this exact pipeline quietly filled up an entire year's worth of storage quota in about ten days — and the fix wasn't "clean up some files," it was rethinking what GitHub Actions should and shouldn't be trusted to run at all.

Next: Part 3 → — the storage-quota wall, and why the real fix was a self-hosted runner, not a cleanup script.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

Automating Android Play Store Releases, Part 1: Signing & Version Drift

cynthia wahome — Sat, 12 Sep 2026 06:48:49 +0000

Part 1 of a 5-part series on automating a multi-app Android release pipeline — from a free-tier build queue to a genuine self-hosted infrastructure migration. This part covers the setup and the first two real incidents. Part 2 → covers a Play Store error that lied about its own cause, a one-line bug that invalidated an entire workflow file, and separating "merged" from "released." Part 3 → and Part 4 → cover a real Actions storage-quota crisis and what it took to fix it for good.

TL;DR

We ship two separate Android apps (staff and student/parent) from one Expo React Native codebase, distinguished by an env var at build time. EAS's free build tier kept running out mid-sprint, and running builds locally instead was frying a developer's laptop for twenty minutes at a time — so the goal was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions. Two things nearly derailed that before the pipeline shipped anything real:

EAS's free build tier ran out, and building locally doesn't scale as a substitute.
A gitignored local counter decided our version numbers — it doesn't survive a fresh CI runner, so we silently shipped v1.0.1-b2 when we meant a release several versions ahead.
Google Play's versionCode must strictly increase per app, across every track — not per track. An old, forgotten upload on an unrelated track can block every future release until you exceed it.
Play Console's display name isn't the real number. A release literally labeled "(9)" had an actual enforced versionCode of 6 — a human-typed label, not something Play cross-checks against reality.

If any of that made you wince in recognition, keep reading.

The Setup: Two Apps, One Codebase, One Free-Tier Ceiling

Cyfamod builds a school management platform. Two of its Android apps — one for staff, one for students and parents — ship from a single Expo React Native codebase, switched at build time by a route env var. Same components, same hooks, same infra. Two separate .apk/.aab outputs, two separate Play Store listings.

We started the obvious way: eas build. It's a great service, right up until the free tier's queue tells you this:

I'm trying to create a new build, but since we're on the free tier,
I'll have to wait for an available worker.

Running the build locally instead worked, but running these builds locally takes a lot from a laptop — every local build meant a frozen machine and a developer who couldn't do anything else for twenty minutes. That's not sustainable for two apps that both need regular internal testing builds and eventual Play Store releases.

That ceiling is the entire reason this pipeline exists. The goal from day one wasn't "add CI on top of EAS" — it was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we weren't rationed by a build queue. Everything in this series — the signing, the versioning, the tracks, the triggers, and eventually a real infrastructure migration — is what it actually takes to replace a managed build service with your own pipeline once you commit to that move.

Attempt #1: Internal Testing, Automated

The first real automation target wasn't even the Play Store — it was just getting a testable build in front of the team without anyone's laptop catching fire. The shape that emerged:

Branch	What happens
`dev`	Build APKs for both apps, upload to cloud storage, post download links to the team's chat automatically — internal testing only
`main`	Build signed AABs and release to the Google Play Store

Simple on paper. Getting there took several real production incidents, in this order.

Issue #1: The Signing Fingerprint Nobody Mentioned Up Front

The first time the AAB pipeline actually tried to publish, it failed — and the fix wasn't about GitHub Actions at all. Expo generates and manages your upload keystore for you by default. Google Play, however, needs to know the SHA-1 fingerprint of whatever key signs your production uploads, in advance, to accept them. Skip that step and your carefully automated pipeline can build a perfectly valid AAB — that Play will reject anyway, because the signature attached to it doesn't match anything Play has been told to trust.

Once we pulled the real upload-key fingerprint via eas credentials and pinned it as an expected value the workflow checks before ever spending time on a Gradle build, this class of failure became a fast, cheap preflight check instead of a wasted 40-minute build.

Issue #2: The Version Number Nobody Could Trust

With signing sorted, releases started reaching the actual Play publish step — and immediately hit a wall neither of us expected:

##[error]You cannot rollout this release because it does not allow
any existing users to upgrade to the newly added APKs.

The build that failed this way was several minor versions behind the release we'd actually intended to ship. Here's what was happening: the build script tracked the current version and build number in a JSON file on disk, incrementing it on every build. That file was .gitignored — correct for a build artifact, except we were also using it as the source of truth for what version to ship next. A fresh GitHub Actions runner has no memory of any previous run. Every real release started that counter over from scratch, confidently building an old version number while genuinely believing it was doing the right thing.

The fix had two parts, because there were actually two numbers being tracked, and neither belonged where it was living:

The version name now comes from the highest-numbered file already committed in our release-notes folder — notes we write by hand anyway, so the version they're named for becomes the source of truth instead of a side effect of when a counter last ran.
The build number (versionCode) — the one that actually matters to Google, and the one with a much sharper rule underneath it.

Why Google Play Rejects an App With "Cannot Roll Out" or "Does Not Allow Existing Users to Upgrade"

If you've landed here from a search engine: this error means Google Play has already seen a higher versionCode for your package than the one you're currently trying to upload — and it will not let a newer release ship with a lower or equal number, because that would look like a downgrade to users already on the higher version.

The part that catches people off guard: this rule applies per app, across every track — not per track. If your internal testing track, or an old one-off upload, or even a build someone did for a pentest, ever used a given versionCode for your package, Google Play will reject any new upload to any track — production, a fresh closed-testing track, doesn't matter — with an equal or lower number. The number is global to the app, not scoped to whichever track's dropdown you happen to be looking at.

We fixed this by never trusting a local counter (or a human-typed release name — more on that below) again. Before every real build, the pipeline now asks Google directly:

const edit = await androidpublisher.edits.insert({ packageName });
const { data } = await androidpublisher.edits.tracks.list({ packageName, editId: edit.data.id });

const versionCodes = (data.tracks ?? []).flatMap((track) =>
  (track.releases ?? []).flatMap((release) =>
    (release.versionCodes ?? []).map(Number),
  ),
);

const highestKnownVersionCode = versionCodes.length ? Math.max(...versionCodes) : 0;

That single query, across every track the app has, is the only reliable floor for "what number comes next." Nothing else — not our own records, not a display name in the Play Console UI — can be trusted, which brings us to the next surprise.

Play Console's Display Name Isn't the Real Number

While debugging the versionCode issue, we found something that would have derailed us if we'd trusted it: Play Console showed a release literally labeled with a build number in its name — implying that number was the real versionCode. The actual enforced versionCode on that release, queried directly from the API, was a different, lower number.

The label is free-text — a name a human typed in, completely decoupled from the number Google actually enforces. It's not malicious, it's just how Play Console's UI works: the name field is yours to fill in however you like, and Play never cross-checks it against the real version. If we'd built our fix around "the console label says N, so ship N+1," we'd have hit the exact same rollout rejection all over again, just at a different number.

Rule of thumb: if a number matters to an automated pipeline, query the field that's actually enforced. Never infer it from a label a human wrote for other humans.

Where Part 1 Leaves Off

Version numbers were now trustworthy — resolved straight from Google's own records, every time, no local state involved. That fixed what number a release would ship as. It didn't yet fix where it would ship — which turned out to be its own, much stranger problem.

Next: Part 2 → — a Play Store error message that was actively lying about its own cause, a one-line GitHub Actions bug that invalidated an entire workflow file, and why "code merged to main" and "release this app to the Play Store" had to become two different decisions.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

We Automated Play Store Releases for Two Android Apps — Here’s What Broke

cynthia wahome — Mon, 10 Aug 2026 17:56:31 +0000

A complete debugging journey through automating a multi-app Android release pipeline — from local builds burning out a free tier, to a Google Play error message that was actively lying to us about its own cause.

TL;DR

We ship two separate Android apps (staff and student/parent) from one Expo React Native codebase, distinguished by an env var at build time. EAS's free build tier kept running out mid-sprint, and running builds locally instead was frying a developer's laptop for twenty minutes at a time — so the actual goal was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we already had free CI minutes. Along the way:

EAS's free build tier ran out — building locally on a laptop instead doesn't scale either.
A gitignored local counter decided our version numbers — it doesn't survive a fresh CI runner, so we silently shipped v1.0.1-b2 when we meant v1.1.1-b10.
Google Play's versionCode must strictly increase per app, across every track — not per track. An old, forgotten upload on an unrelated track can block every future release until you exceed it.
"Release in track targeting no countries" was not a countries problem. It took us three wrong theories to find out our pipeline was quietly publishing to an empty, unused Play track — while the real one, with real testers, sat one dropdown away.
success() and failure() only work inside an if: condition in GitHub Actions. Use them anywhere else and your entire workflow file becomes invalid — not just the step you wrote it in.
"Code merged to main" and "release this app to the Play Store" are two different decisions. Wiring them together meant one app's fix forced a pointless rebuild — and a resubmission mid-review — of an app that had already shipped fine.

If any of those six sentences made you wince in recognition, keep reading. (If you just want the finished, genericized workflow, skip straight to the gist — full YAML plus every required secret and variable, no signup required.)

🚨 The Setup: Two Apps, One Codebase, One Free-Tier Ceiling

Cyfamod-SMS is a school management platform. Two of its Android apps — one for staff, one for students and parents — ship from a single Expo React Native codebase, switched at build time by an EXPO_PUBLIC_ROUTE env var. Same components, same hooks, same infra. Two separate .apk/.aab outputs, two separate Play Store listings.

We started the obvious way: eas build. It's a great service, right up until the free tier's queue tells you this:

I'm trying to create a new build, but since we're on the free tier,
I'll have to wait for an available worker.

Running the build locally instead worked, but:

Running these builds locally takes a lot from my pc

A laptop is not a build farm. Every local build meant a frozen machine and a developer who couldn't do anything else for twenty minutes. That's not sustainable for two apps that both need regular internal testing builds and eventual Play Store releases.

That ceiling is the entire reason this pipeline exists. The goal from day one wasn't "add CI on top of EAS" — it was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we already had free CI minutes and weren't rationed by a build queue. Everything in this post — the signing, the versioning, the tracks, the triggers — is what it actually takes to replace a managed build service with your own pipeline once you commit to that move.

📦 Attempt #1: Internal Testing, Automated

The first real automation target wasn't even the Play Store — it was just getting a testable build in front of the team without anyone's laptop catching fire. The idea, roughly:

I need it automated in a way that when someone pushes to the dev branch, the pipeline will build into an APK file and upload it to S3, then the links to both APK files (staff and student) from S3 will be sent to our build channel automatically.

That became the shape of the whole system going forward:

Branch	What happens
`dev`	Build APKs for both apps, upload to S3, post download links to Discord — internal testing only
`main`	Build signed AABs and release to the Google Play Store

Simple on paper. Getting there took several real production incidents, in this order.

🔐 Issue #1: The Signing Fingerprint Nobody Mentioned Up Front

The first time the AAB pipeline actually tried to publish, it failed — and the fix that came back wasn't about GitHub Actions at all:

You need to get the fingerprint from Expo using EAS before you build the AAB.

Expo generates and manages your upload keystore for you by default. Google Play, however, needs to know the SHA-1 fingerprint of whatever key signs your production uploads, in advance, to accept them. Skip that step and your carefully automated pipeline can build a perfectly valid AAB — that Play will reject anyway, because the signature attached to it doesn't match anything Play has been told to trust.

🧮 Issue #2: The Version Number Nobody Could Trust

With signing sorted, releases started reaching the actual Play publish step — and immediately hit a wall neither of us expected:

##[error]You cannot rollout this release because it does not allow
any existing users to upgrade to the newly added APKs.

The build that failed this way was tagged v1.0.1-b2. The release we'd actually intended to ship was v1.2.1-b10 — a completely different version, several minor releases ahead.

Here's what was happening: the build script tracked the current version and build number in a JSON file on disk — local-builds/.version-state.json — incrementing it on every build. That file was .gitignored, which is correct for a build artifact, except we were also using it as the source of truth for what version to ship next. A fresh GitHub Actions runner has no memory of any previous run. Every real release started that counter over from scratch, confidently building v1.0.1-b2 while genuinely believing it was doing the right thing.

The fix had two parts, because there were actually two numbers being tracked, and neither belonged where it was living:

The version name (1.2.1) now comes from the highest-numbered file already committed in our docs/releases/<app>/ folder — release notes we write by hand anyway, so the version they're named for becomes the source of truth instead of a side effect of when the counter last ran.
The build number (versionCode) — this is the one that actually matters to Google, and it's the one with a much sharper rule underneath it.

❓ Why Does Google Play Reject My App With "Cannot Roll Out" or "Does Not Allow Existing Users to Upgrade"?

The part that catches people off guard: this rule applies per app, across every track — not per track.

If your internal testing track, or an old one-off upload, or even a build someone did for a pentest, ever used versionCode 12 for your package, Google Play will reject any new upload to any track — your production track, a fresh closed-testing track, doesn't matter — with versionCode ≤ 12. The number is global to the app. It is not scoped to whichever track's dropdown you happen to be looking at.

We fixed this by never trusting a local counter (or a human-typed release name — more on that below) again. Before every real build, the pipeline now asks Google directly:

const edit = await androidpublisher.edits.insert({ packageName });
const { data } = await androidpublisher.edits.tracks.list({ packageName, editId: edit.data.id });

const versionCodes = (data.tracks ?? []).flatMap((track) =>
  (track.releases ?? []).flatMap((release) =>
    (release.versionCodes ?? []).map(Number),
  ),
);

const highestKnownVersionCode = versionCodes.length ? Math.max(...versionCodes) : 0;

🕵️ Why This Was Sneaky: Play Console's Display Name Isn't the Real Number

While debugging the versionCode issue, we found something that would have derailed us if we'd trusted it: Play Console showed a release literally labeled "Student v1.2.0 (9)" — implying versionCode 9. The actual enforced versionCode on that release, queried directly from the API, was 6.

The (9) is a free-text release name — a label a human typed in, completely decoupled from the number Google actually enforces. It's not malicious, it's just how Play Console's UI works: the name field is yours to fill in however you like, and Play never cross-checks it against the real version. If we'd built our fix around "the console says 9, so ship 10," we'd have hit the exact same rollout rejection all over again, just at a different number.

Rule of thumb: if a number matters to an automated pipeline, query the field that's actually enforced. Never infer it from a label a human wrote for other humans.

🎯 Issue #3: The Play Store Error That Was Lying About Its Cause

With versioning fixed, the student app shipped cleanly. The staff app kept failing — always the same message:

##[error]Release in track targeting no countries

This is where we burned the most time, because every theory we had was plausible and wrong.

Theory 1: the closed-testing track just needs its countries selected. Checked Play Console — countries were already set, and had been from the very first release.

Theory 2: maybe it's set for one app's track but not the other's, since only staff was failing. Also wrong — nothing had been touched differently between the two apps' track settings.

Theory 3: the API field for a track's country availability (countryTargeting) must only be settable for certain release states. Also a dead end — that field isn't even writable through Google's own API for anything other than the production track.

The actual answer only showed up once we stopped guessing and asked Play directly for every track the staff app had, not just the one our pipeline was configured to publish to:

[diagnostic] com.cyfamod.staff all tracks — "production": []
[diagnostic] com.cyfamod.staff all tracks — "beta": []
[diagnostic] com.cyfamod.staff all tracks — "alpha": []
[diagnostic] com.cyfamod.staff all tracks — "internal": []
[diagnostic] com.cyfamod.staff all tracks — "Cyfamod SMS Staff Beta": [
  { "name": "v1.2.0 (9) - Result PINs nav + freshness fixes", "versionCodes": ["12"], "status": "completed" }
]

Four tracks — production, beta, alpha, internal — all completely empty, never used. And a fifth, custom-named track, Cyfamod SMS Staff Beta, holding the actual live release with real testers and real country availability.

The fix: a single shared "which track do we publish to" setting can't survive two apps that don't happen to share a track name. We split it into one variable per app (PLAY_STORE_TRACK_STAFF, PLAY_STORE_TRACK_STUDENT) instead of one value everyone assumed would fit.

🧨 Issue #4: The One-Line Bug That Invalidated an Entire Workflow

Fixing "which app failed" reporting in our Discord notifications introduced its own bug — one worth calling out because of how completely it hid itself.

- name: Record release result
  if: always()
  run: |
    result="${{ failure() && 'failure' || 'success' }}"

The tool that actually catches this is actionlint, which understands GitHub Actions' schema and expression rules specifically — not a generic YAML parser:

$ actionlint .github/workflows/release-android.yml
# (zero errors, after switching to two if:-gated steps instead)

The fix is the standard pattern for exactly this situation:

- name: Record release result — success
  if: success()
  run: echo "ROUTE_RESULT=success" >> "$GITHUB_ENV"

- name: Record release result — failure
  if: failure()
  run: echo "ROUTE_RESULT=failure" >> "$GITHUB_ENV"

🚦 Issue #5: One Merge, Two Apps, Zero Control

The last one wasn't a bug — it was a design decision we'd made early and outgrown. Merging a PR into main triggered a full build-and-publish of both apps, unconditionally, every time.

That was fine when both apps usually shipped together. It stopped being fine the moment a fix was scoped to just one of them. Landing the staff-only track fix meant main's matrix build would rebuild student too — an app that had already succeeded and was sitting in Play Console under review. Resubmitting it wouldn't just waste 45 minutes of CI time; it would reset a review clock that was already ticking down, for a change that had nothing to do with student at all.

The fix was to stop treating "code is production-ready" and "publish this app to the Play Store" as the same event:

# Every pull_request event is preflight-only now — secrets, signing,
# Play-track diagnostics — regardless of branch, regardless of merged status.
# A real build+publish only ever happens via an explicit workflow_dispatch.
if: >-
  github.event_name == 'workflow_dispatch' ||
  (github.event_name == 'pull_request' && github.event.action != 'closed')

I genericized the actual workflow file (package names, secret names, and track names swapped for placeholders — the structure and every comment explaining why are real) into a gist if you want the full shape of it: per-app Android release pipeline template →

🧾 Lessons Learned

What broke	What we learned
Local `.version-state.json`, gitignored	Never trust local/ephemeral state for anything CI needs to stay in sync with an external system — query the system directly
`versionCode` rejected across an unrelated track	Google Play's version rule is per-app, across every track — not scoped to the one you're looking at
Play Console's release name said "(9)", real versionCode was 6	A human-readable label is never the same thing as the field a platform actually enforces
"No countries" error, countries were already set	When a platform's error message doesn't match reality, get the complete picture (every track) before trusting any one theory
`failure()` used outside `if:`	Some expressions are context-restricted; a generic syntax check won't catch a schema violation — use a tool that understands the platform's own rules
One merge auto-released two apps	"Code is integrated" and "ship this specific thing" are different decisions the moment you have more than one independently-releasable unit

🚀 Where It Stands Now

Both apps release independently, on demand, with the actual versionCode and track resolved from Google Play itself before every build — never from a counter or a label that could quietly drift from reality. Landing code on main is safe by default; shipping to the Play Store is a deliberate, scoped action every time. Failures for one app no longer masquerade as failures for both in our Discord channel, and a workflow file that can't be parsed shows up as a fast, obvious signal instead of a silent no-op.

None of this was obvious going in. Most of it wasn't obvious after the first fix either — three of these six issues required a wrong theory (sometimes two) before the real cause showed up. If you're building something similar and any of these error messages looked familiar on the way in, hopefully this saved you a few of the wrong turns.

The full, genericized workflow — every step, every gotcha commented inline, plus a complete required-secrets-and-variables checklist — is in this gist: per-app Android release pipeline template.

The mobile apps discussed here are closed-source, but the backend and web frontend they talk to are public — school-be-laravel (Laravel 11, multi-tenant) and school-fe-nextjs (Next.js 15) power the same Cyfamod School Management System, alongside school-public-web for the public-facing school sites. If you're curious how the pieces fit together, or want to contribute, they're open for it. More on what we build at cyfamod.com.

Hit a version of any of these? I'd genuinely like to hear which theory you tried first — drop it in the comments.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.