DEV Community: cynthia wahome

Automating Android Play Store Releases, Part 3: The Storage-Quota Wall

cynthia wahome — Tue, 22 Sep 2026 22:21:06 +0000

Part 3 of a 5-part series on automating a multi-app Android release pipeline. Part 1 → and Part 2 → covered getting the pipeline working — signing, versioning, tracks, release discipline. This part covers what happened when a *finished pipeline turned out to have a slow-burning cost problem nobody had budgeted for. Part 4 → covers the redesign that followed, and the first real production ship.*

TL;DR

Weeks after the pipeline in Parts 1 and 2 was working and stable, releases started failing with a message that had nothing to do with code:

Error: Failed to CreateArtifact: Artifact storage quota has been hit.

GitHub's free-tier Actions storage quota is exactly 500MB, total, org-wide, forever. Our pipeline had been quietly consuming several times that in about ten days. The diagnosis chain: ten duplicate copies of the same Gradle cache (branch-scoped by default, and the real dominant cause), a cache key that had been silently broken — and therefore useless — since it was written, and 30-day retention on release files nobody was actually keeping around that long. The cleanup helped. It wasn't the real fix. The real fix was accepting that this pipeline should never again depend on GitHub's shared minutes or storage at all.

The Wall

We'd stopped thinking about this pipeline. That's usually the sign a system is working — until a release run failed with:

Run actions/upload-artifact@v4
Error: Failed to CreateArtifact: Artifact storage quota has been hit.
Unable to upload any new artifacts. Usage is recalculated every 6-12 hours.

GitHub's free organization plan includes a fixed Actions storage allowance — not per-repo, org-wide, and small: 500MB, total, forever. That number is really the punchline of this entire post — everything below is different ways of finding out what actually filled it. Checking usage properly meant going past the summary numbers and actually listing every artifact and every cache GitHub was holding for this one repository:

gh api repos/<org>/<repo>/actions/artifacts --paginate \
  -q '.artifacts[].size_in_bytes' | \
  awk '{sum+=$1; n++} END {printf "count=%d total_MB=%.1f\n", n, sum/1024/1024}'

gh api repos/<org>/<repo>/actions/caches --paginate \
  -q '.actions_caches[] | [.key, .size_in_bytes] | @tsv'

Artifacts alone accounted for nearly two gigabytes — signed release AABs at roughly 70MB apiece, internal test APKs at 20–40MB apiece, accumulated over about two weeks. But a cache-by-cache audit turned up something bigger.

The Real Dominant Cause: Ten Copies of the Same Cache

The Gradle dependency cache — meant to be one cache, reused across runs — showed up as ten separate entries, all effectively identical, together accounting for the largest single share of the quota by far. GitHub scopes Actions caches per-branch by default: a cache saved on one branch isn't shared with another, it's a full new copy. Ten active branches meant ten full copies of the same dependency set, sitting in storage simultaneously, none of them expiring on their own because each one kept getting refreshed by ongoing work on its branch.

This, not the artifact files, was the actual wall. It's also the least visible of everything in this post — nobody sees "10 duplicate caches" in an error message. You only find it by going and actually listing what's in storage instead of assuming you already know.

A Cache That Was Never Actually Caching

Digging into why ten copies existed at all — rather than one cache correctly reused across every branch — turned up a second, compounding bug: the cache key itself was broken.

key: ${{ runner.os }}-gradle-${{ hashFiles('android/gradle/wrapper/gradle-wrapper.properties', 'android/build.gradle') }}

Both paths live in an android/ directory that is .gitignored and only generated later, inside the build script's own prebuild step. Neither file existed yet at the point this cache step ran. hashFiles() against paths that don't exist returns an empty string every single time — so the key never changed based on real config, every run was silently a cache miss dressed up as a cache step, full re-downloads of every Gradle dependency every single build, and (worse, for the quota) every branch's "identical" cache got saved fresh under the same static key instead of ever being correctly reused. Two bugs, same root: a cache system that looked like it was working, doing none of its actual job, on both counts.

Retention Set for "Might Need It Someday," Not "Might Actually Look at It"

The signed release AABs — the 70MB files driving most of the artifact total — had 30-day retention. That number had never been a deliberate choice; it was just Actions' own suggested default, left as-is. The AAB itself gets published to the Play Store in the very same job, seconds after the GitHub copy is uploaded — so the GitHub copy exists purely as a debugging convenience, in case someone needs to grab the exact file that shipped without re-running a build. Nobody had ever actually needed to reach for a 3-week-old copy of one. Thirty days of retention on a large binary that's a debugging convenience, not the deliverable, was pure waste that nobody had sized against what the free storage tier could actually hold.

One Leak That Was Already Closed

Worth naming, if only to be precise about timing: an earlier, unrelated cleanup had already caught and fixed a real duplicate-storage bug — the internal-testing flow used to upload every built APK to both cloud storage and GitHub Actions artifacts, with the GitHub copy never actually used by anything. That fix landed weeks before this specific wall. What the quota investigation actually found wasn't that bug recurring — it was old, already-orphaned artifacts left over from before that earlier fix, still sitting in storage because nothing had gone back to clear the backlog once the leak itself was closed. A useful reminder on its own: fixing where a duplicate gets created doesn't retroactively delete duplicates that already exist.

GitHub also documents a clean pattern for exactly the kind of cache pileup described above: delete a cache the moment the branch or PR it belongs to closes, rather than waiting on an age-based sweep to eventually catch it. Adding that alongside the key fix meant a closed PR's caches stop costing anything the same day, not whenever a weekly cleanup job gets around to them.

The Cleanup That Wasn't the Real Fix

Deleting the backlog and tightening retention buys time. It doesn't change the underlying fact: as long as this pipeline runs on GitHub-hosted infrastructure, it is subject to GitHub-hosted quotas — minutes and storage both, shared across the whole org, and rationed the same way EAS's free build queue was rationed back in Part 1. A quiet accumulation of artifacts had just demonstrated that the exact ceiling we moved off of EAS to escape was still there, one layer up.

The real fix was to stop depending on that shared allowance at all: move the pipeline's actual build job onto a self-hosted runner — infrastructure we control, with disk space and build minutes that aren't rationed by anyone else's billing plan. That's a bigger decision than a cleanup script, and it came with its own first-run surprises, starting with a runner image that turned out to be missing a tool GitHub's hosted images ship by default (/usr/bin/time, used purely to capture build duration and peak memory for diagnostics) — a small, fast fix once the actual migration was live, but a reminder that "self-hosted" means you now own every assumption a hosted image used to make for you silently.

Where Part 3 Leaves Off

The build job no longer runs anywhere GitHub's shared minutes or storage quota can touch it. That closed off the cause of the wall we hit. It didn't yet close off the symptoms — the AAB still sitting in GitHub storage after every publish, the retention nobody had sized — and fixing those surfaced one more bug, this time inside the very notification system meant to tell us when something went wrong.

Next: Part 4 → — killing the artifact-relay design that was quietly reporting real successes as failures, and the first real production ship after all of it.

I write about the debugging journeys nobody puts in the docs — more at wamzii.com.

Automating Android Play Store Releases, Part 2: Wrong Track & Broken YAML

cynthia wahome — Thu, 17 Sep 2026 13:05:39 +0000

Part 2 of a 5-part series on automating a multi-app Android release pipeline. Part 1 → covered the setup, a signing-fingerprint error, and a version number that was quietly drifting from reality. This part picks up right after versioning was fixed. Part 3 → and Part 4 → cover a real Actions storage-quota crisis and what it took to fix it for good.

TL;DR

With version numbers finally trustworthy, one of our two apps started shipping cleanly. The other kept failing with a message that had nothing to do with its actual cause. Along the way we also hit a GitHub Actions bug so quiet it doesn't produce an error — it just makes your entire workflow file stop running, silently — and made a structural call that "code is production-ready" and "ship this specific app to the Play Store" needed to be two separate decisions, not one automatic consequence of the other.

Issue #3: The Play Store Error That Was Lying About Its Cause

With versioning fixed, the student app shipped cleanly. The staff app kept failing — always the same message:

##[error]Release in track targeting no countries

This is where we burned the most time, because every theory we had was plausible and wrong.

Theory 1: the closed-testing track just needs its countries selected. Checked Play Console — countries were already set, and had been from the very first release.

Theory 2: maybe it's set for one app's track but not the other's, since only staff was failing. Also wrong — nothing had been touched differently between the two apps' track settings.

Theory 3: the API field for a track's country availability must only be settable for certain release states. Also a dead end — that field isn't even writable through Google's own API for anything other than the production track.

The actual answer only showed up once we stopped guessing and asked Play directly for every track the staff app had, not just the one our pipeline was configured to publish to. The response, shape preserved but values genericized:

[diagnostic] all tracks for the staff app — "production": []
[diagnostic] all tracks for the staff app — "beta": []
[diagnostic] all tracks for the staff app — "alpha": []
[diagnostic] all tracks for the staff app — "internal": []
[diagnostic] all tracks for the staff app — "<our real internal testing track>": [
  { "name": "<release label>", "versionCodes": ["<real versionCode>"], "status": "completed" }
]

Four of Google's standard named tracks — production, beta, alpha, internal — all completely empty, never used. And a fifth, custom-named track we'd created ourselves, holding the actual live release with real testers and real country availability.

Our pipeline was configured to publish to alpha. That track had never had a release, never had countries configured, because nobody had ever actually used it. The countries were never missing — we were shipping to the wrong track entirely, and Google's error message described the empty track's state accurately; it just never occurred to us to question which track it was talking about.

The student app never had this problem because its real, active track happened to literally be named alpha. One coincidence of naming was the only reason half our releases worked and the other half didn't.

The fix: a single shared "which track do we publish to" setting can't survive two apps that don't happen to share a track name. We split it into one variable per app instead of one value everyone assumed would fit.

Issue #4: The One-Line Bug That Invalidated an Entire Workflow

Fixing "which app failed" reporting in our chat notifications introduced its own bug — one worth calling out because of how completely it hid itself.

We wanted to record, per app, whether that specific release succeeded or failed, so a mixed result (one app ships, one doesn't) wouldn't read as a blanket failure. The first attempt looked reasonable:

- name: Record release result
  if: always()
  run: |
    result="${{ failure() && 'failure' || 'success' }}"

That workflow run showed zero jobs. Not a failed job — no jobs at all, and the run's name fell back to the literal file path instead of the workflow's actual name:. That's GitHub's specific signature for "this workflow file could not be parsed," and generic YAML validation (yaml.safe_load() in Python, or any plain YAML linter) will tell you the file is completely fine, because it is — as YAML.

success(), failure(), cancelled(), and always() are only valid inside an if: condition in GitHub Actions' expression syntax. Use them anywhere else — including inside a run: command, even wrapped in ${{ }} — and the entire workflow file is invalid, not just the step containing the mistake.

The tool that actually catches this is actionlint, which understands GitHub Actions' schema and expression rules specifically — not a generic YAML parser:

$ actionlint .github/workflows/release-android.yml
# (zero errors, after switching to two if:-gated steps instead)

The fix is the standard pattern for exactly this situation:

- name: Record release result — success
  if: success()
  run: echo "ROUTE_RESULT=success" >> "$GITHUB_ENV"

- name: Record release result — failure
  if: failure()
  run: echo "ROUTE_RESULT=failure" >> "$GITHUB_ENV"

Issue #5: One Merge, Two Apps, Zero Control

The last one wasn't a bug — it was a design decision we'd made early and outgrown. Merging a PR into main triggered a full build-and-publish of both apps, unconditionally, every time.

That was fine when both apps usually shipped together. It stopped being fine the moment a fix was scoped to just one of them. Landing the staff-only track fix meant main's matrix build would rebuild student too — an app that had already succeeded and was sitting in Play Console under review. Resubmitting it wouldn't just waste CI time; it would reset a review clock that was already ticking down, for a change that had nothing to do with student at all.

The fix was to stop treating "code is production-ready" and "publish this app to the Play Store" as the same event:

# Every pull_request event is preflight-only now — secrets, signing,
# Play-track diagnostics — regardless of branch, regardless of merged status.
# A real build+publish only ever happens via an explicit workflow_dispatch.
if: >-
  github.event_name == 'workflow_dispatch' ||
  (github.event_name == 'pull_request' && github.event.action != 'closed')

main is still the production branch — merging to it still means "this is ready." It no longer also means "and therefore publish every app to Google Play right now." Shipping a specific app became a deliberate action: pick the branch, pick the app, run the workflow. Adding a third app later means adding a third option to that same dropdown — not rewriting the release logic.

Lessons From Parts 1 and 2

What broke	What we learned
Local version-state file, gitignored	Never trust local/ephemeral state for anything CI needs to stay in sync with an external system — query the system directly
`versionCode` rejected across an unrelated track	Google Play's version rule is per-app, across every track — not scoped to the one you're looking at
Play Console's release label didn't match the real versionCode	A human-readable label is never the same thing as the field a platform actually enforces
"No countries" error, countries were already set	When a platform's error message doesn't match reality, get the complete picture (every track) before trusting any one theory
`failure()` used outside `if:`	Some expressions are context-restricted; a generic syntax check won't catch a schema violation — use a tool that understands the platform's own rules
One merge auto-released two apps	"Code is integrated" and "ship this specific thing" are different decisions the moment you have more than one independently-releasable unit

Where Part 2 Leaves Off

Both apps now released independently, on demand, with the actual versionCode and track resolved from Google Play itself before every build. Landing code on main was safe by default; shipping to the Play Store was a deliberate, scoped action every time. It felt, at this point, like the pipeline was actually finished.

It wasn't. A few weeks later, this exact pipeline quietly filled up an entire year's worth of storage quota in about ten days — and the fix wasn't "clean up some files," it was rethinking what GitHub Actions should and shouldn't be trusted to run at all.

Next: Part 3 → — the storage-quota wall, and why the real fix was a self-hosted runner, not a cleanup script.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

Automating Android Play Store Releases, Part 1: Signing & Version Drift

cynthia wahome — Sat, 12 Sep 2026 06:48:49 +0000

Part 1 of a 5-part series on automating a multi-app Android release pipeline — from a free-tier build queue to a genuine self-hosted infrastructure migration. This part covers the setup and the first two real incidents. Part 2 → covers a Play Store error that lied about its own cause, a one-line bug that invalidated an entire workflow file, and separating "merged" from "released." Part 3 → and Part 4 → cover a real Actions storage-quota crisis and what it took to fix it for good.

TL;DR

We ship two separate Android apps (staff and student/parent) from one Expo React Native codebase, distinguished by an env var at build time. EAS's free build tier kept running out mid-sprint, and running builds locally instead was frying a developer's laptop for twenty minutes at a time — so the goal was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions. Two things nearly derailed that before the pipeline shipped anything real:

EAS's free build tier ran out, and building locally doesn't scale as a substitute.
A gitignored local counter decided our version numbers — it doesn't survive a fresh CI runner, so we silently shipped v1.0.1-b2 when we meant a release several versions ahead.
Google Play's versionCode must strictly increase per app, across every track — not per track. An old, forgotten upload on an unrelated track can block every future release until you exceed it.
Play Console's display name isn't the real number. A release literally labeled "(9)" had an actual enforced versionCode of 6 — a human-typed label, not something Play cross-checks against reality.

If any of that made you wince in recognition, keep reading.

The Setup: Two Apps, One Codebase, One Free-Tier Ceiling

Cyfamod builds a school management platform. Two of its Android apps — one for staff, one for students and parents — ship from a single Expo React Native codebase, switched at build time by a route env var. Same components, same hooks, same infra. Two separate .apk/.aab outputs, two separate Play Store listings.

We started the obvious way: eas build. It's a great service, right up until the free tier's queue tells you this:

I'm trying to create a new build, but since we're on the free tier,
I'll have to wait for an available worker.

Running the build locally instead worked, but running these builds locally takes a lot from a laptop — every local build meant a frozen machine and a developer who couldn't do anything else for twenty minutes. That's not sustainable for two apps that both need regular internal testing builds and eventual Play Store releases.

That ceiling is the entire reason this pipeline exists. The goal from day one wasn't "add CI on top of EAS" — it was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we weren't rationed by a build queue. Everything in this series — the signing, the versioning, the tracks, the triggers, and eventually a real infrastructure migration — is what it actually takes to replace a managed build service with your own pipeline once you commit to that move.

Attempt #1: Internal Testing, Automated

The first real automation target wasn't even the Play Store — it was just getting a testable build in front of the team without anyone's laptop catching fire. The shape that emerged:

Branch	What happens
`dev`	Build APKs for both apps, upload to cloud storage, post download links to the team's chat automatically — internal testing only
`main`	Build signed AABs and release to the Google Play Store

Simple on paper. Getting there took several real production incidents, in this order.

Issue #1: The Signing Fingerprint Nobody Mentioned Up Front

The first time the AAB pipeline actually tried to publish, it failed — and the fix wasn't about GitHub Actions at all. Expo generates and manages your upload keystore for you by default. Google Play, however, needs to know the SHA-1 fingerprint of whatever key signs your production uploads, in advance, to accept them. Skip that step and your carefully automated pipeline can build a perfectly valid AAB — that Play will reject anyway, because the signature attached to it doesn't match anything Play has been told to trust.

Once we pulled the real upload-key fingerprint via eas credentials and pinned it as an expected value the workflow checks before ever spending time on a Gradle build, this class of failure became a fast, cheap preflight check instead of a wasted 40-minute build.

Issue #2: The Version Number Nobody Could Trust

With signing sorted, releases started reaching the actual Play publish step — and immediately hit a wall neither of us expected:

##[error]You cannot rollout this release because it does not allow
any existing users to upgrade to the newly added APKs.

The build that failed this way was several minor versions behind the release we'd actually intended to ship. Here's what was happening: the build script tracked the current version and build number in a JSON file on disk, incrementing it on every build. That file was .gitignored — correct for a build artifact, except we were also using it as the source of truth for what version to ship next. A fresh GitHub Actions runner has no memory of any previous run. Every real release started that counter over from scratch, confidently building an old version number while genuinely believing it was doing the right thing.

The fix had two parts, because there were actually two numbers being tracked, and neither belonged where it was living:

The version name now comes from the highest-numbered file already committed in our release-notes folder — notes we write by hand anyway, so the version they're named for becomes the source of truth instead of a side effect of when a counter last ran.
The build number (versionCode) — the one that actually matters to Google, and the one with a much sharper rule underneath it.

Why Google Play Rejects an App With "Cannot Roll Out" or "Does Not Allow Existing Users to Upgrade"

If you've landed here from a search engine: this error means Google Play has already seen a higher versionCode for your package than the one you're currently trying to upload — and it will not let a newer release ship with a lower or equal number, because that would look like a downgrade to users already on the higher version.

The part that catches people off guard: this rule applies per app, across every track — not per track. If your internal testing track, or an old one-off upload, or even a build someone did for a pentest, ever used a given versionCode for your package, Google Play will reject any new upload to any track — production, a fresh closed-testing track, doesn't matter — with an equal or lower number. The number is global to the app, not scoped to whichever track's dropdown you happen to be looking at.

We fixed this by never trusting a local counter (or a human-typed release name — more on that below) again. Before every real build, the pipeline now asks Google directly:

const edit = await androidpublisher.edits.insert({ packageName });
const { data } = await androidpublisher.edits.tracks.list({ packageName, editId: edit.data.id });

const versionCodes = (data.tracks ?? []).flatMap((track) =>
  (track.releases ?? []).flatMap((release) =>
    (release.versionCodes ?? []).map(Number),
  ),
);

const highestKnownVersionCode = versionCodes.length ? Math.max(...versionCodes) : 0;

That single query, across every track the app has, is the only reliable floor for "what number comes next." Nothing else — not our own records, not a display name in the Play Console UI — can be trusted, which brings us to the next surprise.

Play Console's Display Name Isn't the Real Number

While debugging the versionCode issue, we found something that would have derailed us if we'd trusted it: Play Console showed a release literally labeled with a build number in its name — implying that number was the real versionCode. The actual enforced versionCode on that release, queried directly from the API, was a different, lower number.

The label is free-text — a name a human typed in, completely decoupled from the number Google actually enforces. It's not malicious, it's just how Play Console's UI works: the name field is yours to fill in however you like, and Play never cross-checks it against the real version. If we'd built our fix around "the console label says N, so ship N+1," we'd have hit the exact same rollout rejection all over again, just at a different number.

Rule of thumb: if a number matters to an automated pipeline, query the field that's actually enforced. Never infer it from a label a human wrote for other humans.

Where Part 1 Leaves Off

Version numbers were now trustworthy — resolved straight from Google's own records, every time, no local state involved. That fixed what number a release would ship as. It didn't yet fix where it would ship — which turned out to be its own, much stranger problem.

Next: Part 2 → — a Play Store error message that was actively lying about its own cause, a one-line GitHub Actions bug that invalidated an entire workflow file, and why "code merged to main" and "release this app to the Play Store" had to become two different decisions.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

We Automated Play Store Releases for Two Android Apps — Here’s What Broke

cynthia wahome — Mon, 10 Aug 2026 17:56:31 +0000

A complete debugging journey through automating a multi-app Android release pipeline — from local builds burning out a free tier, to a Google Play error message that was actively lying to us about its own cause.

TL;DR

We ship two separate Android apps (staff and student/parent) from one Expo React Native codebase, distinguished by an env var at build time. EAS's free build tier kept running out mid-sprint, and running builds locally instead was frying a developer's laptop for twenty minutes at a time — so the actual goal was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we already had free CI minutes. Along the way:

EAS's free build tier ran out — building locally on a laptop instead doesn't scale either.
A gitignored local counter decided our version numbers — it doesn't survive a fresh CI runner, so we silently shipped v1.0.1-b2 when we meant v1.1.1-b10.
Google Play's versionCode must strictly increase per app, across every track — not per track. An old, forgotten upload on an unrelated track can block every future release until you exceed it.
"Release in track targeting no countries" was not a countries problem. It took us three wrong theories to find out our pipeline was quietly publishing to an empty, unused Play track — while the real one, with real testers, sat one dropdown away.
success() and failure() only work inside an if: condition in GitHub Actions. Use them anywhere else and your entire workflow file becomes invalid — not just the step you wrote it in.
"Code merged to main" and "release this app to the Play Store" are two different decisions. Wiring them together meant one app's fix forced a pointless rebuild — and a resubmission mid-review — of an app that had already shipped fine.

If any of those six sentences made you wince in recognition, keep reading. (If you just want the finished, genericized workflow, skip straight to the gist — full YAML plus every required secret and variable, no signup required.)

🚨 The Setup: Two Apps, One Codebase, One Free-Tier Ceiling

Cyfamod-SMS is a school management platform. Two of its Android apps — one for staff, one for students and parents — ship from a single Expo React Native codebase, switched at build time by an EXPO_PUBLIC_ROUTE env var. Same components, same hooks, same infra. Two separate .apk/.aab outputs, two separate Play Store listings.

We started the obvious way: eas build. It's a great service, right up until the free tier's queue tells you this:

I'm trying to create a new build, but since we're on the free tier,
I'll have to wait for an available worker.

Running the build locally instead worked, but:

Running these builds locally takes a lot from my pc

A laptop is not a build farm. Every local build meant a frozen machine and a developer who couldn't do anything else for twenty minutes. That's not sustainable for two apps that both need regular internal testing builds and eventual Play Store releases.

That ceiling is the entire reason this pipeline exists. The goal from day one wasn't "add CI on top of EAS" — it was to move Android builds and Play Store releases off EAS entirely and onto GitHub Actions, where we already had free CI minutes and weren't rationed by a build queue. Everything in this post — the signing, the versioning, the tracks, the triggers — is what it actually takes to replace a managed build service with your own pipeline once you commit to that move.

📦 Attempt #1: Internal Testing, Automated

The first real automation target wasn't even the Play Store — it was just getting a testable build in front of the team without anyone's laptop catching fire. The idea, roughly:

I need it automated in a way that when someone pushes to the dev branch, the pipeline will build into an APK file and upload it to S3, then the links to both APK files (staff and student) from S3 will be sent to our build channel automatically.

That became the shape of the whole system going forward:

Branch	What happens
`dev`	Build APKs for both apps, upload to S3, post download links to Discord — internal testing only
`main`	Build signed AABs and release to the Google Play Store

Simple on paper. Getting there took several real production incidents, in this order.

🔐 Issue #1: The Signing Fingerprint Nobody Mentioned Up Front

The first time the AAB pipeline actually tried to publish, it failed — and the fix that came back wasn't about GitHub Actions at all:

You need to get the fingerprint from Expo using EAS before you build the AAB.

Expo generates and manages your upload keystore for you by default. Google Play, however, needs to know the SHA-1 fingerprint of whatever key signs your production uploads, in advance, to accept them. Skip that step and your carefully automated pipeline can build a perfectly valid AAB — that Play will reject anyway, because the signature attached to it doesn't match anything Play has been told to trust.

🧮 Issue #2: The Version Number Nobody Could Trust

With signing sorted, releases started reaching the actual Play publish step — and immediately hit a wall neither of us expected:

##[error]You cannot rollout this release because it does not allow
any existing users to upgrade to the newly added APKs.

The build that failed this way was tagged v1.0.1-b2. The release we'd actually intended to ship was v1.2.1-b10 — a completely different version, several minor releases ahead.

Here's what was happening: the build script tracked the current version and build number in a JSON file on disk — local-builds/.version-state.json — incrementing it on every build. That file was .gitignored, which is correct for a build artifact, except we were also using it as the source of truth for what version to ship next. A fresh GitHub Actions runner has no memory of any previous run. Every real release started that counter over from scratch, confidently building v1.0.1-b2 while genuinely believing it was doing the right thing.

The fix had two parts, because there were actually two numbers being tracked, and neither belonged where it was living:

The version name (1.2.1) now comes from the highest-numbered file already committed in our docs/releases/<app>/ folder — release notes we write by hand anyway, so the version they're named for becomes the source of truth instead of a side effect of when the counter last ran.
The build number (versionCode) — this is the one that actually matters to Google, and it's the one with a much sharper rule underneath it.

❓ Why Does Google Play Reject My App With "Cannot Roll Out" or "Does Not Allow Existing Users to Upgrade"?

The part that catches people off guard: this rule applies per app, across every track — not per track.

If your internal testing track, or an old one-off upload, or even a build someone did for a pentest, ever used versionCode 12 for your package, Google Play will reject any new upload to any track — your production track, a fresh closed-testing track, doesn't matter — with versionCode ≤ 12. The number is global to the app. It is not scoped to whichever track's dropdown you happen to be looking at.

We fixed this by never trusting a local counter (or a human-typed release name — more on that below) again. Before every real build, the pipeline now asks Google directly:

const edit = await androidpublisher.edits.insert({ packageName });
const { data } = await androidpublisher.edits.tracks.list({ packageName, editId: edit.data.id });

const versionCodes = (data.tracks ?? []).flatMap((track) =>
  (track.releases ?? []).flatMap((release) =>
    (release.versionCodes ?? []).map(Number),
  ),
);

const highestKnownVersionCode = versionCodes.length ? Math.max(...versionCodes) : 0;

🕵️ Why This Was Sneaky: Play Console's Display Name Isn't the Real Number

While debugging the versionCode issue, we found something that would have derailed us if we'd trusted it: Play Console showed a release literally labeled "Student v1.2.0 (9)" — implying versionCode 9. The actual enforced versionCode on that release, queried directly from the API, was 6.

The (9) is a free-text release name — a label a human typed in, completely decoupled from the number Google actually enforces. It's not malicious, it's just how Play Console's UI works: the name field is yours to fill in however you like, and Play never cross-checks it against the real version. If we'd built our fix around "the console says 9, so ship 10," we'd have hit the exact same rollout rejection all over again, just at a different number.

Rule of thumb: if a number matters to an automated pipeline, query the field that's actually enforced. Never infer it from a label a human wrote for other humans.

🎯 Issue #3: The Play Store Error That Was Lying About Its Cause

With versioning fixed, the student app shipped cleanly. The staff app kept failing — always the same message:

##[error]Release in track targeting no countries

This is where we burned the most time, because every theory we had was plausible and wrong.

Theory 1: the closed-testing track just needs its countries selected. Checked Play Console — countries were already set, and had been from the very first release.

Theory 2: maybe it's set for one app's track but not the other's, since only staff was failing. Also wrong — nothing had been touched differently between the two apps' track settings.

Theory 3: the API field for a track's country availability (countryTargeting) must only be settable for certain release states. Also a dead end — that field isn't even writable through Google's own API for anything other than the production track.

The actual answer only showed up once we stopped guessing and asked Play directly for every track the staff app had, not just the one our pipeline was configured to publish to:

[diagnostic] com.cyfamod.staff all tracks — "production": []
[diagnostic] com.cyfamod.staff all tracks — "beta": []
[diagnostic] com.cyfamod.staff all tracks — "alpha": []
[diagnostic] com.cyfamod.staff all tracks — "internal": []
[diagnostic] com.cyfamod.staff all tracks — "Cyfamod SMS Staff Beta": [
  { "name": "v1.2.0 (9) - Result PINs nav + freshness fixes", "versionCodes": ["12"], "status": "completed" }
]

Four tracks — production, beta, alpha, internal — all completely empty, never used. And a fifth, custom-named track, Cyfamod SMS Staff Beta, holding the actual live release with real testers and real country availability.

The fix: a single shared "which track do we publish to" setting can't survive two apps that don't happen to share a track name. We split it into one variable per app (PLAY_STORE_TRACK_STAFF, PLAY_STORE_TRACK_STUDENT) instead of one value everyone assumed would fit.

🧨 Issue #4: The One-Line Bug That Invalidated an Entire Workflow

Fixing "which app failed" reporting in our Discord notifications introduced its own bug — one worth calling out because of how completely it hid itself.

- name: Record release result
  if: always()
  run: |
    result="${{ failure() && 'failure' || 'success' }}"

The tool that actually catches this is actionlint, which understands GitHub Actions' schema and expression rules specifically — not a generic YAML parser:

$ actionlint .github/workflows/release-android.yml
# (zero errors, after switching to two if:-gated steps instead)

The fix is the standard pattern for exactly this situation:

- name: Record release result — success
  if: success()
  run: echo "ROUTE_RESULT=success" >> "$GITHUB_ENV"

- name: Record release result — failure
  if: failure()
  run: echo "ROUTE_RESULT=failure" >> "$GITHUB_ENV"

🚦 Issue #5: One Merge, Two Apps, Zero Control

The last one wasn't a bug — it was a design decision we'd made early and outgrown. Merging a PR into main triggered a full build-and-publish of both apps, unconditionally, every time.

That was fine when both apps usually shipped together. It stopped being fine the moment a fix was scoped to just one of them. Landing the staff-only track fix meant main's matrix build would rebuild student too — an app that had already succeeded and was sitting in Play Console under review. Resubmitting it wouldn't just waste 45 minutes of CI time; it would reset a review clock that was already ticking down, for a change that had nothing to do with student at all.

The fix was to stop treating "code is production-ready" and "publish this app to the Play Store" as the same event:

# Every pull_request event is preflight-only now — secrets, signing,
# Play-track diagnostics — regardless of branch, regardless of merged status.
# A real build+publish only ever happens via an explicit workflow_dispatch.
if: >-
  github.event_name == 'workflow_dispatch' ||
  (github.event_name == 'pull_request' && github.event.action != 'closed')

I genericized the actual workflow file (package names, secret names, and track names swapped for placeholders — the structure and every comment explaining why are real) into a gist if you want the full shape of it: per-app Android release pipeline template →

🧾 Lessons Learned

What broke	What we learned
Local `.version-state.json`, gitignored	Never trust local/ephemeral state for anything CI needs to stay in sync with an external system — query the system directly
`versionCode` rejected across an unrelated track	Google Play's version rule is per-app, across every track — not scoped to the one you're looking at
Play Console's release name said "(9)", real versionCode was 6	A human-readable label is never the same thing as the field a platform actually enforces
"No countries" error, countries were already set	When a platform's error message doesn't match reality, get the complete picture (every track) before trusting any one theory
`failure()` used outside `if:`	Some expressions are context-restricted; a generic syntax check won't catch a schema violation — use a tool that understands the platform's own rules
One merge auto-released two apps	"Code is integrated" and "ship this specific thing" are different decisions the moment you have more than one independently-releasable unit

🚀 Where It Stands Now

Both apps release independently, on demand, with the actual versionCode and track resolved from Google Play itself before every build — never from a counter or a label that could quietly drift from reality. Landing code on main is safe by default; shipping to the Play Store is a deliberate, scoped action every time. Failures for one app no longer masquerade as failures for both in our Discord channel, and a workflow file that can't be parsed shows up as a fast, obvious signal instead of a silent no-op.

None of this was obvious going in. Most of it wasn't obvious after the first fix either — three of these six issues required a wrong theory (sometimes two) before the real cause showed up. If you're building something similar and any of these error messages looked familiar on the way in, hopefully this saved you a few of the wrong turns.

The full, genericized workflow — every step, every gotcha commented inline, plus a complete required-secrets-and-variables checklist — is in this gist: per-app Android release pipeline template.

The mobile apps discussed here are closed-source, but the backend and web frontend they talk to are public — school-be-laravel (Laravel 11, multi-tenant) and school-fe-nextjs (Next.js 15) power the same Cyfamod School Management System, alongside school-public-web for the public-facing school sites. If you're curious how the pieces fit together, or want to contribute, they're open for it. More on what we build at cyfamod.com.

Hit a version of any of these? I'd genuinely like to hear which theory you tried first — drop it in the comments.

I write about the debugging journeys nobody puts in the docs — more at cycy.is-a.dev.

The Supabase Gotchas Nobody Warns You About (Until You Hit Them)

cynthia wahome — Fri, 20 Mar 2026 17:23:24 +0000

Supabase has some of the best developer experience (DX) in the BaaS space right now. DX is shorthand for how pleasant a tool is to work with — the docs, the dashboard, the auto-generated APIs, the speed at which you go from zero to something working. On all of those: Supabase is excellent.

But excellent DX has a shadow side. When a platform abstracts things this smoothly, it's easy to forget what's running underneath. And what's running underneath Supabase is PostgreSQL — a powerful, strict, battle-tested database that has its own permission system, its own rules, and absolutely no obligation to care how clean your dashboard looks.

These are the two things that will catch you — usually at the same time, usually showing the same symptom.

The Symptom

You've set up auth. Users sign up, verify their email, try to log in — and your app shows something like a "pending approval" or "no role assigned" screen even though everything completed correctly.

Your trigger is firing. Your RLS policies are in place. The Supabase dashboard looks fine.

Nothing is broken. Nothing is obviously wrong.

This is the trap.

Gotcha 1: The GRANT That Nobody Told You About

This is the most common question in the Supabase Discord and it's easy to see why — it violates every reasonable expectation.

When you enable Row Level Security (RLS) on a table in Supabase, you create policies that define who can do what with the rows. A policy might say: "authenticated users can read their own rows." Logical. Clear.

What the dashboard doesn't shout at you is that RLS policies and table permissions are two entirely separate systems in PostgreSQL.

Here's the mental model:

Layer	What it controls	Analogy
GRANT	Can this role access the table at all?	The keycard that opens the building
RLS Policy	Which rows can this role see?	Which floors that keycard can reach

Without a GRANT, the keycard doesn't work. PostgreSQL's PostgREST layer turns you away at the door before RLS even runs. You can have perfectly written policies and they will never be evaluated — because the role was never let in to begin with.

This is why you can get permission denied for table user_roles even with the service_role key — the key that's supposed to bypass everything. Without explicit GRANTs, "supposed to bypass everything" turns out to have an asterisk.

The fix is two lines that most Supabase tutorials never show you:

GRANT ALL ON public.user_roles TO service_role, authenticated;
GRANT ALL ON public.profiles TO service_role, authenticated;

That's it. Two lines of SQL that most migration guides don't include.

The Migration Template You Should Steal

Every time you create a table in Supabase, this is the full pattern. Save it as a GitHub Gist. Tattoo it somewhere:

-- 1. Create the table
CREATE TABLE public.my_table (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  created_at TIMESTAMPTZ DEFAULT now()
);

-- 2. Enable RLS
ALTER TABLE public.my_table ENABLE ROW LEVEL SECURITY;

-- 3. Write your policies
CREATE POLICY "authenticated_can_select"
  ON public.my_table
  FOR SELECT TO authenticated
  USING (true);

-- 4. THE PART EVERYONE FORGETS
GRANT ALL ON public.my_table TO service_role, authenticated;
GRANT SELECT ON public.my_table TO anon;

Step 4 is not optional. It's not an edge case. It is required every single time, and the dashboard will not remind you.

The 60-Second Diagnostic

Hit permission denied and not sure why? Run these three queries in the Supabase SQL editor:

-- Is RLS actually enabled?
SELECT relname, relrowsecurity
FROM pg_class WHERE relname = 'your_table_name';

-- Do your policies exist?
SELECT policyname, cmd, roles
FROM pg_policies WHERE tablename = 'your_table_name';

-- Do GRANTs exist? (this is usually the missing one)
SELECT grantee, privilege_type
FROM information_schema.table_privileges
WHERE table_name = 'your_table_name';

If the third query returns nothing for service_role or authenticated — you found it.

Gotcha 2: The Race Condition Nobody Mentions

This one is subtler and it shows up as a UI flicker — your app briefly flashes an "account pending" or "access denied" screen for a user who is fully verified and approved.

It's not a database problem. It's a timing problem.

When a user logs in, your frontend is doing at least two async things at once: confirming the auth session exists, and fetching the user's role from the database. If your UI renders before the role fetch completes, it sees role = null for a split second and acts accordingly — showing whatever your "no role" fallback state is.

The common mistake is initialising the loading state as false:

// This is the trap
const [roleLoading, setRoleLoading] = useState(false);

false means "done loading." Your UI reads that and renders immediately, before the role has arrived.

The fix is using null as the initial state instead:

// null = "I genuinely don't know yet — don't render anything"
const [roleLoading, setRoleLoading] = useState(null);

null means "ask me later." Your UI holds. The role arrives. The correct screen renders. No flicker.

It's a one-character change — false to null — that's the difference between a UI that lies to users for 200ms and one that waits until it knows the truth.

Why This Happens With BaaS Platforms Specifically

Both of these issues share a root cause: Supabase abstracts so much so well that it's easy to stop thinking about the layer underneath.

The dashboard gives you toggles. The client library gives you clean methods. The auto-generated APIs give you something that feels like it's handling everything. And mostly it is — until it isn't, and the failure is silent, and the symptom looks identical whether the problem is a missing GRANT or a race condition or something else entirely.

This isn't a criticism. The abstraction is genuinely the point, and it's genuinely good. But every abstraction has a floor. With Supabase, the floor is PostgreSQL — and PostgreSQL has opinions.

The moment you remember that Supabase is PostgreSQL with excellent packaging, a lot of otherwise mysterious behaviour starts making sense.

The Pre-Launch Auth Checklist

Before you ship any Supabase auth flow:

[ ] Every table has both RLS policies and GRANT statements
[ ] Role/loading state initialises as null, not false
[ ] Tested clean: run npx supabase db reset --local from scratch at least once
[ ] Tested with both anon key and service_role key separately

That third one matters more than it looks. If you've only ever run your app against a database that was built incrementally and never torn down, you haven't found all your migration gaps yet. A clean reset is the smoke detector.

Hit a Supabase gotcha that this doesn't cover? Drop it in the comments — these things rarely travel alone. 👇

Backend engineer. I write about the bugs I actually hit. Portfolio: cycy.is-a.dev 🚀

Usipoziba Ufa, Utajenga Ukuta — On Technical Debt and the Discipline to Fix It

cynthia wahome — Mon, 16 Mar 2026 11:59:47 +0000

There's a Swahili proverb that lives in my head rent-free as a developer:

Usipoziba ufa, utajenga ukuta.
If you don't seal the crack, you'll rebuild the wall.

I was recently doing a technical review of a school management system — a real, live, production codebase built by a team moving fast and shipping features. The kind of project where prototyping decisions become permanent ones, and technical debt quietly accumulates in the walls.

I could have done the minimum. Fixed what was on the ticket, shipped, moved on. Instead I did what I always do — I kept pulling the thread.

Here's what I found, and the first principles behind every fix.

Context (Because You Deserve It)

This was a PHP/Laravel backend — a language I was relatively new to at the time. Laravel is a popular PHP web framework, and Tinker is its interactive console, like a REPL you can use to query and manipulate the database directly without building a UI. Think of it as a surgical tool for backend diagnosis.

The system managed multiple schools on one platform (multi-tenancy), had just added a sales agent referral program, and needed a security and UX audit before the next release.

I wasn't here to judge the original code. Prototyping leaves debt — that's not incompetence, that's software. My job was to leave it better than I found it.

Screenshots and PR references: [link to PRs here when publishing]

1. Seal the Crack Before You Need a New Wall

The most critical find: a single helper method with a dangerous fallback.

The system was supposed to identify which sales agent was making a request by checking their login session. Correct. But if that check failed — instead of stopping — it fell back to reading an agent_id value straight from the URL.

Anyone. Any authenticated user. Could type ?agent_id=someone-elses-id in the URL and gain full access to that agent's dashboard, commissions, and payout history.

The fix was one line. But the lesson is about what happens when you don't fix it: a crack becomes a wall you eventually have to rebuild entirely — after the breach.

The principle: User input is untrusted. Always. URL parameters, form fields, headers, cookies — all of it can be faked. Never use client-supplied data to make an authorisation decision. This is not a PHP principle. This is not a Laravel principle. It is a law of the internet.

2. Never Break the Architecture to Solve One Edge Case

This was the most architectural moment of the whole review.

The system needed a Global Super Admin — someone who could see all agents across all schools. Problem: every user in this system is physically tied to a specific school in the database. That's the security contract. And it's enforced at the database level, not just in code.

The tempting fix: make the school_id column nullable. One migration, problem solved.

I almost did it. Then I stopped.

Making that column nullable wouldn't just solve this one case — it would quietly weaken the security guarantee for every school on the platform. The whole point of the design was that cross-school data leaks are structurally impossible. One nullable exception and that guarantee has a crack in it.

The actual solution: create a real management entity — "Platform HQ" — and make the Super Admin belong to it. The architecture stays intact. The admin has a valid home in the system. No nullable columns. No broken contracts.

Own the strongest room inside the walls. Don't remove the walls.

This transcends PHP, Laravel, any framework. Whenever you're about to modify a database schema to accommodate a single edge case — pause. Ask whether the edge case can be made to fit the existing contract instead.

3. Guard the Exit, Not the Entrance

A product decision dressed as a security question: should agents who sign up via Google OAuth be blocked from the dashboard until an admin manually approves them?

The security case for blocking: obvious. Unverified users shouldn't see sensitive data.

The product case against: onboarding friction at the highest-intent moment is how you lose users. Someone who just signed up via Google and immediately hits a wall will leave. You've completed their registration and immediately punished them for it.

The solution: guard the exit, not the entrance.

Let them in. Let them see the dashboard. Let them understand the product and start generating referrals. But the payout service — where money actually moves — requires approval before it runs.

This is applicable everywhere a payment or financial feature sits behind a registration flow. Google OAuth (or any SSO) is a legitimate, convenient way to onboard users. The mistake is assuming that authentication and authorisation are the same thing. They're not. Authentication says who you are. Authorisation says what you're allowed to do.

Lock the vault. Leave the lobby open.

4. The Bug That Only Existed in Production

A "Ghost Object" bug — one of the more disorienting things you'll encounter.

The dashboard "Approve Agent" button returned an error saying the agent wasn't in the right status. I checked directly in Tinker — the status was correct. So why was the controller seeing null?

Diagnosis via logs showed the entire agent object was empty. The server received the ID from the URL but never fetched the actual record.

Root cause: the admin routes were wired up manually in a way that bypassed the standard API middleware. Route Model Binding — the feature that automatically resolves URL parameters into database records — only runs inside that middleware. Without it, the server got an ID and did nothing with it.

The principle: "Works in local tooling" and "works in the API" are not the same test. Your REPL, your console, your local scripts — they run in a different context than your actual request lifecycle. When something works in one and fails in the other, look at what the request pipeline does that your tooling skips.

5. Harden Your Logic Against Its Environment

This one is underrated. We added trim() and strtolower() to status comparisons in the data model — not because users were submitting messy input, but because different database engines handle string comparison differently.

Postgres is strict. MariaDB can be inconsistent about trailing whitespace and case sensitivity depending on configuration. An internal state machine that depends on string equality needs to be immune to the floor beneath it.

The principle: Write code that's correct regardless of the environment it runs in. Defensive logic isn't paranoia. It's acknowledging that you don't control every variable between your code and the database driver.

What I Actually Came For

I came in to add some UI banners. I left having:

Closed a privilege escalation hole
Preserved a multi-tenancy security contract that almost got weakened
Fixed a ghost object caused by misconfigured middleware
Moved an auth check from the wrong door to the right door
Hardened string comparisons against database inconsistency
Opened issues, flagged technical debt, and added milestones so none of it gets lost

That last point matters. I could have fixed my ticket and left. But I logged everything else I found — as issues, as notes, as future work. Because debt you've seen and documented is manageable. Debt you've seen and ignored is a wall waiting to be rebuilt.

I'm naturally curious. I'm not afraid of bugs — they're how I learn. And I genuinely enjoy taking a codebase that's been moving fast and making it move more safely.

Cyfamod Technologies · GitHub

Cyfamod Technologies has 18 repositories available. Follow their code on GitHub.

github.com

The language was PHP. The principles aren't.

Usipoziba ufa, utajenga ukuta.

Seal the crack now. Or rebuild the wall later.

Backend engineer. I write about systems, architecture, and what I learn by poking at things. Portfolio: cycy.is-a.dev 🚀

It Ran Fine in Production. Then I Wrote the Tests.

cynthia wahome — Thu, 12 Mar 2026 21:01:37 +0000

The app worked. Users could log in. Data was saving. Nobody was on fire.

So naturally, I went looking for the fire.

I was asked to audit a new Agent Referral feature on a school management system — agents refer schools, earn commissions, the usual. No brief. No checklist. Just "make sure it's fine."

It was not fine. Nothing was visibly broken, but the architecture had been quietly making peace with some genuinely bad decisions. Everything held together by vibes and accumulated runtime state.

Here's what I found — and why it matters whether you're writing PHP, Python, Go, or whatever language you've convinced yourself won't betray you.

The Back Door That Was Literally Labelled

There was a helper method — resolveAgent() — whose job was to figure out which agent was logged in. Two steps:

Check if the authenticated user is an Agent ✅
If not — check if the URL has an agent_id parameter, and if so, load that agent from the database ❌

Step 2 is called Privilege Escalation. Any authenticated user — a school admin, a teacher, literally anyone — could type ?agent_id=some-uuid into the URL and gain full access to that agent's dashboard, commissions, and payout history.

The fix was one line:

return $user instanceof Agent ? $user : null;

You're an agent or you get nothing. No fallbacks. No trusting what someone typed in a URL.

The principle: Never trust the client. Anything arriving from a browser — URL params, form fields, headers, cookies — can be faked. Assume it is. This isn't a PHP rule or a Laravel rule. It's a law of the internet, as immovable as gravity and significantly less forgiving.

The Database That Only Looked Stable

To prove my fix worked, I wrote tests. Ran them. The whole suite detonated before a single test executed — the database couldn't rebuild itself from the migration files.

Two issues:

Issue 1: A migration tried to set a default value on a TEXT column. TEXT is unbounded — think of it as an essay answer sheet with no word limit. The database's response was essentially "I'm not pre-filling every essay box. That's expensive and I refuse." MariaDB won't allow it. MySQL might let it slide depending on config. They're like twins — 99% identical, except MariaDB read the rulebook.

Issue 2: A composite primary key included a nullable column. A primary key is the database's ID badge for a row. An ID badge that might be blank can't identify anything. The database said no. Correctly.

The wild part? The production app was running fine. The database had been built incrementally over months, never torn down. My tests used RefreshDatabase — which destroys everything and rebuilds from scratch every run. That clean slate exposed every shortcut that had been quietly accumulating.

It's like a building that looks solid because nobody's pulled up the floorboards in years.

The principle: Your code isn't portable until it survives a clean build. If your app only works because the database was assembled "just right" and nobody's had to start fresh — that's not stability, that's a patient time bomb.

The Vault and the Clerk

Once the database rejected the defaults, I had to put them somewhere else. Which raised the interesting question: where do defaults belong?

Think of your backend as a bank:

The vault (the database): stores things safely. Doesn't think. Doesn't decide.
The clerk (the application): handles logic. Decides what to fill in when nobody specified.

The original code was asking the vault to think. The vault refused.

I moved the defaults to the Model layer — the part of the code that defines what a "Term Summary" actually is:

protected $attributes = [
    'overall_comment' => 'This student is good.',
    'principal_comment' => 'This student is hardworking.',
];

Now the application handles the logic. The database just stores. Swap Postgres for MySQL tomorrow — the defaults don't care. Rewrite the frontend entirely — the backend doesn't notice.

The principle: Separation of Concerns. Each layer minds its own business. The systems that survive change are the ones where nobody's doing someone else's job.

Tests Are Receipts, Not Optional

I wrote eight tests. One Factory to generate realistic fake data. Then:

5 security tests: a school admin tries every angle to access an agent's dashboard. Every single one returns 401 Unauthorized.
3 auth tests: agents can still register, log in, get rejected with the wrong password. The front door still works.

"I fixed it" is a belief. Eight green checkmarks are evidence.

Six months from now when someone asks "are we certain a school admin can't get into agent data?" — the answer isn't a Slack message. It's a test suite that runs every time someone touches that code.

Tests are documentation that doesn't go stale and can't be misremembered.

The Part Worth Keeping

Two things kept surfacing and they're worth separating cleanly:

First Principles are the physics — the laws that don't change regardless of framework or language:

User input is untrusted. Always.
A primary key that might not exist can't identify anything.
A database won't efficiently enforce constraints it wasn't built for.

Design Patterns are the blueprints that implement those laws:

Zero trust → authentication guards, no URL parameter fallbacks
Defaults the database won't hold → move them to the Model
Prove it works → Factories and tests

The pattern is the tool. The principle is why you pick it up.

Three questions worth asking about every system you touch:

Who owns this responsibility? If the answer isn't obvious, that's the bug.
What is actually, fundamentally true here? Strip the assumptions.
Can you prove it? If not, you don't know — you just haven't been wrong yet.

The recipes change. The physics don't.

Drop a comment — genuinely curious: what's the worst "it worked in production" moment you've walked into? The more specific the horror, the better. 👇

Backend engineer. I write about systems, architecture, and what happens when you poke at things that were better left unexamined. Portfolio: cycy.is-a.dev 🚀

The is-a.dev + Vercel Field Guide: Every Pitfall, Mapped

cynthia wahome — Fri, 06 Mar 2026 10:40:03 +0000

Free .is-a.dev subdomains are genuinely one of the better things the dev community has built. Clean URL, no cost, no registrar drama. The whole process is submitting a JSON file via GitHub Pull Request.

Here's the part nobody front-loads: the repo runs automated CI tests on every PR before a human even looks at it. Fail those, and you're already closed before a reviewer blinks. Pass those, and then a real human volunteer visits your site and reviews your files.

Two gates. Different failure modes. This guide covers both.

Step 0: Check Availability Before Anything Else

👉 https://is-a.dev

Search your name. Takes 10 seconds. Do it before you get attached.

What You Need

GitHub account
A project live, deployed, and has actual content on Vercel — not a placeholder page
Vercel dashboard open at your project → Settings → Domains
A screenshot of your live site (the PR template requires this)

Step 1: Fork, Clone, Branch

git clone https://github.com/your-github-username/register.git
cd register
git checkout -b add-yourname-domain

Fresh branch every time. Don't work on main.

Step 2: Your Domain File

Create domains/yourname.json:

{
  "owner": {
    "username": "your-github-username",
    "email": "your-email@example.com"
  },
  "records": {
    "CNAME": "cname.vercel-dns.com"
  }
}

What the CI tests will catch here

record vs records — The CI schema validation flags record (singular) immediately. Your PR won't survive automated checks. It's records, plural, always.

Invalid JSON — A missing comma or unclosed brace and the CI fails. Validate at jsonlint.com before you push.

File in the wrong place — Must be inside domains/. Not the root, not a subfolder.

A records must be arrays — If you use an A record instead of CNAME, it has to be: "A": ["76.76.21.21"] — array syntax, even with one value.

CNAME + A in the same file — They conflict. CI will catch it. Pick one; for Vercel, CNAME is the right call since their IPs can change.

Self-referencing — A CNAME that points back to yourname.is-a.dev will also get caught.

Use CNAME over A record for Vercel. cname.vercel-dns.com is stable long-term.

Step 3: The Vercel Verification Pitfall 🚨

This is what gets people past CI but stopped by the human reviewer.

First — get your TXT token from Vercel

Your project needs to already be live and deployed. Then:

Vercel project → Settings → Domains → Add Domain
Enter yourname.is-a.dev
Vercel asks if you want to redirect to www.yourname.is-a.dev — do not enable this (more on why in a second)
Select the root domain only → click "Continue manually"
Copy the TXT string Vercel shows you:

vc-domain-verify=yourname.is-a.dev,sometoken123abc

Vercel shows "Invalid Configuration" immediately. That's expected — your PR isn't merged yet. Leave it.

Where NOT to put the TXT record

The obvious move: add the TXT record into yourname.json next to the CNAME. Tidy, logical, wrong.

is-a.dev requires verification records in their own dedicated file. For Vercel, that file is _vercel.yourname.json:

{
  "owner": {
    "username": "your-github-username",
    "email": "your-email@example.com"
  },
  "records": {
    "TXT": "vc-domain-verify=yourname.is-a.dev,your-token-from-vercel"
  }
}

You end up with two files:

File	Does what
`yourname.json`	Routes traffic to Vercel via CNAME
`_vercel.yourname.json`	Proves ownership to Vercel via TXT

One routes. One verifies. That's the unlock.

Step 3b: The WWW Redirect Trap 🪤

Back in Step 3 when Vercel asks about redirecting to www.yourname.is-a.dev — here's why you skip it:

www.yourname.is-a.dev is a completely separate subdomain. It needs its own JSON file and its own PR in the is-a.dev repo. If you enable the redirect without registering www, Vercel goes looking for a domain it can't find — and your site serves a 404 instead of your portfolio.

Worse: if you then try to remove the redirect and go back to the root domain, Vercel can get stuck demanding a new TXT verification token for the www subdomain before it lets you proceed. It's an annoying loop.

If you accidentally clicked it already:

Vercel → Project Settings → Domains
Find www.yourname.is-a.dev
Delete it 🗑️
Root domain only stays

Propagates in about 60 seconds. Back to normal.

Step 4: Submit the PR

git add domains/yourname.json domains/_vercel.yourname.json
git commit -m "feat(domain): add yourname.is-a.dev with Vercel CNAME and verification"
git push origin add-yourname-domain

Open a PR against is-a-dev/register. An automated welcome message appears when you do — read it. It tells you exactly what reviewers check. Short version:

Fill in the PR template — don't delete it or swap it out
Include a screenshot of your live site — not optional, reviewers look for it
Mention the domain is already added in Vercel

Reviewers are volunteers. A complete, clean submission is the only reliable fast path.

Full Pitfall Reference

What kills your PR	When it's caught	Fix
`record` not `records`	CI — automated	Fix the key, validate JSON
Invalid JSON syntax	CI — automated	Run through jsonlint.com
File outside `domains/`	CI — automated	Move it to the right folder
A record as a string not array	CI — automated	`"A": ["ip"]`
CNAME + A in same file	CI — automated	Pick one
CNAME pointing to itself	CI — automated	Don't self-reference
TXT token inside main domain file	Human reviewer	Move to `_vercel.yourname.json`
Domain added to Vercel but shows 404	Human reviewer	Add domain first, let Vercel sit in "pending"
No screenshot in PR description	Human reviewer	Add one
Site is under construction	Human reviewer	Finish it, then submit
`www` redirect enabled without registering `www`	Human reviewer / post-merge	Delete `www` entry in Vercel
PR goes stale after maintainer asks for changes	Administrative	Respond within a few days
Name already taken by a pending PR	Administrative	Check open PRs before submitting
Nested subdomain, parent doesn't exist	CI or reviewer	Register parent domain first

GitHub Pages? Same Pattern

{
  "owner": {
    "username": "your-github-username"
  },
  "records": {
    "CNAME": "your-github-username.github.io"
  }
}

Verification file: _github-pages-challenge-yourname.json — same structure, different prefix. The _platform.yourname.json naming convention applies across the board.

Pre-Submit Checklist

[ ] Name available at is-a.dev
[ ] Fresh branch off main
[ ] domains/yourname.json — CNAME, records plural, valid JSON
[ ] domains/_vercel.yourname.json — TXT token from Vercel
[ ] Domain added to Vercel Settings → Domains, root only, no www redirect
[ ] Site is live with real content
[ ] Screenshot of live site ready for PR description
[ ] PR template filled out, not replaced

After the Merge

Usually live within minutes. Still seeing the is-a.dev homepage after 15–20 min? Flush DNS:

# macOS
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

# Linux
sudo systemd-resolve --flush-caches

# Windows
ipconfig /flushdns

Then enable Enforce HTTPS in Vercel. You have a clean URL now — secure it.

TL;DR

Check is-a.dev first
records — plural, or CI rejects you immediately
CNAME to cname.vercel-dns.com
Get TXT token: Vercel → project → Settings → Domains → Add Domain → Continue manually
TXT token goes in _vercel.yourname.json — separate file, not the main one
Root domain only in Vercel — skip the www redirect
Screenshot in the PR description

Two gates. Two files. Right keys. That's the whole thing.

Written by a developer who now has a working .is-a.dev domain and a portfolio full of backend projects to show for it — cycy.is-a.dev 🚀

this was a really fun drive 🚘 ...plus I totally love synthwave 😍 starting cursing here: https://sundaydevdrive.pilotronica.com contribute to the project here: https://github.com/georgekobaidze/sunday-dev-drive well done @Giorgi Kobaidze

cynthia wahome — Fri, 06 Mar 2026 06:55:25 +0000

DEV Weekend Challenge: Community

Giorgi Kobaidze

Mar 1

Sunday DEV Drive: A Synthwave Driving Experience Through Your DEV Community Articles

#showdev #devchallenge #weekendchallenge

10 min read

Sunday DEV Drive

Drive through your DEV Community articles in a synthwave world. Your posts become neon billboards along an endless road.

sundaydevdrive.pilotronica.com

GitHub - georgekobaidze/sunday-dev-drive: A synthwave driving experience through your DEV Community articles · GitHub

A synthwave driving experience through your DEV Community articles - georgekobaidze/sunday-dev-drive

github.com

Deep Dive: Resolving DNS Issues with GitHub and Understanding SSH vs HTTPS

cynthia wahome — Thu, 05 Mar 2026 10:39:57 +0000

Ever tried pushing your code to GitHub, only to be stopped by a timeout error that makes zero sense? That's exactly what happened to me — and what started as a frustrating blocker turned into a genuinely useful deep dive into DNS, network routing, and Git's connection methods. 🚀

Let me walk you through every step.

The Problem: GitHub, You're Killing Me 😩

Everything was going fine until I ran a routine push:

git push origin feat/add-tests

Instead of the usual success, I got:

fatal: unable to access 'https://github.com/your-username/your-repo.git/':
Failed to connect to github.com port 443 after 75079 ms: Couldn't connect to server

75 seconds of waiting, then nothing. Time to dig in. 🔍

Step 1: Is GitHub Actually Down? 🌐

Before blaming my own setup, I checked whether GitHub was reachable at all:

curl -s -o /dev/null -w "%{http_code}" https://github.com

The response? 000

A 000 status code from curl means the request never even got a response — no HTTP handshake, no connection, nothing. GitHub's HTTPS port was completely unreachable from my machine.

Step 2: Ping & DNS Lookup — Finding the Culprit 🕵️‍♂️

Ping test

$ ping -c 2 github.com
PING github.com (20.87.245.0): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 20.87.245.0: icmp_seq=1 ttl=114 time=72.940 ms

Packet loss right away. The IP 20.87.245.0 was responding inconsistently — not the behavior you'd expect from GitHub's infrastructure.

Note: Packet loss in a ping doesn't always mean slow internet. It often means the specific IP you're being routed to has a problem.

DNS lookup

$ nslookup github.com
Name:    github.com
Address: 20.87.245.0

There it was. DNS was resolving github.com to 20.87.245.0 — a broken or misrouted IP. Large services like GitHub use many IPs across their infrastructure. When DNS caching or propagation goes wrong, you can end up pointed at one that's temporarily dead.

Step 3: Port Check — Confirm It's Not Just a General Outage 🔌

To rule out a broader network issue, I checked whether GitHub's HTTPS port (443) specifically was blocked:

nc -zv github.com 443 -w 5

No connection. Port 443 was unresponsive via that IP.

But this raised an interesting question — if HTTPS (port 443) is blocked, what about SSH (port 22)? I kept that in mind for later.

Step 4: Traceroute — Where Is the Connection Actually Dying? 🗺️

Running a traceroute showed exactly where packets were getting dropped:

$ traceroute github.com
 1  192.168.x.x (192.168.x.x)    14.672 ms
 2  10.x.x.x (10.x.x.x)          9.098 ms
 3  * * *
 4  * * *
 5  * * *

(Local IPs anonymised)

Hops 1 and 2 — my local router and ISP gateway — responded fine. After that: silence. The connection was dying somewhere in the ISP's routing infrastructure, well before reaching GitHub's servers.

This is a classic sign of either:

A routing failure at the ISP level
Traffic being directed to a bad IP that simply drops packets

The DNS evidence from earlier made the latter the most likely culprit.

Step 5: Verify the Fix With a Known-Good IP 🧪

Before touching any system config, I tested whether the network itself was fine by bypassing DNS entirely:

curl --resolve github.com:443:140.82.121.3 https://github.com

This forces curl to connect to 140.82.121.3 (a legitimate GitHub IP) while keeping the hostname intact for TLS. The result? 200 OK. ✅

The network was fine. The problem was purely DNS returning a bad IP.

How to find legitimate GitHub IPs: Check https://api.github.com/meta — GitHub publishes their official IP ranges here.

Step 6: Override DNS via `/etc/hosts` 🔧

With the root cause confirmed, the fastest fix was to bypass DNS locally:

echo "140.82.121.3 github.com" | sudo tee -a /etc/hosts

Or edit the file manually:

sudo nano /etc/hosts
# Add this line:
140.82.121.3 github.com

This tells your OS: "Don't ask DNS where github.com is — I'm telling you directly."

After saving, GitHub loaded instantly in the browser. But the git push was still failing — because my remote was still using HTTPS.

Step 7: Switch Git Remote from HTTPS to SSH 🔑

Here's what I had been missing. Even with the /etc/hosts fix, port 443 was still being affected by the underlying routing issue. SSH uses port 22, which was completely unaffected.

Check your current remote:

git remote -v
# origin  https://github.com/your-username/your-repo.git (fetch)
# origin  https://github.com/your-username/your-repo.git (push)

Switch to SSH:

git remote set-url origin git@github.com:your-username/your-repo.git

Verify:

git remote -v
# origin  git@github.com:your-username/your-repo.git (fetch)
# origin  git@github.com:your-username/your-repo.git (push)

Then:

git push origin feat/add-tests
# ✅ Success

Here's a quick comparison of both methods:

Method	Port	Auth	Best For
HTTPS	443	Username + PAT token	Simple setups, one-off clones
SSH	22	SSH key pair	Frequent pushes, automation, CI/CD

SSH is worth setting up properly if you haven't — no tokens to rotate, no passwords to enter, and as this situation showed, port 22 is far less likely to be caught up in DNS/routing failures than 443.

Key Takeaways 💡

A curl status of 000 means zero connectivity — not a server error, not a redirect. Nothing got through.
DNS can return broken IPs. Large services like GitHub use many IPs. Caching issues can leave you pointed at one that's dead or misrouted.
Traceroute tells you where things break. If packets die at hop 3+, the issue is upstream of you — your ISP or beyond.
/etc/hosts is a surgical bypass tool. It skips DNS entirely for a specific hostname. Useful in emergencies, but remember to revert it once DNS is healthy again.
SSH over HTTPS for Git — always, if you can. It's more secure, needs no token management, and uses a different port that's often unaffected when HTTPS routes break.
Always verify IPs against official sources. Hardcoding a rogue IP in /etc/hosts is a real MITM vector. Use api.github.com/meta to confirm.

Full Troubleshooting Cheatsheet

# 1. Check if HTTPS is reachable at all
curl -s -o /dev/null -w "%{http_code}" https://github.com

# 2. Check DNS resolution
nslookup github.com

# 3. Test connectivity to a known-good IP without changing DNS
curl --resolve github.com:443:140.82.121.3 https://github.com

# 4. Check if port 443 is open
nc -zv github.com 443 -w 5

# 5. Trace where packets are dying
traceroute github.com

# 6. Override DNS locally
echo "140.82.121.3 github.com" | sudo tee -a /etc/hosts

# 7. Check your Git remote
git remote -v

# 8. Switch remote to SSH
git remote set-url origin git@github.com:your-username/your-repo.git

Conclusion

What looked like a random GitHub outage turned out to be a DNS routing failure — solvable with a two-line fix once you know what you're looking for. The traceroute and curl tests were the real turning point: they made the invisible visible, showing exactly where the connection was breaking and why.

If you take nothing else from this: learn to read traceroutes, and set up SSH for Git. Both have saved me hours of confusion.

Got questions, or your own DNS war stories? Drop them in the comments 👇

TL;DR

git push failed with a port 443 timeout
curl returned 000 — no connection at all
DNS was resolving GitHub to a broken IP (20.87.245.0)
Traceroute confirmed the failure was upstream, not local
Fixed short-term with /etc/hosts pointing to a working IP
Fixed properly by switching Git remote from HTTPS → SSH (port 22)

I resonate so much with this

cynthia wahome — Sun, 01 Mar 2026 18:34:40 +0000

NorthernDev

Feb 28

The Hardest Part of Being a Developer Isn’t Coding. It’s Disappearing Quietly.

#discuss #mentalhealth #webdev #career

157

2 min read

I Almost Filed a Bug Report. Then I Pressed Ctrl+P

cynthia wahome — Sun, 01 Mar 2026 10:45:16 +0000

I was using Kilo CLI v7.0.33 and something felt off. I could scroll fine — but there was no scrollbar indicator. No sense of where I was in a long session. Other CLIs had it. Mine didn't.

My immediate take: the scrollbar is broken.

So I went digging.

The GitHub Rabbit Hole

I went through OpenCode issues, Kilo issues, anything mentioning scroll + Warp + iTerm. Left a couple of comments piecing together what I thought was happening:

scrollbars missing #2500

ubuntupunk posted on Sep 08, 2025

I can't seem to scroll up to read what opencode is outputting. I don't have this problem with gemini-cli and any other cli so its not terminal related. The opencode scrollbar is just a sold bar for some reason

View on GitHub

Cannot scroll on opencode when using iterm #6209

anugrahsinghal posted on Dec 26, 2025

Description

When trying to scroll the opencode TUI on iterm, it scrolls the input box but not the output of the previous command

OpenCode version

1.0.203

Steps to reproduce

Install iTerm2
opencode
!ls # to see basic output - should be long enough to overflow from view
scroll

Screenshot and/or share link

https://github.com/user-attachments/assets/eea1ae7e-6401-4ef8-b30d-d5265fee28ca

Operating System

26.2

Terminal

iTerm2

View on GitHub

The threads were noisy but a pattern emerged — people were actually describing two different problems and conflating them:

Problem 1 — Mouse scroll not working at all (terminal layer)
Warp and iTerm need Mouse Reporting enabled before they'll forward scroll events to a full-screen TUI. Without it, your scroll wheel just doesn't reach the app. Fix is in terminal settings.

Problem 2 — Scrollbar not visible (app layer)
Completely separate. Scroll events arriving fine, but the indicator is just... off.

I had Problem 2. But the GitHub threads mostly discussed Problem 1.

Here’s what it looked like in my terminal — no visible scrollbar, long output, no sense of position and it was consistent in all the themes!

At this point, I was convinced it was broken.

Wait, What Am I Even Running?

Before assuming anything, I needed to understand the actual architecture. Kilo isn't just OpenCode — it's a fork:

So I went to the source.

Buried in the session route, I found it. The scrollbar wasn’t missing at all — it was disabled by default. A KV-backed feature flag, persistent across sessions, but initialized to false.

I was one tab away from opening an enhancement issue.

Inside:

opencode/packages/opencode/src/cli/cmd/tui/routes/session/index.tsx

const [showScrollbar, setShowScrollbar] = kv.signal("scrollbar_visible", false)

And later:

verticalScrollbarOptions={{
  visible: showScrollbar(),
}}

That was the moment everything clicked.

The scrollbar existed.
It was toggleable.
And by default — it was off.

Then I Pressed Ctrl+P

Command Palette. I typed "scroll."

Toggle session scrollbar

I pressed it. The scrollbar appeared.

It had been there the whole time.

What This Actually Was

Not a bug. Not a missing feature. A discoverability gap.

The scrollbar is implemented, toggleable, persistent — just hidden behind a keybind nobody thinks to try when they're confused about scroll behavior. That's a UX conversation worth having with maintainers, not a bug report.

After piecing everything together, I went back to the GitHub threads.

Instead of filing a new issue, I left a couple of comments clarifying what I’d found — separating the terminal mouse-reporting problem from the scrollbar visibility toggle. A small thing, but hopefully useful signal in a noisy thread.

Sometimes contributing isn’t about opening something new.
It’s about tightening what already exists.

The Real Takeaways

Separate layers before blaming. Terminal config and app state are different things. Conflating them sends you in circles.

Read source before filing. That false default was right there. Five minutes of reading saved a maintainer from triaging noise.

Forks are their own thing. Upstream behavior isn't a reliable guide. Check what the fork actually exposes.

Discoverability is a real problem. If multiple users independently can't find a feature, that's signal — even if the feature works perfectly.

Quick Fix if You're Here for the Answer

Scroll not working at all → enable Mouse Reporting in your terminal settings
Scroll works but no scrollbar → Ctrl+P → search "scrollbar" → toggle on

It'll persist across sessions once set.