DEV Community: Sunny JayaRaju

Building a macOS Dev Workstation Toolkit in Bash: 187 Tests, Pinned CI, and Verified Backups

Sunny JayaRaju — Sat, 03 Oct 2026 12:35:34 +0000

Every workstation setup starts the same way: a setup.sh that works. Then it accretes.

Someone adds a second script. The logging block gets copy-pasted. Config lives in three places and nobody can say which one wins. The README describes a flag that was renamed four months ago. Validation is whatever you remember to run before you break something.

The rot isn't dramatic. It's a repo nobody dares to touch, and no idea whether your last change broke anything.

I wanted to fix that on my own machine, in Bash.

What it is

Developer Workstation Framework

An engineering-first framework for building, managing, validating, and evolving a professional macOS development workstation using modern Bash engineering practices.

Ten utilities — install, update, uninstall, sync, backup, restore, check-project, doctor, repo-clean, shell-quality — built on seven shared Bash libraries. Current release is v2.2.0.

git clone https://github.com/SunnyJayaRaju/workstation-framework.git
cd workstation-framework
./scripts/install.sh
./scripts/doctor.sh

The problem it solves

Workstation scripts rot for the same reason any codebase without tests rots. There's no compiler and no failing build, so nothing tells you that you broke something. You find out when the machine is subtly wrong.

The usual failure pattern:

Scripts become duplicated. The same date-formatting logic in five files, one of which is wrong.
Configuration is scattered. No answer to "what is this actually set to on this machine?"
Documentation drifts. The docs describe the previous release, confidently.
Validation is manual. Which means it doesn't happen.
Small changes cause regressions. And you find out weeks later.

This project treats workstation automation as a software project: shared libraries instead of copy-paste, one documented config precedence order, behavioural tests, and CI on every change.

Why it's engineered, not just scripted

Shared libraries, not copy-paste

Seven libraries in scripts/lib/ — logging.sh, config.sh, errors.sh, filesystem.sh, checks.sh, backup_paths.sh, secrets.sh. Common behaviour lives in one place, which means it's fixed in one place.

One documented configuration precedence

Environment variables beat config/user.conf, which beats config/default.conf.

Environment Variables
        ↓
config/user.conf
        ↓
config/default.conf

You can override anything for one command without editing a tracked file, which is also what makes CI able to test paths your laptop will never take.

CI on macOS and Ubuntu, with pinned tools

Every push and pull request runs Bash syntax validation, ShellCheck, shfmt verification, and the Bats suite — on both macos-latest and ubuntu-latest.

The pinning is the part worth copying. Ubuntu gets exact apt versions (shellcheck=0.9.0-1, shfmt=3.8.0-1, bats=1.10.0-1). Homebrew has no per-version pin, so macOS can't do that — instead it asserts the versions after install and fails if they differ.

Both approaches exist for the same reason: unpinned installs had drifted, so the Ubuntu runner ended up on shfmt 3.8.0 while macOS and developer machines were on 3.14.1. That mismatch caused two red pushes.

Bats tests, and a single command to run them

v2.2.0 ships 187 passing Bats tests covering backup, restore, install, update, uninstall, doctor, cleanup, config, shell quality, and repo sync.

make all      # full quality gate — run this before every commit
make test     # just the Bats suite
make lint     # ShellCheck

v2.2.0: making backups actually verifiable

The headline release was backup/restore integrity, and the theme of it was checks that could not fail. The repo had an assert helper with zero callers, a verification harness invoking a flag that never existed, and a standards document mandating a versioning scheme no script used. A test suite could be comprehensively green while a data-loss bug sat in restore.sh.

So the release went through making each safeguard actually do something:

Backups are recorded and verified. Every backup.sh run writes a manifest_<timestamp>.json with the filename, source path, and a SHA-256 of the content. restore.sh verifies a backup against its manifest before overwriting anything — a mismatch names the file, refuses it, and carries on with the rest. --dry-run reports the same verdict, so you can't use a dry run to discover a corrupt backup would have been installed.
A manifest can't redirect a write. It's parsed as text, never evaluated; only the digest is read, and the destination always comes from BACKUP_SOURCES. A tampered manifest can make the tool refuse — it cannot make it write somewhere else.
A 0-byte backup is refused outright. Checked before the manifest, deliberately: the SHA-256 of empty content is a fixed constant, so a manifest written for an empty backup matches perfectly and the hash check alone could never catch it.
Safety copies stop overwriting each other. The pre-restore safety copy is what makes a restore reversible, but its name had one-second resolution. Measured before the fix: one surviving copy in four runs out of five. After: two in five out of five.
Backup names are keyed on path, not basename. ~/.ssh/config and a top-level ~/config no longer collide into a single backup file.
uninstall.sh validates its target before deleting. It fails closed on /, //, relative paths, and shallow paths like /usr/local. The destructive-path tests resolve everything from their own temp directory, and a logging rm stub asserts no removal was even attempted.

That last one is the lesson I'd keep: a test that runs against your real ~/.ssh is a test that will eventually delete your real ~/.ssh.

`doctor.sh` as an honest reporter

doctor.sh reports an absent ShellCheck as an incomplete environment, never as a lint failure — so a missing tool can't be mistaken for a code defect. It also checks that the installed VERSION still matches the source. That check exists because the live machine had silently drifted several releases behind source, which nothing else noticed.

Why this might interest you

If you set up macOS for a living, or for yourself, repeatedly. install.sh writes the utilities to INSTALL_DIR (default ~/.local/bin), update.sh reinstalls in place, doctor.sh tells you what it finds. Your workstation becomes something you can rebuild and audit instead of something you remember.

If you're tired of "just shell scripts." Everything here is Bash, but the discipline isn't language-specific. Tests, a lint gate, pinned CI, documented precedence, and a changelog that reflects reality are all transferable to a shell project you're maintaining right now.

If you enjoy the unglamorous middle. v2.2.0 also removed an unused eval-based assert helper, two existence predicates with no call sites, and an entire lib/colors.sh that duplicated what logging.sh already provided. Deleting dead code that only looked load-bearing is most of the job.

Try it

The roadmap is more automation, broader backup support, cross-platform compatibility, better diagnostics, and more tests. If that sounds like your kind of problem:

Try it — clone it and run ./scripts/install.sh
Star it if it's useful
Open an issue — especially if a utility doesn't fit how you actually work
Contribute — CONTRIBUTING.md has the workflow; the short version is make all must pass

Everything ships through the same gate as everything else: tests, ShellCheck, shfmt, and CI on two platforms. If it merges, it was verified.

Cover Image Description

This section is not part of the article — it is the prompt for generating the cover image.

A clean, minimalist flat-vector illustration of a macOS developer workstation, viewed straight on. A single modern desktop monitor displays a dark terminal window with crisp green monospace text and a tidy vertical list of shell commands, accented by a small green checkmark badge suggesting a passing test run. Beside the monitor sit a laptop and a coffee cup on a light desk. Muted palette: charcoal, soft white, and a restrained green accent, with generous negative space. Flat geometric shapes, no gradients, no photorealism, no text other than abstract terminal glyphs.

Your OpenCode Model List Is Rotting: ocprobe

Sunny JayaRaju — Thu, 01 Oct 2026 03:14:59 +0000

Your OpenCode whitelist is a list of promises made by other people's APIs. This is about the small bash tool I wrote to check them — and one bug in it that turned out to be far more interesting than the feature it came from.

TL;DR

ocprobe is a command-line tool that manages OpenCode model catalogs. It diffs the upstream catalog against your whitelist, probes the models that changed, shows you what it found, and only applies changes after you confirm — with a set of guards that are more interesting than the diffing.

The reason I care about it: running OpenCode against several model providers means your whitelist is a list of promises made by other people's infrastructure. Providers retire models, keys go stale, quotas run out, and a model that worked yesterday can 404 today. The failure is silent until something breaks mid-task, and the tool that reports the problem — the model picker — is not the thing that can tell you which entry is now lying.

Currently at v3.1.3, MIT licensed, installable with Homebrew in two commands.

The section I'd skip to if you only read one thing is the case study — it's about a security allowlist that passed every test on the machine it was written on and was quietly broken on every other platform.

The problem: your catalog rots quietly

Here's the manual loop this replaces, which is straight from the README's own description of the problem:

You diff catalogs by hand, probe models one by one, and risk wiping your whitelist with a bad apply.

Three separate failure modes hide in that sentence:

Models disappear. Providers retire models. An entry that was valid when you added it becomes a 404 later, and it stays in your whitelist until you try to use it.

Models fail without being gone. A key expires, a quota is exhausted, a provider has a bad afternoon. The model exists; your access to it doesn't.

Models change billing behaviour. This is the one that surprised me most, and it's why ocprobe has a status you might not expect. A probe can fail with BILLING_ERROR, which means your account can't currently afford that model's default output size — the model is completely fine and will work the moment credits exist. Blacklisting that would permanently hide a working model on a free or credit-limited key. So BILLING_ERROR is reported for review but never blacklisted, and it doesn't even count toward the failure gate.

That distinction only matters because ocprobe is willing to hide models at all. Which is the part that needs to be dangerous on purpose.

What it actually does

brew tap SunnyJayaRaju/ocprobe
brew install ocprobe

The commands that matter day to day:

Command	What it does
`ocprobe doctor`	Health check: config, DB, auth, disk
`ocprobe audit`	Full cycle: diff → probe → confirm → apply (the default)
`ocprobe check`	Dry-run only; exits 1 if changes are pending
`ocprobe validate`	Probes every model and manages the blacklist. Dry-run by default
`ocprobe probe <model>`	Test one model right now
`ocprobe status`	Whitelisted models plus recent probe results
`ocprobe watch`	`check` + alerts on an interval, never auto-applies
`ocprobe scheduler install`	Same thing as a launchd (macOS) or systemd (Linux) service
`ocprobe session`	`list` / `backup` / `restore` / `cleanup`

A reasonable first run, which is also the sequence I'd suggest:

ocprobe doctor                 # is anything even set up correctly?
ocprobe check                  # what would change, without changing it?
ocprobe audit                  # actually probe, then confirm the apply
ocprobe validate               # what would be blacklisted, without doing it
ocprobe validate --apply       # do it, with a backup and a verification pass

The default probe prompt is Reply with exactly: OK. That's not arbitrary — it's what makes the session-isolation guarantee below checkable.

Blacklisting is gated

ocprobe validate is the part that hides models from your picker, so it has rules about when it's allowed to do that.

Non-terminal failures — TIMEOUT, AUTH_ERROR, ERROR, UNCLEAR — need two consecutive failures before a model is added to the blacklist:

First failure → TENTATIVE. Surfaced for review, not blacklisted.
Second consecutive failure → CONFIRMED. Blacklisted.
A WORKS at any point resets the counter.
EOL / NOT_FOUND → CONFIRMED immediately. Those are terminal, and waiting buys nothing.
BILLING_ERROR → never blacklisted, and doesn't advance the counter.

There are two more brakes on the same path. A modality skip list means embedding, reranking, image/audio/video and moderation models are never probed at all — you didn't configure them for chat, so failing them is meaningless. And an AUTH_ERROR provider-wide abort: if more than 40% of a provider's models come back AUTH_ERROR, the whole provider is skipped, because that pattern means your key or your quota is the problem, not the models.

Lesson #1: the safe default for a tool that hides things is to require a second, independent piece of evidence. One failure is a bad afternoon. Two in a row is a fact.

The guards that keep it from making things worse

Because a tool that edits your config and deletes your sessions can absolutely cause the outage it was installed to prevent:

Session isolation. Only sessions created during this run, carrying the exact probe prompt in their first message, are ever deleted. Real conversations are never touched.
Mass-removal guard. If more than 50% of your whitelist would be removed in one run, ocprobe refuses to apply. Override is opt-in via OCPROBE_ALLOW_MASS_REMOVE=1.
Backups on every apply. Timestamped, next to the file: opencode.json.ocprobe-backup-YYYYMMDD-HHMMSS.
Graveyard cooldown. A model you deliberately removed won't reappear as a "new" model for 24 hours.
Restore is validated per statement. Covered below, because the interesting bug is there.

A case study: an allowlist whose correctness depended on your sqlite build

ocprobe session backup writes your OpenCode sessions to a SQL dump, and ocprobe session restore <file.sql> reads one back. A SQL dump is not something you want to execute by hand — so the restore is enforced through SQLite's authorizer callback, which lets you approve or deny each operation as the statement runs.

The policy ended up small and explicit, living in lib/session_restore.py:

ALLOWED_TABLES = frozenset(("session", "message", "part", "todo"))
ALLOWED_FUNCS  = frozenset(("unistr", "replace", "char"))

Only inserts into the four tables cmd_session_backup actually emits. ATTACH, DETACH, DROP, DELETE, UPDATE, ALTER, every CREATE (which shows up as an insert into sqlite_master, so the table allowlist catches it), every PRAGMA, every other function, and all reads — so INSERT ... SELECT can't quietly copy data out of somewhere. The whole file runs as one transaction, so a rejected dump leaves the database byte-identical, and CI asserts that against the installed package rather than against a working tree where every file is present by definition.

That reads like a solid design. Here's the hole in it.

The replace entry in ALLOWED_FUNCS looks redundant. A real dump doesn't call a function called replace. It's there because of this, quoted from the module's own header:

SQLITE_FUNCTION callback for the conflict target of INSERT OR REPLACE ("replace"), which sqlite 3.54 on macOS does not report at all, so an allowlist of just {"unistr"} passed every local test and the macOS CI leg while breaking restore for every Linux user.

Read that again, because it's the whole lesson. Whether INSERT OR REPLACE causes a function callback at all is a property of your SQLite build, not of the program asking. The allowlist was not just listing which functions were permitted; it was implicitly asserting which functions would ever be offered to it. On macOS, replace never arrives, so its absence from the allowlist was untestable. On the Ubuntu runner, it arrives, isn't on the list, and the restore is denied — for a completely legitimate dump, with an error that names the function rather than the mistake.

There was a second hole of the same shape. INSERT INTO <table> DEFAULT VALUES is a real SQLITE_INSERT on a permitted table, so the authorizer approved it — and it wrote a row of NULLs over real data. It was refused only after a first-keyword check was added ahead of SQLite, which strips a BOM, Unicode whitespace and -- / /* */ comments before looking at the word. That change also stopped the code depending on another untested assumption: REINDEX, ANALYZE and VACUUM were already denied on the builds that had been tested, and after the change that's no longer load-bearing.

Two things make this worth writing down:

An allowlist has two halves — what you permit, and what you expect to be asked about. Only the first is visible in the code. The second is a property of the environment, and the environment is not covered by your tests unless you run your tests in it.
A denier that fires on a legitimate input looks exactly like a denier working. The Linux restore wasn't obviously broken; it was a strict-looking tool refusing a valid file. The only way to tell those apart is to have a test that asserts a known-good input is accepted, on every platform you claim to support.

This is why the project now runs its unit and integration suites on macOS and Ubuntu, plus a separate leg that runs the entire suite under a compiled bash 4.3.30 — the minimum the README advertises. macOS ships bash 3.2, and expanding a declared-but-empty array is an error there before 4.4. Five such expansions aborted. That minimum used to be asserted in the README and never tested.

Lesson #2: "passes on my machine" is not a portability story, it's a measurement you only took once.

Install and try it

brew tap SunnyJayaRaju/ocprobe
brew install ocprobe

ocprobe doctor          # config, DB, auth, disk
ocprobe check           # dry-run: what would change
ocprobe validate        # dry-run: what would be blacklisted

Both of the dry-runs exit 1 when they have something to tell you, so they drop straight into CI or a pre-commit hook if that's useful to you. ocprobe watch is the low-maintenance option — it re-runs check and raises alerts, and never applies anything on its own.

Where it lives

Source, issues, CHANGELOG: github.com/SunnyJayaRaju/oc-model-manager
Homebrew tap: github.com/SunnyJayaRaju/homebrew-ocprobe

MIT licensed. Contributions are welcome — the repo has one rule worth knowing up front, which I think is the right one: any PR that adds a feature, changes a command's behaviour, or changes a flag must update README.md and CHANGELOG.md in the same PR.

What it doesn't do

A few honest limits, so you can decide whether it's for you:

It's a management tool, not a proxy. It probes models on a schedule or on demand. It does not sit in the path of your requests and fail over when one breaks.
The policy engine is experimental and off by default. ocprobe policy gives you declarative never_add / never_remove rules for audit and check, but a missing policy file is a true no-op. It never touches validate.
Restores of older dumps still work, but only because the round-trip test covers the escaping both current and older SQLite builds produce for values containing newlines. That's a compatibility promise with a test behind it, not a general one.
Three config keys are accepted but do nothing: catalog.force_refresh, scheduler.enabled and scheduler.run_at_load. They were declared in the schema and shipped in generated configs, and nothing ever read them — so scheduler.enabled: false did not disable the scheduler. They're still accepted so existing configs keep validating, they log a warning if set to a non-default value, and ocprobe probe --force-refresh is the flag that actually works.
The validate path is the risky one. It's gated, backed up and verifiable, but it does hide models from your picker by design. Start with --apply off and read the diff.

Apigee X Eval Org Stuck at 502? Check These 2 Things First

Sunny JayaRaju — Sat, 05 Sep 2026 07:04:20 +0000

Quick-fix reference. For the full debugging story behind this, see: The 502 That Wouldn't Die

Symptom

Your Apigee X evaluation org shows fully provisioned:

Console wizard: all 4 setup steps green ✅
organizations.get API: "state": "ACTIVE" ✅
instances.list API: "state": "ACTIVE", real host/port ✅
Environment attached, proxy deployed, envgroup hostname bound — all clean ✅

And yet every request — even to a brand-new proxy — returns:

HTTP/2 502
Error: Server Error
The server encountered a temporary error and could not complete your request.

Waiting longer doesn't fix it. Deploying a different proxy doesn't fix it.

First: confirm this is actually your bug

gcloud compute backend-services list
gcloud compute backend-services get-health apigee-proxy-backend --global

If this shows healthState: UNHEALTHY on the apigee-proxy-* instances, keep reading — this is the load-balancer layer that sits between the external HTTPS LB and your actual Apigee runtime, and it's separate from Apigee's own control plane. That's why everything above reports "ACTIVE" while requests still 502: Apigee's config is correct, but the forwarding instances behind the LB aren't actually serving traffic.

Note before you start: the fixes below both work on the instance template, not just the live running instances. These forwarding VMs are preemptible — Google can reclaim and silently replace them at any time, and a replacement rebuilds straight from the template. If you patch a live instance directly instead of the template, the fix will quietly disappear the next time that VM gets swapped, and you'll be back here wondering why it "randomly broke again."

Cause #1: Missing service account on the instance template

Check the boot log of one of the unhealthy instances:

gcloud compute instances get-serial-port-output <INSTANCE_NAME> --zone=<ZONE> | tail -60

Look for:

Instance has service account: false, ...
Failed to download from GCS: ... credentials: cannot fetch token ...
Trying unauthenticated download

Confirm it:

gcloud compute instance-templates describe apigee-proxy-<REGION> \
  --format="yaml(properties.serviceAccounts)"

If this prints null, the template has no service account attached, so the VM can never authenticate to Cloud Storage to pull its real startup script.

Fix — clone the template with a service account attached, then roll the MIG onto it:

# Get every field from your existing template first so you replicate it exactly:
gcloud compute instance-templates describe apigee-proxy-<REGION> --format=yaml

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>

Match every field from your describe --format=yaml output — machine type, disk, network, tags, and especially the scheduling block. --preemptible, --no-restart-on-failure, and --maintenance-policy=TERMINATE must be specified together or gcloud rejects the combination.

Cause #2: Blank `ENDPOINT` metadata

Even after fixing the service account, health checks can still fail. These forwarding VMs don't run a proxy application themselves — they install an iptables DNAT rule redirecting incoming port-443 traffic to your real Apigee runtime instance's internal IP. That IP comes from an instance metadata key called ENDPOINT.

Check it from inside an instance:

gcloud compute ssh <INSTANCE_NAME> --zone=<ZONE>
curl -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ENDPOINT"

If this returns nothing, that's the second bug. Get your runtime instance's real internal IP:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/<ORG>/instances"

Look for the "host" field in the response (e.g. 10.51.204.98).

⚠️ Don't just patch the running instances — patch the template. These forwarding VMs are typically --preemptible, meaning Google can reclaim them at any time (often within 24h). When that happens, the managed instance group silently rebuilds a replacement straight from the template — and if you only patched the live instance's metadata, the new one comes back with ENDPOINT blank again, and you're right back to a 502 with no idea why it "randomly broke." Fix the template, not just the instance, or this will relapse on you.

Fix — bake ENDPOINT into a corrected template, then roll the instance group onto it (this is the fix that actually survives instance replacement):

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed-v2 \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT=<RUNTIME_INTERNAL_IP> \
  --service-account=<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed-v2 --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>

This is the same pattern as the Cause #1 fix — clone the template with the correction included, point the MIG at it, roll. Once ENDPOINT lives in the template itself, any future automatic replacement (preemption, autohealing, whatever) inherits the correct value with zero manual intervention.

(If you only need a same-minute workaround while the rolling-replace above is in flight, you can patch the live instance directly — gcloud compute instances add-metadata <INSTANCE_NAME> --zone=<ZONE> --metadata=ENDPOINT=<RUNTIME_INTERNAL_IP> followed by sudo google_metadata_script_runner startup over SSH — but treat that as temporary only, not the real fix.)

Verify the NAT rule landed on the new instances:

sudo iptables -t nat -L -n -v
# Look for: DNAT tcp dpt:443 to:<RUNTIME_INTERNAL_IP>

Confirm it's fixed

gcloud compute backend-services get-health apigee-proxy-backend --global

Both instances should now show healthState: HEALTHY. Then:

curl "https://<YOUR_HOSTNAME>.nip.io/hello-world"

You should get a real response instead of the 502 page.

This is one specific failure mode out of many possible causes of a 502 on Apigee X — always confirm the backend health check first before assuming this applies to you. Full context and the debugging process that led here: The 502 That Wouldn't Die.

The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up

Sunny JayaRaju — Sat, 05 Sep 2026 06:32:53 +0000

A real-world war story of chasing an Apigee X evaluation organization from "Server Error" to "Hello, Guest!" — the hard way, with gcloud, curl, and a lot of patience.

TL;DR

I spent hours debugging a persistent 502 Server Error on a fresh Apigee X evaluation organization. The Cloud Console showed every provisioning step green. The Apigee Management API confirmed the organization and runtime instance were both ACTIVE. And yet every single request — even to a brand-new proxy — came back with a 502.

The eventual root cause: the Apigee provisioning wizard built the proxy-forwarding instance template without a service account attached, and left a required ENDPOINT metadata key blank. Without a service account, the VMs couldn't authenticate to Cloud Storage to fetch their real startup script. Without ENDPOINT, the fallback script had nothing to forward traffic to. The fix was to clone the broken instance template with the missing service account and the ENDPOINT value both baked in, then roll the managed instance group onto it — fixing the template, not just the live instances, matters, since these forwarding VMs are preemptible and get silently replaced by Google; a fix applied only to a running instance quietly disappears the next time it's swapped (I found this out the hard way — see the Update section near the end).

If you're staring at a 502 on a "successfully" provisioned Apigee eval org, this post is for you.

Background: how I got here

I was setting up Apigee purely to practice — nothing fancy, no production intent, just a personal free-trial Google Cloud project to learn the platform. Along the way I made a very ordinary mistake: while creating a second project to experiment with, it ended up parented under "No organization" instead of my actual Cloud Identity org, because I had several Console tabs open switching between projects mid-setup.

Confused by the mismatch, I deleted that second project. Shortly after, my original project's Apigee evaluation setup started throwing a resource-locked error on the "Apigee evaluation organization" step, with a message like:

the resource is locked by another operation that is 1 percent completed so far
where organization <project-id> is currently being provisioned by operation: <uuid>

That project's setup never fully completed — the "Access routing" step permanently showed "Not configured." At that point, rather than trying to excavate a half-provisioned org with a stuck long-running operation, I made the call to shut down both projects and start clean in a brand-new project. This is a legitimate, low-cost strategy for Apigee eval orgs specifically — they're free, disposable, and tied to your Cloud Billing account (not the project) for trial credits and the 60-day clock, so nuking a broken project costs you nothing.

Lesson #1: if you're mid-setup and juggling multiple projects, do it in separate browser profiles or windows, not tabs. The Cloud Console's single-page-app state can leak between tabs in confusing ways (I saw one project's setup page briefly display another project's Apigee org name — almost certainly a UI caching artifact, not a real resource collision).

The rebuild: a promising start, then déjà vu

With a fresh project, I walked through Apigee → Try Apigee for free → Set up Apigee Evaluation step by step, in a single tab, waiting for each of the four steps to go green before touching anything else:

Enable APIs ✅
Networking ✅
Apigee evaluation organization
Access routing

Step 3 immediately failed with a generic "Something went wrong" and offered a "Try Again" button. Naturally, my first thought was: here we go again.

But clicking "Try Again" resolved it within about nine minutes — which felt suspicious. Org creation is supposed to be a slow, heavyweight operation. Why would a failed attempt succeed almost instantly on retry?

Why the quick retry wasn't fishy

It turns out this is documented, known behavior. Google's own Apigee installation troubleshooting guide calls out this exact failure mode:

If Apigee returns a 403 when you first try to create the new organization, it could mean that one or more of your APIs have not been enabled... If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Org creation is an asynchronous long-running operation (LRO) layered on top of API enablement from the prior step. There's a small window where the APIs report as enabled but haven't fully propagated through Google's internal systems yet. The wizard's org-creation call hits that window, gets rejected, and shows "Something went wrong" — but the expensive networking/VPC-peering work from Step 2 already succeeded, so once the propagation catches up, the retry only has to do the comparatively cheap part (create the org record) and finishes fast.

Don't trust a UI checkmark on faith, though — verify independently. From Cloud Shell:

gcloud config set project YOUR_PROJECT_ID
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_PROJECT_ID"

Look for "state": "ACTIVE" in the response. This hits the Apigee Management API directly, bypassing the Console entirely — the actual ground truth, not a rendered checkmark.

Step 4 (Access routing) subsequently completed cleanly, and the wizard showed the triumphant "Congratulations! Your Apigee organization is ready" message with a "Launch" button pointing at a *.nip.io test URL.

The real bug: green everywhere, 502 everywhere

I clicked Launch. 502 Server Error. The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

I waited the requisite 15–20 minutes for what I assumed was normal load-balancer propagation lag, then tried again. Still 502. I deployed a fresh test proxy just to rule out something proxy-specific. Still 502.

This is the point where "just wait longer" stops being a reasonable answer and it's time to actually verify each layer of the stack independently, from the top down.

Layer 1: Is the request even reaching Apigee's infrastructure?

curl -v "https://YOUR_HOSTNAME.nip.io/hello-world"

The verbose output showed a clean TLS handshake with a valid, freshly-issued Google-managed certificate, HTTP/2 negotiated, and a response that came back structured exactly like Google's generic Global Front End (GFE) "backend unhealthy" page — not a DNS failure, not a connection refusal, not an SSL error. This told me the request was reaching Google's load balancer correctly; the problem was somewhere behind it.

Layer 2: Is the Apigee org/instance actually healthy?

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_ORG/instances"

This came back "state": "ACTIVE" with a real internal host/port and service attachment. So the actual Apigee runtime — the expensive, slow-to-provision Message Processor infrastructure — was completely fine.

Layer 3: Are environment, deployment, and hostname binding all correct?

Three more calls, all clean:

# Is the environment attached to the instance?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../instances/eval-instance/attachments"

# Is a proxy actually deployed?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../environments/eval/deployments"

# Is the hostname bound to an environment group?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../envgroups"

Environment attached, proxy deployed, hostname bound and ACTIVE. Every configuration object in the Apigee control plane was correct. This ruled out the entire Apigee-managed side of the stack. The problem had to be in the one piece of infrastructure that Apigee's "Access routing" step provisions but doesn't directly manage day-to-day: the Compute Engine load balancer and its backing instance group.

Layer 4: The load balancer's backend health

Apigee X's Access Routing step builds a small Compute Engine managed instance group (MIG) of lightweight forwarding VMs that sit between the external HTTPS Load Balancer and the actual Apigee runtime instance (which lives on a private/internal IP). Checking this directly:

gcloud compute backend-services get-health apigee-proxy-backend --global

healthStatus:
- healthState: UNHEALTHY
  instance: .../apigee-proxy-7ztd
- healthState: UNHEALTHY
  instance: .../apigee-proxy-np3w

There it was. Zero of two backend instances were passing their health check. That's a 502 by design — a GFE-class load balancer will never forward traffic to a backend it considers unhealthy, no matter how long you wait.

Root-causing the unhealthy backend

Ruling out the obvious suspect: firewall rules

The most common cause of "backend permanently unhealthy" on a fresh GCP load balancer is a missing firewall rule allowing Google's health-check probe ranges (130.211.0.0/22 and 35.191.0.0/16) through to the target instances. I checked:

gcloud compute firewall-rules list --format="table(name,sourceRanges.list(),allowed[].map().firewall_rule().list(),targetTags.list())"

The rule existed, with the correct source ranges, correct port (443), and a matching target tag (gke-apigee-proxy) — and a quick gcloud compute instances describe confirmed the actual VMs carried that exact tag. Firewall theory eliminated.

Reading the actual boot log

With the network layer cleared, the next move was to read what the instance itself thought happened during boot:

gcloud compute instances get-serial-port-output apigee-proxy-7ztd --zone=asia-south1-b | tail -n 60

Buried in there was the actual smoking gun:

Instance has service account: false, setting ACS client isEnabled to false
...
Failed to download from GCS: downloading object [apigee-envoy-proxy-release/latest/conf/startup-script.sh]...
credentials: cannot fetch token: metadata: GCE metadata "instance/service-accounts/default/token?..." not defined
Trying unauthenticated download

The VM had no service account attached at all. Without one, it can't mint an OAuth token, so it can't authenticate to Cloud Storage to pull down its real startup script (the one that actually sets up the Envoy-based forwarding proxy). It silently fell back to an unauthenticated download, which only pulled a stub containing basic sysctl/iptables scaffolding — nothing that ever configures real port-forwarding to the Apigee runtime.

Confirming this against the instance template sealed it:

gcloud compute instance-templates describe apigee-proxy-asia-south1 \
  --format="yaml(properties.serviceAccounts)"

null

Meanwhile, a healthy default Compute Engine service account existed and was enabled on the project (gcloud iam service-accounts list) — it just wasn't referenced anywhere on this template. This was a genuine defect in how the Apigee provisioning wizard built this specific instance template — not a quota issue, not an org policy, not something I'd misconfigured.

The fix, part 1: rebuild the template with a service account

You can't attach a service account to an existing instance template after the fact — templates are immutable. So the fix is to clone it with the missing piece added, then roll the managed instance group onto the new template:

gcloud compute instance-templates create apigee-proxy-asia-south1-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=asia-south1 \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-asia-south1 \
  --template=apigee-proxy-asia-south1-fixed --region=asia-south1

gcloud compute instance-groups managed rolling-action replace apigee-proxy-asia-south1 \
  --region=asia-south1

Note: copy every field from your existing template's describe --format=yaml output exactly — machine type, disk, network, tags, and especially the scheduling block (preemptible + no-restart-on-failure + maintenance-policy=TERMINATE must all be specified together or gcloud rejects the combination).

After the rolling replace, the new instances came up with the service account correctly attached — the boot log's authentication error was gone. But health checks still failed. One bug down, one to go.

The fix, part 2: the missing `ENDPOINT`

SSHing into a replacement instance and checking what was actually listening confirmed there was still no process bound to port 443:

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo ss -tlnp | grep 443   # → nothing

These forwarding VMs, it turns out, don't run a full proxy application themselves — they run a startup script that installs an iptables DNAT (destination NAT) rule redirecting incoming port-443 traffic straight to the real Apigee runtime instance's internal IP. That destination IP is supplied via an instance metadata key called ENDPOINT. On the original (and my recreated) template, that key existed — but was blank:

metadata:
  items:
  - key: ENDPOINT
    value: ''

With nowhere to forward to, the script had nothing meaningful to do beyond the generic network scaffolding — hence the suspiciously short, error-free-but-incomplete startup log.

The fix: patch the metadata with the real internal IP of the Apigee runtime instance (retrieved earlier from the instances API call in Layer 2), then force each VM to re-run its startup script without a full reboot:

gcloud compute instances add-metadata apigee-proxy-dgz1 \
  --zone=asia-south1-b --metadata=ENDPOINT=10.51.204.98

gcloud compute instances add-metadata apigee-proxy-v4xb \
  --zone=asia-south1-c --metadata=ENDPOINT=10.51.204.98

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo google_metadata_script_runner startup

Checking the NAT table confirmed the fix landed:

sudo iptables -t nat -L -n -v

Chain PREROUTING
DNAT  tcp dpt:443 to:10.51.204.98

Repeated on the second instance, then the moment of truth:

gcloud compute backend-services get-health apigee-proxy-backend --global

healthState: HEALTHY   (apigee-proxy-dgz1)
healthState: HEALTHY   (apigee-proxy-v4xb)

And finally:

curl "https://YOUR_HOSTNAME.nip.io/hello-world"

Hello, Guest!

Done.

Update: it relapsed two days later, and here's why

Two days after publishing this, I opened the project to actually start practicing — and got the exact same 502 again. Same symptom, same 4/4-green setup wizard, same "everything reports ACTIVE" story. My first reaction was that something had regressed on its own, which felt alarming. It hadn't. The real explanation was a gap in the fix above that I'd missed the first time.

Those apigee-proxy-* forwarding VMs are provisioned as --preemptible — cheap, but reclaimable by Google at any time, sometimes within 24 hours. When Google reclaims one, the managed instance group automatically rebuilds a replacement, straight from the instance template — not from whatever state the old VM happened to be in.

Here's the gap: the ENDPOINT fix above patches the metadata on the running instances (gcloud compute instances add-metadata ...), not on the template itself. That's a real, working fix — for as long as those exact VMs stay alive. But the template underneath (apigee-proxy-asia-south1-fixed) still had ENDPOINT blank the whole time. The moment either instance got preempted and silently replaced, the replacement came back from that same broken template — same blank ENDPOINT, same 502, and from the outside it looked like the fix had mysteriously "worn off."

The durable fix is to bake the value into a new template revision, the same pattern as the service-account fix earlier:

gcloud compute instance-templates create apigee-proxy-asia-south1-fixed-v2 \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=asia-south1 \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT=10.51.204.98 \
  --service-account=315668123660-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-asia-south1 \
  --template=apigee-proxy-asia-south1-fixed-v2 --region=asia-south1

gcloud compute instance-groups managed rolling-action replace apigee-proxy-asia-south1 \
  --region=asia-south1

Once ENDPOINT lives in the template rather than on individual instances, any future preemption or autoheal replacement inherits the correct value automatically — no more manual re-patching, ever, for this project.

The lesson generalizes beyond this one bug: on any Compute Engine managed instance group, a fix applied via add-metadata (or any live-instance edit) to a running VM is inherently temporary if that VM can be replaced by the group's own lifecycle — preemption, autohealing, or a manual scale event. The template is the actual source of truth for anything that needs to survive. Patch the instance for an immediate same-minute fix if you're in a hurry; patch the template for the fix that actually sticks.

The debugging methodology, distilled

If you take nothing else from this post, take the layered verification approach — it's the reusable part:

Don't trust a single "it says it's ready" signal. Console checkmarks, API state: ACTIVE fields, and actual traffic serving are three different questions.
Verify each layer of the request path independently, from the outside in:
- TLS/DNS reachability (curl -v)
- Apigee org + runtime instance state (Management API)
- Environment attachment, deployment, and hostname binding (Management API)
- Load balancer backend health (gcloud compute backend-services get-health)
- The actual VM's boot log and listening ports (get-serial-port-output, ss -tlnp)
Read the boot log before assuming propagation delay. A silent early exit with no errors can look identical to "still starting" from the outside — the only way to tell the difference is to read what actually ran.
Understand what's supposed to be running before declaring something broken. My first assumption — that these VMs should have a listening process — was wrong; they're a DNAT relay, not an application server. Getting that model right changed what "healthy" even meant to check for.
When something is auto-provisioned by a wizard, verify its artifacts independently rather than assuming the automation is infallible. In this case, two separate fields on an auto-generated instance template were wrong, and neither surfaced as an explicit error anywhere in the Console.

Should you expect this to happen to you?

Almost certainly not on every setup — this felt like a genuine, if rare, defect in how the Apigee X evaluation wizard built one specific instance template in one specific run. Most eval org setups likely complete without ever seeing this. But if you do hit a 502 that survives 20+ minutes of patient waiting on an otherwise "fully provisioned" eval org, the checklist above will get you to ground truth fast — instead of guessing at propagation delays indefinitely.

If you're setting up Apigee X eval for the first time: budget for the fact that a "Something went wrong" on the org-creation step is often transient and not worth panicking over, but a 502 that survives a proper wait, with all Apigee-side config verified clean, is a signal to go look at the Compute Engine layer underneath — specifically the apigee-proxy-* instance group, its template's service account, and the ENDPOINT metadata key.

Questions or hit something similar? Drop a comment — happy to compare notes.

Building Enterprise-Ready APIs on Google Cloud Apigee X: 5 Hands-On Projects

Sunny JayaRaju — Wed, 12 Aug 2026 18:19:08 +0000

Why I built an Apigee Lab

Enterprise APIs are the backbone of modern platforms. But reading about API gateways is one thing — building them, breaking them, and hardening them is where you actually learn the craft.

That's exactly what I set out to do with Apigee-Lab — an open monorepo of 5 enterprise-inspired projects built on Google Cloud Apigee X.

Each project is fully documented — with architecture diagrams, Mermaid sequence diagrams, deployment guides, and real policy XML. No "hello world" filler. Just production patterns you can actually study and reuse.

Here's what's inside.

1️⃣ Weather Shield Gateway — API Security Layer

A secure, optimized gateway for weather API traffic.

What it demonstrates:

JWT authentication to protect routes
Spike Arrest to stop traffic surges and abuse
Response caching to cut backend load and latency
Monetization quotas (Silver/Gold tiers)

GET /weather-lab?city=London
Authorization: Bearer <jwt>

Request flow: spike arrest → JWT verify → quota check → cache check → backend → transform (XML→JSON).

Read the full architecture in project docs.

2️⃣ Secure Bank Access — OAuth 2.0 Identity Flows

A mock banking API that demonstrates clean identity and access management.

What it demonstrates:

Client Credentials Flow for machine-to-machine security
Conditional proxy flows separating public (/token) and protected (/balance) routes
API Products, Developers, and Apps to enforce scopes and limits

The two-phase pattern: the handshake (get a token) → the access (verify and authorize).

Read the full architecture in project docs.

3️⃣ Retail Mesh Orchestrator — Microservices Aggregation

A composition layer that consolidates multiple backend responses into one payload.

What it demonstrates:

Parallel Service Callouts to cut round-trips
JavaScript transformations for JSON/XML normalization
Fault-tolerant routing during backend failures

The classic Backend-for-Frontend pattern, done with Apigee as the orchestrator.

Read the full architecture in project docs.

4️⃣ Apigee DevOps Pipeline — Full CI/CD Automation

Instead of deploying proxies manually, this project automates the delivery flow.

What it demonstrates:

GitHub Actions workflows that lint, package, and version proxies
apigeelint to catch structural and policy errors before merge
Artifact staging for controlled deployments

Every push runs a quality gate: build structure → lint → package → upload. Bad code never ships.

Read the full pipeline in project docs.

5️⃣ Security Governance — Shared Flows

A centralized security module for global policy enforcement.

What it demonstrates:

Reusable Shared Flows to standardize logic across proxies
Spike Arrests and Security Headers enforced globally via FlowCallout
Governance rules decoupled from individual proxy logic

One shared flow, referenced by many proxies. Consistent security across the entire org.

Read the full architecture in project docs.

What I learned building these

Security-first design: Every proxy needs an identity layer, traffic control, and a governance story — not just routing.
Reusability wins: Shared Flows turn drift-prone duplicate policies into one maintained source of truth.
CI/CD is non-negotiable: Linting and packaging Apigee bundles in CI catches problems before they ever reach eval.
Composition is a superpower: The gateway can be the orchestrator — merging multiple backends into one clean response.

Try it yourself

All projects are documented so you can deploy them to your own Apigee X environment (eval org) and verify them with curl or Postman.

Explore the full lab → github.com/SunnyJayaRaju/Apigee-Lab 🌟

If you build APIs, gateways, or microservices — I'd love your feedback, issues, or a PR. Star it if it helps, and let's keep learning in public.

This article is part of my public learning journey — you can follow more API engineering notes in my Curious-Explorer knowledge base.

DEV Community: Sunny JayaRaju

Building a macOS Dev Workstation Toolkit in Bash: 187 Tests, Pinned CI, and Verified Backups

What it is

The problem it solves

Why it's engineered, not just scripted

Shared libraries, not copy-paste

One documented configuration precedence

CI on macOS and Ubuntu, with pinned tools

Bats tests, and a single command to run them

v2.2.0: making backups actually verifiable

doctor.sh as an honest reporter

Why this might interest you

Try it

Cover Image Description

Your OpenCode Model List Is Rotting: ocprobe

TL;DR

The problem: your catalog rots quietly

What it actually does

Blacklisting is gated

The guards that keep it from making things worse

A case study: an allowlist whose correctness depended on your sqlite build

Install and try it

Where it lives

What it doesn't do

Apigee X Eval Org Stuck at 502? Check These 2 Things First

Symptom

First: confirm this is actually your bug

Cause #1: Missing service account on the instance template

Cause #2: Blank ENDPOINT metadata

Confirm it's fixed

The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up

TL;DR

Background: how I got here

The rebuild: a promising start, then déjà vu

Why the quick retry wasn't fishy

The real bug: green everywhere, 502 everywhere

Layer 1: Is the request even reaching Apigee's infrastructure?

Layer 2: Is the Apigee org/instance actually healthy?

Layer 3: Are environment, deployment, and hostname binding all correct?

Layer 4: The load balancer's backend health

Root-causing the unhealthy backend

Ruling out the obvious suspect: firewall rules

Reading the actual boot log

The fix, part 1: rebuild the template with a service account

The fix, part 2: the missing ENDPOINT

Update: it relapsed two days later, and here's why

The debugging methodology, distilled

Should you expect this to happen to you?

Building Enterprise-Ready APIs on Google Cloud Apigee X: 5 Hands-On Projects

Why I built an Apigee Lab

1️⃣ Weather Shield Gateway — API Security Layer

2️⃣ Secure Bank Access — OAuth 2.0 Identity Flows

3️⃣ Retail Mesh Orchestrator — Microservices Aggregation

4️⃣ Apigee DevOps Pipeline — Full CI/CD Automation

5️⃣ Security Governance — Shared Flows

What I learned building these

Try it yourself

`doctor.sh` as an honest reporter

Cause #2: Blank `ENDPOINT` metadata

The fix, part 2: the missing `ENDPOINT`