DEV Community: Sunny JayaRaju

Apigee X Eval Org Stuck at 502? Check These 2 Things First

Sunny JayaRaju — Sat, 05 Sep 2026 07:04:20 +0000

Quick-fix reference. For the full debugging story behind this, see: The 502 That Wouldn't Die

Symptom

Your Apigee X evaluation org shows fully provisioned:

Console wizard: all 4 setup steps green ✅
organizations.get API: "state": "ACTIVE" ✅
instances.list API: "state": "ACTIVE", real host/port ✅
Environment attached, proxy deployed, envgroup hostname bound — all clean ✅

And yet every request — even to a brand-new proxy — returns:

HTTP/2 502
Error: Server Error
The server encountered a temporary error and could not complete your request.

Waiting longer doesn't fix it. Deploying a different proxy doesn't fix it.

First: confirm this is actually your bug

gcloud compute backend-services list
gcloud compute backend-services get-health apigee-proxy-backend --global

If this shows healthState: UNHEALTHY on the apigee-proxy-* instances, keep reading — this is the load-balancer layer that sits between the external HTTPS LB and your actual Apigee runtime, and it's separate from Apigee's own control plane. That's why everything above reports "ACTIVE" while requests still 502: Apigee's config is correct, but the forwarding instances behind the LB aren't actually serving traffic.

Note before you start: the fixes below both work on the instance template, not just the live running instances. These forwarding VMs are preemptible — Google can reclaim and silently replace them at any time, and a replacement rebuilds straight from the template. If you patch a live instance directly instead of the template, the fix will quietly disappear the next time that VM gets swapped, and you'll be back here wondering why it "randomly broke again."

Cause #1: Missing service account on the instance template

Check the boot log of one of the unhealthy instances:

gcloud compute instances get-serial-port-output <INSTANCE_NAME> --zone=<ZONE> | tail -60

Look for:

Instance has service account: false, ...
Failed to download from GCS: ... credentials: cannot fetch token ...
Trying unauthenticated download

Confirm it:

gcloud compute instance-templates describe apigee-proxy-<REGION> \
  --format="yaml(properties.serviceAccounts)"

If this prints null, the template has no service account attached, so the VM can never authenticate to Cloud Storage to pull its real startup script.

Fix — clone the template with a service account attached, then roll the MIG onto it:

# Get every field from your existing template first so you replicate it exactly:
gcloud compute instance-templates describe apigee-proxy-<REGION> --format=yaml

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>

Match every field from your describe --format=yaml output — machine type, disk, network, tags, and especially the scheduling block. --preemptible, --no-restart-on-failure, and --maintenance-policy=TERMINATE must be specified together or gcloud rejects the combination.

Cause #2: Blank `ENDPOINT` metadata

Even after fixing the service account, health checks can still fail. These forwarding VMs don't run a proxy application themselves — they install an iptables DNAT rule redirecting incoming port-443 traffic to your real Apigee runtime instance's internal IP. That IP comes from an instance metadata key called ENDPOINT.

Check it from inside an instance:

gcloud compute ssh <INSTANCE_NAME> --zone=<ZONE>
curl -H "Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ENDPOINT"

If this returns nothing, that's the second bug. Get your runtime instance's real internal IP:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/<ORG>/instances"

Look for the "host" field in the response (e.g. 10.51.204.98).

⚠️ Don't just patch the running instances — patch the template. These forwarding VMs are typically --preemptible, meaning Google can reclaim them at any time (often within 24h). When that happens, the managed instance group silently rebuilds a replacement straight from the template — and if you only patched the live instance's metadata, the new one comes back with ENDPOINT blank again, and you're right back to a 502 with no idea why it "randomly broke." Fix the template, not just the instance, or this will relapse on you.

Fix — bake ENDPOINT into a corrected template, then roll the instance group onto it (this is the fix that actually survives instance replacement):

gcloud compute instance-templates create apigee-proxy-<REGION>-fixed-v2 \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=<REGION> \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT=<RUNTIME_INTERNAL_IP> \
  --service-account=<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-<REGION> \
  --template=apigee-proxy-<REGION>-fixed-v2 --region=<REGION>

gcloud compute instance-groups managed rolling-action replace apigee-proxy-<REGION> \
  --region=<REGION>

This is the same pattern as the Cause #1 fix — clone the template with the correction included, point the MIG at it, roll. Once ENDPOINT lives in the template itself, any future automatic replacement (preemption, autohealing, whatever) inherits the correct value with zero manual intervention.

(If you only need a same-minute workaround while the rolling-replace above is in flight, you can patch the live instance directly — gcloud compute instances add-metadata <INSTANCE_NAME> --zone=<ZONE> --metadata=ENDPOINT=<RUNTIME_INTERNAL_IP> followed by sudo google_metadata_script_runner startup over SSH — but treat that as temporary only, not the real fix.)

Verify the NAT rule landed on the new instances:

sudo iptables -t nat -L -n -v
# Look for: DNAT tcp dpt:443 to:<RUNTIME_INTERNAL_IP>

Confirm it's fixed

gcloud compute backend-services get-health apigee-proxy-backend --global

Both instances should now show healthState: HEALTHY. Then:

curl "https://<YOUR_HOSTNAME>.nip.io/hello-world"

You should get a real response instead of the 502 page.

This is one specific failure mode out of many possible causes of a 502 on Apigee X — always confirm the backend health check first before assuming this applies to you. Full context and the debugging process that led here: The 502 That Wouldn't Die.

The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up

Sunny JayaRaju — Sat, 05 Sep 2026 06:32:53 +0000

A real-world war story of chasing an Apigee X evaluation organization from "Server Error" to "Hello, Guest!" — the hard way, with gcloud, curl, and a lot of patience.

TL;DR

I spent hours debugging a persistent 502 Server Error on a fresh Apigee X evaluation organization. The Cloud Console showed every provisioning step green. The Apigee Management API confirmed the organization and runtime instance were both ACTIVE. And yet every single request — even to a brand-new proxy — came back with a 502.

The eventual root cause: the Apigee provisioning wizard built the proxy-forwarding instance template without a service account attached, and left a required ENDPOINT metadata key blank. Without a service account, the VMs couldn't authenticate to Cloud Storage to fetch their real startup script. Without ENDPOINT, the fallback script had nothing to forward traffic to. The fix was to clone the broken instance template with the missing service account and the ENDPOINT value both baked in, then roll the managed instance group onto it — fixing the template, not just the live instances, matters, since these forwarding VMs are preemptible and get silently replaced by Google; a fix applied only to a running instance quietly disappears the next time it's swapped (I found this out the hard way — see the Update section near the end).

If you're staring at a 502 on a "successfully" provisioned Apigee eval org, this post is for you.

Background: how I got here

I was setting up Apigee purely to practice — nothing fancy, no production intent, just a personal free-trial Google Cloud project to learn the platform. Along the way I made a very ordinary mistake: while creating a second project to experiment with, it ended up parented under "No organization" instead of my actual Cloud Identity org, because I had several Console tabs open switching between projects mid-setup.

Confused by the mismatch, I deleted that second project. Shortly after, my original project's Apigee evaluation setup started throwing a resource-locked error on the "Apigee evaluation organization" step, with a message like:

the resource is locked by another operation that is 1 percent completed so far
where organization <project-id> is currently being provisioned by operation: <uuid>

That project's setup never fully completed — the "Access routing" step permanently showed "Not configured." At that point, rather than trying to excavate a half-provisioned org with a stuck long-running operation, I made the call to shut down both projects and start clean in a brand-new project. This is a legitimate, low-cost strategy for Apigee eval orgs specifically — they're free, disposable, and tied to your Cloud Billing account (not the project) for trial credits and the 60-day clock, so nuking a broken project costs you nothing.

Lesson #1: if you're mid-setup and juggling multiple projects, do it in separate browser profiles or windows, not tabs. The Cloud Console's single-page-app state can leak between tabs in confusing ways (I saw one project's setup page briefly display another project's Apigee org name — almost certainly a UI caching artifact, not a real resource collision).

The rebuild: a promising start, then déjà vu

With a fresh project, I walked through Apigee → Try Apigee for free → Set up Apigee Evaluation step by step, in a single tab, waiting for each of the four steps to go green before touching anything else:

Enable APIs ✅
Networking ✅
Apigee evaluation organization
Access routing

Step 3 immediately failed with a generic "Something went wrong" and offered a "Try Again" button. Naturally, my first thought was: here we go again.

But clicking "Try Again" resolved it within about nine minutes — which felt suspicious. Org creation is supposed to be a slow, heavyweight operation. Why would a failed attempt succeed almost instantly on retry?

Why the quick retry wasn't fishy

It turns out this is documented, known behavior. Google's own Apigee installation troubleshooting guide calls out this exact failure mode:

If Apigee returns a 403 when you first try to create the new organization, it could mean that one or more of your APIs have not been enabled... If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Org creation is an asynchronous long-running operation (LRO) layered on top of API enablement from the prior step. There's a small window where the APIs report as enabled but haven't fully propagated through Google's internal systems yet. The wizard's org-creation call hits that window, gets rejected, and shows "Something went wrong" — but the expensive networking/VPC-peering work from Step 2 already succeeded, so once the propagation catches up, the retry only has to do the comparatively cheap part (create the org record) and finishes fast.

Don't trust a UI checkmark on faith, though — verify independently. From Cloud Shell:

gcloud config set project YOUR_PROJECT_ID
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_PROJECT_ID"

Look for "state": "ACTIVE" in the response. This hits the Apigee Management API directly, bypassing the Console entirely — the actual ground truth, not a rendered checkmark.

Step 4 (Access routing) subsequently completed cleanly, and the wizard showed the triumphant "Congratulations! Your Apigee organization is ready" message with a "Launch" button pointing at a *.nip.io test URL.

The real bug: green everywhere, 502 everywhere

I clicked Launch. 502 Server Error. The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

I waited the requisite 15–20 minutes for what I assumed was normal load-balancer propagation lag, then tried again. Still 502. I deployed a fresh test proxy just to rule out something proxy-specific. Still 502.

This is the point where "just wait longer" stops being a reasonable answer and it's time to actually verify each layer of the stack independently, from the top down.

Layer 1: Is the request even reaching Apigee's infrastructure?

curl -v "https://YOUR_HOSTNAME.nip.io/hello-world"

The verbose output showed a clean TLS handshake with a valid, freshly-issued Google-managed certificate, HTTP/2 negotiated, and a response that came back structured exactly like Google's generic Global Front End (GFE) "backend unhealthy" page — not a DNS failure, not a connection refusal, not an SSL error. This told me the request was reaching Google's load balancer correctly; the problem was somewhere behind it.

Layer 2: Is the Apigee org/instance actually healthy?

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_ORG/instances"

This came back "state": "ACTIVE" with a real internal host/port and service attachment. So the actual Apigee runtime — the expensive, slow-to-provision Message Processor infrastructure — was completely fine.

Layer 3: Are environment, deployment, and hostname binding all correct?

Three more calls, all clean:

# Is the environment attached to the instance?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../instances/eval-instance/attachments"

# Is a proxy actually deployed?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../environments/eval/deployments"

# Is the hostname bound to an environment group?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../envgroups"

Environment attached, proxy deployed, hostname bound and ACTIVE. Every configuration object in the Apigee control plane was correct. This ruled out the entire Apigee-managed side of the stack. The problem had to be in the one piece of infrastructure that Apigee's "Access routing" step provisions but doesn't directly manage day-to-day: the Compute Engine load balancer and its backing instance group.

Layer 4: The load balancer's backend health

Apigee X's Access Routing step builds a small Compute Engine managed instance group (MIG) of lightweight forwarding VMs that sit between the external HTTPS Load Balancer and the actual Apigee runtime instance (which lives on a private/internal IP). Checking this directly:

gcloud compute backend-services get-health apigee-proxy-backend --global

healthStatus:
- healthState: UNHEALTHY
  instance: .../apigee-proxy-7ztd
- healthState: UNHEALTHY
  instance: .../apigee-proxy-np3w

There it was. Zero of two backend instances were passing their health check. That's a 502 by design — a GFE-class load balancer will never forward traffic to a backend it considers unhealthy, no matter how long you wait.

Root-causing the unhealthy backend

Ruling out the obvious suspect: firewall rules

The most common cause of "backend permanently unhealthy" on a fresh GCP load balancer is a missing firewall rule allowing Google's health-check probe ranges (130.211.0.0/22 and 35.191.0.0/16) through to the target instances. I checked:

gcloud compute firewall-rules list --format="table(name,sourceRanges.list(),allowed[].map().firewall_rule().list(),targetTags.list())"

The rule existed, with the correct source ranges, correct port (443), and a matching target tag (gke-apigee-proxy) — and a quick gcloud compute instances describe confirmed the actual VMs carried that exact tag. Firewall theory eliminated.

Reading the actual boot log

With the network layer cleared, the next move was to read what the instance itself thought happened during boot:

gcloud compute instances get-serial-port-output apigee-proxy-7ztd --zone=asia-south1-b | tail -n 60

Buried in there was the actual smoking gun:

Instance has service account: false, setting ACS client isEnabled to false
...
Failed to download from GCS: downloading object [apigee-envoy-proxy-release/latest/conf/startup-script.sh]...
credentials: cannot fetch token: metadata: GCE metadata "instance/service-accounts/default/token?..." not defined
Trying unauthenticated download

The VM had no service account attached at all. Without one, it can't mint an OAuth token, so it can't authenticate to Cloud Storage to pull down its real startup script (the one that actually sets up the Envoy-based forwarding proxy). It silently fell back to an unauthenticated download, which only pulled a stub containing basic sysctl/iptables scaffolding — nothing that ever configures real port-forwarding to the Apigee runtime.

Confirming this against the instance template sealed it:

gcloud compute instance-templates describe apigee-proxy-asia-south1 \
  --format="yaml(properties.serviceAccounts)"

null

Meanwhile, a healthy default Compute Engine service account existed and was enabled on the project (gcloud iam service-accounts list) — it just wasn't referenced anywhere on this template. This was a genuine defect in how the Apigee provisioning wizard built this specific instance template — not a quota issue, not an org policy, not something I'd misconfigured.

The fix, part 1: rebuild the template with a service account

You can't attach a service account to an existing instance template after the fact — templates are immutable. So the fix is to clone it with the missing piece added, then roll the managed instance group onto the new template:

gcloud compute instance-templates create apigee-proxy-asia-south1-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=asia-south1 \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-asia-south1 \
  --template=apigee-proxy-asia-south1-fixed --region=asia-south1

gcloud compute instance-groups managed rolling-action replace apigee-proxy-asia-south1 \
  --region=asia-south1

Note: copy every field from your existing template's describe --format=yaml output exactly — machine type, disk, network, tags, and especially the scheduling block (preemptible + no-restart-on-failure + maintenance-policy=TERMINATE must all be specified together or gcloud rejects the combination).

After the rolling replace, the new instances came up with the service account correctly attached — the boot log's authentication error was gone. But health checks still failed. One bug down, one to go.

The fix, part 2: the missing `ENDPOINT`

SSHing into a replacement instance and checking what was actually listening confirmed there was still no process bound to port 443:

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo ss -tlnp | grep 443   # → nothing

These forwarding VMs, it turns out, don't run a full proxy application themselves — they run a startup script that installs an iptables DNAT (destination NAT) rule redirecting incoming port-443 traffic straight to the real Apigee runtime instance's internal IP. That destination IP is supplied via an instance metadata key called ENDPOINT. On the original (and my recreated) template, that key existed — but was blank:

metadata:
  items:
  - key: ENDPOINT
    value: ''

With nowhere to forward to, the script had nothing meaningful to do beyond the generic network scaffolding — hence the suspiciously short, error-free-but-incomplete startup log.

The fix: patch the metadata with the real internal IP of the Apigee runtime instance (retrieved earlier from the instances API call in Layer 2), then force each VM to re-run its startup script without a full reboot:

gcloud compute instances add-metadata apigee-proxy-dgz1 \
  --zone=asia-south1-b --metadata=ENDPOINT=10.51.204.98

gcloud compute instances add-metadata apigee-proxy-v4xb \
  --zone=asia-south1-c --metadata=ENDPOINT=10.51.204.98

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo google_metadata_script_runner startup

Checking the NAT table confirmed the fix landed:

sudo iptables -t nat -L -n -v

Chain PREROUTING
DNAT  tcp dpt:443 to:10.51.204.98

Repeated on the second instance, then the moment of truth:

gcloud compute backend-services get-health apigee-proxy-backend --global

healthState: HEALTHY   (apigee-proxy-dgz1)
healthState: HEALTHY   (apigee-proxy-v4xb)

And finally:

curl "https://YOUR_HOSTNAME.nip.io/hello-world"

Hello, Guest!

Done.

Update: it relapsed two days later, and here's why

Two days after publishing this, I opened the project to actually start practicing — and got the exact same 502 again. Same symptom, same 4/4-green setup wizard, same "everything reports ACTIVE" story. My first reaction was that something had regressed on its own, which felt alarming. It hadn't. The real explanation was a gap in the fix above that I'd missed the first time.

Those apigee-proxy-* forwarding VMs are provisioned as --preemptible — cheap, but reclaimable by Google at any time, sometimes within 24 hours. When Google reclaims one, the managed instance group automatically rebuilds a replacement, straight from the instance template — not from whatever state the old VM happened to be in.

Here's the gap: the ENDPOINT fix above patches the metadata on the running instances (gcloud compute instances add-metadata ...), not on the template itself. That's a real, working fix — for as long as those exact VMs stay alive. But the template underneath (apigee-proxy-asia-south1-fixed) still had ENDPOINT blank the whole time. The moment either instance got preempted and silently replaced, the replacement came back from that same broken template — same blank ENDPOINT, same 502, and from the outside it looked like the fix had mysteriously "worn off."

The durable fix is to bake the value into a new template revision, the same pattern as the service-account fix earlier:

gcloud compute instance-templates create apigee-proxy-asia-south1-fixed-v2 \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=asia-south1 \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT=10.51.204.98 \
  --service-account=315668123660-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-asia-south1 \
  --template=apigee-proxy-asia-south1-fixed-v2 --region=asia-south1

gcloud compute instance-groups managed rolling-action replace apigee-proxy-asia-south1 \
  --region=asia-south1

Once ENDPOINT lives in the template rather than on individual instances, any future preemption or autoheal replacement inherits the correct value automatically — no more manual re-patching, ever, for this project.

The lesson generalizes beyond this one bug: on any Compute Engine managed instance group, a fix applied via add-metadata (or any live-instance edit) to a running VM is inherently temporary if that VM can be replaced by the group's own lifecycle — preemption, autohealing, or a manual scale event. The template is the actual source of truth for anything that needs to survive. Patch the instance for an immediate same-minute fix if you're in a hurry; patch the template for the fix that actually sticks.

The debugging methodology, distilled

If you take nothing else from this post, take the layered verification approach — it's the reusable part:

Don't trust a single "it says it's ready" signal. Console checkmarks, API state: ACTIVE fields, and actual traffic serving are three different questions.
Verify each layer of the request path independently, from the outside in:
- TLS/DNS reachability (curl -v)
- Apigee org + runtime instance state (Management API)
- Environment attachment, deployment, and hostname binding (Management API)
- Load balancer backend health (gcloud compute backend-services get-health)
- The actual VM's boot log and listening ports (get-serial-port-output, ss -tlnp)
Read the boot log before assuming propagation delay. A silent early exit with no errors can look identical to "still starting" from the outside — the only way to tell the difference is to read what actually ran.
Understand what's supposed to be running before declaring something broken. My first assumption — that these VMs should have a listening process — was wrong; they're a DNAT relay, not an application server. Getting that model right changed what "healthy" even meant to check for.
When something is auto-provisioned by a wizard, verify its artifacts independently rather than assuming the automation is infallible. In this case, two separate fields on an auto-generated instance template were wrong, and neither surfaced as an explicit error anywhere in the Console.

Should you expect this to happen to you?

Almost certainly not on every setup — this felt like a genuine, if rare, defect in how the Apigee X evaluation wizard built one specific instance template in one specific run. Most eval org setups likely complete without ever seeing this. But if you do hit a 502 that survives 20+ minutes of patient waiting on an otherwise "fully provisioned" eval org, the checklist above will get you to ground truth fast — instead of guessing at propagation delays indefinitely.

If you're setting up Apigee X eval for the first time: budget for the fact that a "Something went wrong" on the org-creation step is often transient and not worth panicking over, but a 502 that survives a proper wait, with all Apigee-side config verified clean, is a signal to go look at the Compute Engine layer underneath — specifically the apigee-proxy-* instance group, its template's service account, and the ENDPOINT metadata key.

Questions or hit something similar? Drop a comment — happy to compare notes.

Building Enterprise-Ready APIs on Google Cloud Apigee X: 5 Hands-On Projects

Sunny JayaRaju — Wed, 12 Aug 2026 18:19:08 +0000

Why I built an Apigee Lab

Enterprise APIs are the backbone of modern platforms. But reading about API gateways is one thing — building them, breaking them, and hardening them is where you actually learn the craft.

That's exactly what I set out to do with Apigee-Lab — an open monorepo of 5 enterprise-inspired projects built on Google Cloud Apigee X.

Each project is fully documented — with architecture diagrams, Mermaid sequence diagrams, deployment guides, and real policy XML. No "hello world" filler. Just production patterns you can actually study and reuse.

Here's what's inside.

1️⃣ Weather Shield Gateway — API Security Layer

A secure, optimized gateway for weather API traffic.

What it demonstrates:

JWT authentication to protect routes
Spike Arrest to stop traffic surges and abuse
Response caching to cut backend load and latency
Monetization quotas (Silver/Gold tiers)

GET /weather-lab?city=London
Authorization: Bearer <jwt>

Request flow: spike arrest → JWT verify → quota check → cache check → backend → transform (XML→JSON).

Read the full architecture in project docs.

2️⃣ Secure Bank Access — OAuth 2.0 Identity Flows

A mock banking API that demonstrates clean identity and access management.

What it demonstrates:

Client Credentials Flow for machine-to-machine security
Conditional proxy flows separating public (/token) and protected (/balance) routes
API Products, Developers, and Apps to enforce scopes and limits

The two-phase pattern: the handshake (get a token) → the access (verify and authorize).

Read the full architecture in project docs.

3️⃣ Retail Mesh Orchestrator — Microservices Aggregation

A composition layer that consolidates multiple backend responses into one payload.

What it demonstrates:

Parallel Service Callouts to cut round-trips
JavaScript transformations for JSON/XML normalization
Fault-tolerant routing during backend failures

The classic Backend-for-Frontend pattern, done with Apigee as the orchestrator.

Read the full architecture in project docs.

4️⃣ Apigee DevOps Pipeline — Full CI/CD Automation

Instead of deploying proxies manually, this project automates the delivery flow.

What it demonstrates:

GitHub Actions workflows that lint, package, and version proxies
apigeelint to catch structural and policy errors before merge
Artifact staging for controlled deployments

Every push runs a quality gate: build structure → lint → package → upload. Bad code never ships.

Read the full pipeline in project docs.

5️⃣ Security Governance — Shared Flows

A centralized security module for global policy enforcement.

What it demonstrates:

Reusable Shared Flows to standardize logic across proxies
Spike Arrests and Security Headers enforced globally via FlowCallout
Governance rules decoupled from individual proxy logic

One shared flow, referenced by many proxies. Consistent security across the entire org.

Read the full architecture in project docs.

What I learned building these

Security-first design: Every proxy needs an identity layer, traffic control, and a governance story — not just routing.
Reusability wins: Shared Flows turn drift-prone duplicate policies into one maintained source of truth.
CI/CD is non-negotiable: Linting and packaging Apigee bundles in CI catches problems before they ever reach eval.
Composition is a superpower: The gateway can be the orchestrator — merging multiple backends into one clean response.

Try it yourself

All projects are documented so you can deploy them to your own Apigee X environment (eval org) and verify them with curl or Postman.

Explore the full lab → github.com/SunnyJayaRaju/Apigee-Lab 🌟

If you build APIs, gateways, or microservices — I'd love your feedback, issues, or a PR. Star it if it helps, and let's keep learning in public.

This article is part of my public learning journey — you can follow more API engineering notes in my Curious-Explorer knowledge base.

DEV Community: Sunny JayaRaju

Apigee X Eval Org Stuck at 502? Check These 2 Things First

Symptom

First: confirm this is actually your bug

Cause #1: Missing service account on the instance template

Cause #2: Blank ENDPOINT metadata

Confirm it's fixed

The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up

TL;DR

Background: how I got here

The rebuild: a promising start, then déjà vu

Why the quick retry wasn't fishy

The real bug: green everywhere, 502 everywhere

Layer 1: Is the request even reaching Apigee's infrastructure?

Layer 2: Is the Apigee org/instance actually healthy?

Layer 3: Are environment, deployment, and hostname binding all correct?

Layer 4: The load balancer's backend health

Root-causing the unhealthy backend

Ruling out the obvious suspect: firewall rules

Reading the actual boot log

The fix, part 1: rebuild the template with a service account

The fix, part 2: the missing ENDPOINT

Update: it relapsed two days later, and here's why

The debugging methodology, distilled

Should you expect this to happen to you?

Building Enterprise-Ready APIs on Google Cloud Apigee X: 5 Hands-On Projects

Why I built an Apigee Lab

1️⃣ Weather Shield Gateway — API Security Layer

2️⃣ Secure Bank Access — OAuth 2.0 Identity Flows

3️⃣ Retail Mesh Orchestrator — Microservices Aggregation

4️⃣ Apigee DevOps Pipeline — Full CI/CD Automation

5️⃣ Security Governance — Shared Flows

What I learned building these

Try it yourself

Cause #2: Blank `ENDPOINT` metadata

The fix, part 2: the missing `ENDPOINT`