DEV Community

Cover image for The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up
Sunny JayaRaju
Sunny JayaRaju

Posted on

The 502 That Wouldn't Die: Debugging a Broken Apigee Eval Org From the Ground Up

#gcp #apigee #cloud #devops

A real-world war story of chasing an Apigee X evaluation organization from "Server Error" to "Hello, Guest!" — the hard way, with gcloud, curl, and a lot of patience.

TL;DR

I spent hours debugging a persistent 502 Server Error on a fresh Apigee X evaluation organization. The Cloud Console showed every provisioning step green. The Apigee Management API confirmed the organization and runtime instance were both ACTIVE. And yet every single request — even to a brand-new proxy — came back with a 502.

The eventual root cause: the Apigee provisioning wizard built the proxy-forwarding instance template without a service account attached, and left a required ENDPOINT metadata key blank. Without a service account, the VMs couldn't authenticate to Cloud Storage to fetch their real startup script. Without ENDPOINT, the fallback script had nothing to forward traffic to. The fix was to clone the broken instance template with the missing service account, roll the managed instance group onto it, then manually patch the ENDPOINT metadata and re-run the startup script on each instance.

If you're staring at a 502 on a "successfully" provisioned Apigee eval org, this post is for you.

Background: how I got here

I was setting up Apigee purely to practice — nothing fancy, no production intent, just a personal free-trial Google Cloud project to learn the platform. Along the way I made a very ordinary mistake: while creating a second project to experiment with, it ended up parented under "No organization" instead of my actual Cloud Identity org, because I had several Console tabs open switching between projects mid-setup.

Confused by the mismatch, I deleted that second project. Shortly after, my original project's Apigee evaluation setup started throwing a resource-locked error on the "Apigee evaluation organization" step, with a message like:

the resource is locked by another operation that is 1 percent completed so far
where organization <project-id> is currently being provisioned by operation: <uuid>
Enter fullscreen mode Exit fullscreen mode

That project's setup never fully completed — the "Access routing" step permanently showed "Not configured." At that point, rather than trying to excavate a half-provisioned org with a stuck long-running operation, I made the call to shut down both projects and start clean in a brand-new project. This is a legitimate, low-cost strategy for Apigee eval orgs specifically — they're free, disposable, and tied to your Cloud Billing account (not the project) for trial credits and the 60-day clock, so nuking a broken project costs you nothing.

Lesson #1: if you're mid-setup and juggling multiple projects, do it in separate browser profiles or windows, not tabs. The Cloud Console's single-page-app state can leak between tabs in confusing ways (I saw one project's setup page briefly display another project's Apigee org name — almost certainly a UI caching artifact, not a real resource collision).

The rebuild: a promising start, then déjà vu

With a fresh project, I walked through Apigee → Try Apigee for free → Set up Apigee Evaluation step by step, in a single tab, waiting for each of the four steps to go green before touching anything else:

  1. Enable APIs ✅
  2. Networking ✅
  3. Apigee evaluation organization
  4. Access routing

Step 3 immediately failed with a generic "Something went wrong" and offered a "Try Again" button. Naturally, my first thought was: here we go again.

But clicking "Try Again" resolved it within about nine minutes — which felt suspicious. Org creation is supposed to be a slow, heavyweight operation. Why would a failed attempt succeed almost instantly on retry?

Why the quick retry wasn't fishy

It turns out this is documented, known behavior. Google's own Apigee installation troubleshooting guide calls out this exact failure mode:

If Apigee returns a 403 when you first try to create the new organization, it could mean that one or more of your APIs have not been enabled... If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Org creation is an asynchronous long-running operation (LRO) layered on top of API enablement from the prior step. There's a small window where the APIs report as enabled but haven't fully propagated through Google's internal systems yet. The wizard's org-creation call hits that window, gets rejected, and shows "Something went wrong" — but the expensive networking/VPC-peering work from Step 2 already succeeded, so once the propagation catches up, the retry only has to do the comparatively cheap part (create the org record) and finishes fast.

Don't trust a UI checkmark on faith, though — verify independently. From Cloud Shell:

gcloud config set project YOUR_PROJECT_ID
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_PROJECT_ID"
Enter fullscreen mode Exit fullscreen mode

Look for "state": "ACTIVE" in the response. This hits the Apigee Management API directly, bypassing the Console entirely — the actual ground truth, not a rendered checkmark.

Step 4 (Access routing) subsequently completed cleanly, and the wizard showed the triumphant "Congratulations! Your Apigee organization is ready" message with a "Launch" button pointing at a *.nip.io test URL.

The real bug: green everywhere, 502 everywhere

I clicked Launch. 502 Server Error. The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

I waited the requisite 15–20 minutes for what I assumed was normal load-balancer propagation lag, then tried again. Still 502. I deployed a fresh test proxy just to rule out something proxy-specific. Still 502.

This is the point where "just wait longer" stops being a reasonable answer and it's time to actually verify each layer of the stack independently, from the top down.

Layer 1: Is the request even reaching Apigee's infrastructure? 

curl -v "https://YOUR_HOSTNAME.nip.io/hello-world"
Enter fullscreen mode Exit fullscreen mode

The verbose output showed a clean TLS handshake with a valid, freshly-issued Google-managed certificate, HTTP/2 negotiated, and a response that came back structured exactly like Google's generic Global Front End (GFE) "backend unhealthy" page — not a DNS failure, not a connection refusal, not an SSL error. This told me the request was reaching Google's load balancer correctly; the problem was somewhere behind it.

Layer 2: Is the Apigee org/instance actually healthy? 

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  "https://apigee.googleapis.com/v1/organizations/YOUR_ORG/instances"
Enter fullscreen mode Exit fullscreen mode

This came back "state": "ACTIVE" with a real internal host/port and service attachment. So the actual Apigee runtime — the expensive, slow-to-provision Message Processor infrastructure — was completely fine.

Layer 3: Are environment, deployment, and hostname binding all correct?

Three more calls, all clean:

# Is the environment attached to the instance?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../instances/eval-instance/attachments"

# Is a proxy actually deployed?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../environments/eval/deployments"

# Is the hostname bound to an environment group?
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  ".../envgroups"
Enter fullscreen mode Exit fullscreen mode

Environment attached, proxy deployed, hostname bound and ACTIVE. Every configuration object in the Apigee control plane was correct. This ruled out the entire Apigee-managed side of the stack. The problem had to be in the one piece of infrastructure that Apigee's "Access routing" step provisions but doesn't directly manage day-to-day: the Compute Engine load balancer and its backing instance group.

Layer 4: The load balancer's backend health

Apigee X's Access Routing step builds a small Compute Engine managed instance group (MIG) of lightweight forwarding VMs that sit between the external HTTPS Load Balancer and the actual Apigee runtime instance (which lives on a private/internal IP). Checking this directly:

gcloud compute backend-services get-health apigee-proxy-backend --global
Enter fullscreen mode Exit fullscreen mode 
healthStatus:
- healthState: UNHEALTHY
  instance: .../apigee-proxy-7ztd
- healthState: UNHEALTHY
  instance: .../apigee-proxy-np3w
Enter fullscreen mode Exit fullscreen mode

There it was. Zero of two backend instances were passing their health check. That's a 502 by design — a GFE-class load balancer will never forward traffic to a backend it considers unhealthy, no matter how long you wait.

Root-causing the unhealthy backend

Ruling out the obvious suspect: firewall rules

The most common cause of "backend permanently unhealthy" on a fresh GCP load balancer is a missing firewall rule allowing Google's health-check probe ranges (130.211.0.0/22 and 35.191.0.0/16) through to the target instances. I checked:

gcloud compute firewall-rules list --format="table(name,sourceRanges.list(),allowed[].map().firewall_rule().list(),targetTags.list())"
Enter fullscreen mode Exit fullscreen mode

The rule existed, with the correct source ranges, correct port (443), and a matching target tag (gke-apigee-proxy) — and a quick gcloud compute instances describe confirmed the actual VMs carried that exact tag. Firewall theory eliminated.

Reading the actual boot log

With the network layer cleared, the next move was to read what the instance itself thought happened during boot:

gcloud compute instances get-serial-port-output apigee-proxy-7ztd --zone=asia-south1-b | tail -n 60
Enter fullscreen mode Exit fullscreen mode

Buried in there was the actual smoking gun:

Instance has service account: false, setting ACS client isEnabled to false
...
Failed to download from GCS: downloading object [apigee-envoy-proxy-release/latest/conf/startup-script.sh]...
credentials: cannot fetch token: metadata: GCE metadata "instance/service-accounts/default/token?..." not defined
Trying unauthenticated download
Enter fullscreen mode Exit fullscreen mode

The VM had no service account attached at all. Without one, it can't mint an OAuth token, so it can't authenticate to Cloud Storage to pull down its real startup script (the one that actually sets up the Envoy-based forwarding proxy). It silently fell back to an unauthenticated download, which only pulled a stub containing basic sysctl/iptables scaffolding — nothing that ever configures real port-forwarding to the Apigee runtime.

Confirming this against the instance template sealed it:

gcloud compute instance-templates describe apigee-proxy-asia-south1 \
  --format="yaml(properties.serviceAccounts)"
Enter fullscreen mode Exit fullscreen mode 
null
Enter fullscreen mode Exit fullscreen mode

Meanwhile, a healthy default Compute Engine service account existed and was enabled on the project (gcloud iam service-accounts list) — it just wasn't referenced anywhere on this template. This was a genuine defect in how the Apigee provisioning wizard built this specific instance template — not a quota issue, not an org policy, not something I'd misconfigured.

The fix, part 1: rebuild the template with a service account

You can't attach a service account to an existing instance template after the fact — templates are immutable. So the fix is to clone it with the missing piece added, then roll the managed instance group onto the new template:

gcloud compute instance-templates create apigee-proxy-asia-south1-fixed \
  --machine-type=e2-micro \
  --image-project=debian-cloud --image-family=debian-12 \
  --boot-disk-size=20GB \
  --network=default --subnet=default --region=asia-south1 \
  --tags=https-server,apigee-proxy,gke-apigee-proxy \
  --metadata=startup-script-url=gs://apigee-5g-saas/apigee-envoy-proxy-release/latest/conf/startup-script.sh,ENDPOINT= \
  --service-account=YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com \
  --scopes=cloud-platform \
  --preemptible --no-restart-on-failure --maintenance-policy=TERMINATE

gcloud compute instance-groups managed set-instance-template apigee-proxy-asia-south1 \
  --template=apigee-proxy-asia-south1-fixed --region=asia-south1

gcloud compute instance-groups managed rolling-action replace apigee-proxy-asia-south1 \
  --region=asia-south1
Enter fullscreen mode Exit fullscreen mode

Note: copy every field from your existing template's describe --format=yaml output exactly — machine type, disk, network, tags, and especially the scheduling block (preemptible + no-restart-on-failure + maintenance-policy=TERMINATE must all be specified together or gcloud rejects the combination).

After the rolling replace, the new instances came up with the service account correctly attached — the boot log's authentication error was gone. But health checks still failed. One bug down, one to go.

The fix, part 2: the missing ENDPOINT

SSHing into a replacement instance and checking what was actually listening confirmed there was still no process bound to port 443:

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo ss -tlnp | grep 443   # → nothing
Enter fullscreen mode Exit fullscreen mode

These forwarding VMs, it turns out, don't run a full proxy application themselves — they run a startup script that installs an iptables DNAT (destination NAT) rule redirecting incoming port-443 traffic straight to the real Apigee runtime instance's internal IP. That destination IP is supplied via an instance metadata key called ENDPOINT. On the original (and my recreated) template, that key existed — but was blank:

metadata:
  items:
  - key: ENDPOINT
    value: ''
Enter fullscreen mode Exit fullscreen mode

With nowhere to forward to, the script had nothing meaningful to do beyond the generic network scaffolding — hence the suspiciously short, error-free-but-incomplete startup log.

The fix: patch the metadata with the real internal IP of the Apigee runtime instance (retrieved earlier from the instances API call in Layer 2), then force each VM to re-run its startup script without a full reboot:

gcloud compute instances add-metadata apigee-proxy-dgz1 \
  --zone=asia-south1-b --metadata=ENDPOINT=10.51.204.98

gcloud compute instances add-metadata apigee-proxy-v4xb \
  --zone=asia-south1-c --metadata=ENDPOINT=10.51.204.98

gcloud compute ssh apigee-proxy-dgz1 --zone=asia-south1-b
sudo google_metadata_script_runner startup
Enter fullscreen mode Exit fullscreen mode

Checking the NAT table confirmed the fix landed:

sudo iptables -t nat -L -n -v
Enter fullscreen mode Exit fullscreen mode 
Chain PREROUTING
DNAT  tcp dpt:443 to:10.51.204.98
Enter fullscreen mode Exit fullscreen mode

Repeated on the second instance, then the moment of truth:

gcloud compute backend-services get-health apigee-proxy-backend --global
Enter fullscreen mode Exit fullscreen mode 
healthState: HEALTHY   (apigee-proxy-dgz1)
healthState: HEALTHY   (apigee-proxy-v4xb)
Enter fullscreen mode Exit fullscreen mode

And finally:

curl "https://YOUR_HOSTNAME.nip.io/hello-world"
Enter fullscreen mode Exit fullscreen mode 
Hello, Guest!
Enter fullscreen mode Exit fullscreen mode

Done.

The debugging methodology, distilled

If you take nothing else from this post, take the layered verification approach — it's the reusable part:

  1. Don't trust a single "it says it's ready" signal. Console checkmarks, API state: ACTIVE fields, and actual traffic serving are three different questions.
  2. Verify each layer of the request path independently, from the outside in:
    • TLS/DNS reachability (curl -v)
    • Apigee org + runtime instance state (Management API)
    • Environment attachment, deployment, and hostname binding (Management API)
    • Load balancer backend health (gcloud compute backend-services get-health)
    • The actual VM's boot log and listening ports (get-serial-port-output, ss -tlnp)
  3. Read the boot log before assuming propagation delay. A silent early exit with no errors can look identical to "still starting" from the outside — the only way to tell the difference is to read what actually ran.
  4. Understand what's supposed to be running before declaring something broken. My first assumption — that these VMs should have a listening process — was wrong; they're a DNAT relay, not an application server. Getting that model right changed what "healthy" even meant to check for.
  5. When something is auto-provisioned by a wizard, verify its artifacts independently rather than assuming the automation is infallible. In this case, two separate fields on an auto-generated instance template were wrong, and neither surfaced as an explicit error anywhere in the Console.

Should you expect this to happen to you?

Almost certainly not on every setup — this felt like a genuine, if rare, defect in how the Apigee X evaluation wizard built one specific instance template in one specific run. Most eval org setups likely complete without ever seeing this. But if you do hit a 502 that survives 20+ minutes of patient waiting on an otherwise "fully provisioned" eval org, the checklist above will get you to ground truth fast — instead of guessing at propagation delays indefinitely.

If you're setting up Apigee X eval for the first time: budget for the fact that a "Something went wrong" on the org-creation step is often transient and not worth panicking over, but a 502 that survives a proper wait, with all Apigee-side config verified clean, is a signal to go look at the Compute Engine layer underneath — specifically the apigee-proxy-* instance group, its template's service account, and the ENDPOINT metadata key.

Questions or hit something similar? Drop a comment — happy to compare notes.

Top comments (0)