How to Run a Real-Time API Across 10 Global Regions on GKE — Without Duplicating Your Manifests

When you're building a real-time speech translation product, latency isn't a performance metric — it's the product itself. A 300ms round-trip to process an audio chunk is the difference between a conversation that feels natural and one that feels broken. That's the constraint that pushed us to design a multi-cluster, multi-region Kubernetes setup across 10 GCP regions.

This article walks through the full architecture: every layer, the decision behind each one, and the gotchas that will bite you if you're not careful. I'll use a fictional app called SpeakFlow — a real-time speech translation SaaS — to illustrate the patterns.

---

The Problem That Forces Multi-Region

SpeakFlow processes real-time audio in a tight loop: speech-to-text, translation, text-to-speech, under strict latency budgets. A user speaking in Tokyo, routed to a cluster in us-central1, adds ~150ms of pure network latency before any compute happens. For a real-time product, that's a broken experience.

The solution is obvious in theory: run the stack close to users. 10 GCP regions, covering every major market — North America, Europe (including a GDPR-dedicated Germany cluster), Asia Pacific, South America, Australia.

The engineering challenge: do it without creating 10 independently drifting copies of your infrastructure. One team, one source of truth, 10 regions.

---

The Stack

Before going into the how, here's what gets deployed to every region:

• translate-api — main API, handles REST and WebSockets (Socket.IO)
• translate-agg — aggregates real-time transcription events
• translate-worker — async batch processing for offline audio jobs
• manager-api — admin operations and organization management
• dashboard-app — the React SPA served via nginx

Each runs in its own GKE cluster per region, backed by regional Memorystore Redis and a paired Azure Cognitive Services instance (STT, Translator, Language).

The dual-cloud pattern — GCP for compute, Azure for AI services — is a deliberate decision. Azure's speech and translation APIs are best-in-class for our use case. Running them in the same geographic zone as the GKE cluster keeps cross-cloud latency under 20ms.

---

The Core Pattern: One Base Manifest, Ten Overlays

The most important architectural decision is using Kustomize with a strict base/overlay separation.

prod.yml is the single source of truth for all deployments, services, HPAs, and security configurations across all 10 regions. It is never applied directly to a cluster. It defines everything that is identical everywhere: image tags, replica counts, resource limits, probe paths, security contexts, and all secrets that are shared across regions.

k8s/manifests/speakflow/multiregion/
├── prod.yml                ← base — READ ONLY, never apply directly
├── external-secret.yml     ← base ExternalSecret (ESO v1)
├── gateway-prod.yml        ← GKE Gateway + HTTPRoutes (once, not per-region)
└── per-region/
    ├── us-central1/kustomization.yml
    ├── europe-west2/kustomization.yml
    ├── europe-west3/kustomization.yml
    ├── northamerica-northeast1/kustomization.yml
    ├── asia-northeast1/kustomization.yml
    ├── southamerica-east1/kustomization.yml
    ├── australia-southeast1/kustomization.yml
    ├── asia-southeast1/kustomization.yml
    ├── asia-northeast3/kustomization.yml
    └── asia-south1/kustomization.yml

Each per-region/{region}/kustomization.yml patches exactly two things:

A. The REGION environment variable + topology.kubernetes.io/region annotation on each backend Deployment. Services use REGION to route pub/sub messages and tag metrics correctly.

B. The first 9 entries in the ExternalSecret — the ones that differ per region: Memorystore Redis connection details and Azure Cognitive Services credentials for that region's paired Azure location.

Everything else is inherited from the base.

What a per-region overlay looks like

Here's the europe-west2 overlay (GCP London / Azure uksouth):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: speakflow-prod
resources:
  - ../../prod.yml
  - ../../external-secret.yml
patches:
  # [A1] translate-api — REGION env (index 28) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/28/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-api

  # [A2] translate-agg — REGION env (index 7) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/7/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-agg

  # [B] ExternalSecret — regional Redis + Azure Cognitive Services (uksouth)
  - patch: |-
      - op: replace
        path: /spec/data/0/remoteRef/key
        value: speakflow-prod-europe-west2-redis-url
      - op: replace
        path: /spec/data/1/remoteRef/key
        value: speakflow-prod-europe-west2-redis-pubsub-host
      - op: replace
        path: /spec/data/2/remoteRef/key
        value: speakflow-prod-europe-west2-redis-password
      - op: replace
        path: /spec/data/3/remoteRef/key
        value: speakflow-prod-europe-west2-stt-region
      - op: replace
        path: /spec/data/4/remoteRef/key
        value: speakflow-prod-europe-west2-stt-key
      - op: replace
        path: /spec/data/5/remoteRef/key
        value: speakflow-prod-europe-west2-translator-key
      - op: replace
        path: /spec/data/6/remoteRef/key
        value: speakflow-prod-europe-west2-translator-region
      - op: replace
        path: /spec/data/7/remoteRef/key
        value: speakflow-prod-europe-west2-language-endpoint
      - op: replace
        path: /spec/data/8/remoteRef/key
        value: speakflow-prod-europe-west2-language-key
    target:
      kind: ExternalSecret
      name: translate-api

The overlay is a pure diff. It contains no credentials — only the names of secrets in GCP Secret Manager.

The index stability trap you must not ignore

JSON Patch addresses array elements by position. env/28 means the 29th environment variable in the container spec. This is load-bearing.

Deployment	REGION env index
translate-api	env[28]
translate-agg	env[7]
translate-worker	env[12]
manager-api	env[17]

If you add a new environment variable before the REGION entry in prod.yml, every overlay patch breaks silently. The REGION value gets applied to the wrong variable, and your services start routing pub/sub messages to the wrong region — a failure mode that looks like an application bug, not an infrastructure one.

Rules to live by:

• Always append new env vars after REGION, never insert before it.
• Do not reorder the first 9 entries in the ExternalSecret.
• If you must change the order, update all 10 overlay patches before merging.

Document the indices in your README and treat that table as load-bearing infrastructure.

---

Secrets Management: External Secrets Operator + GCP Secret Manager

Hardcoded secrets in manifests are a non-starter for a multi-region setup. We use the External Secrets Operator (ESO) with GCP Workload Identity to sync secrets from GCP Secret Manager into Kubernetes Secrets at runtime.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: translate-api
  namespace: speakflow-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: translate-api
    creationPolicy: Owner
  data:
    # ── Regional secrets (indices 0–8) — patched per region by Kustomize ────
    - secretKey: redisUrl              # index 0
      remoteRef:
        key: speakflow-prod-us-central1-redis-url   # ← patched by overlay
    - secretKey: redisPubsubHost       # index 1
      remoteRef:
        key: speakflow-prod-us-central1-redis-pubsub-host
    - secretKey: redisPassword         # index 2
      remoteRef:
        key: speakflow-prod-us-central1-redis-password
    - secretKey: sttRegion             # index 3
      remoteRef:
        key: speakflow-prod-us-central1-stt-region
    - secretKey: sttKey                # index 4
      remoteRef:
        key: speakflow-prod-us-central1-stt-key
    - secretKey: translatorKey         # index 5
      remoteRef:
        key: speakflow-prod-us-central1-translator-key
    - secretKey: translatorRegion      # index 6
      remoteRef:
        key: speakflow-prod-us-central1-translator-region
    - secretKey: languageEndpoint      # index 7
      remoteRef:
        key: speakflow-prod-us-central1-language-endpoint
    - secretKey: languageKey           # index 8
      remoteRef:
        key: speakflow-prod-us-central1-language-key

    # ── Shared secrets (index 9+) — same value across all regions ───────────
    - secretKey: jwtSecret             # index 9
      remoteRef:
        key: speakflow-prod-jwt-secret
    - secretKey: dbUri                 # index 10
      remoteRef:
        key: speakflow-prod-db-uri
    # ... additional shared secrets

The design splits secrets into two groups:

Regional (indices 0–8): Memorystore Redis private IP addresses, and Azure Cognitive Services API keys and region names. These are unique to each GCP+Azure region pair, provisioned in Secret Manager under a speakflow-prod-{region}-{key} naming convention.

Shared (index 9+): JWT secret, database URI, AI provider keys, email service, auth provider. Provisioned once, read by all 10 regions.

The ClusterSecretStore named gcp-secret-store is backed by GCP Workload Identity — provisioned by Terraform, not by these manifests.

Rotating a regional secret

# 1. Update the value in GCP Secret Manager (via console or gcloud)

# 2. Force an immediate ESO sync without waiting for refreshInterval
kubectl annotate externalsecret translate-api -n speakflow-prod \
  force-sync=$(date +%s) --overwrite

# 3. Pods read secrets from env on startup — rolling restart required
kubectl rollout restart deployment/translate-api -n speakflow-prod
kubectl rollout restart deployment/translate-agg -n speakflow-prod
kubectl rollout restart deployment/translate-worker -n speakflow-prod
kubectl rollout restart deployment/manager-api -n speakflow-prod

---

The Dual-Cloud Region Pairing

Each GCP region is paired with the geographically nearest Azure region for AI services. This is what keeps cross-cloud latency under 20ms:

GCP region	Azure Cognitive Services region
us-central1	eastus (STT) / westus2 (Translator)
europe-west2	uksouth
europe-west3 (GDPR)	germanywestcentral
northamerica-northeast1	canadacentral
asia-northeast1	japaneast
southamerica-east1	brazilsouth
australia-southeast1	australiaeast
asia-southeast1	southeastasia
asia-northeast3	koreacentral
asia-south1	centralindia

The asymmetry in us-central1 (different Azure regions for STT vs. Translator) is a real-world quirk — not every Azure service has the same regional availability. This asymmetry lives in the Secret Manager values, not in the manifests. The per-region secret speakflow-prod-us-central1-stt-region stores eastus and speakflow-prod-us-central1-translator-region stores westus2. The manifest is unaware of the difference.

---

Multi-Cluster Routing: GKE Fleet + ServiceExport/Import

The Gateway layer is where the multi-cluster routing happens. All 10 regional clusters are enrolled in a GKE Fleet, federated into a single logical mesh. The GKE multi-cluster Gateway controller (gke-l7-global-external-managed-mc) exposes services across it.

The key primitives are ServiceExport and ServiceImport.

In prod.yml, each service that needs to be reachable cross-cluster gets a ServiceExport:

apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: translate-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: manager-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: dashboard-app
  namespace: speakflow-prod

The Fleet control plane creates corresponding ServiceImport resources cluster-wide automatically.

The Gateway and HTTPRoutes are deployed once — not per region. The gke-l7-global-external-managed-mc GatewayClass handles global traffic distribution:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: speakflow-prod-gateway
  namespace: speakflow-prod
  annotations:
    networking.gke.io/certmap: "speakflow-cert-map"
spec:
  gatewayClassName: gke-l7-global-external-managed-mc
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
  addresses:
    - type: NamedAddress
      value: speakflow-prod-static-ip

HTTPRoutes reference services by ServiceImport — this is what tells the load balancer to route to the nearest healthy backend across all clusters:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: translate-api-route
  namespace: speakflow-prod
spec:
  parentRefs:
    - name: speakflow-prod-gateway
      sectionName: https
  hostnames:
    - "api.speakflow.io"
  rules:
    # REST API traffic
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: translate-api
          kind: ServiceImport      # ← multi-cluster aware
          group: net.gke.io
          port: 80
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /

    # WebSocket traffic (Socket.IO)
    - matches:
        - path:
            type: PathPrefix
            value: /socket.io
      backendRefs:
        - name: translate-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

    # Manager API
    - matches:
        - path:
            type: PathPrefix
            value: /manager
      backendRefs:
        - name: manager-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

The GKE load balancer routes each request to the cluster with the lowest latency healthy backends — no application-level routing logic needed. The user in Tokyo hits asia-northeast1. The user in São Paulo hits southamerica-east1.

One thing that surprised us: GKE's managed multi-cluster Gateway handles WebSocket connection persistence correctly without extra configuration. We initially expected to write sticky-session logic ourselves; the -mc GatewayClass handled it.

---

Health Probes: The Live/Ready Split That Matters

One decision that significantly improved our production stability was enforcing a strict separation between liveness and readiness probe behavior.

readinessProbe:
  httpGet:
    path: /health/ready   # checks Redis + DB connectivity
    port: 3000
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /health/live    # in-process only — event loop, heap
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 15
  failureThreshold: 3

The principle: liveness must never fail due to a dependency outage.

If the managed Redis instance goes down, we don't want Kubernetes restarting all pods in a cascade — the pods are healthy, the dependency is the problem. GET /health/live checks only in-process state: is the Node.js event loop responsive, is heap usage within bounds.

GET /health/ready checks Redis and the database. When those fail, the pod is removed from the load balancer — traffic is redirected to other regions or healthy pods — but the pod is not killed. Recovery is automatic when the dependency comes back.

We also configure heap size explicitly:

env:
  - name: NODE_OPTIONS
    value: '--max-old-space-size=896'

Without this, the healthcheck library derives its heap threshold from the V8 default (1200MB), not from the container's memory limit (1024Mi). Setting it to ~87% of the limit prevents false health alerts when the container approaches its ceiling.

---

The SPA Init Container Pattern

The dashboard-app is a React SPA built at compile time — no runtime environment access. But we need different API endpoints, auth server URLs, and feature flag keys per environment. An init container solves this by performing token substitution on the compiled JS bundles at pod startup:

initContainers:
  - name: env-substitution
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsUser: 0  # root required to read image files and write to emptyDir
    command:
      - /bin/sh
      - -c
      - |
        set -e
        cp -rp /usr/share/nginx/html/assets/. /mnt/assets/
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__API_URL_PLACEHOLDER__,'"$API_URL"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__AUTH_CLIENT_ID_PLACEHOLDER__,'"$AUTH_CLIENT_ID"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__FEATURE_FLAGS_KEY_PLACEHOLDER__,'"$FF_KEY"',g' {} \;
        chmod -R a+r /mnt/assets
    volumeMounts:
      - name: html-assets
        mountPath: /mnt/assets
    env:
      - name: API_URL
        value: https://api.speakflow.io
      - name: AUTH_CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: authClientId
      - name: FF_KEY
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: featureFlagsKey
volumes:
  - name: html-assets
    emptyDir: {}

containers:
  - name: dashboard-app
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 10002
      allowPrivilegeEscalation: false
      capabilities:
        drop: ['ALL']
    volumeMounts:
      - name: html-assets
        mountPath: /usr/share/nginx/html/assets

The init container copies the static assets to an emptyDir volume shared with the main container, substitutes placeholders with values from Kubernetes Secrets, then exits. nginx serves the patched files. One Docker image works in any environment without rebuilding.

The runAsUser: 0 on the init container is intentional — it needs root to read image-owned files and write to the shared volume. The main container runs with full non-root restrictions. Document this explicitly; it will come up in security reviews.

---

HPA + Cost-Aware Replica Scaling

The base manifest sets minReplicas: 2 for genuine HA, scaling up to 10 on CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: translate-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: translate-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Not every region has the same traffic volume. For regions still ramping up, per-region overlays patch minReplicas down to 1 and reduce replica counts on lower-priority services:

- patch: |-
    - op: replace
      path: /spec/minReplicas
      value: 1
  target:
    kind: HorizontalPodAutoscaler
    name: translate-api-hpa
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 1
  target:
    kind: Deployment
    name: translate-agg

High-traffic regions stay at the 2-replica baseline. The per-region patch is a cost control that you revisit as real traffic data comes in — not a permanent configuration.

---

Operations

Deploying to a region

# Render and inspect before applying
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone

# Apply (ensure kubectl context is set to that cluster)
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone \
  | kubectl apply -n speakflow-prod -f -

--load-restrictor=LoadRestrictionsNone is required because the overlay references files two levels above it. This relaxes kustomize's path check only — no effect on Kubernetes RBAC.

Verifying a region after deploy

# ExternalSecret sync status (should be Ready)
kubectl get externalsecret translate-api -n speakflow-prod

# Confirm the synced Secret exists
kubectl get secret translate-api -n speakflow-prod

# Check the REGION env var in a running pod
kubectl exec -n speakflow-prod deploy/translate-api -- printenv REGION

# Check topology annotation on pods
kubectl get pods -n speakflow-prod -l app=translate-api \
  -o jsonpath='{.items[*].metadata.annotations.topology\.kubernetes\.io/region}'

Bumping image tags across all regions

Image tags live in prod.yml. Change them once; all overlays inherit on next apply:

sed -i 's|:1.4.2|:1.5.0|g' k8s/manifests/speakflow/multiregion/prod.yml

Adding a new region

REGION=europe-north1
cp -r k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
      k8s/manifests/speakflow/multiregion/per-region/$REGION

sed -i "s/europe-west2/$REGION/g" \
  k8s/manifests/speakflow/multiregion/per-region/$REGION/kustomization.yml

Then provision the 9 regional secrets in GCP Secret Manager, render and verify, apply.

CI validation — render all regions on every PR

#!/usr/bin/env bash
set -euo pipefail

REGIONS=(
  us-central1 europe-west2 europe-west3
  northamerica-northeast1 asia-northeast1
  southamerica-east1 australia-southeast1
  asia-southeast1 asia-northeast3 asia-south1
)
BASE=k8s/manifests/speakflow/multiregion/per-region

for region in "${REGIONS[@]}"; do
  echo "Rendering $region..."
  kubectl kustomize "$BASE/$region" --load-restrictor=LoadRestrictionsNone > /dev/null
  echo "  OK"
done

echo "All regions rendered successfully."

Run this in CI on every PR that touches anything under the multiregion directory. It catches broken JSON Patch paths before they reach a cluster.

---

Key Takeaways

The dual-cloud pairing table is as important as the Kubernetes config. Running GCP compute and Azure AI services in the same geographic zone is the actual latency win. Getting the pairings wrong means your speech-to-text requests are still crossing oceans.

Kustomize JSON Patch indices are silent failures. When env/28 patches the wrong variable because you inserted something earlier in the list, services misbehave in ways that look like application bugs. Treat the index table in your README as load-bearing infrastructure.

ESO's refreshInterval is not the whole rotation story. Secrets sync every hour, but pods pick up new values only on restart. Build your rotation runbook around annotate-to-force-sync + rolling restart, not just the Secret Manager update.

The init container pattern for SPAs scales better than build-time env injection. One image, any environment. Substitution happens in under a second at pod startup and survives deployments without rebuilding.

gke-l7-global-external-managed-mc handles WebSocket routing. We expected to write sticky-session logic for WebSocket connections. The multi-cluster GatewayClass did the work. Try the managed path before building custom routing.

Scale replicas per region, not globally. The cost difference between minReplicas: 1 and minReplicas: 2 across 8 secondary regions adds up. Use traffic data to decide when to promote a region to full HA baseline.

---

If you're building something similar and have questions, the comments are open.