A user in Frankfurt hitting an API hosted in us-central1 pays a fixed tax: roughly 90–110 ms of round-trip time before a single byte of application logic runs. Add TLS handshake and you have burned 300 ms before your handler is invoked. No amount of code optimization fixes the speed of light.
Google Cloud's global external Application Load Balancer solves part of this problem by terminating the connection at the closest of ~190 Google Front End (GFE) points of presence, then carrying the request over Google's private backbone to the nearest healthy backend. The entry point is a single anycast IPv4/IPv6 address — one DNS record, no GeoDNS, no per-region hostnames.
This article walks through building that front end: multi-region backends (MIGs and serverless NEGs), health checks tuned for fast failover, gcloud and Terraform implementations, GKE standalone NEGs, and a Cloud Armor baseline.
Context: why DNS-based geo-routing is not enough
The classic pattern is a weighted/geo DNS record pointing at regional load balancers: api.example.com resolves to the EU IP for European resolvers, the US IP for American ones. It works, but it has three structural problems.
TTL is your failover budget. If a region dies, clients keep hammering the dead IP until their resolver cache expires. Browsers and JVMs cache aggressively and ignore your 30s TTL. Real failover time is measured in minutes.
Resolver location ≠ client location. A client using a public DNS resolver may be geolocated hundreds of kilometres away. EDNS Client Subnet helps, but coverage is partial.
TCP and TLS still start at the origin. With DNS routing, the three-way handshake and the TLS handshake travel the full distance to the region. With an anycast front end, the handshake terminates at the PoP a few milliseconds away; only the HTTP request crosses the ocean — and it does so on a warm, multiplexed connection over Google's backbone, not the public internet.
Measured effect on a Paris → us-central1 path: a cold HTTPS request drops from roughly 480 ms to 180–220 ms, mostly because handshake round-trips are localised.
Architecture overview
The target topology:
api.example.com → 34.x.x.x (global anycast)
│
┌────────┴────────┐
│ GFE / PoP │ TLS termination, Cloud Armor, Cloud CDN
└────────┬────────┘
│ Google backbone
┌──────────────────────┼──────────────────────┐
│ │ │
europe-west1 us-central1 asia-southeast1
MIG / GKE NEG MIG / GKE NEG Cloud Run NEG
One global backend service holds all regional backends. The GFE picks the closest backend that still has capacity and passes health checks; when a region is unhealthy or saturated, traffic spills to the next-closest one automatically.
Components, in dependency order: health check → backend service → backends (NEGs/MIGs) → URL map → target HTTPS proxy → global forwarding rule + static IP.
Building the front end with gcloud
1. Reserve the anycast address
export PROJECT_ID=my-project
gcloud config set project "$PROJECT_ID"
# Global external IPv4 — anycast, announced from every Google PoP
gcloud compute addresses create api-global-ipv4 \
--ip-version=IPV4 \
--network-tier=PREMIUM \
--global
gcloud compute addresses create api-global-ipv6 \
--ip-version=IPV6 \
--network-tier=PREMIUM \
--global
gcloud compute addresses describe api-global-ipv4 --global --format='value(address)'
PREMIUM tier is mandatory for the global ALB. Standard tier gives you regional routing over the public internet — exactly what we are trying to avoid.
2. A health check that actually fails fast
Default health check settings (5s interval, 2 failures) mean up to 15 seconds of blackholing. Tighten it, and point it at a dedicated endpoint that checks dependencies — not at /.
gcloud compute health-checks create http hc-api-readyz \
--port=8080 \
--request-path=/readyz \
--check-interval=2s \
--timeout=1s \
--healthy-threshold=2 \
--unhealthy-threshold=2 \
--enable-logging \
--global
With 2s × 2 = 4s detection plus propagation, a region is drained in under 10 seconds. --enable-logging emits a Cloud Logging entry on every health state transition — invaluable during incidents:
gcloud logging read \
'resource.type="gce_instance_group" AND jsonPayload.healthCheckProbeResult.detailedHealthState!=""' \
--limit=20 --format='table(timestamp, jsonPayload.healthCheckProbeResult.detailedHealthState)'
Do not go below 1s interval: probes come from every GFE region, so a 1s interval on a 50-instance MIG generates meaningful load.
3. Backend service with multi-region backends
gcloud compute backend-services create api-backend \
--load-balancing-scheme=EXTERNAL_MANAGED \
--protocol=HTTP \
--port-name=http \
--health-checks=hc-api-readyz \
--global-health-checks \
--timeout=30s \
--connection-draining-timeout=60s \
--enable-logging --logging-sample-rate=1.0 \
--global
# Europe: managed instance group
gcloud compute backend-services add-backend api-backend \
--instance-group=api-mig-euw1 \
--instance-group-region=europe-west1 \
--balancing-mode=RATE \
--max-rate-per-instance=120 \
--capacity-scaler=1.0 \
--global
# US: managed instance group
gcloud compute backend-services add-backend api-backend \
--instance-group=api-mig-usc1 \
--instance-group-region=us-central1 \
--balancing-mode=RATE \
--max-rate-per-instance=120 \
--capacity-scaler=1.0 \
--global
EXTERNAL_MANAGED selects the modern (Envoy-based) global ALB rather than the classic one — required for advanced traffic management such as outlier detection and weighted routing.
--balancing-mode=RATE with --max-rate-per-instance defines when a region is "full". Once Europe exceeds its aggregate capacity, overflow goes to the US instead of queuing. --capacity-scaler=0.0 drains a region without removing it — your manual kill switch during a regional rollout.
4. Serverless NEGs for Cloud Run regions
A serverless NEG maps a region's Cloud Run (or App Engine/Cloud Functions) service into the same backend service:
for REGION in europe-west1 us-central1 asia-southeast1; do
gcloud compute network-endpoint-groups create neg-api-"$REGION" \
--region="$REGION" \
--network-endpoint-type=SERVERLESS \
--cloud-run-service=api
gcloud compute backend-services add-backend api-backend \
--network-endpoint-group=neg-api-"$REGION" \
--network-endpoint-group-region="$REGION" \
--global
done
Important gotcha: serverless NEG backends do not support health checks, balancing modes, or capacity scalers. Availability is handled by the Cloud Run control plane, and the GFE routes to the closest region with a deployed revision. If you need explicit health-based failover semantics, use MIGs or GKE NEGs — or mix: serverless NEGs for low-traffic regions, MIGs where you need capacity control.
5. URL map, proxy, forwarding rule
gcloud compute url-maps create api-urlmap --default-service=api-backend --global
gcloud compute ssl-certificates create api-cert \
--domains=api.example.com --global
gcloud compute target-https-proxies create api-https-proxy \
--url-map=api-urlmap \
--ssl-certificates=api-cert \
--ssl-policy=tls12-modern \
--global
gcloud compute forwarding-rules create api-fr-https \
--load-balancing-scheme=EXTERNAL_MANAGED \
--address=api-global-ipv4 \
--target-https-proxy=api-https-proxy \
--ports=443 \
--global
Repeat the forwarding rule for the IPv6 address. Add an HTTP→HTTPS redirect URL map on port 80; never leave port 80 serving content.
GKE: standalone zonal NEGs
If your workloads run on GKE in several regions, expose each cluster's pods as standalone NEGs and attach them to the same global backend service. Pods become load balancer endpoints directly — no extra hop through a node port.
apiVersion: v1
kind: Service
metadata:
name: api
namespace: production
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080": {"name": "neg-api-gke-euw1"}}}'
cloud.google.com/backend-config: '{"default": "api-backendconfig"}'
spec:
type: ClusterIP
selector:
app: api
ports:
- name: http
port: 8080
targetPort: 8080
protocol: TCP
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: api-backendconfig
namespace: production
spec:
timeoutSec: 30
connectionDraining:
drainingTimeoutSec: 60
healthCheck:
type: HTTP
requestPath: /readyz
port: 8080
checkIntervalSec: 2
timeoutSec: 1
healthyThreshold: 2
unhealthyThreshold: 2
securityPolicy:
name: api-armor-policy
logging:
enable: true
sampleRate: 1.0
The NEG controller creates one zonal NEG per zone where pods run. Attach them all:
for ZONE in europe-west1-b europe-west1-c europe-west1-d; do
gcloud compute backend-services add-backend api-backend \
--network-endpoint-group=neg-api-gke-euw1 \
--network-endpoint-group-zone="$ZONE" \
--balancing-mode=RATE --max-rate-per-endpoint=100 \
--global
done
Make the readiness probe meaningful, since it drives both Kubernetes and GFE decisions:
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: production
spec:
replicas: 6
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api
containers:
- name: api
image: europe-docker.pkg.dev/my-project/api/api:1.8.3
ports:
- containerPort: 8080
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 2
failureThreshold: 2
lifecycle:
preStop:
exec: { command: ["sleep", "15"] }
resources:
requests: { cpu: "250m", memory: "256Mi" }
limits: { memory: "512Mi" }
The preStop sleep of 15s covers the window between pod termination and NEG endpoint removal — without it you will see 502s on every rollout.
Terraform
The same stack, declaratively. This is the version you actually want in production.
resource "google_compute_global_address" "api" {
name = "api-global-ipv4"
ip_version = "IPV4"
}
resource "google_compute_health_check" "api" {
name = "hc-api-readyz"
check_interval_sec = 2
timeout_sec = 1
healthy_threshold = 2
unhealthy_threshold = 2
http_health_check {
port = 8080
request_path = "/readyz"
}
log_config { enable = true }
}
resource "google_compute_region_network_endpoint_group" "api" {
for_each = toset(["europe-west1", "us-central1", "asia-southeast1"])
name = "neg-api-${each.key}"
region = each.key
network_endpoint_type = "SERVERLESS"
cloud_run { service = "api" }
}
resource "google_compute_backend_service" "api" {
name = "api-backend"
load_balancing_scheme = "EXTERNAL_MANAGED"
protocol = "HTTP"
timeout_sec = 30
security_policy = google_compute_security_policy.api.id
dynamic "backend" {
for_each = google_compute_region_network_endpoint_group.api
content {
group = backend.value.id
}
}
log_config {
enable = true
sample_rate = 1.0
}
}
resource "google_compute_url_map" "api" {
name = "api-urlmap"
default_service = google_compute_backend_service.api.id
}
resource "google_compute_managed_ssl_certificate" "api" {
name = "api-cert"
managed { domains = ["api.example.com"] }
}
resource "google_compute_target_https_proxy" "api" {
name = "api-https-proxy"
url_map = google_compute_url_map.api.id
ssl_certificates = [google_compute_managed_ssl_certificate.api.id]
ssl_policy = google_compute_ssl_policy.modern.id
}
resource "google_compute_ssl_policy" "modern" {
name = "tls12-modern"
profile = "MODERN"
min_tls_version = "TLS_1_2"
}
resource "google_compute_global_forwarding_rule" "api" {
name = "api-fr-https"
load_balancing_scheme = "EXTERNAL_MANAGED"
ip_address = google_compute_global_address.api.address
port_range = "443"
target = google_compute_target_https_proxy.api.id
}
For MIG or GKE NEG backends, add balancing_mode, max_rate_per_instance and capacity_scaler inside the backend block, plus health_checks = [google_compute_health_check.api.id] on the backend service.
Testing geographic routing and failover
Have your application echo the serving region (K_SERVICE / GOOGLE_CLOUD_REGION on Cloud Run, the node label on GKE) in a response header such as X-Served-By.
Measure from VMs in several regions:
for R in europe-west1-b us-central1-a asia-southeast1-a; do
gcloud compute instances create probe-$R --zone=$R \
--machine-type=e2-micro --image-family=debian-12 \
--image-project=debian-cloud --no-address 2>/dev/null
done
gcloud compute ssh probe-europe-west1-b --zone=europe-west1-b --command \
'curl -s -o /dev/null -D - -w "ttfb=%{time_starttransfer}s total=%{time_total}s\n" https://api.example.com/healthz | grep -Ei "x-served-by|ttfb"'
Expected: the European probe reports a European backend with a TTFB under 30 ms; the Asian probe reports asia-southeast1.
Force a failover by blocking health check probes in one region. Google's probes come from 35.191.0.0/16 and 130.211.0.0/22:
gcloud compute firewall-rules create deny-hc-euw1 \
--network=prod-vpc --action=DENY --rules=tcp:8080 --priority=900 \
--source-ranges=35.191.0.0/16,130.211.0.0/22 \
--target-tags=api-euw1
# Watch the backend flip to UNHEALTHY
watch -n 2 'gcloud compute backend-services get-health api-backend --global \
--format="table(status.healthStatus[].instance, status.healthStatus[].healthState)"'
While the rule is active, European clients should transparently receive responses from us-central1 with a higher TTFB. Delete the rule and verify traffic returns. Do this in a staging project first, and script the rollback.
A less invasive drain, for rollouts:
gcloud compute backend-services update-backend api-backend \
--instance-group=api-mig-euw1 --instance-group-region=europe-west1 \
--capacity-scaler=0.0 --global
Security best practices
Put Cloud Armor in front of every backend service. A global anycast IP is reachable by every scanner on the internet within minutes of creation.
gcloud compute security-policies create api-armor-policy \
--description="Baseline WAF + rate limiting"
# OWASP preconfigured rules (sensitivity 1 = low false positives)
gcloud compute security-policies rules create 1000 \
--security-policy=api-armor-policy \
--expression="evaluatePreconfiguredExpr('sqli-v33-stable', ['owasp-crs-v030301-id942251-sqli'])" \
--action=deny-403
gcloud compute security-policies rules create 1100 \
--security-policy=api-armor-policy \
--expression="evaluatePreconfiguredExpr('xss-v33-stable')" \
--action=deny-403
# Per-IP rate limiting with temporary ban
gcloud compute security-policies rules create 2000 \
--security-policy=api-armor-policy \
--action=rate-based-ban \
--rate-limit-threshold-count=600 \
--rate-limit-threshold-interval-sec=60 \
--ban-duration-sec=300 \
--conform-action=allow \
--exceed-action=deny-429 \
--enforce-on-key=IP \
--src-ip-ranges='*'
gcloud compute backend-services update api-backend \
--security-policy=api-armor-policy --global
Always deploy new WAF rules with --preview first, inspect jsonPayload.enforcedSecurityPolicy.outcome in the load balancer logs, then enforce.
Lock down the backends. Instances and GKE nodes must only accept traffic from GFE ranges. Everything else is denied:
gcloud compute firewall-rules create allow-gfe-to-api \
--network=prod-vpc --action=ALLOW --rules=tcp:8080 --priority=1000 \
--source-ranges=35.191.0.0/16,130.211.0.0/22 --target-tags=api
gcloud compute firewall-rules create deny-all-ingress-api \
--network=prod-vpc --action=DENY --rules=all --priority=65000 \
--source-ranges=0.0.0.0/0 --target-tags=api
Backends should have no external IP; use Cloud NAT for egress. For GKE, use private clusters with authorized networks on the control plane.
TLS and transport. Enforce a MODERN SSL policy with TLS_1_2 minimum, enable HSTS at the application layer, and use managed certificates so renewal is not a human task. If your backends require encryption in transit, switch the backend protocol to HTTPS or HTTP2.
Identity. For internal tooling behind the same LB, enable Identity-Aware Proxy instead of IP allowlists. For service-to-service calls into Cloud Run, keep the service private (--no-allow-unauthenticated) and let the serverless NEG handle ingress with --ingress=internal-and-cloud-load-balancing.
Observability as a control. Enable LB logging at 100% during rollouts, export to BigQuery, and alert on backend_latencies p95 per backend_target_name and on request_count filtered by response_code_class=500. A latency regression in one region is the earliest signal of a partial outage.
Key takeaways
- A single global anycast IP removes DNS TTL from your failover path and moves TLS termination within a few milliseconds of the user. That is the bulk of the perceived latency win.
-
One global backend service, many regional backends. The GFE handles proximity routing and overflow; you only decide capacity through
--balancing-mode,--max-rate-per-instanceand--capacity-scaler. -
Health checks are the failover SLA. A 2s interval with a 2-failure threshold, probing a dependency-aware
/readyz, gives sub-10-second drains. Enable health check logging before you need it. - Serverless NEGs trade control for simplicity — no health checks, no capacity scalers. Mix them with MIG or GKE NEG backends where you need deterministic failover behaviour.
-
GKE standalone NEGs route directly to pods; pair them with
BackendConfig, topology spread constraints and apreStopdelay to avoid 502s on rollout. -
Ship the whole thing as Terraform, and keep
capacity-scaleras an operational escape hatch you can flip from a runbook. - Cloud Armor is not optional on an internet-facing anycast IP. Preconfigured OWASP rules in preview mode, rate-based bans, and GFE-only firewall rules are the minimum baseline.
Deploy it, then prove it: measure TTFB from three continents, kill a region's health checks, and watch traffic move. An untested failover is a hypothesis, not an architecture.
Top comments (0)