Amitesh0512

Posted on Oct 22 Originally published at amiteshsurwar.com

Designing High-Availability Load Balancing for API Gateways at Scale

#apigateway #microservices #scalability #observability

Quick Answer

Learn how to build high-availability load balancing for API gateways that survives regional outages, scales horizontally, and keeps latency low for microservices in the US and India.

In practice, the choice between a single global LB vs regional ones hinges on your latency budget and failure domain. If you can tolerate 50‑ms extra hops, a single Azure Front Door with health probes may be simpler; if you need <1 ms, a dual‑layer approach is warranted. When I’d choose a single edge LB, I’m willing to accept slower failover but lower operational overhead. When I’d choose regional LBs, I’m trading a slightly higher cost for instant regional failover.

Single Point of Failure in API Gateways

In a global microservice landsCAPe, the API gateway is the single source of truth for routing, authentication, and policy enforcement. When a gateway node goes down, the effect ripples: downstream services receive fewer requests, rate limits are bypassed, and clients see spikes in latency. Traditional round‑robin DNS or a single edge load balancer is a single point of failure. The question is not *if* a failure will happen, but *when* and *how quickly* the system can recover while keeping the user experience intact.

A key design decision is whether to treat the gateway as a stateless proxy or embed stateful auth/ratelimiting. Embedding state trades scalability for simplicity; pulling state out to Redis or a distributed cache keeps the gateway lean but introduces network hops. In a production environment, I would avoid embedding stateful rate limits in the gateway because it quickly becomes a bottleneck when you scale.

Real‑World Example

Consider a SaaS platform that serves 1.2 M active users across North America, Europe, and Asia. The gateway runs as a Kubernetes deployment behind Azure Front Door. Each region has 3 pods behind a regional NLB. A recent incident showed that an AZ outage caused all traffic to the NA region to bounce back to the edge, which then re‑directed to the EU region. The edge’s latency‑based routing kept sending traffic to EU until the NA DNS record expired, leading to a 30‑second thundering‑herd and a 15 % spike in 5xx errors. The incident report highlighted that the gateway’s health checks were only probing /healthz, missing the fact that downstream auth service was throttling.

The incident underscored that health checks must model the full request path; probing only /healthz missed the downstream auth throttling that caused the 15 % spike. In a real deployment, I would instrument synthetic traffic that mirrors real traffic patterns and includes downstream dependencies.

Trade‑Offs

Latency vs. Consistency: An active‑active cluster spreads traffic globally but requires a shared configuration store (e.g., Consul KV). The cost of eventual consistency can lead to stale routes.
Operational Overhead vs. Resilience: Multi‑region clustering demands separate databases, cross‑region replication, and more complex observability. The upside is instant failover; the downside is higher ops cost.
Weight vs. Dynamism: Static weights simplify deployment but ignore real‑time pod health. Dynamic weights based on CPU or request latency add complexity but improve throughput.
Statelessness vs. Session Affinity: Stateless gateways scale horizontally but lose session affinity. Using token‑based routing (consistent hashing on JWT sub) preserves session semantics without sticky cookies.

The central tension is between immediate failover and consistent routing. A shared config store gives you global consistency but can become a bottleneck; a per‑region config gives you faster failover but risks split‑brain routing. When I’d choose a shared KV, I’m prioritizing consistency; when I’d choose per‑region config, I’m prioritizing latency and resilience.

Latency Budgets, Consistency, and Algorithm Selection

Define latency budget per region. If <1 ms is required, lean toward active‑active with shared config.
Assess data plane consistency needs. If routing tables change rarely, a passive sync via ConfigMap is enough; if they change often, use a distributed KV.
Determine failure domain granularity. For large AZs, use regional LBs; for smaller setups, a single edge LB with health probes may suffice.
Pick a load‑balancing algorithm that matches traffic profile: least‑connections for streaming, consistent hashing for auth‑heavy endpoints, weighted round‑robin for premium traffic.
Implement proactive health checks that include downstream dependencies. Combine passive telemetry with active synthetic requests.
Define graceful shutdown hooks (SIGTERM handling, graceful drain) to avoid 502 bursts during pod evictions.
Set up observability pipelines (OpenTelemetry + Jaeger + Prometheus) that capture gateway.instance.id, upstream.service.name, status codes, and error types.

When you decide, ask: how many failure domains can you tolerate? If you can only afford one AZ outage, a regional LB with health probes is enough; if you need cross‑region resilience, you must pay for cross‑region replication.

When This Fails in Production

Health checks only probe /healthz while downstream queues are filling, leading to a false healthy state.
Edge LB caches stale health status due to long DNS TTLs, delaying failover.
Dynamic scaling (HPA) triggers on CPU but not latency, causing pods to be added after a spike has already caused request backpressure.
Consistent hashing on JWT sub is disabled during a config reload, breaking session affinity and causing cache misses.
Graceful shutdown is not implemented; pods terminate abruptly, producing 502s during drain.

Failover delays are often caused by DNS caching or health probe lag. A common pitfall is setting a high DNS TTL; keep it low if you need rapid recovery. When I’d choose a 30‑second TTL, I’m willing to trade a few extra DNS queries for faster failover.

Common Mistakes Engineers Make

Assuming a single edge LB is sufficient; neglecting a regional LB creates a second single point of failure.
Hard‑coding weights; ignoring real‑time pod health leads to over‑loading some nodes.
Storing rate‑limit counters inside the gateway; scaling becomes a race condition.
Using passive health checks alone; missing slow‑degrading downstream services.
Over‑reliance on default HPA metrics; missing custom latency or request‑rate metrics.

Avoid embedding rate limits in the gateway; that turns the gateway into a stateful bottleneck. Instead, use a distributed rate‑limiter like Redis or a dedicated service. I’d also avoid a monolithic gateway that does auth, routing, and rate‑limiting in one container; separate concerns into a lightweight API gateway + sidecar for policy enforcement.

Better Approach Based on Experience

In a production environment where uptime is non‑negotiable, I would:

Deploy a dual‑layer LB: Azure Front Door (global) + regional NLB. Front Door performs latency‑based routing; NLB distributes to a stateless gateway cluster.
Use Consistent Hashing on JWT sub for auth endpoints, falling back to least‑connections for generic traffic.
Implement hybrid health checks: passive metrics trigger an active synthetic request that walks the full request path.
Expose a custom HPA metric for request_latency_ms and request_rate via OpenTelemetry.
Run graceful drain scripts that close new connections after a SIGTERM and wait for in‑flight requests to finish.
Externalize all state: rate limits in Redis, config in Consul KV, and use sidecar Envoy for outlier detection.
Set DNS TTL to 30–60 s and configure Front Door health probes to 1 s intervals.

When I’d choose sidecar Envoy, I’m prioritizing fine‑grained outlier detection and traffic shaping; when I’d choose a simple HTTP proxy, I’m reducing complexity and memory footprint.

Aspect	Single Global LB (Azure Front Door)	Regional LBs (one per region)
Latency Impact	≈50 ms extra hops for cross‑regional traffic	No inter‑regional hops; local traffic stays within region
Failure Domain Resilience	Single point for health checks; regional outages propagate to global LB but mitigated by probes	Isolates outages to region; requires cross‑region coordination for failover
Operational Complexity	Simple: one LB, single configuration, fewer health probes	Higher: multiple LBs, separate health probes, coordination logic
Health Probe Strategy	Global set of health probes covers all regions	Region‑specific probes; allows fine‑grained health status

Performance Considerations

Gateway instance memory footprint: keep sidecar Envoy stateless; container size < 200 MiB to reduce startup time.
Connection multiplexing: enable HTTP/2 and keep‑alive to reduce TLS overhead.
Batching of downstream calls: use gRPC with request aggregation when possible.
Cache validation headers aggressively to avoid unnecessary downstream roundtrips.

When I’d enable HTTP/2, I’m targeting a 10 % reduction in per‑request latency for high‑volume endpoints. If my traffic is mostly short, single‑request, I may opt for HTTP/1.1 to keep the gateway lighter.

Scaling Notes

Start with 3 replicas per region; scale up to 12 replicas when latency > 200 ms or request rate > 5k/s.
Use cluster autoscaler to add nodes when CPU > 70% for > 2 min; adjust thresholds after load testing.
For multi‑region deployments, keep a minimum of 4 pods per region to avoid a single pod becoming a bottleneck.
Monitor pod_cpu_usage and pod_memory_usage to catch memory leaks before they cause OOM kills.

In a multi‑region cluster, use pod anti‑affinity to avoid all pods landing in the same AZ; otherwise, an AZ outage kills the entire region.

What to Ship

Deploy at least two API gateway instances in separate availability zones and register them with a global load balancer that uses a least‑connections algorithm tuned to your defined latency budget.
Expose a health‑check endpoint that reports the average request latency over the last minute; fail the instance if latency exceeds the threshold or if the error rate exceeds the acceptable limit.
Implement consistent‑hash routing to downstream services so that a single gateway failure causes minimal cache misses; fall back to round‑robin when the hash ring is empty.
Add a request‑level circuit breaker that opens after three consecutive 5xx responses from a backend and automatically redirects traffic to a backup backend.
Store routing rules in a distributed configuration store (e.g., etcd) and configure gateways to reload the rules on change without a restart.
Configure a secondary DNS failover that switches traffic to a healthy region if the primary zone’s gateway health‑check fails for more than 30 seconds.

Conclusion

High‑availability load balancing for API gateways is not a set‑and‑forget feature. It requires a layered approach, dynamic health checks, and careful trade‑offs between latency, consistency, and operational cost. By treating the gateway as a stateless, horizontally scalable component with a robust observability stack, you can achieve sub‑second failover, predictable scaling, and a resilient user experience.

Remember that high‑availability is a moving target; continuously validate that your health checks, observability, and failover policies still match your traffic profile. When I’d choose to revisit the health probe strategy, I do it after every major traffic shift or infrastructure change.

DEV Community