An API gateway looks like a thin proxy until the night it becomes the outage.
Centralization is the point: one place for TLS termination, authentication, rate limiting, routing, and policy enforcement.
Centralization is also the risk.
Every request shares the same event-loop budget, connection pools, memory, and blast radius.
Microsoft's Gateway Offloading guidance says it plainly: keep the gateway highly available, ensure it never becomes the bottleneck, and don't put business logic inside it.
Canva's public incident report from November 2024 demonstrated what happens when that shared front door saturates: approximately 1.5 million requests per second—around 3× normal peak traffic—combined with a telemetry lock that blocked Netty's event loop until Linux began OOM-killing gateway instances.
This article walks through six production problems API gateways commonly suffer from—and how to prevent them before the next traffic spike.
What an API Gateway Is (and Isn't)
An API gateway is the edge facade sitting in front of backend services.
Typical responsibilities include:
- TLS termination
- Authentication
- Rate limiting
- Routing
- Metrics and tracing
- Header or path transformation
It is not responsible for:
- Business authorization
- UI composition
- Unlimited retries
- Product-specific business rules
Treating the gateway as "just a proxy" hides its failure modes.
Treating it like an application server creates new ones.
1. Shared Blast Radius
Symptom
Everything appears offline even though backend services remain healthy.
Autoscaling launches new gateway instances that immediately fail.
Why it happens
One gateway cluster fronts every public request.
A traffic spike, blocking plugin, or memory leak impacts the entire platform simultaneously.
Microsoft explicitly warns against allowing the gateway to become a single bottleneck.
Canva provides a textbook example.
A delayed CDN asset caused over 270,000 clients to reconnect simultaneously.
Once the asset became available, every client resumed requests at nearly the same moment.
The gateway fleet saturated.
Off-heap memory exploded.
Linux OOM-killed containers faster than autoscaling could replace them.
Recovery only became possible after traffic was blocked at the CDN.
How to fix it
- Keep baseline capacity well above average load
- Load test the gateway as a Tier-0 service
- Add load shedding (HTTP 503) before queues consume the fleet
- Isolate listener pools or gateways by product area
- Practice traffic blocking at the CDN instead of relying only on cluster scaling
Saturation isn't simply "high latency."
It's when the gateway itself becomes the system bottleneck.
2. Broken Timeout Hierarchies
Symptom
Clients receive HTTP 504 while backend requests continue executing.
Connection pools fill.
Latency increases until the entire gateway stalls.
Why it happens
Timeouts are configured independently instead of forming a hierarchy.
If the gateway waits longer than the client, it continues consuming resources after users have already disconnected.
Slow upstreams then exhaust connection pools.
Queues continue growing.
New requests begin failing.
AWS documents this behavior in API Gateway timeout guidance.
Increasing timeout values without redesigning timeout relationships simply keeps abandoned requests alive longer.
Correct timeout ordering
Client timeout
>
Gateway → Backend timeout
>
Per-retry timeout
Recommendations
- Gateway timeouts should always be shorter than client timeouts
- Propagate deadlines downstream
- Cap pending request queues
- Prefer early HTTP 503 over extremely long waits
- Long-running work belongs in asynchronous workflows, not synchronous gateway paths
3. Retry Storms
Symptom
A minor upstream failure becomes a platform-wide outage.
CPU utilization spikes.
Request volume multiplies.
Recovery takes longer than the original incident.
Why it happens
Retries amplify traffic.
AWS Well-Architected explicitly warns against retries without:
- exponential backoff
- jitter
- retry limits
- centralized retry ownership
Multiple retry layers compound.
For example:
Client retries × Gateway retries × Service Mesh retries
A single failed request can become dozens.
Envoy recommends using retry budgets rather than unlimited retry counts.
Recommendations
Retry only:
- connection failures
- transient timeouts
- selected HTTP 5xx
- HTTP 429
Never retry:
- validation errors
- authentication failures
- authorization failures
Also:
- use exponential backoff
- add jitter
- cap retry attempts
- configure retry budgets
- retry in one layer only
4. The God Gateway
Symptom
Every product change requires modifying gateway configuration.
Gateway plugins begin parsing JSON.
Business rules appear inside Lua or JavaScript extensions.
Latency slowly increases.
Why it happens
Cross-cutting concerns gradually become business logic.
Microsoft's Gateway Offloading documentation is unambiguous:
Business logic should never be offloaded to the gateway.
Good gateway responsibilities:
- TLS
- authentication
- routing
- API versioning
- correlation IDs
Poor gateway responsibilities:
- entitlement rules
- product workflows
- screen composition
- response shaping
- domain validation
Rule of thumb
If a Product Manager is approving gateway changes more often than an SRE, the gateway is doing too much.
5. Blind Observability
Symptom
Monitoring shows "API latency."
Nobody knows whether the delay comes from:
- TLS
- authentication
- plugins
- gateway overhead
- backend services
Or worse:
Everything works until load increases.
Then a seemingly harmless metrics library blocks the event loop.
Why it happens
Observability itself becomes part of the request path.
Canva's postmortem identified a telemetry lock that reduced Netty throughput precisely when the traffic spike arrived.
Recommendations
Measure separately:
- Gateway processing time
- Upstream latency
- Pending queue depth
- Active connections
- Retry overflow
- Circuit breaker events
Prefer:
- local JWT verification
- cached JWKS
- asynchronous logging
- non-blocking plugins
If you can't answer
"Is the gateway slow, or is Payments slow?"
within one minute, observability needs improvement.
6. Weak Edge Hardening
Symptom
Rate limits work inconsistently.
Backends remain publicly accessible.
Administrative APIs are internet-exposed.
Why it happens
Three common mistakes:
Per-node rate limiting
Each gateway replica enforces its own counters.
Four replicas effectively quadruple the intended limit.
Backend bypass
Clients can reach backend services directly.
Gateway security policies become optional.
Exposed control plane
Administrative APIs remain publicly reachable.
For products like Kong, exposing the Admin API effectively grants full control over the gateway.
Recommendations
- Keep backend services private
- Enforce authentication at the gateway
- Use shared rate limiting for cluster-wide quotas
- Protect administrative APIs with private networking and RBAC
- Continuously scan for routes bypassing the gateway
How These Problems Reinforce Each Other
Traffic spike
│
▼
Gateway saturation
│
├── Timeout inversion
│ │
│ ▼
│ Pool exhaustion
│
├── Retry storms
│ │
│ ▼
│ Traffic amplification
│
├── Business logic
│ │
│ ▼
│ Reduced capacity
│
▼
Observability failures
│
▼
Security and control-plane gaps
Fixing only retries while telemetry still blocks the event loop doesn't solve the outage.
These issues reinforce one another.
FAQ
Is an API Gateway always a single point of failure?
Not necessarily.
Multiple gateway instances remove a single process as a SPOF.
However, shared configuration, shared telemetry, or synchronized traffic spikes can still create a shared failure domain.
Should the gateway perform retries?
Sometimes.
Only for idempotent operations and only within a defined retry budget.
Avoid retries simultaneously in the client, gateway, and service mesh.
Gateway vs BFF vs Service Mesh
| Component | Responsibility |
|---|---|
| API Gateway | North–south traffic, TLS, authentication, routing |
| BFF | Product-specific aggregation and UI contracts |
| Service Mesh | East–west resilience, mTLS, retries, service communication |
How do I know my timeout hierarchy is wrong?
Typical indicators include:
- backend work continues after clients disconnect
- growing pending queues
- frequent HTTP 504 responses
- connection pool exhaustion
Instrument client timeout, gateway timeout, and upstream duration in the same trace.
Final Thoughts
Production gateways rarely fail because routing is difficult.
They fail because every concern eventually accumulates at the edge:
- shared saturation
- inverted timeout budgets
- retry amplification
- business logic
- opaque observability
- weak control-plane security
Each problem has well-established architectural guidance.
The goal isn't building a smarter gateway.
It's building one that is boring, observable, predictable, and capable of failing in small pieces instead of taking the entire platform down.
References
- Canva incident report: API Gateway outage — Canva Engineering (Dec 2024)
- Gateway Offloading pattern — Microsoft Azure Architecture Center
- API gateways in microservices — Microsoft Azure Architecture Center
- Gateway Routing pattern — Microsoft Azure Architecture Center
- Circuit breaking — Envoy Proxy documentation
- Outlier detection — Envoy Proxy documentation
- Circuit breakers (proto) — retry_budget — Envoy Proxy API
- REL05-BP03 Control and limit retry calls — AWS Well-Architected
- REL05-BP05 Set client timeouts — AWS Well-Architected
- Troubleshoot API Gateway HTTP 504 timeout errors — AWS re:Post
- Amazon API Gateway integration timeout limit increase beyond 29 seconds — AWS News (Jun 2024)
- Secure the Admin API — Kong Gateway documentation
- Kong API Gateway Misconfigurations: An API Gateway Security Case Study — Trend Micro
- Timeouts, retries, and backoff with jitter — Amazon Builders’ Library
- Exponential Backoff And Jitter — AWS Architecture Blog
- 6 problems your API gateway is already suffering from - Personal Portfolio Article
Top comments (0)