With two services in a system, letting clients call each of them directly feels completely reasonable, there's not much to get wrong about a client hitting the users service here and the orders service there. Then the system grows. A payments service shows up, then search, then several different client applications, each needing authentication, staying under a rate limit, targeting the right API version, and all of it ideally traceable when something goes wrong. Instances of every one of those services are also constantly scaling up and down, so their actual addresses keep changing underneath everything.
At that point, "which server should receive this request" stops being the real question. The real question becomes which service should receive this request, whether the caller is even allowed to make it, whether they're within their quota, and how you're supposed to observe the whole journey once it's scattered across half a dozen services. That's the specific problem an API gateway exists to solve. I put together a visual walkthrough of the request flow through a gateway, routing, auth, rate limiting, the whole pipeline, on SeeItFlow, if you'd like to see it step by step rather than read it linearly.
The basic shift: one door instead of many
Instead of exposing every backend service directly to clients:
Client ──→ Users Service
──→ Orders Service
──→ Payments Service
a gateway sits in front of all of them:
┌──→ Users Service
Client ──→ API Gateway ──┼──→ Orders Service
└──→ Payments Service
The client now sees exactly one API endpoint. The gateway is the thing that actually knows the internal topology, so a request like GET /orders/42 gets routed internally to wherever the orders service actually lives, without the client ever needing to know or care.
Routing alone would already be a reasonable thing to centralize, but it's really just the entry point into a bigger idea. Because literally every external request has to pass through this one layer, it becomes the natural place to put anything that would otherwise need to be reimplemented, slightly differently, inside every single service: authentication and authorization, rate limiting, API versioning, service discovery, request aggregation, tracing and metrics. Without a gateway, each service tends to end up with its own slightly-off version of the same edge logic, and "slightly off" is exactly where inconsistent security behavior creeps in.
This isn't the same job as a load balancer
These two get confused constantly because they both physically sit between clients and backend systems, but they're answering genuinely different questions. A load balancer asks which replica of a given service should handle this request:
┌──→ Orders #1
Client → LB ──┼──→ Orders #2
└──→ Orders #3
Every box there is running the same service. An API gateway asks a different question entirely, which service should handle this, and what policy needs to apply before it even gets there:
/users/* ──→ Users
Client → API Gateway ─── /orders/* ──→ Orders
/payments/*──→ Payments
In practice these two usually work together rather than substituting for each other:
Clients
│
▼
Load Balancer
│
▼
API Gateway Cluster
│
├──→ Users
├──→ Orders
└──→ Payments
The load balancer's job is spreading traffic across a pool of interchangeable gateway instances. The gateways then do the actual application-aware routing and policy enforcement on top of that.
What actually happens to one request
Thinking of the gateway as a pipeline, rather than a single box that "does gateway stuff," makes its behavior much easier to reason about. A request typically moves through something like this sequence: terminate TLS and parse the HTTP request, assign or propagate a trace ID, match the route, authenticate the caller, authorize the caller, apply rate limits, transform or aggregate the request or response if needed, discover a currently-healthy backend instance, forward the request, and finally record metrics before returning the response.
The order here isn't arbitrary. Authentication has to happen before authorization, since you can't decide what someone's allowed to do before you know who they are. Rate limiting needs to happen before any expensive downstream work, otherwise you're doing the expensive work first and only then deciding it shouldn't have been allowed. And the trace ID needs to be assigned near the very beginning, so every later stage in the pipeline, and every downstream service the request eventually touches, can be tied back to the same originating request. This ordering is exactly why an API gateway is better understood as a structured, sequential pipeline rather than a loose bag of unrelated features that happen to live in one process.
Why authentication sits early in that pipeline
Say a request arrives carrying a JWT. The gateway can verify its signature, check its expiry, confirm the issuer and audience, all before the request ever reaches a backend service:
Client
│
│ Bearer token
▼
Gateway
│
├── invalid token ──→ 401
│
└── valid token
│
▼
Service
Authentication is answering "who are you." Authorization is a separate question, "are you allowed to do this specific thing." A logged-in user attempting an admin-only action and someone presenting an outright invalid token are genuinely different situations, one gets rejected before their identity is even established, the other gets identified successfully and then denied for a completely different reason. Centralizing both checks in the gateway also means individual backend services don't each need their own slightly divergent implementation of the same edge policy, which is exactly the kind of duplication that quietly drifts out of sync over time.
Rate limiting gets genuinely interesting once you scale the gateway itself
A token bucket is a common approach: each client gets a bucket that refills at some configured rate, requests consume tokens from it, and once the bucket's empty, the gateway responds with 429 Too Many Requests. Straightforward, right up until there's more than one gateway instance.
Picture a limit of 100 requests per minute enforced across three separate gateway instances:
┌── Gateway 1 → counter = 100
Client ────┼── Gateway 2 → counter = 100
└── Gateway 3 → counter = 100
If each gateway keeps its own independent counter, a client hitting all three effectively gets 300 requests per minute instead of 100, three separate quotas instead of one shared one. The fix is coordinating that rate-limit state somewhere all the gateway instances can see it, commonly Redis:
Gateway 1 ─┐
Gateway 2 ─┼──→ Shared rate-limit state
Gateway 3 ─┘
This is one of those details that looks completely trivial on a whiteboard, "just add a rate limiter", and turns out to be considerably more interesting the moment you actually run more than one instance of the thing enforcing it.
Services don't sit at fixed addresses
Hardcoding something like orders-service = 10.0.4.17 works for approximately as long as nothing autoscales. Instances come up, go down, fail health checks, and generally move around constantly in any system that scales dynamically. Instead, the gateway can rely on service discovery:
Gateway
│
▼
Service Registry
│
├── orders-1 ✓
├── orders-2 ✓
└── orders-3 ✗
The registry tells the gateway which instances are currently actually healthy, which means a brand-new instance can start receiving traffic the moment it's registered, with no client change needed and no redeploy of the gateway itself carrying a new hardcoded address.
Aggregation trades client round trips for gateway complexity
Say a mobile home screen needs a user's profile, their recent orders, and their current payment status, three genuinely separate pieces of data from three separate services. The client could fire off three separate requests. Or the gateway could expose one endpoint:
GET /home
and fan that single request out internally:
┌──→ Users
Client → Gateway ─┼──→ Orders
└──→ Payments
running those backend calls in parallel and combining the results into one response. That reduces the number of round trips the client makes, which matters more than it sounds like it should on higher-latency mobile networks. The trade-off is real, though: push too much aggregation and response-shaping logic into the gateway and it slowly turns into a place genuinely full of business logic, which is exactly the kind of responsibility a shared, cross-cutting layer shouldn't be accumulating. For anything beyond simple fan-out, a dedicated Backend-for-Frontend tends to be a cleaner home for that complexity than continually teaching the shared gateway more application-specific behavior.
The gateway can't be a single point of failure
Putting one single gateway process in front of everything creates an obvious problem:
Clients → ONE Gateway → Everything
💥
If that one process dies, the entire API disappears at once, every service behind it becomes unreachable regardless of how healthy they individually are. A real production setup looks more like:
┌── Gateway 1
Clients → LB ───────┼── Gateway 2
└── Gateway 3
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Users Orders Payments
Gateway instances should generally be stateless and fully interchangeable with each other. Routes, keys, and policy configuration get pulled from a shared control plane rather than baked into any one instance, and any runtime state that genuinely needs to be coordinated across instances, distributed rate limits being the obvious example from earlier, lives outside the individual gateway processes entirely. With that in place, any single gateway instance can disappear and the load balancer in front of the cluster simply routes around it, no different in principle from losing one replica of any other stateless service.
The gateway is also a natural place to see the whole request
Because every external request enters through this one layer, it's a genuinely good place to originate distributed tracing rather than trying to stitch it together afterward from scattered logs:
Client
│
▼
Gateway trace=8f2a
│
├──→ Users trace=8f2a
└──→ Orders trace=8f2a
The same trace ID follows the request through every downstream call it triggers, so instead of separately eyeballing unrelated log lines from several different services and trying to guess which ones belong to the same user action, you can reconstruct the entire request as one coherent story. The gateway is also well positioned to expose useful per-route metrics directly, request rate, error rate, latency percentiles, broken down by route rather than averaged across the whole system.
None of this is free
Every request now takes an extra hop through the gateway before it reaches the service that actually handles it. That's added latency, one more system that has to be operated and kept healthy, and one more thing sitting in the critical path that the whole API now depends on. For a small system with a couple of services, calling them directly may genuinely be simpler and entirely sufficient, there's no rule saying every architecture needs a gateway on principle.
The gateway earns its cost as the number of services, clients, and shared policies grows, specifically once those things start needing to be enforced consistently across many services rather than once. The more useful framing isn't "should every microservice architecture have an API gateway." It's whether the cost of duplicating routing, authentication, rate limiting, versioning, and observability across every individual service has become more expensive than the cost of operating one shared gateway layer that does all of it consistently.
The mental model worth keeping
Three separate layers are answering three separate questions here, and keeping them separate is what keeps the whole architecture reasoning-friendly rather than tangled. The load balancer is asking which replica should handle this. The API gateway is asking which service should handle this, and whether the request is even allowed to happen. And the individual service is asking what business operation should actually occur now that it's arrived. Once those three questions stop being conflated into one blurry "handle the request" step, a surprising amount of what makes distributed systems hard to reason about gets a lot more tractable.
References
This post covers the core request pipeline and production concerns of an API gateway. The full visual walkthrough on SeeItFlow covers the fundamentals in more depth, there's a dedicated production engineering guide covering gateway clustering, distributed rate limiting, and service discovery in production, and an engineering insights guide focused on failure modes and debugging a gateway layer under real traffic.
Top comments (0)