Part 1 — Why we built an Auth Gateway instead of putting auth in every service

If you've been on a platform team long enough, you've probably watched this slow-motion failure:

You ship an auth library. Three services adopt it. Six months later, two of them are still on v1.0, one forked it to add a custom claim, and a fourth service rolled its own because the library "didn't fit their use case." A CVE drops. Now you're hunting through repos to find every place that decodes a JWT.

We've been running a multi-tenant platform on Kubernetes for a while, and we kept ending up there. So a couple of years ago we made a call: stop trying to protect every service and start making the decision once — at the edge.

This is the first post in a 10-part series about that gateway. The actual gateway is two pieces:

• NGINX, packaged as a Helm chart, that fronts every authenticated route.
• Auth Service, a small Go service that exposes a single POST /auth endpoint. NGINX hits it as a subrequest on every protected request.

I'll skip the marketing in this series. I'll show real code, real config, and the parts that hurt.

The decision: three things, three different homes

The mistake we kept making was treating "auth" as one thing. It's three:

1. Authentication — who is this caller? (Authorization: Bearer ..., signature, expiry, revocation)
2. Authorization — are they allowed to call this endpoint? (role, access level, tenant)
3. Routing — where does this request go? (multi-tenant DNS, single-tenant vs multi-tenant upstream, header propagation)

If you put all three inside every service, every service ends up with its own opinion. So we split:

• NGINX owns routing and fail-closed posture. It already sits in the request path. It's the cheapest place on earth to say "no."
• Auth Service owns the decision — token validity, endpoint classification, access level check.
• Upstream services own the business logic and trust the identity headers NGINX injects.

What this looks like on a single request

Here's the happy path, as it actually runs in production. A user calls GET /user-management/users/me with a bearer token and a tenant header.

A few things worth noticing:

• The subrequest is a POST, not a GET. NGINX's auth_request always uses proxy_method POST in our chart. The Auth Service doesn't need a body — it decides from X-Original-URI, X-Original-Method, X-Tenant-ID, and the bearer token.
• The Auth Service responds with identity headers. NGINX pulls them out with auth_request_set and re-injects them into the upstream proxy call. Upstream services never look at the JWT — they trust X-Identity-ID because they trust NGINX.
• X-Request-ID is propagated end-to-end. Every log line on every hop carries the same id. (More on that in Chapter 9.)

The deny path is where centralization actually pays off

Two things that we got for free the moment we centralized:

1. Identical error envelopes. Whether the failure is "no tenant header," "expired token," "wrong access level," or "the Auth Service itself is down," the client sees the same shape: {"source":"auth","message":"...","code":"...","error":"..."}. We didn't have to coordinate this across 30 services.
2. Upstream services never run on a bad token. They aren't even invoked. That alone fixed a long tail of "service X returned 200 with weird data because the token didn't validate but the framework didn't care."

The corresponding NGINX config is small and worth showing in full. Trimmed to the parts that matter:

# auth.conf — the subrequest endpoint
location = /auth {
  internal;                                # only callable from auth_request

  proxy_pass_request_body off;             # auth doesn't need the body
  proxy_method POST;
  proxy_set_header Content-Length "";

  proxy_set_header X-Original-URI    $request_uri;
  proxy_set_header X-Original-Method $request_method;
  proxy_set_header X-Original-Host   $host;
  proxy_set_header X-Request-ID      $request_id;
  proxy_set_header X-Tenant-ID       $tenant_id;
  proxy_pass_request_headers on;            # forward Authorization etc.

  proxy_connect_timeout 5s;
  proxy_read_timeout    10s;
  proxy_next_upstream   error timeout http_500 http_502 http_503 http_504;
  proxy_next_upstream_tries 2;
  proxy_next_upstream_timeout 15s;

  error_page 502 503 504 = @auth_unavailable;   # fail-closed: 503 to the client

  proxy_pass http://auth_service/auth;
}

And here's how a regular service location uses it:

location /user-management/ {
  set $product "incore";
  set $microservice "user-service";

  auth_request     /auth;
  auth_request_set $identity_id        $upstream_http_x_identity_id;
  auth_request_set $identity_type      $upstream_http_x_identity_type;
  auth_request_set $session_id         $upstream_http_x_session_id;
  auth_request_set $auth_error_message $upstream_http_x_auth_error_message;
  auth_request_set $auth_error_code    $upstream_http_x_auth_error_code;

  error_page 401 = @unauthorized;
  error_page 403 = @forbidden;
  error_page 502 503 504 = @upstream_unavailable;

  proxy_set_header X-Identity-ID  $identity_id;
  proxy_set_header X-Tenant-ID    $tenant_id;
  proxy_set_header X-Request-ID   $request_id;

  rewrite ^/user-management/(.*)$ /$1 break;
  proxy_pass http://user-service;
}

That's roughly 25 lines per route, generated by a Helm range over the services dictionary. Adding a new service is a values change — no NGINX expert required.

Why NGINX, specifically

We didn't pick NGINX because of opinions. We picked it because of one directive: auth_request.

auth_request lets you tell NGINX: before you proxy the main request, fire a subrequest to this internal location. If the subrequest returns 200, continue. If it returns 401 or 403, stop and run my error handler. If it returns 5xx, run my "auth is down" error handler.

That sounds boring. It's not. It means:

• Your upstream services don't see unauthenticated traffic at all.
• You can change auth logic by deploying one service. No client SDK update, no library bump in 30 repos.
• You get a single observable choke-point. auth_request_time_ms is one log field. We graph it. We page on it.
• You can implement fail-closed by default with one line: error_page 502 503 504 = @auth_unavailable;. If the Auth Service is unhealthy, NGINX returns 503 to the client instead of letting the request through. We pay this cost on purpose. Allowing traffic through a broken auth check is how data leaks happen.

We'll dissect auth_request in Chapter 2.

Where the Auth Service fits

The Auth Service is intentionally small. It does five things:

1. Reads a few request headers.
2. Resolves the tenant.
3. Matches the request path to an endpoint metadata record (we call this the trie — Chapter 4).
4. Classifies the endpoint as OPEN, AUTHENTICATED, or ACCESS_CONTROLLED and runs the right validation pipeline.
5. Emits exactly one structured AUTH_DECISION log line with the timing, identity, decision reason, and outcome.

It does not store sessions. It does not mint tokens (a separate service does). It does not know about your business logic. It's a policy decision point — the thing whose only job is to answer "yes or no, and why."

Here's the controller, paraphrased:

func (c *AuthController) Auth(ctx *gin.Context) {
    log := helpers.GetAuthDecisionLog(ctx)

    log.URI    = ctx.GetHeader("X-Original-URI")
    log.Method = ctx.GetHeader("X-Original-Method")
    log.TenantID = ctx.GetHeader("X-Tenant-ID")

    slug, trieKey, perms, trieExists, found := c.routeResolver.
        ResolveEndpointWithMetrics(ctx, log.URI, log.Method)

    if !trieExists { /* 503: trie not initialized */ }
    if !found      { /* 404: no such API found  */ }

    switch endpointType(perms) {
    case OPEN:
        log.DecisionReason = ReasonOpenEndpoint
        log.Outcome = "allow"
        ctx.Status(http.StatusOK)
    case AUTHENTICATED:
        c.runAuthN(ctx, log)
    case ACCESS_CONTROLLED:
        c.runAuthN(ctx, log)
        c.runAuthZ(ctx, log, perms)
    }
}

Five branches. That's the entire shape of the gateway. The next nine posts are about what's inside each branch and what we learned operating it at ~50k RPS.

What centralizing actually costs

I'd be lying if I said this was free. Three real costs:

One extra hop on the hot path. Every authenticated request now does an in-cluster RPC to the Auth Service before it goes anywhere. We make this cheap with caching (Chapter 8) and with Ristretto-backed JWT verification, but the hop is still there. Median auth_request_time_ms is in the low single digits in our environment, but it's a budget you have to keep.

The Auth Service has to be HA. When it's down, everything is 503. We chose fail-closed on purpose — a permissive default would mean unauthenticated traffic could hit business services during an outage — but it raises the bar on availability. We run it with an HPA (2–10 replicas, 75% CPU/mem), keep alive 64 connections per worker, and gate readiness on the trie being loaded. Even with that, the Auth Service is the single most carefully operated thing on the platform.

auth_request is not cached. This surprises people. NGINX does not cache auth subrequest responses by default, and the obvious caching workarounds (caching by Authorization header) are dangerous in a multi-tenant world. Every protected request hits the auth pod. So everything inside the auth pod has to be fast. That constraint is what shaped the entire internal design — and is why Chapters 7 and 8 spend a lot of time on caches that live inside the auth process, not in NGINX.

Before vs after, at a glance

flowchart LR
    subgraph Before["Before — auth in every service"]
      C1[Client] --> S1[Service A<br/>auth lib v1.2] & S2[Service B<br/>auth lib v1.0] & S3[Service C<br/>custom auth]
    end
    subgraph After["After — Auth Gateway"]
      C2[Client] --> NX[NGINX<br/>auth_request] --> AU[Auth Service]
      NX --> SA[Service A] & SB[Service B] & SC[Service C]
    end

What's coming

This series moves from primitive to production-grade:

• Chapter 2 — auth_request in depth. Subrequest lifecycle, auth_request_set, named error_page locations.
• Chapter 3 — inside the Auth Service. JWT validation, the per-tenant RSA key cache, the decision-reason model.
• Chapter 4 — endpoint classification (OPEN / AUTHENTICATED / ACCESS_CONTROLLED) and the trie that drives it.
• Chapter 5 — multi-tenant routing. SNI server blocks, X-Tenant-ID vs X-Tenant-Host, MT vs ST upstreams, and why we 400 on no tenant.
• Chapter 6 — authorization at scale. Role → access level → endpoint, and the bitmap fast path that replaced string-set matching.
• Chapter 7 — token revocation without killing performance. Redis ZSET + Stream + local cache, the race-condition fixes, and service-account rotation.
• Chapter 8 — every cache in the hot path and how each one is invalidated.
• Chapter 9 — operating the gateway. The AUTH_DECISION log, OpenTelemetry, health probes, and degraded-mode alerts.
• Chapter 10 — what we'd build differently on day one.

Chapter 2 is up next: auth_request is a 12-character directive that quietly does most of the work in this post. I want to show you exactly why it's the right primitive — and what its sharp edges are.

If you're working on something similar and want to compare notes, drop a comment. We made plenty of mistakes; happy to share which ones bit hardest.