Part 5 — Multi-tenant auth and routing in Kubernetes

In the first four chapters of this series I've talked about what the Auth Gateway decides. This chapter is about who it decides for.

We run a multi-tenant platform. Every request, on every endpoint, belongs to one tenant. Get tenant resolution wrong and you don't have a security incident — you have a cross-tenant data leak incident, which is a category of bad you don't recover from.

This chapter is the boring, careful, paranoid story of how NGINX and the Auth Service cooperate to never let a request through without a clear tenant identity.

The two questions

Every multi-tenant request raises two questions:

1. Which tenant is this for? (resolution)
2. Where does the request go for that tenant? (routing)

We answer #1 at the NGINX layer, before auth. We answer #2 partly at NGINX (path-based routing) and partly inside the upstream service (tenant-scoped queries). The Auth Service sits between them: it makes sure the token's tenant matches the request's tenant before either service sees the request.

Resolution: two valid inputs, one explicit failure mode

We accept two ways to identify a tenant:

• X-Tenant-ID header. Explicit. Used by service-to-service calls and SDKs that know who they're for.
• Host header (mapped via X-Tenant-Host). Implicit. Used by per-tenant DNS like tenant1.example.com.

We do not accept a third way: a default tenant. There is no fallback. If both inputs are missing or unknown, NGINX returns 400 before the Auth Service is even called.

Why so strict? Because a default tenant is the most expensive bug you can ship. Every "wait why is data showing up in tenant X?" post-mortem starts the same way: somebody added a fallback "for convenience" and somebody else's request hit it without a tenant header.

We removed our default tenant on day 90 and have never looked back.

Per-tenant SNI server blocks

For tenants with their own DNS, NGINX uses server blocks to short-circuit resolution. The Helm chart templates one server per tenant from global.tenants:

{{- range .Values.global.tenants }}
server {
  listen {{ $.Values.containers.containerPort }};
  server_name {{ .tenant_dns }};

  set $tenant_id        {{ .tenant_id }};
  set $tenant_namespace {{ .tenant_namespace }};

  include /etc/nginx/auth.conf;
  include /etc/nginx/locations.conf;
  include /etc/nginx/custom_error_locations.conf;
}
{{- end }}

A request to tenant1.example.com matches server_name tenant1.example.com, lands in this block, and $tenant_id is already set before any other directive runs. There is no header parsing, no map lookup, no opportunity for ambiguity. Tenant identity is pinned at SNI time.

This is also nice for TLS: the per-tenant ingress can attach per-tenant certificates if you want them, and the SNI selection happens before any HTTP processing.

The default server block: header-based fallback

Not every tenant has its own DNS. Many service-to-service calls hit a shared in-cluster ingress with X-Tenant-ID set explicitly. For those, the default server block handles resolution:

server {
  listen {{ .Values.containers.containerPort }};
  server_name _;                            # match anything not matched above

  set $tenant_id "";

  if ($http_x_tenant_id) {
    set $tenant_id $http_x_tenant_id;       # priority 1
  }

  if ($tenant_id = "") {
    set $tenant_id $tenant_id_from_host;    # priority 2: map of X-Tenant-Host
  }

  if ($tenant_id = "") {
    return 400 "Tenant not specified";      # priority 3: hard fail
  }

  include /etc/nginx/auth.conf;
  include /etc/nginx/locations.conf;
  include /etc/nginx/custom_error_locations.conf;
}

$tenant_id_from_host is a map populated from the same global.tenants list:

map $http_x_tenant_host $tenant_id_from_host {
  {{- range .Values.global.tenants }}
  "{{ .tenant_dns }}" "{{ .tenant_id }}";
  {{- end }}
}

A few subtleties worth highlighting:

• The order matters. Header beats host. We picked header priority because programmatic clients should be explicit.
• $tenant_id_from_host defaults to empty string if the host isn't in the map. We then 400 — same as if the header was missing entirely.
• if directives in NGINX are deeply weird. We confined them to this block and resisted the temptation to put ifs anywhere else in the config.

Tenant binding inside the JWT

Once NGINX has set $tenant_id, it forwards it as X-Tenant-ID to the Auth Service. But the token also carries a tenant claim. The Auth Service must check they match:

if userToken.TenantID != log.TenantID {
    c.fail(ctx, log, 401, ReasonTenantMismatch, "tenant mismatch")
    return
}

This is the line that saves you when a malicious actor copies a token from tenant A and replays it against tenant B's hostname. The token signature is valid. The token isn't expired. The token isn't revoked. But the tenant in the token is tenantA and the request is for tenantB. We 401.

Three things make this work:

1. The token is bound to a tenant at issuance. Our token issuer puts tid: "tenantA" in the JWT claims when it mints the token. We sign with the per-tenant RSA key (Chapter 3), so a token from tenant A can't be re-signed for tenant B without the private key.
2. The gateway picks the verification key by X-Tenant-ID. If the request says it's for tenant B, we verify the token's signature with tenant B's public key. A tenant A token signed with tenant A's key fails signature validation, not tenant binding — but either way it's denied.
3. The tenant claim is also checked. Even if the keys were the same, the explicit userToken.TenantID != log.TenantID check would catch reuse.

Belt and suspenders. We've never regretted having both.

Sequence: tenant flow end-to-end

Routing: MT vs ST upstreams

Once we know the tenant, we have to route the request. We have two upstream models:

• MT (multi-tenant) services. One deployment, serves all tenants. Tenant comes in as X-Tenant-ID, the service queries data with WHERE tenant_id = ?.
• ST (single-tenant) services. One deployment per tenant, in the tenant's own Kubernetes namespace. The service doesn't even need to know about other tenants — it can't see them.

This is purely a per-service architectural choice. Some products are happy with MT; some have stricter isolation requirements (or run heavy per-tenant data) and want ST.

The location loop in locations.conf handles both with one branch:

proxy_pass http://{{ if eq $type "ST" }}{{ $serviceDict.SERVICE_HOST }}.$tenant_namespace.svc.cluster.local{{ else }}{{ $serviceDict.SERVICE_HOST }}{{ end }};

Unrolled:

• MT: proxy_pass http://user-service — the bare service name. CoreDNS resolves it to the service's ClusterIP in whatever namespace the gateway lives in.
• ST: proxy_pass http://api-service.tenant1-ns.svc.cluster.local — the FQDN includes the tenant namespace. Each tenant has its own copy of the service in their own namespace.

A few subtleties:

• For ST, the tenant namespace is part of the DNS name. NGINX's resolver kicks in at request time, not at config-load time. Adding a new tenant means deploying its services in tenantN-ns, then adding it to global.tenants. NGINX picks it up on the next config reload.
• For MT, all tenants hit the same upstream IP. The upstream service is responsible for tenant scoping. We trust it because we forward X-Tenant-ID and the upstream service's auth library re-checks the header against the token's tenant. (Yes, double-checking. After the first cross-tenant near-miss, we added it.)
• ST is more expensive operationally — N deployments of every service — but radically simpler to reason about. Two services, two answers; pick what your compliance team can defend.

Headers we propagate to upstream

After auth passes, NGINX sends a defined set of headers to the upstream. From locations.conf:

proxy_set_header X-Identity-ID       $identity_id;
proxy_set_header X-Identity-Type     $identity_type;
proxy_set_header X-Identity-Name     $identity_name;
proxy_set_header X-Session-ID        $session_id;
proxy_set_header X-Tenant-ID         $tenant_id;
proxy_set_header X-Tenant-Namespace  $tenant_namespace;
proxy_set_header X-Request-ID        $request_id;

The upstream contract:

• X-Identity-ID is the principal. Treat it as the user's primary key.
• X-Identity-Type is USER or SERVICE_ACCOUNT. Some endpoints reject service accounts, some require them.
• X-Tenant-ID is the tenant. Always scope queries by it.
• X-Tenant-Namespace is the Kubernetes namespace, useful for diagnostics and per-tenant Kafka topic naming.
• X-Session-ID is an opaque session correlation ID. Useful for logging, never for auth.
• X-Request-ID is the trace correlation ID. Forward it to your downstream calls so the whole graph stitches together.

We do not forward Authorization. The upstream service has no business looking at the JWT. If it needs to know who's calling, it uses X-Identity-ID. If it needs to make a downstream call, it gets a fresh service-account token — it does not replay the user's token.

Stripping Authorization was one of those changes that everyone agrees is a great idea in principle and fights tooth-and-nail when their service breaks during the rollout. Worth the fight.

Skipping tenant for a few endpoints

A handful of endpoints genuinely don't have a tenant: NGINX /healthz, public OAuth callbacks, JWKS endpoints, version probes. For these, tenant resolution must not run.

In NGINX:

location /healthz {
  access_log off;
  return 204;
}

This location is matched before the tenant-resolving if block in the default server, because NGINX processes more-specific locations first. /healthz returns 204 without ever evaluating $tenant_id.

Inside the Auth Service, the equivalent pattern shows up: the trie has rows with endpoint_type=OPEN and no tenant requirement. Even if NGINX did pass through, the Auth Service would allow without checking the tenant. Belt and suspenders again.

The Ingress regex: another layer of opt-in

Our cluster's Ingress controller routes some paths through the Auth Gateway and some paths around it. The chart's multi-tenant Ingress uses a negative-lookahead regex to express "send everything to NGINX except these specific exempt paths":

- path: "/(?!{{ $exemptedPattern }}).*"
  pathType: ImplementationSpecific
  backend:
    service:
      name: {{ $shortName }}
      port:
        number: 80

$exemptedPattern is a long alternation built from values.yaml:

ingress:
  exemptedPaths:
    - api/
    - login/
  exemptedPrefixes:
    - ui
  exemptedExtensions:
    - js
    - css
    - ico
    # ... static assets

Anything matching the regex bypasses the gateway and goes to a legacy ingress. This is how we rolled the gateway out one service at a time — and how we keep it manageable today as services migrate at different speeds.

What we'd warn future-us about

A few real lessons:

• Default tenants are forever. If you ship one, every subsequent design decision will assume it exists, and removing it later is a multi-quarter project.
• Tenant-aware logging is non-negotiable. Every log line must carry tenant_id. We don't grep by user — we grep by tenant first, then narrow. Chapter 9 has the log format.
• Keep the tenant model boring. "What is a tenant?" should have a 1-sentence answer. The moment "tenant" starts meaning different things in different services, your isolation guarantees evaporate.
• MT and ST are different operating models, not different security models. The same auth contract should hold. If your ST services can be looser because "they're isolated anyway," you have a problem.
• Never derive tenant from the user. "User belongs to tenant X, so I'll use tenant X" sounds reasonable until you have users in multiple tenants. The tenant comes from the request, not from the user.

What's next

Chapter 6 stays on tenant boundaries but zooms in on the authorization side: roles, access levels, the role → access-level → endpoint mapping, and the bitmap fast path that replaced our original string-set matching. We'll see why a JWT should not be the source of truth for a user's full permission set, and how to encode permissions densely enough that the gateway can decide in O(1) bitwise.