DEV Community: Kaustav Sarkar

Patterns, Anti-Patterns, and Operational Reality

Kaustav Sarkar — Tue, 01 Sep 2026 19:10:35 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous posts covered what the architecture does and why each layer exists. This post covers what matters when the system is running: the patterns that compose cleanly, the mistakes that widen blast radius, and the operational work that falls out of the design.

Pattern: The Sidecar Contract

The simplest pattern in this architecture is the service contract. A protected application service reads two sidecar-projected headers (x-ms<N>-user, x-ms<N>-role) and does its domain logic. It doesn't validate JWTs. It doesn't call Keycloak. It doesn't manage TLS certificates. It doesn't need to know whether the user authenticated with a session cookie or a bearer token.

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

This has practical consequences. To add a protected service, you deploy it with a service account, add destination-side header stripping, configure RequestAuthentication, write an AuthorizationPolicy, and read the two projected headers. Gateway-facing services also need a route-to-audience entry in Auth Service. The application still avoids an auth library, OIDC issuer config, and a runtime dependency on Keycloak.

When the identity infrastructure changes (new signing algorithm, new identity provider, new token format), application code doesn't change. The mesh config updates, and services keep reading the same two headers. Resource checks still live in application code, but the identity input stays stable.

The service does not trust arbitrary incoming headers. Each destination sidecar has a Lua filter that strips existing x-ms<N>-user and x-ms<N>-role headers before JWT validation runs. After stripping, outputClaimToHeaders writes fresh values from the verified token. If a compromised internal service sends a request with a spoofed x-ms2-user header, the destination sidecar removes it and replaces it with the value from the signed JWT. The protected service only sees headers authored by its own sidecar.

The tradeoff is dependency on the mesh. If sidecar injection fails, the service has no independent way to verify identity. PeerAuthentication: STRICT and workload AuthorizationPolicy are what make that failure visible instead of silently accepting plaintext traffic. Security is centralized at the infrastructure layer, so the infrastructure has to be treated as part of the application contract.

Pattern: Namespace Segregation by Function

Four project namespaces, plus istio-system for the mesh control plane and ingress gateway:

zt-apps — application services (Profile Aggregator, Employee Records, Device Inventory, Holiday Calendar, Office Directory, Auth Service)
zt-identity — Keycloak (OIDC provider)
zt-security — Vault, Cerbos
zt-data — PostgreSQL

Namespaces are RBAC boundaries, network policy boundaries, and resource quota boundaries. An attacker with kubectl access scoped to zt-apps cannot read Secrets in zt-security, where the POC stores the Vault root token for tests. Auth Service's scoped Vault token does live in zt-apps, because it needs signing authority at runtime. That token can sign through Transit and read public keys. It cannot rotate keys, configure Vault, or access other secrets engines. Istio's SPIFFE identity includes the namespace, so AuthorizationPolicy can distinguish "any workload in zt-apps" from "Auth Service running as auth-service-sa."

With everything in default, accidental privilege broadening is easier. RBAC rules, Secrets, service accounts, and network policy exceptions all share the same namespace scope, so blast radius becomes harder to reason about during reviews.

Pattern: Signed Tokens Over Forwarded Credentials

Auth Service (auth-service) validates the user's session cookie or bearer token and then mints a short-lived, purpose-specific mesh token. Downstream services do not receive the original credentials. The session cookie and Keycloak JWT stop at the gateway/Auth Service boundary.

Why this matters:

If a downstream service is compromised, the original credential is absent. Depending on the compromise point, the attacker may see projected identity headers or the short-lived mesh assertion in transit, but they do not get the user's session cookie or Keycloak bearer token. The mesh assertion has a 5-minute TTL and still has to match the destination's aud, act, and source-principal checks.
The mesh token carries the platform claims this architecture uses: user, role, audience, delegation, groups, department, and request metadata. It does not carry session IDs or refresh tokens.
Each token is scoped to a specific set of destination services via the aud claim. A token minted for the profile flow (Profile Aggregator, Employee Records, Device Inventory) is invalid at Holiday Calendar or Office Directory.

The mistake this avoids is forwarding the original bearer token or session cookie to every downstream service. That gives each service in the call chain access to authentication material that should have stopped at the boundary.

Pattern: Fail-Closed by Default

Protected paths are designed so that missing configuration or failed infrastructure denies access instead of opening it.

Protected workloads have scoped ALLOW AuthorizationPolicy resources. In Istio, traffic is allowed by default when no policy targets a workload. Once an ALLOW policy applies, unmatched requests are denied.
Auth Service down? ExtAuthz cannot approve protected routes, so the gateway returns 401, 403, or 503 depending on where the failure surfaces.
Vault unreachable? Auth Service cannot mint a new mesh token. The JWKS endpoint returns 500 if it cannot read public keys from Vault, while sidecars may keep using cached keys until refresh.
Cerbos unreachable? Employee Records and Device Inventory fail closed and return 403. Holiday Calendar and Office Directory allow read-only public-data actions during Cerbos outages and deny writes. That is a product decision, not a universal security property.
Sidecar not injected? PeerAuthentication: STRICT means mesh peers reject plaintext connections to that pod. Traffic from outside the mesh still needs Kubernetes NetworkPolicy or another boundary if the cluster allows it.

This is verified by a negative-path test that scales the Auth Service deployment to zero replicas and confirms that protected requests do not return 200:

# Scale auth to 0, confirm requests fail-closed
kubectl scale deployment auth-service -n zt-apps --replicas=0
sleep 10
response=$(curl -s -o /dev/null -w "%{http_code}" https://app.localtest.me/api/profile/1)
assert_not_equal "$response" "200"
kubectl scale deployment auth-service -n zt-apps --replicas=1

For protected routes, a failure that returns application data is a security bug.

Anti-Pattern: mTLS-Only Security

mTLS verifies that the calling pod is who it claims to be (via its SPIFFE certificate) and encrypts the connection. That's necessary but not sufficient.

If Holiday Calendar is compromised and your only security layer is mTLS, the attacker has a valid mesh certificate. They can make encrypted, authenticated connections to any other service in the mesh. mTLS tells Employee Records "this connection is from Holiday Calendar," but without AuthorizationPolicy, Employee Records accepts it. The encrypted tunnel between a compromised pod and a sensitive service is still a compromised path.

mTLS is the foundation. It's the transport layer identity that AuthorizationPolicy references. But treating it as the entire security model means any single pod compromise gives the attacker lateral movement across the entire mesh.

Anti-Pattern: Security in Application Code

One common alternative is implementing identity verification and coarse authorization in every service. Each service validates JWTs, wires its own policy checks, and owns the failure behavior for missing or malformed credentials.

The problems compound at scale:

One service has a bug in JWT validation. Maybe it doesn't check exp. Maybe it accepts tokens from the wrong issuer. That service is now a bypass.
The signing algorithm needs to change. You roll it out to 47 services. Three of them are on the old version for a week because the team is busy. During that week, your security posture is inconsistent.
A new developer adds an endpoint and forgets the auth middleware. The code review misses it. That endpoint is now unprotected, and the mesh has no way to catch it because it doesn't know the endpoint should be protected.

With mesh-based security, identity verification and coarse enforcement (mTLS, AuthorizationPolicy, header projection) are expressed in mesh config and enforced by the sidecar. The application developer does not need to remember those checks on every endpoint because they run before application code. Resource-level authorization (Cerbos calls) and response shaping (field masking) still live in application code, but the identity those decisions rely on comes from the sidecar.

Anti-Pattern: Flat Trust Zones

The cluster-internal trust assumption is simple: anything inside the cluster can talk to anything else. The perimeter becomes the main security boundary.

A single compromised pod then becomes a useful foothold. Container escapes, supply-chain issues, and vulnerable dependencies all put the attacker inside the trusted zone. Without service-to-service policy, the network gives them too many next hops.

The tiered architecture limits those next hops. Tier 1 can reach the specific Tier 2 services it needs. Approved services can reach PostgreSQL. Reverse and lateral paths are not part of the allow-list. Istio AuthorizationPolicy enforces that at each destination workload.

Operational Reality: Debugging Auth Failures

Auth failures are easiest to debug by locating the first boundary that rejected the request. The same HTTP status can come from different layers, so start with where the request stopped.

401, 403, or 503 from the gateway: ExtAuthz denied the request, Auth Service was unavailable, Vault signing failed, or the route was not in the coarse policy map. Check the ingress gateway response code first, then Auth Service logs.

401 from the destination service: The sidecar validated the JWT but the service didn't find projected headers. This usually means RequestAuthentication is misconfigured (wrong fromHeaders name, wrong outputClaimToHeaders claim path) or the JWT is missing the expected claims.

403 before the destination service logs anything: The mTLS identity or JWT claims did not match AuthorizationPolicy. Check istioctl proxy-config listener <pod> and the policy when conditions. A common cause is a token whose aud claim does not include the destination service name.

Cerbos denial: The user passed the route and network checks, but the resource-level policy denied the action. Check Cerbos with the exact principal, resource, action, and resource attributes.

Empty result set (RLS): The query returns zero rows. Check what app.current_user_id was set to on the transaction and compare with the RLS policy conditions.

The rule of thumb: start from the gateway and move inward. If the request never reaches the service, investigate ExtAuthz, routing, JWT validation, and AuthorizationPolicy. If it reaches the service, investigate projected headers, Cerbos inputs, and RLS transaction context.

Operational Reality: Sidecar Overhead

Every pod in this mesh runs an Envoy sidecar. That adds CPU and memory per pod, latency per request path, and one more place to inspect when behavior does not match the application logs.

For this POC on a Kind cluster, the overhead is negligible. In production, it is a real cost. Istio ambient mesh (ztunnel + waypoint proxies) addresses part of that cost by moving L4 processing to a node-level daemon and making L7 processing opt-in.

The tradeoff is where L7 policy runs. Sidecar-per-pod gives each workload local L7 enforcement. Ambient mesh moves L4 security and policy to ztunnel; L7 features such as path and method matching, CUSTOM authorization, JWT claim handling, and header projection require waypoint proxies. This architecture depends on L7 behavior at the gateway and protected service hops, so an ambient version would need waypoints for those workloads. The resource savings come from workloads that only need L4 mTLS, peer identity, and connection-level authorization.

Operational Reality: Testing Security Properties

The tests that matter most verify that incorrect behavior is denied. The main negative-path script checks these cases:

Unauthenticated access to protected routes returns 401/403.
External requests with spoofed internal headers are rejected.
Direct access to Tier 2 services from the gateway is impossible.
A non-Profile Aggregator pod calling Employee Records in-cluster is rejected.
Tokens with wrong audience claims are rejected.
Expired tokens are rejected.
Profile Aggregator cannot connect directly to PostgreSQL.
Auth Service outage fails closed for protected routes.

Each test encodes a specific assumption about the security model. If any test returns application data where denial is expected, a security property has been violated. The repository also has a broader 12-category attack-test harness for spoofing, JWT abuse, token replay, path normalization, SSRF, RLS leakage, ExtAuthz fail-open behavior, Cerbos spoofing, mTLS, and Istio config audit. For this POC, I treat these as local release-gate scripts; production needs them wired into CI before mesh config changes can be treated as guarded.

The rotation test verifies the mint-and-rotate path: mint a token, record its kid, rotate the Vault Transit key, mint another token, and confirm the new kid changed. That proves Auth Service fetches the current key version at mint time. It does not, by itself, prove every sidecar refreshed JWKS without a temporary rejection window.

What I'd Do Differently

Vault storage: emptyDir means Vault's state is lost on pod restart. The bootstrap job re-initializes everything, which works for a POC but is unacceptable for production. Persistent storage with auto-unseal is the minimum for real use.

JWKS caching: Auth Service fetches public keys from Vault on every JWKS request. Under load, this is unnecessary. A short-lived cache with forced invalidation on rotation would reduce Vault load. The cache duration should be tied to the token TTL and the rotation process, because stale JWKS is part of the key-rotation failure mode.

Token TTL tuning: 5 minutes is conservative. For internal-only calls from Profile Aggregator to Employee Records, a 60-second TTL would be sufficient and would reduce the replay window. The tradeoff is more frequent token minting for slower operations.

Ambient mesh: I would evaluate ambient mesh service by service. Workloads that only need mTLS and peer authorization are good candidates for ztunnel without sidecars. Workloads that depend on JWT validation, header projection, path/method policy, or ExtAuthz need waypoints. In this architecture, most protected paths depend on L7 behavior, so ambient mesh would reduce overhead only where the workload boundary can stay at L4.

Wrapping Up

Five layers. Eight request checkpoints. The config is complex, but the idea is simple: every hop verifies independently, each layer covers a different failure mode, and application code stays focused on domain logic.

The patterns that make it work are all forms of separation: security enforcement from application code, identity creation from identity verification, coarse authorization from fine-grained authorization, network boundaries from data boundaries. Each separation limits how far one mistake can propagate.

The anti-patterns all collapse those separations: treating one layer as sufficient, spreading identity verification across services, or trusting cluster networking as the security boundary.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Layered Authorization: Network to Database

Kaustav Sarkar — Tue, 01 Sep 2026 19:06:31 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post covered how Auth Service signs the mesh identity token that carries a user through the mesh. Once a request arrives with a verified identity, the next question is what that identity is allowed to do.

Authentication tells you who someone is. Authorization tells you what they're allowed to do. In this architecture, authorization runs at four granularities: network, request, resource, and field. Each operates at a different enforcement point with different information available, and each catches failures the others can't. A separate layer, PostgreSQL row-level security, sits at the database and backstops all of them when application code has a bug.

Why Four Granularities

A single policy engine can't efficiently handle every authorization decision. Some decisions require only the service identity (which pod is calling). Others require the user's role. Others require knowing the specific resource being accessed and the relationship between the user and that resource. And field visibility only makes sense at query time, when you know the actual data attributes.

Different information is available at different enforcement points:

At the sidecar: mTLS identity and JWT claims. No application context. This is where network-level authorization runs (Istio AuthorizationPolicy).
At the gateway: user role and route. No specific resource. This is where request-level authorization runs (ExtAuthz).
At the application: full request context, resource attributes, user-resource relationships. This is where Cerbos makes resource-level and field-level decisions.
At the database: transaction-level identity. No application logic. This is where row-level security runs. Strictly it's a data-access control rather than authorization, and it's the final enforcement point that backstops the others.

Trying to push all decisions to one point either means that point needs all context (which defeats the separation) or that some decisions can't be made at all.

Network (Sidecar): Istio AuthorizationPolicy

The first question: can this service even talk to that service?

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

This policy says: only Profile Aggregator's service account (ms1-profile-aggregator-sa) can reach Employee Records, and only with a token that has the correct audience and delegation claims. Everything else is denied.

Important nuance: Istio's default behavior when no AuthorizationPolicy targets a workload is to allow all traffic. Default-deny only kicks in once at least one ALLOW policy exists for that workload. This means every service that should be protected needs an explicit policy. A newly deployed service without a matching policy is open until you write one. In this architecture, every workload in zt-apps has at least one scoped ALLOW policy, which triggers deny-by-default for unmatched requests.

What this catches: lateral movement. If Holiday Calendar is compromised, the attacker cannot reach Employee Records. The connection is refused at the sidecar level before any application code processes the request. The attacker would need both the correct service account identity (the mTLS certificate) and a validly-signed token with the right claims.

The full boundary map:

Source	Can reach	Cannot reach
Ingress Gateway	Profile Aggregator, Holiday Calendar, Office Directory, Auth Service	Employee Records, Device Inventory, Vault, Postgres
Profile Aggregator	Employee Records, Device Inventory	Cerbos, Vault, Keycloak, Postgres
Employee Records, Device Inventory	Cerbos, Postgres (own schema only)	Profile Aggregator, Holiday Calendar, Office Directory, Vault
Holiday Calendar, Office Directory	Cerbos, Postgres (own schema only)	Profile Aggregator, Employee Records, Device Inventory, Vault
Auth Service	Vault, Postgres	Employee Records, Device Inventory, Holiday Calendar, Office Directory

Every cell in this table is an explicit decision. Unmatched traffic is denied because each workload has at least one ALLOW policy targeting it.

Request (Gateway): ExtAuthz

The second question: is this user, with this role, allowed to call this route?

Auth Service (auth-service) acts as an external authorization check at the gateway. Before minting a mesh token, it checks a route-role policy map:

Route	Method	Allowed Roles
`/api/profile/*`	any	employee, manager, hr_admin, it_admin
`/api/holidays`	GET	employee, manager, hr_admin, it_admin, public_data_admin, security_auditor
`/api/holidays`	POST/PUT/DELETE	hr_admin, public_data_admin
`/api/offices/*`	any	(bypasses ExtAuthz entirely; see below)

The system has six roles total (employee, manager, hr_admin, it_admin, public_data_admin, security_auditor). The core access model for the profile/HR flow is built around the first four. public_data_admin manages office and holiday data. security_auditor has read access to non-sensitive endpoints.

Office routes are the exception: the gateway's AuthorizationPolicy exempts /api/offices and /api/offices/* from ExtAuthz entirely. Reads are public, and write authorization (public_data_admin only) is enforced downstream by Cerbos at Office Directory, not at the gateway. Everything else routes through the map above.

A public_data_admin trying to access the profile endpoint gets a 403 at the gateway. They never hit Profile Aggregator. The token is never minted. Downstream services never see the request.

This is coarse authorization. It knows the user's role and the route they're hitting, but it knows nothing about which specific resource they want to access. A manager can access the profile endpoint, but that doesn't mean they should see every employee's salary; they should only see pay data for their own direct reports. That's a different layer's job.

The key decision here is what gets denied early (saving downstream resources and reducing attack surface) versus what gets deferred to finer-grained layers that have more context.

Resource and Field (Application): Cerbos

The third and fourth granularities: can this user access this specific resource? And which fields should they see?

Cerbos evaluates context-aware policies. It receives the principal (user ID, roles, attributes), the resource (type, ID, owner, attributes), and the action. It returns an allow/deny decision and a list of visible fields.

# cerbos/policies/employee_profile.yaml (abbreviated: view/list and hr_admin rules omitted)
apiVersion: api.cerbos.dev/v1
resourcePolicy:
  version: default
  resource: employee_profile
  rules:
    - actions: ["view_sensitive", "update"]
      effect: EFFECT_ALLOW
      roles: ["employee"]
      condition:
        match:
          expr: request.principal.id == request.resource.attr.id
      output:
        expr: |-
          {"visible_fields": ["name", "title", "department", "salary_band", "base_salary", "ssn"]}

    - actions: ["view_sensitive"]
      effect: EFFECT_ALLOW
      roles: ["manager"]
      condition:
        match:
          expr: request.principal.id == request.resource.attr.manager_id
      output:
        expr: |-
          {"visible_fields": ["name", "title", "department", "salary_band"]}

What this means concretely: Alice (employee) views her own profile and sees everything including salary and SSN. Bob (manager) views Alice's profile and sees the salary band but not the exact salary or SSN. Carol (another employee) views Alice's profile and sees only name, title, and department.

Same endpoint. Same authentication. Same route-level authorization. Different data, determined by the relationship between the user and the resource.

What Cerbos catches that the previous layers cannot: role alone isn't sufficient. A manager should see sensitive data for their direct reports, not for every employee in the company. This requires knowing who the resource belongs to and what the manager's relationship is to that person. The gateway doesn't have that information. Only the application, at query time, knows these relationships.

Field Masking

Cerbos doesn't filter data itself. It returns a list of visible fields, and the application code uses that list to shape the response. This is a contract between policy and application:

result = cerbos.check_resource(
    principal={"id": user_id, "roles": [role]},
    resource={"kind": "employee_profile", "id": employee_id, "attr": {"manager_id": ...}},
    actions=["view_sensitive"]
)
visible = result.outputs["visible_fields"]
# Application filters the response to only include these fields

The application must respect this output. If it doesn't, row-level security on the sensitive tables (PII and financials) still limits which rows the database returns, though the directory table is open and relies on Cerbos for field masking. Field-level masking specifically depends on correct application implementation. This is one place where the "security as infrastructure" model has a boundary. The mesh can't mask JSON response fields for you.

Database (Backstop): PostgreSQL Row-Level Security

The final question: even if everything above passes, should the database return this row?

Row-level security here is deliberately split. The employee directory table (hr.employees, holding name, title, department, manager) is readable by any authenticated principal, because the directory itself isn't sensitive and Cerbos handles field masking on top of it. The sensitive tables (hr.employee_pii, hr.employee_financials) carry the strict row-level policy:

CREATE POLICY employee_financials_visibility ON hr.employee_financials
  USING (
    current_setting('app.current_roles', true) LIKE '%hr_admin%'
    OR employee_id::text = current_setting('app.current_user_id', true)
    OR EXISTS (
      SELECT 1 FROM hr.employees e
      WHERE e.id = employee_id
        AND e.manager_id::text = current_setting('app.current_user_id', true)
    )
  );

ALTER TABLE hr.employee_financials ENABLE ROW LEVEL SECURITY;
ALTER TABLE hr.employee_financials FORCE ROW LEVEL SECURITY;

This policy says: you can see a financial record if you're an hr_admin, if it's your own, or if you manage that employee. Nothing else. A query with no WHERE clause still returns only the rows this user is allowed to see. The PII table (hr.employee_pii) uses the same shape, and it.hardware_assets uses self-or-it_admin.

RLS uses current_setting('app.current_user_id') and current_setting('app.current_roles'), which are set by the application at the start of each transaction from the sidecar-projected headers. The database has no knowledge of JWTs, sidecars, or mesh tokens. It just enforces visibility based on the transaction context it's given.

Why Cerbos and RLS

The two overlap, but they catch different failures:

Cerbos fails (policy bug, service misconfiguration, Cerbos unreachable): The services fail closed. If Cerbos is unreachable, the application returns 403 (the client catches all Cerbos errors and denies by default). If a policy bug returns an incorrect allow decision, RLS on the sensitive tables still drops the PII and financial rows the user can't see. The response might leak directory fields that should have been masked, but the sensitive records stay protected.

Application fails (developer forgets to call Cerbos, broken filter logic, new endpoint without an auth check): RLS still enforces on the sensitive tables. SELECT * FROM hr.employee_financials without any Cerbos check still returns only rows matching the RLS policy. The mistake doesn't become a financial-data breach.

RLS fails (misconfigured policy, wrong user ID set on transaction): The query order is: the service loads the employee record first (to obtain attributes like manager_id for the Cerbos policy), then calls Cerbos, then queries the sensitive tables only after Cerbos approves. That initial directory read is intentionally open, so Cerbos is what gates the sensitive query. A broken sensitive-table RLS policy widens which rows come back, but Cerbos already gated access to those tables. A broken Cerbos policy widens what's returned, but RLS still limits sensitive-row visibility.

Neither Cerbos nor RLS alone is sufficient. Together, an application developer's mistake stops short of full exposure of PII or financial data.

Schema Isolation

RLS is complemented by PostgreSQL GRANT-level isolation. Each service has a dedicated database role:

ms2_hr_role can only access hr.* tables (employees, PII, financials)
ms3_it_role can only access it.hardware_assets
ms4_public_readwrite_role can only access public_data.*

Even without RLS, Device Inventory physically cannot query hr.employee_financials. The GRANT isn't there. This is a separate enforcement layer from RLS. If Device Inventory is compromised and the attacker somehow bypasses Istio AuthorizationPolicy (they'd need the mTLS cert and a valid token), the database still won't serve them HR data because the connection role lacks the privilege.

How the Granularities Compose: A Concrete Example

Alice (employee) requests her own profile with GET /api/profile/{alice_id}:

Network: Gateway can reach Profile Aggregator. Profile Aggregator can reach Employee Records. ✓
Request: Alice has the employee role, /api/profile/* allows employees. Token minted. ✓
Resource and field (Cerbos): Alice is requesting her own record (principal.id == resource.attr.id). Allowed, with visible fields that include salary and SSN.
Database (RLS): When the app reads her financials and PII, app.current_user_id matches the employee_id on those rows, so they're returned.

Bob (employee) tries to see Alice's financial data. His request also comes in as GET /api/profile/{alice_id}, and Profile Aggregator fans out toward Alice's sensitive records:

Network: Same path. ✓
Request: Bob has the employee role, /api/profile/* allows employees. Token minted. ✓
Resource and field (Cerbos): When the app tries to read Alice's financials, it checks the view_sensitive action. Bob is not Alice and not Alice's manager. Denied. The /financials and /pii endpoints on Employee Records are all-or-nothing, so the response is a 403 with no partial data.
Database (RLS), if Cerbos had a bug and allowed it: The RLS policy on hr.employee_financials requires employee_id == current_user_id or that the caller manages the employee. Bob is neither. Zero rows returned. The bug at the resource layer is caught at the database.

Bob viewing Alice's basic profile is a different action on the directory: the view action is allowed for any employee, and Cerbos returns visible_fields: [name, title, department]. Same request path, separate actions, separate policies, and the directory row is readable while the sensitive tables are not.

What's Next

The final post covers the patterns and anti-patterns across all layers: what makes this architecture compose well, what operational pain points exist, and what you'd do differently at scale.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Signing Mesh Tokens with Hashicorp Vault

Kaustav Sarkar — Tue, 01 Sep 2026 19:01:59 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post traced one request's identity flow from gateway to database. It left one piece unexplained: how Auth Service actually signs the identity token that carries a user through the mesh. That signing step is the system's root of trust. Steal the signing key and you can forge a token for any user and any role, and every downstream service will believe it.

A quick recap, so this post stands on its own. The system is an HR employee directory running on Istio. A gateway-facing service, Profile Aggregator, fans out to internal-only services like Employee Records (HR data) and Device Inventory (hardware assignments). Auth Service (auth-service) sits at the gateway: it authenticates the external user and mints a short-lived signed JWT, x-mesh-identity (the "mesh token"), that downstream service sidecars verify on every request. It signs that token with Vault, HashiCorp's secrets manager, using Vault's Transit engine, so the private key never touches application code. This post covers why that matters, how it works, and what happens when things go wrong.

The Problem: Secrets in Pods

The default Kubernetes pattern for secrets is to mount them into pods. Environment variables, files in /var/run/secrets, ConfigMaps with sensitive values. Assume any compromised pod leaks whatever secrets it holds. If your signing key is in the Auth Service environment, a container escape gives the attacker the key. They can mint tokens for any user, any role, indefinitely, from anywhere, until you detect the breach and rotate.

The blast radius of a compromised pod should be limited to what that pod can do right now, not what it can do forever with stolen material.

What Vault Solves, and What It Does Not

Token signing is only one use case. In a production cluster, Vault is the central control point for secret access, audit, and rotation across the system. The larger pattern is that applications should not carry durable secret material when the platform can issue short-lived authority from workload identity.

The production version removes static Vault tokens from application config. Auth Service runs under a dedicated Kubernetes service account with a projected, short-lived service-account token. Vault Kubernetes Auth maps that service account to the narrow signing policy. A Vault Agent sidecar authenticates, renews the resulting Vault token, and keeps the raw token inside the Agent boundary. Auth Service calls Vault through the Agent instead of reading a long-lived token from its own environment.

There is still a credential. The difference is that it is platform-issued workload identity, not a manually provisioned Vault token sitting in app config. The remaining credential is issued by Kubernetes, rotated by the platform, scoped by Vault policy, and auditable at Vault.

This POC uses the simpler version: Auth Service receives VAULT_TOKEN directly. That keeps the signing flow easy to see, but it is a demo shortcut, not the endpoint of the pattern.

Vault also does not make a compromised pod harmless. If the Auth Service pod is compromised, the attacker can use whatever runtime authority that pod has until you revoke or cut it off. What Vault changes is the value of that authority. Instead of stealing the RSA private key and signing offline forever, the attacker gets access to a narrow, revocable, auditable signing path. The private key stays inside Vault, every signing operation goes through a central API, and rotation is controlled in one place.

Vault Transit: Signing Without Holding Keys

Vault's Transit engine is a cryptography-as-a-service API. You send it bytes, it signs them with a key it manages, and returns the signature. The key material never leaves Vault's control.

In this architecture, Auth Service uses Transit for one specific operation: signing mesh identity tokens with RS256 (RSA-2048, PKCS#1 v1.5).

The flow:

Auth Service builds the JWT header and payload (claims assembled from the authentication result).
Auth Service encodes the signing input: base64url(header).base64url(payload).
Auth Service sends those bytes to Vault: POST /v1/transit/sign/mesh-identity.
Vault signs with the current RSA private key, returns vault:v<version>:<base64_signature>.
Auth Service strips the prefix, re-encodes as base64url, and assembles the final JWT.

def sign_payload(payload_bytes: bytes) -> str:
    """Signs a payload using Vault Transit and returns the raw signature"""
    url = f"{settings.VAULT_URL}/v1/transit/sign/mesh-identity"
    headers = {"X-Vault-Token": settings.VAULT_TOKEN}

    encoded_payload = base64.b64encode(payload_bytes).decode("utf-8")
    data = {
        "input": encoded_payload,
        "hash_algorithm": "sha2-256",
        "signature_algorithm": "pkcs1v15",
    }

    response = _client.post(url, headers=headers, json=data)
    response.raise_for_status()
    result = response.json()

    # Vault returns the signature as "vault:v<version>:<base64sig>"; strip the prefix for JWS.
    sig = result["data"]["signature"]
    parts = sig.split(":")
    if len(parts) >= 3:
        return parts[2]
    return sig

Two Vault round-trips happen per token mint: one to fetch the current key version (for the kid header claim), one to sign. In this POC, Auth Service mints a fresh mesh token for every protected gateway request. That keeps the behavior simple and security-first: every request gets a new iat, exp, jti, and signing decision, and key rotation is observed immediately.

The tradeoff is that Vault sits in the synchronous request path. Under production load, you may choose to cache mesh tokens inside Auth Service until their 5-minute expiry, keyed by session, subject, roles, audience, and delegation context. The token still stays inside the platform boundary: it is not returned to the browser, stored in a user cookie, or exposed to frontend code. Caching removes most signing calls for repeated requests from the same user flow.

The cost is freshness: revocation, role changes, and key rotation take effect at token-expiry boundaries instead of per request.

Caching also constrains the token shape. Keep request-specific data such as x-request-id outside the token so the same mesh assertion can safely travel with multiple requests. That would require changing the current POC token, because this implementation includes request_id in the JWT payload.

The JWKS Endpoint: Public Key Distribution

Every service sidecar needs to verify mesh tokens. Auth Service exposes GET /auth/jwks, which fetches all active public key versions from Vault and formats them as a standard RFC 7517 JWKS response:

def get_jwks() -> dict:
    """Format Vault public keys as JWKS"""
    keys = vault_client.get_public_keys()
    jwks = {"keys": []}
    for version, key_info in keys.items():
        public_key = serialization.load_pem_public_key(key_info["public_key"].encode("utf-8"))
        if isinstance(public_key, RSAPublicKey):
            numbers = public_key.public_numbers()
            jwks["keys"].append({
                "kty": "RSA",
                "kid": str(version),
                "use": "sig",
                "alg": "RS256",
                "n": int_to_base64url(numbers.n),
                "e": int_to_base64url(numbers.e),
            })
    return jwks

There's no version filtering in this code, and none is needed. get_public_keys() reads the keys map from vault read transit/keys/mesh-identity, and Vault only returns the working set there: versions from min_decryption_version through the latest. Anything below min_decryption_version is archived and dropped from that response, so it never reaches the JWKS output. The filtering happens inside Vault, not in application code. That makes min_decryption_version the single switch controlling which keys sidecars will accept, and it's why rotation and invalidation are two separate operations, covered next.

This cluster runs istiod with PILOT_JWT_ENABLE_REMOTE_JWKS enabled, so each Envoy sidecar fetches this endpoint directly over mTLS, rather than istiod fetching it once and pushing the keys. A sidecar caches the response and refreshes on its own schedule, on the order of 5 minutes. During rotation, there's a window where a sidecar still holds the old key set. More on that below.

Key Rotation

Rotation happens in two steps: create a new key version, then invalidate old versions.

#!/bin/bash
# Step 1: Rotate — creates a new key version, old versions remain valid
vault write -f transit/keys/mesh-identity/rotate

# Step 2: Invalidate — bumps min_decryption_version so old keys can't verify
vault write transit/keys/mesh-identity/config \
  min_decryption_version=$NEW_LATEST

The --grace flag in the rotation script introduces a 300-second (5-minute) wait between these steps. Why 5 minutes? Because mesh tokens have a 5-minute TTL. After one full TTL passes, all tokens signed with the old key have expired. No valid old-key tokens remain in flight, so invalidating the old key causes no disruption.

if [ "$1" = "--grace" ]; then
    echo "Grace period: waiting 300s for in-flight tokens to expire..."
    sleep 300
fi
vault write transit/keys/mesh-identity/config \
    min_decryption_version=$NEW_LATEST

Bumping min_decryption_version removes the old version from the JWKS endpoint, but sidecars don't pick that up instantly. Each one validates tokens against the JWKS it last fetched. A failed validation does not trigger an immediate refetch. The sidecar keeps using that cached key set until its next scheduled refresh, on the order of the 5-minute cache interval above.

That cache lag is exactly why the grace period exists. By waiting one full 5-minute token TTL before bumping the minimum, every token signed with the old key has already expired. After that, it no longer matters whether a given sidecar has refreshed its JWKS. If it still has the old public key cached, there are no valid old-key tokens left to accept. If it has refreshed, it only sees the new key. Skip the grace period and you risk rejecting still-valid old-key tokens the moment a sidecar refreshes mid-flight. Either way, bumping the minimum only archives a version; it stays recoverable by lowering min_decryption_version again, and is gone for good only once it drops below min_available_version.

The grace period covers the old key. The new key has a separate window. Auth Service sets kid to the latest version on every mint, so it starts issuing tokens signed with the new key the instant you rotate. A sidecar whose JWKS cache is still stale has no public key for that kid yet, and rejects those freshly-minted tokens with a 403. So rotation should also trigger a JWKS cache invalidation, forcing every sidecar to refetch immediately and pick up the new public key before any new-key token reaches it. That closes the window so clients never see a 403.

Vault Token Scoping

In this POC, Auth Service holds a Vault token with exactly two capabilities:

update on transit/sign/mesh-identity (can sign)
read on transit/keys/mesh-identity (can read public keys and current version)

It cannot rotate keys, cannot modify key configuration, cannot access any other Vault path. This is the minimum privilege needed for signing and JWKS serving.

The token has a 32-day renewable period. Rotation of the Vault token itself (revoking and reissuing) is a separate operational concern from key rotation. In the production model described earlier, this same policy would be attached to the Vault token issued through Kubernetes Auth and managed by Vault Agent.

Blast Radius: What Happens When Things Break

Auth Service compromised:

The attacker has the pod's signing authority: the direct Vault token in this POC, or access to the Agent-mediated signing path in the production model. They can sign arbitrary payloads under mesh-identity and read public keys. They can mint tokens for any user, any role, with any audience. That breaks the user-identity layer for any service that accepts x-mesh-identity.

It does not automatically bypass every mesh policy. Downstream AuthorizationPolicy rules still check the caller's mTLS workload identity, and the token still has to carry the expected aud (audience: which services may accept the token) and act (the delegation claim naming the authorized intermediary) claims for that route. A forged token from the wrong pod can still fail the source-principal check. The real danger is that the compromised Auth Service can mint tokens that look legitimate to the JWT layer, including tokens with audiences and delegation claims chosen by the attacker.

But the damage is bounded. The attacker cannot rotate keys or modify Vault configuration. You revoke the direct Vault token, or revoke the Agent-issued token and cut off the Kubernetes Auth path for that workload, and their signing ability stops. In-flight tokens they've already minted expire within 5 minutes. You rotate the key and bump min_decryption_version, which drops the compromised version from the JWKS endpoint so sidecars stop accepting anything signed with it once they refresh their cache.

Detection: Vault audit logs show every signing request with the token accessor. Anomalous patterns (sudden spike in signing, unusual audiences) are detectable if audit logging is enabled.

Vault compromised:

This is the worst case, but the severity depends on the compromise depth. If the attacker gains root token access or breaks into the Vault API, they can sign arbitrary tokens through Transit. There's still an audit trail (Vault audit log records each API call), and you recover by rotating the Transit key and revoking the compromised token.

If the attacker extracts the actual key material (storage backend compromise, memory dump, unsealed Vault snapshot), they can sign tokens offline, from anywhere, without making Vault API calls. No audit trail. Tokens they produce are indistinguishable from legitimate ones until you rotate to an entirely new key.

In this POC, Vault uses emptyDir storage (data lives only in the pod's ephemeral filesystem). In production, Vault would use encrypted backend storage with auto-unseal via a cloud KMS. The private key's residency model depends entirely on how Vault is deployed.

A downstream service compromised (Employee Records, Device Inventory, etc.):

The attacker can read projected headers for any request that arrives at that service. They see user IDs and roles for requests they handle. They cannot mint new mesh tokens (no Vault access) and cannot call other internal-only (Tier 2) services (AuthorizationPolicy blocks it).

For database access, the compromised service has its scoped DB role (Employee Records can access hr.* tables, Device Inventory can access it.*). A malicious process could call set_config('app.current_user_id', ...) with an arbitrary value to bypass row-level security (RLS) within its own schema. RLS stops application bugs, not a fully malicious process controlling its own DB session. The containment here is schema isolation (Employee Records can't query it.* tables) and AuthorizationPolicy (Employee Records can't reach Device Inventory's database path).

The Anti-Patterns

Local key pairs: Auth Service generates and holds an RSA key pair. Compromise means the attacker has the key forever (until you detect and rotate). No audit trail of signing operations. No centralized rotation procedure.

Shared HMAC secrets: All services share a symmetric key for signing/verification. Any compromised service can mint tokens. The blast radius of a single pod compromise becomes total. Rotation requires coordinated rollout to every service simultaneously.

Secrets in environment variables: The key material is readable by any process in the container (via /proc/self/environ), visible in crash dumps, and exposed through the container runtime debug surface. Even secretKeyRef values end up as plaintext in the process environment once the pod starts. This is the Kubernetes default for "sensitive config."

Each of these works fine at small scale with a trusted team. They become unacceptable when the question shifts from "can someone on our team see this secret" to "what happens when one pod in a 50-service mesh is compromised."

What's Next

The next post covers authorization in depth: how network-level, request-level, resource-level, and field-level authorization compose into a system where each layer catches what the one above it misses.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

The Identity Flow: From Gateway to Database

Kaustav Sarkar — Tue, 01 Sep 2026 18:55:31 +0000

This blog is a part of the Zero-Trust Security on Istio Series, if you haven't read the previous article, please do, it will provide better context.

The previous post laid out the five security layers as a static architecture. This post follows a single request through them, down to the question that matters in practice: what does a service actually trust, and why is trusting it safe?

The system is an HR employee directory, built to be attacked. Five microservices sit behind the mesh, numbered ms1 through ms5: Profile Aggregator (ms1) is the gateway-facing orchestrator; Employee Records (ms2) holds PII and salary data and Device Inventory (ms3) holds hardware assignments, both internal-only; Holiday Calendar (ms4) and Office Directory (ms5) serve holiday and office-location data, most of it public. Auth Service mints a signed identity token, x-mesh-identity, that carries the user through the mesh, and Vault signs it. This post traces one concrete request: an employee, Alice, requests her own profile with GET /api/profile/{id}. That request reaches Profile Aggregator, which fans out to Employee Records and Device Inventory. I'll follow the ms1 to ms2 hop, Profile Aggregator calling Employee Records, so every ms2-* resource below is that running example, not a special case.

Employee Records reads the user's identity from a plain HTTP header, eg x-ms2-user. The header is named per service (ms2 reads x-ms2-user, ms3 reads x-ms3-user) so one service can never consume another's identity headers. Employee Records does no JWT parsing and makes no callback to an auth service. It reads that header and treats the value as the authenticated user. That should look fragile, because headers are trivially spoofable. Any external client can send x-ms2-user: alice, and so can any compromised pod inside the cluster.

It works because of everything that happens before a service reads that header. A single request moves through eight steps from the gateway to the database, each adding one guarantee:

Header stripping at the gateway.
Authentication of the caller's credentials.
Route-level authorization against a coarse policy map.
Token minting, signed via Vault.
Token validation and header projection at the destination sidecar.
AuthorizationPolicy, binding peer identity to token claims.
The service contract, the two headers the application actually reads.
Database identity context for row-level enforcement.

Six of those steps run in the mesh before the application sees the request. By the time Employee Records reads x-ms2-user, its value has been stripped of anything the caller sent and re-derived from the signed token the destination sidecar verified. The last two steps are the service's own contract and the database scoping it then sets up.

The Trust Boundary

The Istio Ingress Gateway is where external input is converted into internal identity. Everything before it is untrusted. An external client can send any header it wants, including headers that look exactly like internal identity assertions. The gateway's job is to strip those, authenticate the caller, and produce a cryptographically signed token that downstream services can verify independently.

After the gateway, a service trusts only the headers its own destination sidecar wrote. The sidecar does the verification itself: it strips any inbound copy of the projected headers (like x-ms2-user), validates the JWT signature, checks the issuer, and verifies expiry, then writes the claims into fresh headers. The application reads those headers directly and runs no verification of its own, because the sidecar already ran it.

That trust is narrowly scoped. It covers only the headers the sidecar itself wrote after validating the token. Anything that merely arrived on the request, from an external client or an internal pod, is ignored. So if a compromised internal pod sends a spoofed x-ms2-user header, the destination sidecar strips it before the JWT filter runs and re-projects the value from the verified token. The spoofed header never reaches the application.

This model has a clear contract: the gateway is responsible for identity creation, sidecars are responsible for identity verification, and services are responsible for business logic. If any of those three fail in isolation, the other two still limit the blast radius.

Step 1: Header Stripping

Before authentication, before routing, before anything else, a Lua EnvoyFilter removes every internal trust header from the incoming request:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-prestrip
  namespace: istio-system
spec:
  priority: -100
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.ext_authz"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.prestrip
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                local headers_to_remove = {}
                for key, value in pairs(request_handle:headers()) do
                  local lower_key = string.lower(key)
                  if string.match(lower_key, "^x%-platform%-") or
                     string.match(lower_key, "^x%-role%-") or
                     lower_key == "x-mesh-identity" or
                     string.match(lower_key, "^x%-ms%d%-user") or
                     string.match(lower_key, "^x%-ms%d%-role") then
                    table.insert(headers_to_remove, lower_key)
                  end
                end
                for _, key in ipairs(headers_to_remove) do
                  request_handle:headers():remove(key)
                end
              end

The filter runs at priority -100 with INSERT_BEFORE the ExtAuthz filter. The order is load-bearing: if ExtAuthz ran first and injected x-mesh-identity based on a valid session, but a spoofed x-ms2-user header also survived from the client, the downstream service would read the attacker-controlled value.

The two-pass approach (collect keys, then remove) is required because mutating headers during iteration is undefined behavior in Envoy's Lua API.

What gets stripped: x-platform-*, x-role-*, x-mesh-identity, and all x-ms<N>-user / x-ms<N>-role headers. Case-insensitive matching ensures no variant slips through.

Step 2: Authentication (ExtAuthz)

After stripping, the request hits the ExtAuthz filter. This calls Auth Service (auth-service), which resolves the caller's identity from one of two sources:

Bearer token: Extracted from the Authorization header, validated against Keycloak's JWKS endpoint. Roles come from realm_access.roles in the Keycloak JWT.

Session cookie: An opaque zt_session cookie, looked up in PostgreSQL. The session record holds the user's roles from the original OIDC flow.

Both paths produce the same output: a subject identifier, username, email, roles, groups, and department. The subject is deterministic across both auth paths. It's computed as uuid5(NAMESPACE_URL, "istio-security://users/<username>"), which means the same user always gets the same sub regardless of whether they authenticated via cookie or bearer token. This is important for RLS (which filters by sub) and for audit trails.

Step 3: Route-Level Authorization

Before minting a token, Auth Service checks a coarse policy map. This is the first authorization gate:

# Coarse route/role policy map (ext_authz.py)
if path.startswith("/api/profile/"):
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin"}
    decision["audience"] = ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"]
elif path.startswith("/api/holidays") and method == "GET":
    allowed_roles = {"employee", "manager", "hr_admin", "it_admin", "public_data_admin", "security_auditor"}
    decision["audience"] = ["ms4-holiday-calendar"]
elif path.startswith("/api/holidays") and method in ("POST", "PUT", "DELETE", "PATCH"):
    allowed_roles = {"public_data_admin", "hr_admin"}
    decision["audience"] = ["ms4-holiday-calendar"]
else:
    decision["reason"] = "Route not defined in coarse policy map"
    return decision

user_roles = set(decision["roles"])
if not allowed_roles.intersection(user_roles):
    decision["reason"] = "Unauthorized role for route"
    return decision

If the user's role doesn't match the route, the request is denied with 401. No token is minted. The audience field determines which services this token will be valid for. A token for the profile endpoint is valid at Profile Aggregator (ms1-profile-aggregator), Employee Records (ms2-employee-details), and Device Inventory (ms3-hardware-assets). A token for the holidays endpoint is valid only at Holiday Calendar (ms4-holiday-calendar). This prevents a valid token from one flow being replayed against a different service.

Any path not in the policy map is denied by default. GET /api/offices is deliberately absent, because those reads are public: the gateway's AuthorizationPolicy exempts them from ExtAuthz, so an unauthenticated read reaches Office Directory (ms5-office-locations) without a token and is treated as anonymous/public. Writes to /api/offices do go through the map above and require public_data_admin.

Step 4: Token Minting

If authentication and route authorization both pass, Auth Service mints a signed mesh token. The JWT is built manually (no library) and signed via Vault Transit:

header = {"alg": "RS256", "typ": "JWT", "kid": str(vault_key_version)}

payload = {
    "iss": "auth-service",
    "sub": subject_uuid,
    "preferred_username": username,
    "roles": ["employee", "manager"],
    "roles_csv": "employee,manager",
    "aud": ["ms1-profile-aggregator", "ms2-employee-details", "ms3-hardware-assets"],
    "act": {"sub": "ms1-profile-aggregator"},
    "exp": now + 300,  # 5 minutes
    "jti": str(uuid4()),
    "request_id": request_id
}

The signing input (header_b64.payload_b64) is sent to Vault Transit's /v1/transit/sign/mesh-identity endpoint. Vault signs it with the RSA-2048 private key and returns the signature. Auth Service never holds the private key.

Key details:

roles_csv exists because Istio's outputClaimToHeaders can only project string claims, not arrays.
act (the delegation claim from RFC 8693) is set to ms1-profile-aggregator when Profile Aggregator is in the audience. Auth Service sets this at mint time because it already knows (from the route-policy map) that the profile flow will pass through Profile Aggregator to reach Employee Records and Device Inventory. The act claim asserts that Profile Aggregator is the authorized intermediary for this request. When Profile Aggregator forwards the token downstream, the receiving AuthorizationPolicy verifies both the mTLS identity (is this actually Profile Aggregator?) and the act claim (was Profile Aggregator designated as the intermediary?). If a different service somehow obtained this token, the mTLS check would fail.
kid is fetched live from Vault on every mint. When keys rotate, new tokens immediately reference the new version.
TTL is 5 minutes. Short enough to limit replay windows, long enough that a fan-out from Profile Aggregator to Employee Records and Device Inventory completes comfortably.
The JWT is assembled manually rather than via a library like PyJWT. This is because the signing step goes through Vault Transit's API (not a local key), and standard JWT libraries assume they hold the private key. Building the token by hand keeps the signing interface clean: encode header and payload, send bytes to Vault, attach the returned signature.

Step 5: Token Validation and Header Projection

The signed token travels as the x-mesh-identity header. The destination sidecar handles it in two stages before the application sees the request.

First, a Lua EnvoyFilter strips any inbound x-ms2-user / x-ms2-role headers. It runs at priority: 10 with INSERT_BEFORE the JWT authentication filter, so spoofed values are removed before any claim is projected:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-projection-ms2
  namespace: zt-apps
spec:
  priority: 10
  workloadSelector:
    labels:
      app: ms2-employee-details
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.jwt_authn"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua.projection
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
            inlineCode: |
              function envoy_on_request(request_handle)
                request_handle:headers():remove("x-ms2-user")
                request_handle:headers():remove("x-ms2-role")
              end

This is the destination-side counterpart to the gateway pre-strip described in The Trust Boundary: the same strip-then-project pattern, now enforced at the workload. Any inbound copy of these headers is removed before the JWT filter runs.

Then the RequestAuthentication resource validates the token and projects the verified claims:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: mesh-identity-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  jwtRules:
    - issuer: "auth-service"
      jwksUri: "http://auth-service.zt-apps.svc.cluster.local:8000/auth/jwks"
      forwardOriginalToken: false
      fromHeaders:
        - name: "x-mesh-identity"
      outputClaimToHeaders:
        - header: "x-ms2-user"
          claim: "sub"
        - header: "x-ms2-role"
          claim: "roles_csv"

The sidecar fetches public keys from the Auth Service JWKS endpoint, validates the JWT signature, checks iss and exp, and then projects sub into x-ms2-user and roles_csv into x-ms2-role. Because the strip filter already cleared any inbound copies, these headers can only carry claims from a validated token. These are the only headers the application reads.

One Istio nuance is easy to miss: RequestAuthentication validates a JWT when one is present. It does not, by itself, mean "a JWT is required." The requirement comes from the AuthorizationPolicy in the next step. That policy checks claims that only exist after successful validation, so a missing or invalid token has no matching rule and is denied.

Profile Aggregator has forwardOriginalToken: true because it needs to propagate x-mesh-identity to Employee Records and Device Inventory. All other services have it set to false since they're terminal and don't make downstream calls that need identity propagation.

Step 6: AuthorizationPolicy

After token validation, Istio's AuthorizationPolicy enforces token presence, token scope, and peer identity:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-ms1-to-ms2
  namespace: zt-apps
spec:
  selector:
    matchLabels:
      app: ms2-employee-details
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/zt-apps/sa/ms1-profile-aggregator-sa
      to:
        - operation:
            ports: ["8000"]
      when:
        - key: request.auth.claims[aud]
          values: ["ms2-employee-details"]
        - key: request.auth.claims[act][sub]
          values: ["ms1-profile-aggregator"]

Three things must be true simultaneously: the request must come from ms1-profile-aggregator-sa over mTLS, the token must be valid for ms2-employee-details, and the token must name ms1-profile-aggregator as the authorized intermediary. The peer identity comes from the client certificate. The aud and act values come from the signed JWT. These are independent cryptographic assertions. Compromising one doesn't give you the other.

The aud claim is a list because a single profile request legitimately reaches more than one service. Istio's claim matching checks whether the configured value is present in that claim, so a token minted for Profile Aggregator, Employee Records, and Device Inventory can match ms2-employee-details here. A token minted only for Holiday Calendar would not match.

This is confused-deputy prevention: stopping a privileged intermediary from being tricked into using its authority on behalf of an attacker. If Profile Aggregator's service account is somehow used by a different process, the process would still need a validly-signed token that authorizes Profile Aggregator to call Employee Records for this user flow. If an attacker replays a legitimate token from a different pod, the mTLS identity check fails. Both bindings must match.

The deny behavior depends on having an AuthorizationPolicy that targets the workload. Istio allows traffic by default when no policy applies. In this architecture, every protected workload has at least one scoped ALLOW policy, so unmatched traffic is denied. A request with no token, a bad signature, the wrong audience, or the wrong caller identity has no matching rule and never reaches the application.

Fine-grained authorization (can this user see this specific record, and which fields?) also runs at this point, via Cerbos. That's the resource and field-level layer, covered in a later post. This post stays on the identity flow.

Step 7: The Service Contract

After all of this, what does the service actually see? Two headers:

async def get_ms2_headers(
    x_ms2_user: str | None = Header(None),
    x_ms2_role: str | None = Header(None),
    x_request_id: str | None = Header(None),
):
    if not x_ms2_user or not x_ms2_role:
        raise HTTPException(status_code=401, detail="Missing required legacy headers")
    return {"user": x_ms2_user, "role": x_ms2_role, "request_id": x_request_id}

The service reads x-ms2-user and x-ms2-role. It returns 401 if they're missing. That's the entire identity contract. The 401 message calls them "legacy" headers, but the name only refers to this plain-header interface: the service reads simple headers as if it had no awareness of the mesh, and the mesh does the real verification upstream. The service does no JWT parsing, OIDC configuration, or Vault calls of its own. It trusts that if these headers are present, the sidecar has already done the verification.

Profile Aggregator forwards x-mesh-identity and x-request-id to downstream services. It explicitly does not forward x-ms2-user or x-ms2-role since the receiving sidecar projects those independently from the JWT. Profile Aggregator cannot influence what identity Employee Records sees.

Step 8: Database Identity Context

The final step. Before executing any query, the service sets the transaction context:

async def set_rls_context(session: AsyncSession, user_id: str, roles: str, request_id: str):
    await session.execute(
        text("SELECT set_config('app.current_user_id', :user_id, true)"),
        {"user_id": user_id},
    )
    await session.execute(
        text("SELECT set_config('app.current_roles', :roles, true)"),
        {"roles": roles},
    )
    if request_id:
        await session.execute(
            text("SELECT set_config('app.request_id', :request_id, true)"),
            {"request_id": request_id},
        )

PostgreSQL RLS policies read app.current_user_id and app.current_roles to filter rows; app.request_id is set for audit correlation. The true third argument makes each setting transaction-local, so it doesn't leak across pooled connections.

At this point, the database sees the same user identity that the sidecar projected from the verified token. In normal operation, the service is a pass-through for identity. It reads the sidecar-projected header and sets it on the transaction.

A fully compromised service process could call set_config with an arbitrary value. RLS doesn't protect against a malicious process controlling its own DB session. But the layers above constrain this: the compromised service only has its own scoped DB role (Employee Records can only access hr.* tables, Device Inventory only it.*), and AuthorizationPolicy prevents it from reaching other services' databases. The blast radius of a compromised Employee Records service is limited to the HR data it already has access to, scoped by RLS to the identities it actually receives requests for.

Failure Behavior

The secure path matters, but so does the failure path. A missing or invalid mesh token fails at AuthorizationPolicy because the required claims are absent. If Auth Service is down, ExtAuthz cannot approve the request and no mesh token is minted. If Vault is unreachable, Auth Service cannot sign a token. If the sidecar cannot refresh JWKS, it continues using its cached keys until the cache expires; after that, validation fails closed for tokens it cannot verify.

The one operational caveat is key rotation. Istio sidecars cache JWKS, so there is a short window where a sidecar may still verify tokens with a cached old public key. The next post covers that rotation window and why the token TTL is only 5 minutes.

The Complete Chain

Gateway strips external headers -> Auth Service validates credentials -> Auth Service checks route-level policy -> Auth Service mints a signed token -> sidecar validates the token and projects claims -> AuthorizationPolicy checks peer identity and token claims -> service reads projected headers -> database enforces row-level visibility.

Eight steps. By the time Employee Records reads x-ms2-user, that value has passed through header stripping, credential validation, route authorization, token signing, sidecar validation, and peer binding. The service receives a narrow identity contract and passes that identity into PostgreSQL for row-level enforcement.

What's Next

The next post covers the signing infrastructure: why Vault Transit instead of local keys, how key rotation works without downtime, and what the blast radius looks like when different components are compromised.

Code

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth

Service Mesh Zero-Trust Architecture

Kaustav Sarkar — Tue, 01 Sep 2026 18:37:58 +0000

Zero-trust is a property that emerges when several independent checks each verify a request on their own, without assuming the layers around them already did the work. It often gets described as a product you install or a setting you switch on, which badly undersells how much has to line up for it to actually hold.

I learned that building an HR directory designed to be attacked. The rule I wanted it to hold was simple: if any single pod gets compromised, it still can't read data it has no business touching. Salaries, bank details, device serial numbers, the things an attacker would actually go after.

No single mechanism enforces that rule. mTLS proves which service is on the other end of a connection, but it has nothing to say about whether that service should be making the call. A signed identity token establishes who the user is, yet a service that trusts raw headers will believe whatever an attacker sends it. Authorization decides what a user is allowed to do, and then an application bug skips the check and the database returns the whole table anyway. Every layer has a blind spot, and the thing that covers it is another layer that doesn't share the same one.

So the architecture is a stack: every hop independently verified, every request authenticated and authorized, each layer built on the assumption that the one before it might fail. Remove a single layer and the others are left exposed in the exact spots they were relying on it to cover.

This post is the architecture overview for the series: the application, the five security layers, how they compose into a single request flow, and what specifically breaks when you remove one. It all runs end-to-end on Istio on a local Kind cluster with real config. Later posts go deep into the YAML, Python, and policy files.

What you need to know going in: Istio is a service mesh for Kubernetes. The key idea is that every pod gets an Envoy sidecar proxy. All inbound and outbound traffic passes through the sidecar, which means identity verification, routing rules, and policy enforcement happen at the infrastructure layer before your application code sees the request. If you understand containers and basic networking, the rest will make sense as we go.

The Application

The domain is deliberately simple: an HR Employee Directory. Five microservices, a few database tables, a handful of roles. The security architecture is the interesting part. In the demo there are 4 roles used.

Alice (employee) — on her own profile she sees full PII and exact salary; everyone else only gets name, title, and department.
Mary (manager) — on Alice (her direct report) she sees salary band, not exact pay or SSN; on others, same public view as any employee.
Henry (HR admin) — full HR fields on all employees (salary, SSN, etc.), but no full IT asset serial numbers.
Ivan (IT admin) — only basic employee fields everywhere, but full hardware/asset details including serials.

The services:

Profile Aggregator (ms1-profile-aggregator) is a fan-out orchestrator that calls Employee Records and Device Inventory, merges the results, and returns a single profile response. Gateway-facing, requires authentication.
Employee Records (ms2-employee-details) is the source of truth for employee records, PII, and financial data (salary, bank details). Internal only, not directly reachable from outside.
Device Inventory (ms3-hardware-assets) tracks which devices are assigned to which employees (serial numbers, MAC addresses). Internal only, same as Employee Records.
Holiday Calendar (ms4-holiday-calendar) handles company holidays. Gateway-facing, requires authentication. Read by everyone, written by HR.
Office Directory (ms5-office-locations) serves office location data. Gateway-facing, no authentication required for reads. The only fully public endpoint in the system.

Six roles : employee, manager, hr_admin, it_admin, public_data_admin, and security_auditor. The first four drive the HR/profile flow and are the focus of this series. public_data_admin manages office and holiday data, and security_auditor has read-only access for compliance. The point that matters for security: each role sees a different slice of data from the same endpoints, controlled entirely by policy.

The tiering matters for security:

Tier 1 (gateway-facing): Profile Aggregator, Holiday Calendar, Office Directory. Reachable from the Istio ingress gateway.
Tier 2 (internal only): Employee Records, Device Inventory. Reachable exclusively from Profile Aggregator. No direct external path exists.
Tier 3 (data): PostgreSQL. Reachable only from specific service identities.

Holiday Calendar and Office Directory cannot reach Employee Records or Device Inventory regardless of what code runs inside them. Profile Aggregator also cannot connect directly to PostgreSQL. The mesh intercepts every connection attempt and checks whether the calling service is allowed to talk to the destination. If it's not on the allow-list, the connection is rejected before any application code even sees the request.

An attacker who compromises Holiday Calendar cannot reach Employee Records. The network topology itself is a security boundary, enforced by Istio AuthorizationPolicy.

The Threat Model

Every security layer exists because of a specific assumption about what can go wrong. These are the threats we're designing against:

An external user may send arbitrary headers, including internal trust headers, in their HTTP request.
Any single application pod may be compromised and act maliciously within whatever network access it has.
Application code may have bugs: missing authorization checks, overly broad queries, endpoints that skip validation.
Database queries may return more data than intended if the application doesn't filter correctly.
Signing keys and secrets must not live inside application pods, because any compromised pod leaks whatever it holds.

Each layer in the architecture addresses one or more of these. When we get to "What Happens When You Skip a Layer," you can map each failure directly back to this list.

The Layers

Five distinct security layers, each addressing an attack surface the others can't cover.

1. Transport Security (mTLS)

Mutual TLS encrypts data in transit and verifies the identity of both ends of the connection. Every pod in the mesh presents a SPIFFE identity certificate, and the receiving pod validates it before accepting any bytes.

This gives you encrypted communication and peer identity verification. A rogue service without a valid mesh certificate cannot establish a connection. But mTLS has a specific scope: it verifies that Service A is Service A. It does not determine whether Service A is allowed to call Service B with this particular payload for this particular user. That's a different problem entirely.

2. Identity Propagation (Signed Mesh Tokens)

Once a request enters the mesh and passes authentication at the gateway, the system needs to carry the user's verified identity through every downstream hop. A request from Alice hits Profile Aggregator, which then fans out to Employee Records and Device Inventory. Both downstream services need to know it's Alice, what role she holds, and that the request is legitimate.

This is done via a cryptographically signed JWT, the x-mesh-identity token, minted by Auth Service (auth-service) and signed by Vault Transit. The token carries who the user is (sub), what roles they hold (roles_csv), which services this token is valid for (aud), and who is acting on behalf of the user (act, for service-to-service delegation).

Any downstream service can verify this token using the public key from the Auth Service JWKS endpoint. No callback to Auth Service needed. But the token alone isn't sufficient. If projected headers like x-ms2-user arrive from outside the mesh without being stripped, a service reading those headers trusts attacker-controlled values. That's why header stripping at the gateway is non-negotiable, and why each destination sidecar also strips and re-projects these headers from the verified JWT before the request reaches the application.

3. Request Authorization (Policy Enforcement)

Knowing who someone is doesn't mean they're allowed to do what they're asking. Authorization in this architecture operates at four granularities, and each one catches what the layer above misses:

Network-level: Can this pod even reach that pod? Istio AuthorizationPolicy enforces which service identities are allowed to initiate connections.
Request-level: Is this user, with this role, allowed to call this endpoint? The ExtAuthz check at the gateway applies coarse route-level decisions.
Resource-level: Can this manager see this specific employee's record? Cerbos evaluates fine-grained, context-aware policies.
Field-level: Can this role see salary data, or should it be masked? Cerbos again, with policies that return which fields to include per role.

Why four layers instead of one? Because a single policy engine can't efficiently handle all of these. Network-level happens at the sidecar before any application code runs. Field-level requires knowing the actual resource attributes at query time. Different layers, different information available, different enforcement points.

4. Data Access Security (Row-Level Security)

The database is the last line of defense. PostgreSQL Row-Level Security policies enforce visibility at the engine level based on the current transaction context. Even if application code has a bug, a missing authorization check or a broken filter, RLS ensures the database itself drops rows the user shouldn't see.

Application developers make mistakes. A query like SELECT * FROM employees hitting production without a proper WHERE clause should still not leak the entire company directory. RLS makes that guarantee at the database layer, independent of whatever the application did or didn't check.

There is a trust boundary here: the application sets app.current_user_id on the transaction, and RLS uses that value. If the application sets it incorrectly, RLS enforces the wrong identity. In normal operation, the identity comes from sidecar-projected headers, so the application is a pass-through, not a source. But RLS does not protect against a fully compromised application process that sets arbitrary transaction context. RLS catches bugs (missing filters, broad queries). It does not stop a malicious process that controls its own DB session. The defense against that scenario is the layers above: AuthorizationPolicy limiting which pods can reach the database, scoped DB roles limiting which tables they can query, and Vault ensuring compromised pods can't escalate signing privileges.

5. Secrets Management (Vault)

Secrets in pods are the most common attack surface in Kubernetes. Environment variables, mounted files, ConfigMaps. Assume any compromised pod leaks whatever secrets it holds.

In this architecture, Vault serves a specific critical function: the Transit engine signs mesh identity tokens without ever exposing the private key to any service. Auth Service sends bytes to Vault, Vault signs them, returns the signature. The private key is never exposed to Auth Service or any application pod. If Auth Service is compromised, the attacker can ask Vault to sign things (until you revoke access), but they never get the key itself. They can't take it offline and mint tokens indefinitely.

Beyond signing, Vault is the natural home for database credentials, API keys, TLS certificates, and anything that would otherwise be scattered across pod specs. The pattern matters more than our specific usage: services request secrets through controlled APIs rather than holding them in memory at rest.

How the Layers Compose

These layers form a chain where each one depends on the guarantees of the ones around it. Here's what happens when a request enters the system.

An external request hits the Istio Gateway. TLS terminates. Before anything else, an EnvoyFilter strips all internal trust headers (x-mesh-identity, x-ms*-user, x-role-*), ensuring nothing spoofable survives from outside. This is the single most important security mechanism in the entire architecture. Without it, everything downstream can be bypassed by setting a header.

The gateway's ExtAuthz filter calls Auth Service. Auth Service validates either the session cookie (opaque, database-backed) or the Bearer token (validated against Keycloak's JWKS). If valid, it calls Vault Transit to sign a short-lived mesh identity token with a 5 minute TTL. This token comes back to Envoy as a response header and gets injected into the request.

Envoy routes the request to the target service. At the destination, a RequestAuthentication resource validates the x-mesh-identity token against the Auth Service JWKS endpoint. Istio's native outputClaimToHeaders projects the validated claims into service-specific headers (x-ms2-user, x-ms2-role). The service never touches the JWT directly. It just reads simple headers that the sidecar guarantees are legitimate.

AuthorizationPolicy at the destination checks both the mTLS peer identity (is the calling pod's service account allowed?) and the token claims (is the audience correct? is the delegation chain valid?). A token minted for Employee Records cannot be replayed against Device Inventory because the AuthorizationPolicy checks the aud claim. Both the source principal and the token audience must match for the request to proceed.

The service calls Cerbos for fine-grained authorization. Can this user, with this role, perform this action on this specific resource? Cerbos returns both an allow/deny decision and a list of visible fields.

Finally, the database query executes with RLS active. The application calls set_config('app.current_user_id', ...) on the transaction before querying, and PostgreSQL enforces row visibility at the engine level.

Header stripping, authentication, token minting, token validation with header projection, authorization policy, Cerbos, and RLS. Seven checkpoints. Each one reduces the blast radius of failures in the others.

What Happens When You Skip a Layer

Each layer exists because the others have blind spots.

Skip header stripping? An external attacker sets x-mesh-identity: <forged-token> in their HTTP request. The token won't pass signature validation at the destination sidecar. But the projected headers (x-ms2-user, x-ms2-role) are the real danger. If these arrive from outside without being stripped, and the service reads them directly, the attacker controls the identity. Header stripping ensures no internal trust header survives from outside the mesh, regardless of whether it's a signed token or a raw projected header.

Skip signed tokens (use raw headers instead)? Any compromised pod can forge headers. Without cryptographic signing, there's no way to distinguish a legitimate identity assertion from one injected by a rogue process.

Skip authorization policies? A valid user with a valid token can call any service and access any resource, regardless of whether they should. Authentication without authorization is an access-all-pass.

Skip RLS? A bug in your application's authorization logic, a missing check or a broken filter, becomes a data breach. The database has no independent enforcement, so it returns whatever the query asks for.

Skip Vault (use local keys)? If Auth Service is compromised, the attacker has the signing key. They can mint tokens for any user with any role, indefinitely, until you detect the breach and rotate. With Vault Transit, the key never leaves Vault. You revoke Auth Service access and rotate immediately.

Why a Service Mesh

The core idea: your application service has a simple contract. It reads x-ms2-user and x-ms2-role from request headers and does its business logic. It doesn't know about mTLS certificates, JWT validation, OIDC flows, or policy engines. The sidecar proxy handles all of that.

This matters for three reasons.

First, you can add services to the mesh without rewriting their auth logic. The security infrastructure wraps around them.
Second, when the security architecture evolves (new signing algorithm, new policy engine, new identity provider) application code doesn't change. Only mesh configuration does.
Third, application developers focus on domain logic while security is configured at the infrastructure layer. These concerns don't bleed into each other.

The complexity is real. You need Kubernetes knowledge, you need to understand Envoy, you need to debug sidecar injection and filter ordering. But the tradeoff is concrete: instead of implementing auth in every single service (and maybe getting it wrong in one of them), you configure it once at the infrastructure layer. The complexity is upfront and centralized instead of distributed across every service where it's invisible until it fails.

What's Next

The next post peels back the identity flow, from the gateway EnvoyFilter that strips headers, through the ExtAuthz check, to the signed mesh token landing on a downstream service. Real YAML, real Python, real attack scenarios.

Code:

https://github.com/Kaustav-Sarkar/Istio-Defense-In-Depth