Namespaces are just labels on a door: building a zero-trust lab on my laptop

Two namespaces, two tenants, problem solved. It's the assumption I see most often when teams start sharing a cluster, and it's wrong.

A namespace is a naming boundary. A pod in tenant-b can still reach a Service in tenant-a by DNS name, and Kubernetes won't object, because by default the network is flat.

[TODO: if you used to believe this yourself, say so in one sentence here. If not, delete this line.]

I wanted to see the problem and then close it, so I built a small lab on my machine. This post walks through what's in it, what each layer actually does, and the places where the behaviour surprised me. Everything is in the repo: kubernetes-multi-tenant-zero-trust-lab.

One caveat up front. The lab runs on Kind with Docker Desktop on Windows: one control-plane node, two workers, Istio on top. It is not GKE. I give a talk on this topic that covers GKE Enterprise and Binary Authorization, but this repo demonstrates the ideas locally, and I'll point out where that difference matters.

The setup

Each tenant gets its own namespace with Istio sidecar injection on. tenant-a runs a frontend and a backend, two replicas each, and every pod has an Envoy sidecar next to the app container. Each workload runs under its own ServiceAccount: frontend-sa and backend-sa.

That last detail matters more than it looks. In a mesh, a workload's identity is its ServiceAccount. If everything runs as default, you can't tell workloads apart, and every policy you write afterwards is vague. "One ServiceAccount per workload" is step one of the rollout checklist at the end of this post.

An IP address is not an identity

The old way to restrict traffic is an IP allow-list. In Kubernetes, pod IPs get recycled within seconds, so an IP tells you where a packet came from and nothing about who sent it.

Istio gives each workload a short-lived certificate that encodes who it is: trust domain, namespace and service account. The private key stays in the pod's memory and the certificate rotates on its own (typically about a day), so there is no long-lived credential to leak.

Turning on enforcement takes one small resource:

kubectl get peerauthentication -n tenant-a
NAME      MODE
default   STRICT

STRICT means the sidecars refuse any connection that isn't mutual TLS. Both sides have to prove who they are.

mTLS does not authorize anything

This is the part that catches people out, and the demo exists to show it.

tenant-b is also in the mesh, so its pods have perfectly valid certificates. mTLS tells the backend who is calling. It says nothing about whether that caller should be allowed in. Authentication and authorization are separate jobs, and mTLS only does the first.

[TODO: one or two sentences on your own reaction the first time you saw tenant-b get through the handshake, or what you expected to happen.]

The second job belongs to AuthorizationPolicy. Mine says four things about the backend:

• it applies to the backend pods
• the caller must be the frontend-sa identity
• the request must carry a valid JWT principal
• only GET is allowed

Then I ran a throwaway curl pod from tenant-b against the backend:

kubectl run tenant-b-demo -n tenant-b --image=curlimages/curl --restart=Never --command -- curl -sS -o /dev/null -w "%{http_code}\n" http://backend-service.tenant-a.svc.cluster.local

403

The request reached the backend, and the backend's sidecar turned it away. The network said yes and the identity layer said no. That's the moment in the demo where people tend to nod.

401 and 403 mean different things here

RequestAuthentication validates JWTs, so it's natural to assume it rejects requests that have no token. It doesn't. It only checks a token if one is present. Here's what the frontend pod gets back:

Request from the frontend pod	Result
No token	403
Garbage token (invalidToken)	401
Valid token	200
Valid token, DELETE method	403

A missing token gets a 403 because the AuthorizationPolicy demands an authenticated principal and finds none. A broken token gets a 401 because it fails validation before authorization is even evaluated. So when you're debugging a mesh and see a 403, you can't tell from the code alone whether the caller is wrong, the token is missing or the method is disallowed. You have to look further.

[TODO: how long the 401/403 difference took you to work out, or what you checked first. Delete if it didn't trip you up.]

The last case shows that even the right workload with a valid token can't do everything. frontend-sa can GET and that's all. This is the granularity NetworkPolicy can't give you, because it works at layers 3 and 4 and can't tell a GET from a DELETE. I'd still keep NetworkPolicy as an outer fence and add the mesh on top.

For the demo I used Istio's public sample JWT. That's fine in a lab and not something to copy into production.

Reading what the sidecar saw

Encrypted traffic is opaque on the wire. But the sidecar terminates the TLS, so it sees the method, the path and the caller, and it can log them:

kubectl logs $backendPod -n tenant-a -c istio-proxy --since=30m

The 200s and 403s are right there in those logs. Envoy also keeps counters, and I like these because they show enforcement actually happening:

kubectl exec $backendPod -n tenant-a -c istio-proxy -- pilot-agent request GET stats | Select-String "rbac.allowed|rbac.denied"

Send another request from tenant-b and rbac.denied goes up. One gotcha: it goes up on whichever replica handled the request, so with two backend pods you may need to check both.

Pinning images, and what that does and doesn't prove

The last layer is about what's allowed to run. If an attacker can deploy their own container under a trusted ServiceAccount, they inherit that identity. Identity controls only work if the workloads behind them are trustworthy.

In the lab, both Deployments reference the image by SHA-256 digest instead of a tag like latest, which pins the exact image content. A ValidatingAdmissionPolicy then rejects any Deployment in tenant-a whose images don't use a digest:

kubectl create deployment image-test --image=nginx:latest -n tenant-a --dry-run=server

All container images must use a SHA-256 digest.

The properly pinned manifest passes a server-side dry run.

I want to be careful about how much this shows. Digest pinning tells you which bytes you're running. It doesn't tell you who built them or whether you should trust them. My admission policy doesn't verify signatures, attestations or provenance, and it is not Binary Authorization. That product checks signed attestations on the exact digest at admission time, with signing keys held in a KMS. In a real GKE setup the pipeline builds the image, scans it, signs the digest, and the admission controller refuses anything without a valid attestation. The lab covers only the first, simplest step of that chain, and I'd rather say so plainly than let the demo imply more.

If you roll this out for real

I'd go in this order:

1. One ServiceAccount per workload, before anything else.
2. Inject sidecars and run mTLS in PERMISSIVE mode first.
3. Watch the telemetry to find who still sends plaintext, then go STRICT namespace by namespace.
4. Default-deny each tenant namespace, then grant access explicitly.
5. Add layer-7 and JWT rules, using Istio's dry-run annotation first so a new policy only logs what it would deny.
6. Bring in image signing in audit mode before you enforce it.

Each step is reversible and observable, which is how you do this in production without taking something down on a Friday.

The short version

Identity beats IP addresses. mTLS authenticates but doesn't authorize. And a trusted identity is only as safe as the image running behind it.

The commands, manifests and a rehearsal runbook are in the repo. If you spot something I got wrong, open an issue.

Part 2 with GKE coming up...