A regulated launch in New Jersey needed every service-to-service connection encrypted and mutually authenticated. The requirement had been open for a year and a half. I joined the team to close it, and three months later the mesh was ready as a building block: any team joins in about ten minutes, without changing a line of application code.
Here is how it is built, the details that decided whether it worked, and a clean-room reference implementation: github.com/devopsmher/istio-ambient-building-block.
The requirement in engineering terms
The launch fell under New Jersey’s internet gaming rules and the GLI-19 standard. For the platform that meant four things:
- Every connection between services encrypted and mutually authenticated.
- Deny by default : a service talks only to callers that are explicitly allowed.
- No bypass : plain-text traffic must not reach a workload, even from inside the cluster.
- Evidence : something an auditor can check, and that we can re-check after every change.
Why ambient mode
The classic Istio data plane puts a sidecar proxy into every pod. That works, but for dozens of teams it turns the mesh into an application change: sidecar injection, rolling restarts, extra CPU and memory in every pod, and every team in the critical path.
Ambient mode moves the proxy out of the pod:
- ztunnel , one per node, carries mTLS for every pod of an enrolled namespace and enforces L4 authorization by workload identity.
- Waypoints add L7 features (HTTP authorization, retries, header routing) only for the namespaces that need them.
- Joining is a label : istio.io/dataplane-mode=ambient on the namespace. Leaving is removing it. No restarts in either direction.
That made the mesh an infrastructure change, owned by the platform, instead of a migration that every team had to schedule.
The pieces
Terraform module (istio_enabled)
└─ istiod · istio-cni · ztunnel · STRICT mTLS · shared Gateway · Kiali
Team values file → Argo CD ApplicationSet
├─ mesh-namespace : ambient label · default-deny · allow rules · NetworkPolicy
├─ mesh-routes : Gateway API HTTPRoutes to the shared gateway
└─ mesh-egress : external hosts as ServiceEntries
The whole mesh sits behind one Terraform feature flag, istio_enabled, off by default. Turning it on installs Istio and changes nothing for running workloads: namespaces still opt in one by one.
North-south traffic uses the Kubernetes Gateway API: one shared Gateway, and an HTTPRoute per team route. The same resources work with other Gateway API implementations, so teams learn one model.
Two layers of defense
Identity. Every workload has a SPIFFE identity such as cluster.local/ns/orders/sa/checkout. Each team namespace starts with a default-deny policy and allows only the callers listed in its values file:
allowFrom:
- namespace: payments
- namespace: istio-ingress
serviceAccounts: [public-istio]
- namespace: orders
serviceAccounts: [checkout]
Transport. A mesh-wide PeerAuthentication enforces STRICT mTLS, and a NetworkPolicy in every enrolled namespace admits only mesh traffic:
ingress:
- ports:
- port: 15008 # HBONE: the mTLS tunnel between ztunnels
protocol: TCP
- from:
- ipBlock:
cidr: 169.254.7.127/32 # kubelet probes, IPv4
- ipBlock:
cidr: fd16:9254:7127:1337:ffff:ffff:ffff:ffff/128 # kubelet probes, IPv6
Istio decides who may connect; the CNI (Calico in our case) makes sure nothing reaches a pod around the mesh.
Self-service onboarding
A team writes one values file and opens a pull request. CI renders the three charts with it and validates the result against the Kubernetes, Istio and Gateway API schemas, so a typo fails in the pull request, not in the cluster. After the merge, an Argo CD ApplicationSet creates the team’s Application, and within minutes:
- the namespace is in ambient mode;
- default-deny and the allow rules are in force;
- the routes are attached to the shared gateway;
- the external hosts are registered as ServiceEntries.
Leaving the mesh is the same file with enrolled: false. In the reference implementation the team namespace is also protected from deletion: it carries helm.sh/resource-policy: keep and an Argo CD Delete=false sync option, because a mistake in a GitOps repository should never delete a team's workloads.
A validation plan as a deliverable
Before the first team joined, I wrote a validation plan: a table of scenarios, each with a command and an expected result.
- ingress through the shared gateway;
- a route from a namespace that is not allowed to attach to the gateway;
- east-west traffic from an allowed caller and from an unknown one;
- plain text from outside the mesh;
- egress to an external API and to an external database;
- health probes after enrolment;
- the same checks through a waypoint;
- rollback.
It became the most useful document of the project: we ran it after the PoC and before handing the mesh over, it is the checklist for every upgrade, and its output is exactly the evidence an auditor asks for.
Details that decide whether it works
- NetworkPolicy and ambient. Mesh traffic arrives on port 15008, and kubelet probes come from fixed link-local addresses after istio-cni rewrites them. Allow both families of probe addresses, or pods go unready the moment they join the mesh.
- Waypoints change who the caller is. With a waypoint, ztunnel sees the waypoint’s identity, not the original client’s. Put the caller rules on the waypoint, and let ztunnel admit only the waypoint, so nothing can skip it.
- Ingress bypasses waypoints by default. Label the namespace with istio.io/ingress-use-waypoint=true and enable ENABLE_INGRESS_WAYPOINT_ROUTING in istiod; otherwise ingress traffic misses the L7 policy, and a ztunnel that admits only the waypoint rejects it.
- Egress is explicit. Every external API and database gets a ServiceEntry visible only in its namespace. Databases speak TCP, so test them separately from HTTPS APIs.
- No big bang. A feature flag for the cluster, an opt-in label per namespace and a one-line rollback made it possible to start with one team and grow without a freeze.
- Pin versions. One Istio minor version at a time, the Gateway API CRDs that this Istio release documents, and the validation plan after every upgrade.
Observability and handover
The mesh came with Kiali behind single sign-on, Istio dashboards in the shared Grafana, tracing through waypoints and gateways, and alerts with short runbooks, for example: istiod unavailable, ztunnel not ready on a node, a spike of denied connections, 5xx growth at the gateway.
At the end, I wrote the HOWTO and handed the mesh over to the platform team as a ready building block.
Results
- The service mesh the launch had been waiting for 1.5 years, delivered in 3 months.
- About ten minutes for a team to join, with no application code changes.
- STRICT mTLS and default-deny across the mesh, with a validation plan as evidence.
Try it
The reference implementation is on GitHub: devopsmher/istio-ambient-building-block. It is a clean-room rewrite of the pattern with no code from the original project: the Terraform module behind the feature flag, the three team charts with JSON schemas, the Argo CD ApplicationSet, the validation plan and the runbooks. Everything renders and validates in CI without a cluster.
Mher Poghosyan is a Senior DevOps & Platform Engineer in Yerevan, Armenia. He builds Kubernetes platforms on AWS and GCP. LinkedIn · GitHub _· [_Medium](https://medium.com/@mher.poghosyan)
Top comments (0)