AWS is retiring App Mesh, with end of support in September 2026. Our whole platform ran on it, so we had no choice. We had to move.
We did not pick the replacement by instinct. We ran an evaluation first and wrote it up as a decision record. We listed the candidates, agreed on the criteria that mattered to us, gave each criterion a weight, and scored every option against them. Then we built a proof of concept for the shortlist. Istio Ambient Mesh came out on top.
Choosing the tool was the easy part. Before we changed anything in any cluster, we wrote the plan. Not a rough idea in someone's head. A real document, reviewed by the team.
It answered a small number of questions, in detail. Which service moves in which order, and why. What has to be ready before the first service moves. What the exact steps are for one single service. How to roll back at every step. And which step is the first one we cannot undo.
Writing that plan took a long time. It was the best investment in the whole project. Almost every problem in this post was found while planning, not while deploying. Finding them on paper is much cheaper than finding them in production.
Then came the migration. Around 40 microservices, three environments, multiple AWS regions. We went slowly over a few months, and that was on purpose. Safety mattered more than speed. We spent the time on validating each option before deploying, and on verifying the system after every change. The hard part was working out how two service meshes can live next to each other while we move services one by one.
We finished with no outage and no major incident.
This post is about the problems we hit and how we solved them.
Where we started
Our setup was a normal AWS App Mesh deployment:
- Every pod ran an Envoy sidecar, injected by the App Mesh controller.
- Services talked to each other with mTLS. The certificates were files mounted into the sidecar.
- Service discovery used Cloud Map. A service was reachable at a name like
service.env.internal. - External traffic came in through API Gateway, then a VPC Link, then an internal NLB, then an App Mesh Virtual Gateway, then the service.
One thing in that list is easy to miss. App Mesh was doing two different jobs for us.
The first job was the service mesh. Mutual TLS between services, and rules about which service is allowed to call which. That is the inside of the cluster.
The second job was the gateway. Taking external traffic from API Gateway and routing it to the right service. That is the edge.
One product owned both. We never thought about it that way until we had to replace it.
This worked well for years. The sidecar model was heavy, but it was stable and we knew it.
Where we wanted to go
The replacement is not one tool. It is two, because we split the two jobs apart.
The edge job moved to a Kubernetes gateway built on the Gateway API. The mesh job moved to Istio Ambient Mesh.
Why this combination:
- No sidecar. Ambient mode runs one proxy per node, called ztunnel, and it handles mTLS for every pod on that node. Instead of an Envoy container inside every pod, there is one shared proxy. That removes a lot of duplicated CPU and memory across the fleet, and it removes the sidecar lifecycle problems we had in jobs.
- Standard routing. Gateway API is a Kubernetes standard, not a cloud provider format. Our routing config is portable and any engineer can read it. It also gives us things we did not have before, like traffic mirroring and A/B testing.
- The two layers are now independent. We can upgrade or replace the gateway without touching the mesh, and the other way around. While one product owned both jobs, that was not possible.
Two phases
We split the work into two phases, and we did not mix them.
Phase 1: gateway cutover. Move external traffic from the App Mesh Virtual Gateway to a Kubernetes gateway that speaks the Gateway API. Every sidecar stays exactly as it is. Service to service traffic does not change at all.
Phase 2: mesh migration. Move each service off its App Mesh sidecar and onto ambient mesh, one wave at a time.
The order matters. Phase 1 does not touch the data plane of any service. If something goes wrong, we point the ingress back and the inside of the cluster is untouched. It also left the old mesh gateway running with no traffic on it for a long time, which is a very cheap rollback.
The plan looked simple on paper. It was not.
Problem 1: the two meshes cannot share a pod
This was the first blocker, and it shaped everything else.
The old mesh authenticates with certificate files issued by its own CA. Ambient mesh uses SPIFFE identities, which are certificates issued by Istio's CA, and wraps the connection in an HTTP/2 tunnel called HBONE.
The two CAs do not trust each other. A connection from an App Mesh Envoy to a ztunnel fails the TLS handshake.
So there is no "run both and shift traffic slowly" option for a single service. Each service flips in one step: old mesh off, new mesh on, at the same time.
That is fine for one service. The hard part is that during the migration, the cluster has both kinds of services at once, and they still need to talk to each other.
Problem 2: making the two meshes talk during the migration
We needed both directions to work while the migration was in progress.
Old mesh calling new mesh. In App Mesh, a caller lists its backends, and each backend entry can require TLS. If we remove the callee from that list, the caller falls back to plain TCP. On the other side, we set the new service to PERMISSIVE mode. In PERMISSIVE mode, ztunnel accepts both HBONE connections and plain TCP. So the call goes through.
New mesh calling old mesh. ztunnel always wraps outbound traffic in HBONE. To the old service, this arrives as a normal TCP connection. We set every App Mesh listener to PERMISSIVE, which makes Envoy accept both its own TLS and plain TCP. So this direction also works.
PERMISSIVE mode comes with a tradeoff. While one side is still on the old mesh, that connection does not get the mutual TLS that both sides normally enforce. We planned for this and kept it small. It applied only to internal pod to pod traffic inside a private network. It applied only to the pair of services in the middle of a flip. And it lasted only until the caller moved too, usually a day or two. Small waves kept the window short.
Problem 3: the authorization policy that silently blocks everything
This one nearly caused an incident, and it is not obvious.
Istio lets you add an L4 authorization policy that only allows specific SPIFFE identities to reach a workload. We had tooling that could generate this policy for a service.
The trap is how Istio evaluates it. An ALLOW policy is deny by default. Anything that does not match a rule is rejected.
Old mesh services have no SPIFFE identity. Their connections carry no identity at all. So they match nothing, and they get denied. This happens even in PERMISSIVE mode, because PERMISSIVE controls encryption, not authorization. They are two different layers, checked in order.
The rule we followed:
Do not create any authorization policy while any caller is still on the old mesh. Add them only after everything is migrated.
We only turned on identity based authorization at the very end, as one batch, once every caller could present a SPIFFE identity.
Problem 4: the front door (phase 1)
External traffic reached us through API Gateway, and API Gateway connects to an internal NLB through a VPC Link.
A VPC Link is bound to a specific NLB. You cannot repoint it. Our VPC Links pointed at the App Mesh Virtual Gateway NLB, and the new gateway had its own NLBs. So we created new VPC Links for the new gateway.
Before switching anything, we added the routes to the new gateway and left the old gateway routes in place. Both gateways were live at the same time. That let us call each service directly on the new gateway NLB and confirm it answered correctly, while real traffic still went the old way.
The new NLB was internal only, with no public exposure. Only the team running the migration could reach it from inside the network to do these checks. There was no way for a user to land on the new path before we switched. So the new gateway could be fully live and fully tested while it still served nobody.
Most of the risk was removed before the switch itself.
Then we changed the API Gateway integrations to point at the new VPC Links. Non production first, then production, with verification between them.
Public DNS did not change at all. DNS points at the API Gateway custom domain, and the swap happened inside API Gateway. Rolling back meant pointing the integrations at the old VPC Links again.
We did all of this before touching any sidecar. After the cutover, the old Virtual Gateway was still running but served no traffic. That gave us an easy rollback for a long time.
Problem 5: hardcoded service addresses
Services did not talk to each other by Kubernetes service name. They looked up an address from a parameter store, and that parameter held the Cloud Map name of the old mesh.
If we had left this until each service migrated, every flip would also change the address callers use. Too many moving parts at once.
So we separated it. We changed the parameter value to the Kubernetes service DNS name for every service first, before enrolling anything into the new mesh. Kubernetes DNS works whether the service is in the old mesh, the new mesh, or no mesh at all. So this change was safe on its own.
Two things we learned here:
- The indirection saved us. Because addresses came from a parameter store, we changed them without touching application code.
- Consumers must restart to pick up a new value. That step is easy to forget, and it is silent when you forget it.
The order of migration (phase 2)
We drew the dependency graph from our deployment config and migrated in waves, leaves first.
- Leaf services. Nothing calls them inside the mesh. No coordination needed.
- Shared foundation services. Called by many, call few.
- Mid tier services.
- Core hubs, including the main platform API.
- The auth service, last.
The auth service was last for a specific reason. An external authorizer component called it directly, outside the cluster, through the old mesh gateway. If we had migrated auth early, that path would break and every request would fail authorization. So we updated that component to use the new gateway first, deployed it, verified it, and only then migrated the auth service, inside a maintenance window.
Each service got a soak period, 48 to 72 hours, before we moved on.
The result
The platform now runs fully on ambient mesh with strict mTLS and identity based authorization. The old mesh is being decommissioned.
We migrated a complex, live system with around 40 services across all environments and regions, with no outage and no major incident. Users did not notice.
What I would tell someone starting this
Write the plan before you touch anything. This is the one that mattered most. The order of services, the prerequisites, the exact steps for one service, the rollback for every step, and the point where rollback is no longer possible. Writing it feels slow, and it feels like you are not making progress. You are. Each change on its own is small. The hard part is knowing what must be ready first, so that the change is safe. The plan is where you work that out.
Find out what the old system actually does. Ours was doing two separate jobs, edge routing and service to service security. We only saw that clearly once we started planning. After we split the jobs, we could migrate each one on its own, with its own rollback. If we had treated it as one replacement, it would have been one big cutover.
Do not choose the replacement by instinct. Write down what matters to you, give each criterion a weight, and score the options against it. Then build a proof of concept for the shortlist. The scoring narrows the list. The proof of concept is what actually decides.
Change one layer at a time. Gateway first, mesh second, never both together. When something broke, we always knew which layer to look at.
Accept the atomic switch. You cannot slowly shift a single service between two meshes. Plan the compatibility window instead of trying to design the constraint away.
Encryption mode and authorization are separate layers. PERMISSIVE does not make an ALLOW policy permissive. This is the one that will surprise you.
Leaves first, hubs last. And find the service that something outside the cluster depends on. That one goes last, and it needs its own plan.
A forced deadline is not only bad news. We did not choose to do this. But the stack we have now is better than what we had, and we would not have moved on our own.
Top comments (0)