DEV Community

Cover image for Kubernetes Networking Was Easy — Until Production Got Real
Kubernetes with Naveen
Kubernetes with Naveen

Posted on

Kubernetes Networking Was Easy — Until Production Got Real

#kubernetes #devops #microservices #iservicemesh

When I first started managing microservices at scale, Kubernetes felt like magic — until it didn’t. Pods restarted, requests vanished into the void, and tracing cross-namespace traffic became a weekend sport.

Spotify

At first, the networking model seemed almost too simple.

A Pod gets an IP. A Kubernetes Service gives that Pod a stable virtual endpoint. CoreDNS handles service discovery. The CNI provides pod-to-pod connectivity. kube-proxy handles service traffic using mechanisms such as iptables or IPVS.

For a small cluster, that model works remarkably well. Then you have 150 services. Then 500.

Then teams start deploying independently, services communicate across namespaces, traffic crosses availability zones, a few workloads become extremely chatty, and somebody introduces three layers of retries because the network is sometimes flaky.

That's when the networking stops being infrastructure you can ignore. You start asking questions that Kubernetes Services alone don't really answer:

  • Which service is calling this endpoint?
  • Why did this request get a 503?
  • Which version received it?
  • Is the connection failing, or is the application returning the error?
  • Why did latency jump only for traffic crossing namespaces?
  • Is this workload actually talking to the service it thinks it is?

This is where a service mesh starts making sense. And this is also where you discover that a service mesh isn't free.

Twitter

The Architectural Shift: From Kubernetes Services to a Service Mesh

Kubernetes gives you the basic network primitives. The CNI establishes the network between Pods. A Service provides a stable virtual IP and load-balancing abstraction. kube-proxy programs the node networking rules required to direct Service traffic toward backend Pods.

That's enough to answer:

How does service-a reach service-b?

But production systems eventually ask:

How should service-a reach service-b?

Those are very different questions. Suppose payments has two versions:

payments-v1 → 90%
payments-v2 → 10%
Enter fullscreen mode Exit fullscreen mode

Kubernetes Services don't natively give you application-aware traffic splitting based on HTTP headers, cookies, weights, or request properties. You can create separate Services, manipulate Deployments, introduce an ingress controller, or build application-level routing logic. But now routing logic starts leaking into multiple layers. Istio moves much of that policy into the networking layer.

The basic architecture becomes:

Istio Control Plane

                               │
                  Configuration / Certificates
                               │
                               ▼
                        ┌─────────────┐
                        │    Envoy    │
                        │   Proxies   │
                        └─────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
   service-a               service-b               service-c
   + Envoy                 + Envoy                 + Envoy
Enter fullscreen mode Exit fullscreen mode

The important distinction is between the control plane and data plane. The Istio control plane manages configuration and security material. The Envoy proxies sit in the traffic path and actually handle requests. That distinction matters enormously when debugging.

The control plane can be struggling while existing Envoy configuration continues serving traffic perfectly well. Understanding that separation saves a lot of unnecessary panic during incidents.

The Kubernetes Layer Still Matters

One mistake I see repeatedly is treating Istio as if it replaces Kubernetes networking. It doesn't. Your underlying Kubernetes networking still needs to work.

A typical request might look like:

Pod A
  │
  ▼
Envoy sidecar
  │
  ▼
ClusterIP
  │
  ▼
Kubernetes networking
  │
  ▼
Envoy sidecar
  │
  ▼
Pod B
Enter fullscreen mode Exit fullscreen mode

The mesh adds another layer of behavior on top of Kubernetes networking. That means there are now more places where things can go wrong.

DNS → Service → CNI → routing → Envoy listener → Envoy cluster → application

When someone says:

The service is reachable.

That statement is almost meaningless without knowing from where, through what path, and at which layer.

Ingress Isn't Service-to-Service Networking

Another production lesson is keeping north-south and east-west traffic conceptually separate. North-south traffic is traffic entering or leaving the cluster. East-west traffic is traffic between workloads inside the cluster.

An Istio Ingress Gateway handles the former.

For instance:

Internet
   │
   ▼
Load Balancer
   │
   ▼
Istio Ingress Gateway
   │
   ▼
VirtualService
   │
   ▼
service-a
Enter fullscreen mode Exit fullscreen mode

Inside the cluster:

service-a
   │
   ▼
Envoy
   │
   ▼
service-b
   │
   ▼
Envoy
Enter fullscreen mode Exit fullscreen mode

These are different traffic-management problems. The ingress gateway is your controlled entry point. The sidecars or ambient data plane handle service-to-service communication. Mixing those responsibilities makes architecture and troubleshooting unnecessarily difficult.

1. Securing the Perimeter: Ingress and Egress

A production cluster shouldn't have every workload freely reaching the internet. Direct Pod egress looks convenient:

Pod
 │
 ├── api.example.com
 ├── payment-provider.com
 ├── random-third-party.com
 └── anything-else
Enter fullscreen mode Exit fullscreen mode

The problem isn't simply security. It's control.

When something goes wrong, you want to know:

  • Which workload made the connection?
  • Where did it connect?
  • Was the destination approved?
  • What protocol was used?
  • Can we block it centrally?
  • Can we observe the traffic?

An Istio Egress Gateway gives you a controlled exit point:

Pod
 │
 ▼
Envoy
 │
 ▼
Egress Gateway
 │
 ▼
External Service
Enter fullscreen mode Exit fullscreen mode

Now outbound traffic can be governed at a predictable boundary. That doesn't mean every organization needs to force every packet through an egress gateway. You pay for centralized inspection with additional hops, infrastructure, configuration, and failure modes.

The right question isn't:

Can we put everything through the egress gateway?

It's:

Which external traffic actually needs centralized policy and visibility?

That distinction matters.

2. mTLS: Zero Trust Without Breaking Everything

One of Istio's strongest capabilities is mutual TLS.

Instead of:

service-a ───── HTTP ─────> service-b
Enter fullscreen mode Exit fullscreen mode

you can have:

service-a
   │
 Envoy
   │
   │ mTLS
   ▼
 Envoy
   │
service-b
Enter fullscreen mode Exit fullscreen mode

The application doesn't necessarily need to manage certificates itself. The mesh handles identity and encryption between workloads. But switching an existing production environment directly to:

mode: STRICT
Enter fullscreen mode Exit fullscreen mode

can turn a quiet Tuesday into a very long night. Why? Because not everything is necessarily inside the mesh.

You might have:

mesh workload
     │
     ▼
legacy service
     │
     X
   TLS required
Enter fullscreen mode Exit fullscreen mode

The legacy workload doesn't have an Envoy sidecar and therefore cannot participate in mesh mTLS in the same way.

This is where PeerAuthentication modes matter.

PERMISSIVE

Accept both plaintext and mTLS. Useful during migration.

mTLS ────────┐
             ├──> workload
plaintext ───┘
STRICT
Enter fullscreen mode Exit fullscreen mode

Require mTLS.

plaintext ───> rejected
mTLS ────────> accepted
Enter fullscreen mode Exit fullscreen mode

A safer migration looks like:

Phase 1
PERMISSIVE
   ↓
Inject sidecars
   ↓
Verify workload communication
   ↓
Identify legacy clients
   ↓
Migrate dependencies
   ↓
STRICT
Enter fullscreen mode Exit fullscreen mode

Don't turn on STRICT because the architecture diagram says everything is meshed. Production traffic doesn't care what the architecture diagram says.

3. Traffic Shifting Without DNS Games

Canary deployments are another area where Istio becomes extremely useful. Without mesh-level routing, teams sometimes create:

payments-v1.example.com
payments-v2.example.com
Enter fullscreen mode Exit fullscreen mode

and manipulate DNS or load balancers. DNS isn't designed to provide precise request-level traffic control. Caching, TTLs, resolvers, client behavior, and connection reuse all get involved. Istio lets you shift traffic directly at the request-routing layer.

A VirtualService can express something conceptually like:

payments
   │
   ├── v1 → 90%
   │
   └── v2 → 10%
Enter fullscreen mode Exit fullscreen mode

Then:

90% → payments-v1
10% → payments-v2
Enter fullscreen mode Exit fullscreen mode

You can gradually move:

100 / 0
  ↓
95 / 5
  ↓
90 / 10
  ↓
75 / 25
  ↓
50 / 50
  ↓
0 / 100
Enter fullscreen mode Exit fullscreen mode

The interesting part isn't the YAML. It's what you can do without changing application code or DNS. But traffic splitting introduces another responsibility: knowing what you're actually measuring. If v2 receives 10% of traffic but happens to receive the most expensive customer requests, raw request percentages can become misleading.

Traffic management is easy. Traffic management with meaningful telemetry is the real engineering problem.

DestinationRules: Where Routing Gets Interesting

A VirtualService describes how requests should be routed. A DestinationRule describes policies applied to traffic going toward a destination.

This is where things such as subsets, connection pools, circuit breakers, outlier detection, and TLS behavior start becoming relevant.

For instance:

payments
   │
   ├── subset: v1
   │
   └── subset: v2
Enter fullscreen mode Exit fullscreen mode

VirtualService: 

route 90% → v1
route 10% → v2
Enter fullscreen mode Exit fullscreen mode

DestinationRule: 

v1 → connection policy
v2 → connection policy
Enter fullscreen mode Exit fullscreen mode

That separation becomes valuable once routing rules grow beyond simple send traffic here. It also becomes a source of configuration complexity. We'll get to that.

The Battle Scars: The Istio Tax

Let's talk about the part nobody gets excited about during the architecture presentation. Every sidecar consumes resources. One proxy doesn't sound like much. Now multiply it.

500 workloads
×
1 Envoy proxy
=
500 additional processes
Enter fullscreen mode Exit fullscreen mode

At low traffic, this can look harmless. Under high concurrency, it isn't. Envoy maintains connections, buffers data, processes HTTP, performs TLS operations, tracks metrics, handles filters, and maintains configuration.

Memory consumption can become particularly painful. Imagine a cluster where application Pods were sized carefully:

Application:
500Mi memory

Envoy:
250Mi memory
Enter fullscreen mode Exit fullscreen mode

Suddenly your 500 MiB application workload isn't a 500 MiB workload anymore. It's closer to: 750MiB+. And that changes: node packing, autoscaling, eviction pressure, cluster cost, and pod startup behavior

CPU can also spike under heavy request rates, TLS operations, logging, or complex filters. This is why blindly enabling sidecars everywhere is dangerous.

Control the scope

Istio's Sidecar resource can be used to constrain the configuration visibility available to workloads. That's important in larger environments.

A proxy doesn't necessarily need configuration for every service in the cluster. If a workload can only communicate with payments, orders, and identity there's little reason for its Envoy to carry unnecessary configuration for hundreds of unrelated services.

Reducing configuration scope can improve both resource usage and operational clarity.

The exact optimization depends on the cluster, but the principle is simple:

Don't make every proxy understand the entire universe if it only needs to understand three services.

The Battle Scars: The Danger of Naive Retries

his one has caused some spectacular incidents. Let's imagine service-a → service-b, service-b becomes slow. So service-a retries.

Now imagine 100 clients doing the same thing. Then each client has three retries. You can accidentally transform 1,000 requests into 4,000 requests when the dependency is already struggling.

That's a retry storm. And it gets uglier with multiple layers.

Frontend
  ↓ retry × 3
Service A
  ↓ retry × 3
Service B
  ↓ retry × 3
Service C
Enter fullscreen mode Exit fullscreen mode

One failed request can explode into a ridiculous number of downstream attempts. The service mesh makes retries easy to configure. That doesn't mean you should configure them everywhere.

Retries should be deliberate. So use bounded retry counts, exponential backoff, jitter, appropriate timeout budgets, retry budgets, circuit breaking, and outlier detection

The idea behind a retry budget is particularly important: retries should consume only a controlled fraction of normal traffic rather than being allowed to multiply without bound. And never retry operations blindly. Retrying a failed GET may be reasonable. Retrying a payment operation without understanding idempotency can be a financial incident.

Circuit Breaking and Outlier Ejection

A service that's returning failures shouldn't necessarily continue receiving traffic indefinitely. DestinationRules can define connection-pool and outlier-detection behavior.

Conceptually:

service-b
   │
   ├── healthy instance
   ├── healthy instance
   └── failing instance
             │
             ▼
       outlier detection
             │
             ▼
          ejected
Enter fullscreen mode Exit fullscreen mode

The unhealthy endpoint can temporarily be removed from load balancing. This can stop one broken instance from poisoning every caller.

But again, thresholds matter. Make them too aggressive and healthy instances can get ejected during normal traffic variation. Make them too lenient and the protection arrives too late.

There is no magical production value. You need to understand your workload.

The Battle Scars: Debugging the Ghost 503s

Few things are more irritating than: The application is healthy. while users are receiving:

503 Service Unavailable
Enter fullscreen mode Exit fullscreen mode

This is where Envoy's response flags become incredibly useful. One example you'll eventually encounter is: 503 NR (NR means No Route). The request reached Envoy, but Envoy couldn't find a valid route for it.

That is very different from:

application returned HTTP 503
Enter fullscreen mode Exit fullscreen mode

The HTTP status looks similar from the outside. The failure is completely different. When debugging, I want to know who generated the 503? Then:

  • Did Envoy have a route?
  • Did it have a cluster?
  • Did the cluster have endpoints?
  • Could it establish a connection?
  • Did TLS negotiation work?
  • Did the upstream respond?
  • Did the application return the status?

That's the difference between randomly restarting Pods and actually debugging the system.

Read the Envoy Access Logs

A useful Envoy access log can tell you considerably more than an application log. You want fields such as downstream address, request method, request path, response code, response flags, upstream host, request duration, and upstream service time.

Suppose you see:

HTTP 503
response_flags=NR
Enter fullscreen mode Exit fullscreen mode

Start looking at routing configuration. Check:

VirtualService
DestinationRule
Service
ServiceEntry
Gateway
Enter fullscreen mode Exit fullscreen mode

If instead you see something indicating an upstream connection failure, your investigation moves toward:

  • endpoint health
  • network connectivity
  • TLS
  • connection limits
  • upstream availability

And if Envoy successfully reaches the upstream and the application itself returns 503, stop blaming the mesh. The application is returning the error. This sounds obvious. During an outage, it's surprisingly easy to forget.

Configuration Drift Is a Silent Killer

Another lesson from large clusters: The configuration you think exists and the configuration Envoy is actually running are not necessarily the same thing. Istio configuration flows through the control plane before reaching the proxies.

When configuration changes, you need to know What was applied? What did Istio accept? What configuration did Envoy receive? and What configuration is Envoy actually using? This is why operational tooling matters.

Commands such as:

istioctl proxy-status
Enter fullscreen mode Exit fullscreen mode

and:

istioctl proxy-config
Enter fullscreen mode Exit fullscreen mode

become part of your everyday debugging toolkit.

For instance:

istioctl proxy-status
Enter fullscreen mode Exit fullscreen mode

can help identify proxies that aren't synchronized correctly.

Then you can inspect specific configuration categories on a problematic workload rather than staring at YAML for an hour.

The important lesson is:

Source configuration is not the same thing as runtime configuration.

That's true in Kubernetes generally. Istio just gives you another layer where drift can happen.

Sidecars vs. Ambient Mesh

Sidecar-based Istio is the model most engineers first encounter:

Pod
├── Application
└── Envoy
Enter fullscreen mode Exit fullscreen mode

It works, it's mature, and it's conceptually straightforward. But it also means every workload carries proxy overhead. Istio's ambient mesh changes the data-plane architecture by moving away from requiring an Envoy sidecar in every Pod.

The important point isn't that ambient is "better." It's that it changes the operational trade-offs.

With sidecars:

Every workload
      ↓
Envoy sidecar
Enter fullscreen mode Exit fullscreen mode

With Ambient:

Workloads
    │
    ▼
Node / shared mesh components
    │
    ▼
Optional higher-level L7 processing
Enter fullscreen mode Exit fullscreen mode

That can reduce some per-Pod overhead and simplify certain adoption scenarios. But it introduces a different set of components and operational concepts. If you're already operating a mature sidecar mesh successfully, moving to ambient isn't automatically an upgrade worth doing.

Infrastructure decisions should be driven by an actual problem, not architectural fashion.

The Control Plane Is Part of Your Production Dependency Chain

There's another uncomfortable reality. Once you depend heavily on Istio, the control plane becomes part of your platform's operational surface.

Upgrading it isn't:

kubectl apply
Enter fullscreen mode Exit fullscreen mode

You need to think about:

  • API compatibility
  • proxy compatibility
  • configuration changes
  • CRD behavior
  • gateway behavior
  • certificate management
  • control-plane resource usage
  • rollout sequencing
  • rollback strategy

And don't assume that because the control plane upgrade succeeded, every proxy is healthy. Check proxy synchronization, check gateways, check workloads, and check actual traffic.

A control-plane upgrade that looks green in Kubernetes can still leave a subset of proxies running unexpected configuration.

What I Would Actually Do?

If I were building a new Kubernetes platform today, I wouldn't start by installing Istio everywhere. I'd start with Kubernetes networking and make sure that CNI works, DNS works, Services work, NetworkPolicies are understood, Ingress is predictable. Observability exists before adding another layer. Then I'd identify the problems that justify a mesh.

If the organization needs:

  • workload-to-workload mTLS
  • consistent service identity
  • advanced traffic splitting
  • request-level telemetry
  • standardized retries and circuit breaking
  • controlled east-west traffic
  • centralized egress policy

then Istio starts earning its operational cost. But if you have 20 services, 3 engineers, low traffic, simple architecture and the biggest production problem is that nobody understands Kubernetes Services yet, installing a service mesh probably isn't going to save you.

It may actually make things worse.

The Platform Engineer's Verdict

Istio is one of those tools that becomes extremely valuable once your Kubernetes environment has problems that Kubernetes itself wasn't designed to solve. When you're operating hundreds of services, managing service-to-service encryption, running canary deployments, debugging cross-service latency, and trying to understand why a request disappeared somewhere between two namespaces, having a consistent networking layer can make an enormous difference. mTLS, traffic splitting, circuit breaking, outlier detection, service identity, and detailed Envoy telemetry are not just nice features when you're dealing with a large production platform. They can turn an otherwise opaque failure into something you can actually investigate.

But I've also seen teams introduce Istio far earlier than they needed it and end up spending more time debugging the mesh than solving their original networking problems. Every sidecar consumes CPU and memory. Every VirtualService and DestinationRule becomes another piece of configuration that someone has to understand. Every control-plane upgrade needs planning and validation. A badly configured retry policy can turn a small downstream failure into a cluster-wide incident, and a single routing mistake can produce a wall of mysterious 503s. The mesh gives you more control, but it also gives you more ways to shoot yourself in the foot.

My rule is fairly simple: don't adopt Istio because your architecture diagram looks more impressive with it. Adopt it because you have a networking problem that justifies the operational cost. Start with solid Kubernetes networking, clear service ownership, sensible observability, and NetworkPolicies. Introduce the mesh gradually around the workloads that benefit from it, keep routing policies simple, put strict limits around retries and resources, and make Envoy debugging part of your team's operational knowledge. If you do that, Istio becomes a useful platform capability rather than another layer of infrastructure that everyone is afraid to touch when production breaks.

Top comments (0)