In 2020, I gave a talk about Linkerd at DevOpsDays São Paulo. One of the recommendations in the official documentation contained the following statement: "You should not use Linkerd alongside Kafka solutions."
At the time, Gomex pulled me aside to discuss this very phrase: "But Carlos, why can't we use the two together?"
That conversation came back to me recently, after I ran into a similar situation, only this time in a real world scenario. So I decided to write about it.
Why You Shouldn't Put Kafka Traffic in Your Service Mesh
The rule that "you should never use a service mesh with Kafka" is a common piece of advice, but it is an oversimplification. It hides a more important and nuanced truth: using a service mesh like Istio or Linkerd to manage the core data plane traffic of Apache Kafka is typically a bad idea and a recognized anti-pattern. It is not strictly forbidden, but the benefits are negligible while the costs and risks are substantial. The fundamental issue is a collision of protocols and architectural philosophies.
Let's break down why this is the case, clearly separating what you should never do from the limited scenarios where a service mesh can be helpful alongside Kafka.
The Central Problem: Kafka's Protocol is Not HTTP
Most service meshes are designed and highly optimized for HTTP/2, gRPC, and generic TCP traffic. The proprietary binary protocol of Apache Kafka is an entirely different entity, and this mismatch creates several critical failures.
First, layer 7 routing becomes impossible. A service mesh's true power lies in its ability to read HTTP headers and route traffic based on the URL path, method, or custom headers. The Kafka protocol is a continuous, stateful binary stream. The Envoy or Linkerd sidecar proxy has no way to parse a ProduceRequest from a FetchRequest or a MetadataRequest. To the mesh, this communication is an opaque TCP bytestream, making any intelligent, application layer routing decisions impossible.
Second, a mesh fundamentally breaks Kafka's stateful nature. Kafka is inherently stateful. A consumer must connect to a specific broker and partition. This connection is not ephemeral. A layer 7 proxy like Envoy is designed to load balance identical, stateless requests across multiple servers. If you place it between a consumer and a Kafka cluster, it may unknowingly try to load balance a Fetch request to a broker that is not the partition leader. This will result in an error, increased latency, and cascading failures. The Kafka client is meticulously designed to perform this specific routing on its own. Furthermore, the sticky connection a consumer establishes with a broker must be maintained. The sidecar proxy adds an extra hop that can experience timeouts, be restarted, or misroute the connection, interrupting the continuous message stream.
The Practical Nightmare: Advertised Listeners and mTLS
The single biggest operational problem is the mechanism of advertised.listeners. This is how a Kafka broker tells a client which IP address and port to use for a direct connection. Within a service mesh, a broker might have an internal Pod IP, but a client outside the cluster needs an external address. When a mesh is introduced, the client's traffic is intercepted by its own sidecar, encrypted with mTLS, and then sent to the broker's sidecar. The question then becomes: what address should the broker advertise? Its own Pod IP? The IP of its sidecar proxy? A Kubernetes service name? Correctly configuring this mesh wrapped communication loop, especially with mutual TLS enabled, becomes a nightmare of complexity. It almost always ends in connection refused errors, traffic blackholes, or infinite routing loops.
Consider a disaster scenario: a producer sending a message with acks=all through an Istio sidecar. The application sends a Produce request to the local sidecar port. The producer's sidecar intercepts it, applies mTLS, and sends it to the sidecar of Broker A. Broker A's sidecar decrypts the traffic and hands it to the Broker A application. Broker A, as the partition leader, then needs to replicate the data to followers Broker B and Broker C. It becomes a "client" itself, and its replication request is again intercepted by its own sidecar, undergoes mTLS, and is sent to the sidecar of Broker B. The result is a storm of network hops, added latency on every single step, and a debugging environment of maddening complexity. The critical low latency of Kafka is severely degraded, and any observability gain is lost in the noise created by this design.
A Clear Separation of Concerns: What to Do Instead
The conclusion is not "never use a service mesh." It is "never place the raw Kafka protocol traffic inside the data plane mesh." The separation of responsibilities must be clear.
What Not to Do
- Do not inject sidecar proxies into the pods of your Kafka brokers with the goal of managing their inter broker or client facing data traffic.
- Do not inject sidecars into your client application pods, producers or consumers, for the purpose of managing their traffic to the Kafka cluster.
- Do not create Istio
DestinationRulesorServiceEntriesthat attempt L7 routing or load balancing for the Kafka bootstrap service. If you must use aServiceEntry, configure it as a simple TCP passthrough solely to allow mesh resident clients to resolve the DNS name and connect directly, bypassing any L7 logic.
What to Do
You can and should use a service mesh for everything that is not the Kafka binary protocol. This applies perfectly to management APIs and auxiliary tooling.
- Management Interfaces: If you use an administration interface like Conduktor, AKHQ, or the Confluent REST Proxy, these are ideal HTTP/gRPC applications that benefit greatly from the mTLS, authorization, and traffic routing policies of a service mesh.
- Schema Registry: The Confluent Schema Registry is another perfect candidate, as it communicates via a standard HTTP/REST API. You can place the sidecar in the Schema Registry pod and manage all access to it with the mesh's policies.
- Kafka Connect with HTTP Connectors: For Kafka Connect, you might use the mesh to secure the HTTP based connectors, but the internal communication between the Connect framework and the Kafka brokers must remain outside the mesh.
- Microservice Orchestration: You can use the mesh to manage the HTTP/gRPC traffic between your microservices, while those same services independently produce and consume messages to Kafka over a direct, mesh free TCP connection.
Summary
| Characteristic | Service Mesh (Istio/Linkerd) | Apache Kafka | Conflict? |
|---|---|---|---|
| Primary Protocol | HTTP/2, gRPC, Generic TCP | Proprietary Binary Protocol | Yes |
| Routing Model | L7 (headers, paths) or L4 (TCP) | Custom L7 logic in the client | Yes |
| Connection State | Stateless, ephemeral | Stateful, long lived, sticky | Yes |
| Load Balancing | Proxy side (Envoy) | Client side (partition aware) | Yes |
| TLS Management | Automated mTLS via sidecar | Native Kafka TLS or direct mTLS | Possible |
| Service Discovery | Mesh DNS ( svc.cluster.local) |
advertised.listeners and Metadata |
Yes |
In summary, forcing Kafka's data plane traffic through a service mesh is a fundamental architectural mistake. The service mesh is designed for ephemeral, stateless, HTTP driven workloads, while Kafka relies on a custom binary protocol with stateful, long lived connections and client driven routing. The two systems have contradictory models for load balancing, connection management, and service discovery. To use them successfully together, you must respect their boundaries: use the service mesh for your REST and gRPC APIs, and let Apache Kafka handle its own data streaming with direct, unproxied TCP connections.
Top comments (1)
Thanks for sharing. As a complement to service meshes for non-Kafka traffic, Kafka-aware proxies are becoming more common. They understand the Kafka protocol (L7) and use port or SNI binding to preserve broker mappings (as you raise the issue, Kafka traffic is stateful here), with a goal of adding security, governance, failover, and much more as they intercept everything going through, as a service mesh.