Kafka addressing & VPC peering

If you've ever tried to connect a Kafka client that lives in a different VPC than the cluster, you've probably hit this issue where there's a route between the two networks, telnet works, and yet the client still can't consume a single record. It feels like a networking bug. It isn't.

The issue is that reaching Kafka across VPCs is an addressing problem, not a networking one. Once that clicks, why peering, transit gateways, and per-broker load balancers don't work will make sense.

What your client actually does when it connects

Most services are easy to reach across a networking boundary. Put a load balancer in front, give it one address, point clients at it. Done.

Kafka doesn't work like that. A client connects in two stages:

1. Bootstrap. The client talks to any broker and asks one question: who's in this cluster?
2. Direct connection. The broker answers with metadata — a list of every broker, each named by its own advertised.listeners address. The client then opens direct connections to specific brokers: the leader for each partition it reads or writes (or the nearest replica, if you're using follower fetching, KIP-392, Kafka 2.4+).

That second stage is the most important to remember. One shared address in front of the cluster is not working, because the client has to resolve and route to each broker's advertised address, exactly as the cluster hands it back.

So if a broker advertises broker-1.cluster.internal:9092, a private name that only resolves inside the cluster's VPC, a remote client connects to the bootstrap fine, then fail when it tries to connect directly to one of the brokers:

docker exec kafka-consumer-a kcat -b kafka:9092 -L -m 5
# -> Failed to resolve 'kafka:9092'

A network path between two VPCs is not the same as Kafka reachability. The client still has to resolve and route to every broker's advertised address from where it sits.

"Then just make the broker advertise something reachable". Sure, brokers support multiple listeners, one internal and one external. But every listener is another port to open on every broker, and managed Kafka (MSK, Confluent Cloud, Aiven) won't let you touch them anyway. Listeners don't scale to a dozen independent networks across accounts and clouds. Which is exactly the situation you're in.

Why peering collapses the moment you have more than two networks

VPC peering is genuinely great for connecting two VPCs. Past two, here be dragons:

• It's point-to-point and non-transitive. Peering A and B each to the cluster does not let A talk to B. Every new client network needs its own peering straight into the cluster's VPC.
• Your cluster's VPC becomes an accidental hub. Ten client networks means ten peerings, ten route-table entries, and ten security-group conversations on a VPC that was never designed to be a hub.
• CIDRs have to be coordinated. Peered VPCs can't overlap ranges, so every team negotiates address space.
• Managed Kafka caps it. Providers limit how many peerings or PrivateLink attachments you get, and none of them let you rewrite broker listeners.

The recurring pain is one line: every new client forces another change onto the cluster's VPC, the one piece of infra you least want to keep editing.

The fix: let a Kafka-aware proxy rewrite the addresses

The solution: put something in the connection path that actually understands the Kafka protocol instead of just shuffling packets, and have it rewrite the broker addresses in the metadata response before the client ever sees them.

That's what Conduktor Gateway does. A broker advertises kafka-internal:9092; the Gateway rewrites that to an address the client can reach. Because clients connect to specific brokers (the partition leaders), the Gateway maps each broker to its own port so traffic still routes deterministically to the right one behind it.

From the client's side, the only change is a single line:

# before: pointing straight at a broker it can't actually reach
bootstrap.servers=broker-1.cluster.internal:9092

# after: pointing at the proxy, which hands back addresses it can
bootstrap.servers=conduktor-gateway.hub:9092

Credentials don't change. The client presents the same SASL credentials it already has, the Gateway forwards them to the broker, and Kafka ACLs (or Confluent RBAC) still decide what it's allowed to do. The proxy is stateless, it stores no credentials and adds no new auth model.

The nice architectural payoff: the Gateway lives in its own hub VPC. Peerings or PrivateLink attachments land on the hub, never directly between a client and the cluster. New networks attach to the hub; the cluster keeps a single attachment no matter how many clients reach in, and its listeners are never touched.

The Gateway isn't a substitute for a route as you still need connectivity between the hub and each VPC. What it removes is the per-client peering into the cluster, the broker reconfiguration, and the requirement that every advertised address be reachable from every client.

"But what about a transit gateway? Or an LB per broker?"

Won't work.

A cloud transit gateway is one hub many VPCs attach to for any-to-any IP connectivity. But it operates at L3 so it moves IP packets with zero awareness of Kafka. It doesn't change what brokers advertise, so you're back to sharing private hosted zones across accounts. Overlapping CIDRs still break it without NAT, it's confined to one cloud, and on managed Kafka you still can't touch the listeners.

A load balancer per broker can't be round-robined, because clients connect to specific leaders. So it's one NLB (or target group, or PrivateLink endpoint) per broker, plus advertised.listeners reconfigured: N pieces of plumbing to keep in sync, each with its own port, DNS record, and TLS SAN. Scale or replace a broker and the mapping churns, forcing a rolling restart. And on managed Kafka you usually can't set advertised listeners at all — so it's off the table before you begin.

Approach	Works at	Fixes the advertised-address problem?	Managed Kafka?	Cross-cloud?
VPC peering	L3	No	Capped by limits	No
Transit gateway	L3	No	No	On-prem yes, other cloud no
NLB per broker	L4	Only if you reconfigure listeners	Usually not	Partial
Kafka-aware proxy	Kafka L7	Yes, automatically	Yes	Yes

Reproduce the whole thing on one machine

The free Gateway Community quickstart stands up the exact scenario: a Kafka cluster in a private Docker network the clients can't reach, two consumers in two separate "VPC" networks, and the Gateway as the only container joined to all three.

bash <(curl -fsSL https://releases.conduktor.io/gateway-community-quickstart)

Consumer A can't even resolve the broker directly — different network:

docker exec kafka-consumer-a kcat -b kafka:9092 -L -m 5
# -> Failed to resolve 'kafka:9092'

Same consumer, same SASL credentials, one different bootstrap address — now it reads:

docker exec kafka-consumer-a kcat -b conduktor-gateway:9092 -t customers -C -e -c 3 \
  -s value=avro -r http://karapace:8081 \
  -X security.protocol=SASL_PLAINTEXT -X sasl.mechanism=PLAIN \
  -X sasl.username=consumer-a -X sasl.password=consumer-a-secret
# -> readable records

And the log line that proves why:

docker logs conduktor-gateway 2>&1 | grep "Rewriting METADATA"
# kafka:9092 -> conduktor-gateway:9092

That's the whole trick. The broker advertises an address the client can't reach; the proxy rewrites it to one it can.

Wrapping up

If cross-VPC Kafka has been a recurring issue on your platform team, try rethinking it: you don't have a networking problem, you have an addressing problem, and a Kafka-aware proxy is the one thing that fixes the address without you touching the cluster.

If you later want to encrypt fields, enforce schemas, or isolate tenants on that same path, this same proxy will do it too.

If you want to poke at it, the Gateway Community quickstart is free and runs on one machine. Curious how you're solving this today, peering everything everywhere, transit gateways, something else? Let me know.