DEV Community

Cover image for Kubernetes Networking [Level-2: Pod-to-Pod Communication]
ADITYA RAJ
ADITYA RAJ

Posted on

Kubernetes Networking [Level-2: Pod-to-Pod Communication]

#docker #linux #kubernetes #networking

This is Level 2 of our Kubernetes networking series. In Level 0, we covered core Linux networking primitives, and in Level 1, we learned how a Pod gets its own network namespace, eth0 interface, and IP address.

Now it's time to answer one of the most important questions in Kubernetes networking:

How does Pod A actually communicate with Pod B?

We'll start with the simplest possible case and work our way up to a full hop-by-hop trace of cross-node Pod traffic — including routing, overlay networking, VXLAN, and BGP.

Table of Contents

  1. Our Scenario
  2. The Kubernetes Networking Expectation
  3. First Case: Same-Node Communication
  4. How Pod A Knows Where to Send the Packet
  5. Crossing the Node Boundary
  6. The Node Routing Table
  7. Who Creates These Routes?
  8. Two Major Approaches: Routing vs Overlay
  9. Overlay Networking: A Simple Analogy
  10. Why We Need Encapsulation
  11. VXLAN in a Nutshell
  12. Routing Without an Overlay
  13. Why BGP Shows Up Here
  14. The Complete Cross-Node Packet Journey
  15. Pod IP vs Node IP
  16. Why You Can't Just Use Pod IPs on the Internet
  17. Kubernetes Doesn't Mandate an Overlay
  18. Same-Node vs Cross-Node: A Quick Reference
  19. The Big Problem We've Just Created
  20. What's Next: Kubernetes Services

Our Scenario

Let's set up a concrete example. We have two Pods:

Pod A
IP: 10.244.1.5
Node: node-1
Enter fullscreen mode Exit fullscreen mode 
Pod B
IP: 10.244.2.5
Node: node-2
Enter fullscreen mode Exit fullscreen mode

Visually:

              Kubernetes Cluster

   Node 1                              Node 2
┌───────────────┐                  ┌───────────────┐
│ Pod A         │                  │ Pod B         │
│ 10.244.1.5    │                  │ 10.244.2.5    │
└───────┬───────┘                  └───────▲───────┘
        │                                  │
        └──────────── Network ─────────────┘
Enter fullscreen mode Exit fullscreen mode

Pod A wants to send traffic to Pod B: 10.244.1.5 → 10.244.2.5.

The Kubernetes Networking Expectation

Kubernetes enforces an important networking model:

Every Pod should be able to communicate with every other Pod, without anyone needing to manually manage routes for each Pod.

Ideally, from the application's point of view:

Pod A (10.244.1.5) → HTTP → Pod B (10.244.2.5)
Enter fullscreen mode Exit fullscreen mode

The application shouldn't need to know:

  • Is Pod B on Node 1 or Node 2?
  • Is the traffic going over VXLAN?
  • Is it using BGP?
  • Is it using eBPF?

All of that complexity belongs to the networking layer, not the application.

First Case: Same-Node Communication

Let's start with the easier scenario: both Pods live on the same Node.

Node 1
Pod A: 10.244.1.5
Pod B: 10.244.1.6
Enter fullscreen mode Exit fullscreen mode

Conceptually, traffic flows like this:

Pod A
  ↓
eth0
  ↓
veth
  ↓
Node networking
  ↓
veth
  ↓
eth0
  ↓
Pod B
Enter fullscreen mode Exit fullscreen mode

The exact implementation depends on the CNI plugin, but the core idea holds:

Pod A → Node networking → Pod B
Enter fullscreen mode Exit fullscreen mode

How Pod A Knows Where to Send the Packet

Remember ip route from Level 0? Every Pod has its own routing table. For example:

default via 10.244.1.1 dev eth0
10.244.1.0/24 dev eth0
Enter fullscreen mode Exit fullscreen mode

When Pod A wants to reach 10.244.1.6, Linux checks: do I have a route for this? Yes — it falls within 10.244.1.0/24. So Linux sends the packet out through eth0.

Crossing the Node Boundary

Now the interesting case: Pod A and Pod B are on different Nodes.

Pod A (10.244.1.5) on Node 1
              ↓
Pod B (10.244.2.5) on Node 2
Enter fullscreen mode Exit fullscreen mode

Pod A sends a packet with:

Source:      10.244.1.5
Destination: 10.244.2.5
Enter fullscreen mode Exit fullscreen mode

The packet leaves the Pod through eth0, crosses the veth pair, and lands on Node 1's networking stack. Now Node 1 has to answer a critical question:

Where is 10.244.2.5?

The Node Routing Table

Node 1 needs to know how to reach the Pod network living on Node 2. Conceptually, it might have a routing table like:

10.244.1.0/24 → local
10.244.2.0/24 → Node 2
Enter fullscreen mode Exit fullscreen mode

In plain terms:

10.244.1.x → my local Pods
10.244.2.x → Pods on Node 2
Enter fullscreen mode Exit fullscreen mode

So the flow becomes:

Pod A → Node 1 → Route → Node 2 → Pod B
Enter fullscreen mode Exit fullscreen mode

This is the fundamental shape of all cross-node Pod traffic in Kubernetes.

Who Creates These Routes?

This is exactly where the CNI implementation becomes important. Different networking plugins solve this problem differently, using mechanisms such as:

  • Routing
  • Overlay networking
  • VXLAN
  • Geneve
  • BGP
  • Cloud-native routing
  • eBPF

We'll dig into specific CNI implementations properly in Level 7. For now, just remember: CNI's job is to make sure the network knows how to reach every Pod IP.

Two Major Approaches: Routing vs Overlay

There are two broad strategies Kubernetes networking implementations use to make cross-node Pod traffic work.

Approach 1 — Routing

The underlying physical (or cloud) network is taught how to reach each Pod network directly:

Pod A → Node 1 → Router → Node 2 → Pod B
Enter fullscreen mode Exit fullscreen mode

Approach 2 — Overlay

The Pod's packet gets encapsulated inside another packet addressed between Nodes:

Original packet:
10.244.1.5 → 10.244.2.5

        ↓ encapsulation

Outer packet:
Node1-IP → Node2-IP
    |
    | carries
    ↓
10.244.1.5 → 10.244.2.5
Enter fullscreen mode Exit fullscreen mode

We'll explore both approaches in more depth below.

Overlay Networking: A Simple Analogy

Imagine sending a letter between two buildings, but the postal system only understands building addresses — not individual recipients. So you put your letter (addressed to a specific person) inside an outer envelope addressed to the building itself:

Outer envelope: Node 1 → Node 2
Inside:         Pod A → Pod B
Enter fullscreen mode Exit fullscreen mode

That's essentially what packet encapsulation does:

Outer packet
┌──────────────────────────────┐
│ Node 1 → Node 2              │
│                              │
│   Inner packet               │
│   Pod A → Pod B              │
└──────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Why We Need Encapsulation

Suppose the physical network only knows about Node IPs, like 192.168.1.10 and 192.168.1.20 — it has no idea what 10.244.1.5 or 10.244.2.5 even are. The physical network understands Nodes, not Pods.

So the trick is to communicate at the level the network does understand (Node to Node), while carrying the Pod-level conversation inside:

Node 1 (192.168.1.10) → Node 2 (192.168.1.20)
       carries
Pod A (10.244.1.5) → Pod B (10.244.2.5)
Enter fullscreen mode Exit fullscreen mode

VXLAN in a Nutshell

VXLAN is one of the most common technologies used to implement overlay networking. You don't need to master every detail yet — just the high-level flow:

Pod packet (10.244.1.5 → 10.244.2.5)
       ↓
VXLAN encapsulation
       ↓
Node 1 → Node 2
       ↓
VXLAN decapsulation
       ↓
Pod B
Enter fullscreen mode Exit fullscreen mode

Or more generally:

Pod → Overlay → Node → Network → Node → Overlay → Pod
Enter fullscreen mode Exit fullscreen mode

Routing Without an Overlay

An alternative approach is to make the underlying network itself understand Pod networks directly — no encapsulation required.

Node 1 — Pod CIDR: 10.244.1.0/24
Node 2 — Pod CIDR: 10.244.2.0/24
Enter fullscreen mode Exit fullscreen mode

If the network knows:

10.244.1.0/24 → Node 1
10.244.2.0/24 → Node 2
Enter fullscreen mode Exit fullscreen mode

...then traffic can be routed directly, with no wrapping/unwrapping step:

Pod A → Node 1 → Router → Node 2 → Pod B
Enter fullscreen mode Exit fullscreen mode

Why BGP Shows Up Here

You'll eventually run into BGP (Border Gateway Protocol) in Kubernetes networking discussions — don't let the name intimidate you. At a high level, BGP is simply a protocol for distributing routes across the network.

For example:

Node 1 announces: "10.244.1.0/24 is reachable through me"
Node 2 announces: "10.244.2.0/24 is reachable through me"
Enter fullscreen mode Exit fullscreen mode

Once these routes propagate, the network can route Pod traffic directly — no overlay needed. Some CNI implementations use BGP specifically for this purpose.

The Complete Cross-Node Packet Journey

Let's trace a full request from Pod A (10.244.1.5) to Pod B (10.244.2.5) step by step.

Step 1 — Application: Pod A's application sends GET /api to 10.244.2.5:8080.

Step 2 — Pod network namespace: Linux sees source 10.244.1.5, destination 10.244.2.5.

Step 3 — Pod eth0: The packet leaves the Pod via eth0.

Step 4 — veth: The packet crosses the veth pair from the Pod namespace into the Node namespace.

Step 5 — Node routing: Node 1 asks "where is 10.244.2.5?" and finds that 10.244.2.0/24 → Node 2.

Step 6 — Cross-node network: Depending on the CNI implementation, this hop uses routing, an overlay, eBPF, or cloud-native networking.

Step 7 — Node 2: The packet arrives and Node 2 determines that 10.244.2.5 belongs to Pod B.

Step 8 — Pod B: The packet travels Node 2 → veth → Pod B network namespace → eth0 → application.

Putting it all together:

Pod A (10.244.1.5)
   ↓
eth0 → veth → Node 1
   ↓
Routing / Overlay / eBPF
   ↓
Node 2 → veth → eth0
   ↓
Pod B (10.244.2.5)
Enter fullscreen mode Exit fullscreen mode

Pod IP vs Node IP

This distinction trips up a lot of people, so let's be explicit. Suppose:

Node 1: 192.168.1.10 → Pod A: 10.244.1.5
Node 2: 192.168.1.20 → Pod B: 10.244.2.5
Enter fullscreen mode Exit fullscreen mode

These live in two entirely separate networks:

Node network: 192.168.1.0/24
Pod network:  10.244.0.0/16
Enter fullscreen mode Exit fullscreen mode

A Pod's IP is not the same as, or derived from, its Node's IP. Keep this mental separation clear — it will save you a lot of confusion later.

Why You Can't Just Use Pod IPs on the Internet

A private Pod IP like 10.244.1.5 isn't reachable directly from the public internet. Instead, traffic typically flows like this:

Internet
   ↓
Load Balancer / Ingress
   ↓
Kubernetes
   ↓
Service
   ↓
Pod
Enter fullscreen mode Exit fullscreen mode

This is precisely why Kubernetes has dedicated abstractions — Services, Ingress, and Gateway API — which we'll cover soon.

Kubernetes Doesn't Mandate an Overlay

Here's an important point, especially if you're prepping for interviews: it's a common misconception that "Kubernetes networking uses an overlay network." That's not universally true.

Kubernetes defines a networking model (every Pod can reach every other Pod by IP), but it doesn't dictate how that model is implemented:

Kubernetes networking model
           |
     ┌─────┼─────┐
     ↓     ↓     ↓
 Routing Overlay eBPF
Enter fullscreen mode Exit fullscreen mode

The specific CNI plugin you choose determines the actual implementation.

Same-Node vs Cross-Node: A Quick Reference

Traffic What Happens Conceptually
Pod → Pod, same Node Node-local networking
Pod → Pod, different Nodes Cross-node routing/overlay/dataplane
Pod → Internet Node/network egress path
Internet → Pod Usually LoadBalancer/Ingress/Gateway + Service
Pod → Service Service dataplane selects a backend Pod

That last row is exactly the problem we tackle next.

The Big Problem We've Just Created

So far, Pod A → Pod B works fine. But there's a serious practical problem lurking here.

Suppose we have three backend Pods:

Backend Pod 1: 10.244.1.5
Backend Pod 2: 10.244.2.5
Backend Pod 3: 10.244.3.5
Enter fullscreen mode Exit fullscreen mode

The frontend needs to call the backend — but which IP should it use? And what happens when Backend Pod 1 dies and Kubernetes replaces it with a brand-new Pod at a brand-new IP, say 10.244.4.8?

We clearly don't want applications hardcoding or tracking individual Pod IPs.

The Problem

Pod IPs are:

  • Dynamic
  • Temporary
  • Tied to individual Pod lifecycles

Applications shouldn't have to manage any of that.

The Solution

A Kubernetes Service — a stable abstraction sitting in front of a changing set of Pods:

Frontend
   ↓
Service
   ├──→ Pod 1
   ├──→ Pod 2
   └──→ Pod 3
Enter fullscreen mode Exit fullscreen mode

What's Next: Kubernetes Services

In Level 3, we'll build up Services from first principles, starting with the exact question this article ends on:

Why can't I simply use a Pod IP?

From there, we'll progressively cover:

What is a Service?
       ↓
ClusterIP
       ↓
Selectors
       ↓
Endpoints / EndpointSlices
       ↓
How Service traffic actually reaches Pods
       ↓
Service types: NodePort, LoadBalancer, ExternalName
Enter fullscreen mode Exit fullscreen mode

We'll then trace a complete request:

Frontend Pod → Service IP → Kubernetes dataplane → Backend Pod
Enter fullscreen mode Exit fullscreen mode

That's where components like kube-proxy, iptables, IPVS, and eBPF will finally start making sense — though we'll save their deep internals for Level 8, right on schedule with the rest of this roadmap.

Conclusion

Pod-to-Pod communication is the connective tissue of Kubernetes networking. Same-node traffic is fairly straightforward — it stays within the Node's networking stack via veth pairs. Cross-node traffic is where things get interesting: the network needs a way to route or encapsulate Pod IPs that it may not natively understand, whether through direct routing, VXLAN-based overlays, or BGP-distributed routes.

The most important takeaway is this: Kubernetes defines what should happen (any Pod can reach any other Pod by IP), not how — that's the CNI plugin's job. And because Pod IPs are inherently unstable, we need a better abstraction for talking to a group of Pods reliably. That abstraction is the Service, and it's exactly where we're headed next.

Enjoyed this deep dive? Follow along for Level 3, where we build Kubernetes Services from the ground up and trace a real request from a frontend Pod all the way to a backend Pod. Drop your questions in the comments, and share this with a teammate still wondering why their Pod IP keeps changing. Let's keep demystifying Kubernetes networking together. 🚀

Top comments (0)