DEV Community

pathvector-dev
pathvector-dev

Posted on • Originally published at blog.pathvector.dev

Policy Routing: When the Source Address Picks the Path, Not the Destination

#network #linux #containerlab #routing

Originally published at https://blog.pathvector.dev/protocol-lab-pbr-38/ — part of the free Protocol Lab series.

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

Every routing lab so far has picked the path by destination. Policy routing breaks that assumption: the forwarding decision can also depend on the source address (or the incoming interface, a firewall mark, ToS…). The router puts a rule database in front of its routing tables, and a rule decides which table to consult for each packet.

Reading guide: rfc-notes/policy-routing.md

Prerequisite: Lab 34: OSPF — Flood the Map, Compute the Shortest Path

Expected time: 40–55 minutes.

The Goal

Two source hosts sit behind a router with two uplinks, and both uplinks host the same service address (10.0.100.1):

  • Baseline (destination-based): the router's main table sends 10.0.100.1 via up1, so both srcA and srcB reach up1.
  • Add one ip rule from srcB lookup 200 (table 200's default goes via up2).
  • Now srcA still reaches up1, but srcB reaches up2 — the same destination routed over different uplinks by source.

By the end, you should be able to explain this table:

srcA → 10.0.100.1 srcB → 10.0.100.1
baseline (destination-based) up1 up1
with ip rule from srcB up1 up2

What You Will Learn

  • That Linux has multiple routing tables and a rule database in front of them.
  • How ip rule selects a table by source (or iif, fwmark, ToS).
  • Why ordinary routing sends the same destination one way regardless of source.
  • How policy routing implements multi-homing ("this LAN uses ISP-A, that one ISP-B").
  • Rule priority order and how unmatched packets fall through to main.

This lab does not cover:

  • fwmark-based rules (iptables mark → ip rule fwmark) in depth.
  • Return-path asymmetry / RPF interplay beyond a mention.
  • Dynamic multipath (ECMP, Lab 32) — that hashes, this selects by rule.

Where to Read Up

Reference What to focus on
ip-rule(8) The rule database, priorities, from/iif/fwmark selectors
ip-route(8) table Creating and consulting multiple routing tables
RFC 3704 Why source matters in multihomed networks
RFC 5737 / RFC 1918 Confirming the addresses used here are local/documentation-only

The Big Picture

srcA and srcB sit behind router r, which has two uplinks: up1 and up2. Both uplinks carry the same 10.0.100.1 on lo and each runs an HTTP responder that returns its own name.

 srcA (10.0.1.2) --- r --- up1 (lo 10.0.100.1)  → "up1"
 srcB (10.0.5.2) --/ \--- up2 (lo 10.0.100.1)  → "up2"
Enter fullscreen mode Exit fullscreen mode 
flowchart LR
  A["srcA (10.0.1.2)"] --> R["r<br/>rule: from srcB → table 200"]
  B["srcB (10.0.5.2)"] --> R
  R -->|"main table: default via up1"| U1["up1 → 'up1'"]
  R -->|"table 200: default via up2<br/>(only for srcB)"| U2["up2 → 'up2'"]
Enter fullscreen mode Exit fullscreen mode

Note: 10.0.0.0/8 is a local, closed range (RFC 1918), so nothing in this lab touches the real internet.

What You Need

Recommended environment:

  • Linux / WSL2 / a Linux VM
  • Docker
  • containerlab

Images used:

  • nicolaka/netshoot:latest — bundles ip, curl, and python3.

No additional images are required.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run pbr-38
Enter fullscreen mode Exit fullscreen mode

Or step through it manually:

1. Move into the working directory 

cd protocol-lab/examples/pbr-38
Enter fullscreen mode Exit fullscreen mode

2. Deploy and start the responders 

sudo containerlab deploy -t pbr-38.clab.yml
docker exec -d clab-pbr-38-up1 python3 /responder.py up1
docker exec -d clab-pbr-38-up2 python3 /responder.py up2
Enter fullscreen mode Exit fullscreen mode

r's main table has default via 10.0.2.2 (up1). Both uplinks carry 10.0.100.1.

3. Observe the baseline (destination-based) 

docker exec clab-pbr-38-srcA curl -s http://10.0.100.1/   # up1
docker exec clab-pbr-38-srcB curl -s http://10.0.100.1/   # up1
Enter fullscreen mode Exit fullscreen mode

Different sources, same destination — so both land on up1.

4. Add the policy route (send srcB via table 200 → up2) 

docker exec clab-pbr-38-r sh -c '
  ip route replace default via 10.0.3.2 table 200
  ip rule add from 10.0.5.2 lookup 200
'
docker exec clab-pbr-38-r ip rule
docker exec clab-pbr-38-r ip route show table 200
Enter fullscreen mode Exit fullscreen mode

5. Look again 

docker exec clab-pbr-38-srcA curl -s http://10.0.100.1/   # up1 (unchanged)
docker exec clab-pbr-38-srcB curl -s http://10.0.100.1/   # up2 (changed by source)
Enter fullscreen mode Exit fullscreen mode

Expected Output

  • Baseline: srcA=up1, srcB=up1.
  • ip rule: from 10.0.5.2 lookup 200 sits ahead of main (prio 32765).
  • ip route show table 200: default via 10.0.3.2 (up2).
  • After the policy: srcA=up1, srcB=up2 — same destination 10.0.100.1.

Why It Works

Policy routing means "let the routing decision depend on more than the destination."

  • Ordinary forwarding. The destination address is looked up in a single table (main) using longest-prefix match. Whoever the sender is, the same destination gets the same route. In the baseline, both srcA and srcB resolve 10.0.100.1 through main's default via up1, so both hit up1.
  • Multiple tables + a rule database. Linux can hold multiple routing tables (ip route ... table 200). In front of them sits the rule database (ip rule), which decides — in priority order — which packets consult which table. The first matching rule's table is used for route resolution.
  • Steering by source. ip rule add from 10.0.5.2 lookup 200 says "if the source is srcB, consult table 200." Table 200 has default via up2, so srcB's packets go out via up2. srcA doesn't match the rule and falls through to main (via up1). The same destination 10.0.100.1 ends up on different uplinks depending on the source.
  • Selectors. Beyond source, you can steer by incoming interface (iif), firewall mark (fwmark, in tandem with iptables), or ToS/DSCP — bringing non-destination information into the routing decision.
  • Where it's used. Multi-homing ("this LAN uses ISP-A, that one ISP-B"), VPN split tunneling, bandwidth separation. Return traffic should also exit via the egress matching its source, to avoid asymmetry (RFC 3704).

The key insight: put rules in front of the routing tables and select a table by source (or other selectors), and the same destination can take different paths. It's the extension of a routing decision that used to be destination-only.

Common Pitfalls

  • Assuming routes are decided by destination alone. Policy routing also decides by source, iif, or mark.
  • Confusing rules with routes. A rule picks which table to consult; a route is an entry inside a table. It's a two-stage process.
  • Rule priority. Rules are evaluated lowest-number first and the first match wins. If your rule isn't placed ahead of main (32766), it never takes effect.
  • Return-path asymmetry. Even if you steer outbound by source, an asymmetric return path can get dropped by RPF checks or stateful devices (Lab 36). In production, design the return path too.
  • fwmark integration. To steer per L4 flow or per application, mark packets with iptables, then match with ip rule fwmark.
  • Cleaning up tables. Forgetting ip rule del / ip route flush table N leaves stale rules behind.

Cleanup 

sudo containerlab destroy -t pbr-38.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

If you used labctl.sh run pbr-38, the script runs destroy for you at the end.

Check Your Understanding

  1. What does ordinary forwarding base its routing decision on? What can policy routing add?
  2. What do ip rule and ip route ... table N each define?
  3. Why does only srcB go to up2 in this lab? Why does srcA stay on up1?
  4. How does rule priority work? What happens if a rule is placed after main?
  5. Besides source, what selectors can steer packets? Name two.
  6. In a multi-homed setup, what goes wrong if you don't also design the return path?

References

Verified Run Log (2026-07-08)

This lab has been confirmed reproducible on real hardware.

Environment:

  • Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
  • Docker 29.1.3
  • containerlab 0.77.0
  • srcA / srcB / r / up1 / up2: nicolaka/netshoot:latest (ip, curl, python3)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run pbr-38 performed deploy → verify → destroy, and verification.json returned "status": "verified".

Baseline (destination-based) — both hit up1 

baseline: srcA=up1 srcB=up1
Enter fullscreen mode Exit fullscreen mode

r's main table is default via up1. Even with different sources, the destination is the same 10.0.100.1, so both requests land on up1 — the route is decided by destination alone.

After the policy — split by source

Adding ip rule add from 10.0.5.2 lookup 200 plus table 200's default via up2:

# ip rule
0:     from all lookup local
32765: from 10.0.5.2 lookup 200
32766: from all lookup main
32767: from all lookup default

# ip route show table 200
default via 10.0.3.2 dev eth4

after: srcA=up1 srcB=up2
Enter fullscreen mode Exit fullscreen mode

srcB's source address (10.0.5.2) matches the prio-32765 rule and is steered into table 200, exiting via up2. srcA matches no rule and stays on main (up1). The same destination 10.0.100.1 now takes a different uplink per source (srcA→up1, srcB→up2) — a split that destination-based routing alone could never produce.

Cleanup 

containerlab destroy -t pbr-38.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

That's policy routing: a rule database in front of the routing tables, letting source addresses — not just destinations — decide the path.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll keep pushing on traffic steering — from static rules toward marking flows with fwmark and per-application routing.

Top comments (0)