DEV Community

pathvector-dev
pathvector-dev

Posted on • Originally published at blog.pathvector.dev

Path MTU Discovery: Why Blocking ICMP Makes Big Pages Hang

#mtu #icmp #network #containerlab

Originally published at https://blog.pathvector.dev/protocol-lab-mtu-25/ — part of the free Protocol Lab series.

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

Every link has a maximum frame size — its MTU. A packet bigger than a link's MTU cannot cross it. If the packet is marked Don't Fragment (DF), a router that can't forward it must drop it and send back an ICMP "fragmentation needed" message carrying the MTU it can handle. The sender uses that to learn the Path MTU — the largest packet the whole path allows. This is Path MTU Discovery (PMTUD), and in this lab we build a path with a bottleneck link and watch it happen.

Reading guide: rfc-notes/mtu-path-mtu-discovery.md

Prerequisites: Lab 19: traceroute and TTL, TCP Lab 08: Retransmission, Windowing, and Loss

Expected time: 45–60 minutes.

The Goal

We set up client — router — server, where the router's link to the server has MTU 1400, and then:

  • the client sends a 1500-byte DF packet toward the server,
  • the router replies ICMP fragmentation-needed (mtu 1400) — it cannot forward such a big packet,
  • the client caches path MTU 1400 for the server, and a packet that fits gets through.

By the end, you should be able to explain this table:

Client sends Router (1400 link) Result
1500 bytes, DF too big, cannot fragment ICMP frag-needed (mtu 1400)
— — client caches path MTU = 1400
1300 bytes, DF fits delivered

What You Will Learn

  • What an MTU is and why a smaller-MTU link in the path is a bottleneck.
  • What the DF (Don't Fragment) bit does, and why modern IP relies on it.
  • How ICMP fragmentation-needed (type 3, code 4) carries the next-hop MTU.
  • How the sender discovers and caches the Path MTU (ip route get shows it).
  • Why a black-holed ICMP frag-needed (filtered) causes the classic "small pages load, big ones hang" bug.

This lab does not cover:

  • IPv4 fragmentation without DF, or reassembly details.
  • IPv6 PMTUD (routers never fragment; the mechanism is similar via ICMPv6 Packet Too Big).
  • TCP MSS clamping (a common workaround).

Where to Read in the RFCs

RFC Sections What to focus on
RFC 1191 3–4 How Path MTU Discovery works (DF + the MTU in ICMP frag-needed)
RFC 791 2.3, 3.2 IP fragmentation and the DF flag
RFC 792 Destination Unreachable ICMP type 3, code 4 (fragmentation needed)
RFC 5737 3 Confirming the addresses used here are documentation-only

The Big Picture

Three nodes: client, router, and server. Only the router→server link has MTU 1400.

client ---10.0.1.0/24 (MTU 1500)--- router ---10.0.2.0/24 (MTU 1400)--- server
10.0.1.1                        10.0.1.2   10.0.2.1                  10.0.2.2
Enter fullscreen mode Exit fullscreen mode

When the client sends a large DF packet, the router hits the narrow link (1400) and returns ICMP frag-needed.

sequenceDiagram
  participant C as client
  participant R as router
  participant S as server

  C->>R: IP 1500 bytes, DF set, to server
  Note over R: eth2 MTU=1400 < 1500, DF -> can't forward
  R-->>C: ICMP frag needed, next-hop MTU 1400
  Note over C: cache path MTU(server)=1400
  C->>R: IP 1300 bytes, DF set
  R->>S: fits, forward
  S-->>C: reply
Enter fullscreen mode Exit fullscreen mode

Note: The 10.0.0.0/8 subnets here are a local, closed range — nothing in this lab touches the real internet.

What You Need

Recommended environment:

  • Linux / WSL2 / a Linux VM
  • Docker
  • containerlab

Images used:

  • nicolaka/netshoot:latest — bundles ping (with -M do support), ip, and tcpdump.

No additional images are required. The narrow link's MTU is set by an exec in the topology (ip link set eth2 mtu 1400).

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run mtu-25
Enter fullscreen mode Exit fullscreen mode

Or step through it manually:

1. Move into the working directory 

cd protocol-lab/examples/mtu-25
Enter fullscreen mode Exit fullscreen mode

2. Deploy and set up routes 

sudo containerlab deploy -t mtu-25.clab.yml
docker exec clab-mtu-25-router sysctl -w net.ipv4.ip_forward=1
docker exec clab-mtu-25-client ip route add 10.0.2.0/24 via 10.0.1.2
docker exec clab-mtu-25-server ip route add 10.0.1.0/24 via 10.0.2.1
docker exec clab-mtu-25-router ip -br link show   # eth2 should show MTU 1400
Enter fullscreen mode Exit fullscreen mode

3. Send a big DF packet (it gets rejected) 

docker exec clab-mtu-25-client ping -M do -s 1500 -c1 10.0.2.2
Enter fullscreen mode Exit fullscreen mode

Expected:

From 10.0.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1400)
Enter fullscreen mode Exit fullscreen mode

The router (10.0.1.2) is telling us "I can only take up to 1400." The -M do flag sets DF.

4. See that the Path MTU was cached 

docker exec clab-mtu-25-client ip route get 10.0.2.2
Enter fullscreen mode Exit fullscreen mode 
10.0.2.2 via 10.0.1.2 dev eth1 src 10.0.1.1
    cache expires 597sec mtu 1400
Enter fullscreen mode Exit fullscreen mode

The client has learned — and cached — "the path to the server has MTU 1400."

5. A size that fits gets through 

docker exec clab-mtu-25-client ping -M do -s 1300 -c1 10.0.2.2
Enter fullscreen mode Exit fullscreen mode

1300 + 28 = 1328 < 1400, so it passes.

Expected Output

  • ping -s 1500 -M do: Frag needed and DF set (mtu = 1400).
  • Capture: ICMP ... unreachable - need to frag (mtu 1400) (type 3, code 4).
  • ip route get 10.0.2.2: mtu 1400 (the cached path MTU).
  • ping -s 1300 -M do: success.

Why It Works

The MTU (Maximum Transmission Unit) is the largest number of bytes a link can carry in one frame. Ethernet is normally 1500. If a link along the path has a smaller MTU, that link becomes the narrowest gate.

  • DF (Don't Fragment). A flag in the IP header that says "do not split this packet." In the old days, routers along the way would fragment oversized packets and pass them through — but fragmentation is inefficient and problematic. So the modern default is to set DF and never fragment in transit. TCP uses DF by default.
  • What happens when it can't pass. When a DF packet is larger than a link's MTU, the router cannot forward it (fragmentation is forbidden). So the router drops the packet and sends ICMP fragmentation-needed (type 3, code 4) back to the source. That ICMP carries the MTU the router can handle — the next-hop MTU (RFC 1191).
  • Path MTU Discovery. On receiving this ICMP, the sender learns "the path to this destination allows at most 1400" and caches it (the mtu in ip route get). From then on it sends at or below that size, so it never gets stuck again. If an even narrower link exists further along the path, another ICMP arrives and the sender learns an even smaller value.
  • Why it matters — the famous bug. If a firewall along the way drops ICMP frag-needed, the sender learns neither that it got stuck nor what the right size is. Large DF packets just keep silently vanishing. The result is the classic, hard-to-diagnose failure: "small requests (the page's HTML) go through, but large responses (images, a big TLS handshake) hang." It's the textbook example of why you must not blindly drop all ICMP.

The key insight: the sender learns the narrowest MTU on the path via DF and ICMP frag-needed, and sizes its packets to match. Block that ICMP, and the whole mechanism breaks.

Common Pitfalls

  • Confusing MTU with packet size. The MTU is the link's ceiling; packets must fit inside it. The N in ping -s N is the payload — add the IP/ICMP headers for the real on-wire size.
  • Forgetting DF. Without DF, a router (in the legacy behavior) would fragment the packet and pass it through, and PMTUD never happens. Use -M do to set DF.
  • Blocking ICMP. Drop frag-needed and PMTUD breaks — you get a blackhole.
  • Thinking path MTU is global. It's learned and cached per destination (per path). A different destination is a different entry.
  • The first packet fails. PMTUD is a "get stuck once, receive the ICMP, learn" mechanism. The first oversized packet is lost (TCP recovers by retransmitting smaller).
  • The IPv6 difference. IPv6 routers never fragment, so PMTUD is always mandatory there (via ICMPv6 Packet Too Big).

Cleanup 

sudo containerlab destroy -t mtu-25.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

If you used labctl.sh run mtu-25, the script runs destroy for you at the end.

Check Your Understanding

  1. What is an MTU? What happens when the path contains a link with a smaller MTU?
  2. What does the DF (Don't Fragment) bit instruct? Why does modern IP use DF?
  3. When a DF packet can't pass, what does the router send back? What does it contain?
  4. How does the sender learn the Path MTU, and where does it keep it?
  5. If a firewall drops ICMP frag-needed, what symptoms appear? Why is the failure hard to diagnose?
  6. How does the need for PMTUD differ between IPv4 and IPv6?

References

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

  • Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
  • Docker 29.1.3
  • containerlab 0.77.0
  • client / router / server: nicolaka/netshoot:latest (ping, tcpdump)

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run mtu-25 performed deploy → verify → destroy, and verification.json returned "status": "verified". The router's eth2 (server side) has MTU 1400.

The big DF packet is rejected 

$ docker exec clab-mtu-25-client ping -M do -s 1500 -c1 10.0.2.2
From 10.0.1.2 icmp_seq=1 Frag needed and DF set (mtu = 1400)
Enter fullscreen mode Exit fullscreen mode

Capture (ICMP on the client side):

10.0.1.2 > 10.0.1.1: ICMP 10.0.2.2 unreachable - need to frag (mtu 1400), length 556
Enter fullscreen mode Exit fullscreen mode

The router (10.0.1.2) returns type 3, code 4 (fragmentation needed) carrying next-hop MTU 1400.

The client learns and caches the Path MTU 

$ docker exec clab-mtu-25-client ip route get 10.0.2.2
10.0.2.2 via 10.0.1.2 dev eth1 src 10.0.1.1
    cache expires 597sec mtu 1400
Enter fullscreen mode Exit fullscreen mode

The client has learned and cached "the path to the server has MTU 1400."

A size that fits gets through 

$ docker exec clab-mtu-25-client ping -M do -s 1300 -c1 10.0.2.2
1 packets transmitted, 1 received, 0% packet loss
Enter fullscreen mode Exit fullscreen mode

1300 + 28 = 1328 < 1400, so it passes. The sender learns the narrowest MTU on the path via DF and ICMP frag-needed, and sizes its packets to match — that is Path MTU Discovery. Drop the ICMP somewhere along the way, and this learning breaks into a blackhole.

Cleanup 

containerlab destroy -t mtu-25.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

That's Path MTU Discovery: one deliberately dropped packet, one honest ICMP message, and the sender knows exactly how big it's allowed to go — as long as nobody filters the messenger.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, we'll look at the workarounds operators reach for when PMTUD can't be trusted — starting with TCP MSS clamping.

Top comments (0)