Reproducing a flaky network in Linux namespaces with netem

Faking a flaky network takes one Linux box. You need three network namespaces, a veth pair on each side of the middle one, and tc netem on the egress of the middle hop. No VMs and no spare hardware. Everything below runs as root and leaves nothing behind once you delete the namespaces.

This post is the recipe and the measurement method. We use the same setup in our lab to put a VPN client under impairment and watch what it reports. Here the setup is the subject, and any program that sends packets will do.

What we are building

Three namespaces in a line:

client  --  mid  --  server

mid forwards packets and is the only place where we break things. The client and server never know. That matters, because a test that touches the endpoints directly measures your test, not the path.

Build the topology

for n in c m s; do ip netns add $n; ip -n $n link set lo up; done

ip link add c0 netns c type veth peer name m0 netns m

# second pair, server side
ip link add s0 netns s type veth peer name m1 netns m

for spec in c:c0:10.10.1.2 m:m0:10.10.1.1 m:m1:10.10.2.1 s:s0:10.10.2.2; do
  set -- ${spec//:/ }
  ip -n $1 addr add $3/24 dev $2 && ip -n $1 link set $2 up
done

# routes, then let the middle namespace forward
ip -n c route add default via 10.10.1.1 && ip -n s route add default via 10.10.2.1 && ip netns exec m sysctl -w net.ipv4.ip_forward=1

Check that the path works before you break anything:

ip netns exec c ping -c 3 10.10.2.2

If that fails, stop here. Every later result depends on a clean baseline, and a typo in a route will look exactly like packet loss.

Record a baseline first

Run each measurement once with no impairment and keep the output. When a number looks odd under fault, the baseline tells you whether the fault or the machine produced it.

We use two tools. ping gives a per-packet view of delay and loss. iperf3 gives sustained throughput.

ip netns exec s iperf3 -s -D
sleep 1
ip netns exec c iperf3 -c 10.10.2.2 -t 10 --json > baseline-iperf.json
# and the per-packet view
ip netns exec c ping -c 50 -i 0.2 10.10.2.2 > baseline-ping.txt

The veth pair is fast, so the baseline will be limited by your CPU and not by any link. That is fine. You are measuring change against yourself, never against an absolute figure.

A logger that does not lie

Summary output from ping hides the shape of a failure. A path that drops every packet for three seconds and then recovers can show the same loss percentage as a path that drops one packet in twenty. Those are different problems.

So log each probe on its own line, with a timestamp:

ip netns exec c bash -c '
while true; do
  t=$(date +%s.%N)
  if ping -c1 -W1 10.10.2.2 >/dev/null 2>&1; then r=ok; else r=fail; fi
  echo "$t,$r"
  sleep 0.2
done' > probe.csv &
echo $! > probe.pid

One row per probe means you can line the log up against the moment you applied each impairment. Write the time you applied it into a second file. Do this by hand if you must, but do it every time.

Impairment one: delay and jitter

Apply netem on the egress of m1, the interface facing the server:

ip netns exec m tc qdisc add dev m1 root netem delay 80ms 20ms

That adds a base delay with random variation around it. Note that netem acts on outgoing packets only. A single qdisc on m1 delays the forward direction. To delay the return path as well, add a matching qdisc on m0. Without it, only the forward leg is delayed, so a ping's round trip grows by the configured delay once, not twice.

Change values in place with change, not add:

ip netns exec m tc qdisc change dev m1 root netem delay 200ms 50ms

Remove it with:

ip netns exec m tc qdisc del dev m1 root

Impairment two: random loss

ip netns exec m tc qdisc add dev m1 root netem loss 5%

Random loss is the gentle case. Real networks tend to lose packets in bursts. netem's optional correlation value does little to produce real bursts, so use its Gilbert-Elliott model instead:

ip netns exec m tc qdisc change dev m1 root netem loss gemodel 1% 20%

The first number is the chance of moving from a good state into a bad one, the second the chance of moving back. Every packet in the bad state is dropped, so this loses about 5% of packets on average, in runs of about five. Compare the two runs in your probe log. The count of failed probes may be similar, but the runs of consecutive failures will not be. Whatever you test, test it against both shapes. Retransmission logic that copes with scattered loss can struggle with a burst.

Impairment three: a narrow link

Netem has a rate option, but a token bucket filter gives you more honest queueing behaviour. Put netem under it if you want delay as well:

# remove the loss qdisc from the previous step first
ip netns exec m tc qdisc del dev m1 root
ip netns exec m tc qdisc add dev m1 root handle 1: tbf rate 5mbit burst 32kbit latency 400ms
# netem hangs off the bucket as a child
ip netns exec m tc qdisc add dev m1 parent 1:1 handle 10: netem delay 40ms

Now run iperf3 again and compare it with the baseline file. Throughput should settle close to the configured rate. If it does not, check that you applied the filter on the interface the traffic actually leaves through.

Impairment four: the path that silently shrinks

This one deserves its own section, because it is the hardest to spot from the endpoints.

A packet with the don't-fragment bit set that is too big for a hop should trigger an ICMP "fragmentation needed" message back to the sender. The sender then lowers its packet size. If that message never arrives, the sender keeps sending large packets that vanish. Small packets still pass. Pings work, the connection opens, and then anything bulky stalls.

Reproduce it in two steps. First, shrink the MTU on the far-side interface:

ip -n m link set m1 mtu 1400

Second, drop the ICMP errors that the middle hop would generate. Save this as drop-icmp.nft:

table inet t {
  chain out {
    type filter hook output priority 0;
    ip protocol icmp icmp type destination-unreachable drop
  }
}

Load it inside the middle namespace:

ip netns exec m nft -f drop-icmp.nft

Now compare a small probe with a large one:

# small probe
ip netns exec c ping -c 3 -M do -s 100 10.10.2.2
# large probe
ip netns exec c ping -c 3 -M do -s 1472 10.10.2.2

The first succeeds. The second gets no answer and no error message either. That silence is the whole point. A program that only checks whether the host answers will report a healthy path while real traffic goes nowhere.

To undo it, delete the nft table and set the MTU back to 1500:

ip netns exec m nft delete table inet t && ip -n m link set m1 mtu 1500

Reading the log

When a run is over, stop the logger and look at three things in probe.csv.

First, the time from the moment you applied an impairment to the first failed probe. Second, the longest outage. Third, the time from the moment you removed the impairment to the first success. Recovery time is the number people forget to measure, and it often tells you more than the failure itself.

This short awk script covers the second item:

awk -F, '$2=="fail"{if(!n)s=prev; n++; next} {if(n){d=$1-s; if(d>m)m=d} n=0; prev=$1} END{if(n){d=$1-s; if(d>m)m=d}; print m+0}' probe.csv

It prints the longest outage in seconds, measured from the last successful probe before the failures to the first success after them. It takes the duration from the timestamps, not from a count of rows, because a failed probe waits up to a second for its reply on top of the 0.2 second sleep. A probe cycle then takes about 1.2 seconds, so say in your notes that the figure is only as fine as that. The script assumes the first row is a success, which the baseline ping gives you.

What this tells you and what it does not

This is a lab. Namespaces on one machine share a clock and a CPU, so timing under heavy load will wobble. Netem's delay and loss are statistical models, not recordings of any real network. Run each condition several times and keep every log, because a single run can mislead you. We publish no results here for that reason: the useful part is the method, and your numbers will depend on your hardware.

It also tests one direction of cause at a time. Real faults combine. Add delay and loss together, then add the narrow link, and look again. The interactions are often where the surprises are.

If you test software that reports its own view of the connection, line its reports up against the probe log. In our lab we do this with a VPN client, and the useful question is whether what the client says matches what the probe log shows. When the two disagree, one of them is wrong, and the log is the easier one to audit.

Clean up

kill $(cat probe.pid)
ip netns pids s | xargs -r kill
for n in c m s; do ip netns del $n; done

Deleting the namespaces removes the veth pairs and every qdisc with them.

Checklist

• Build the path, then ping it before breaking anything.
• Record a baseline for every tool you plan to use.
• Log one line per probe, with a timestamp.
• Note the time you apply and remove each impairment.
• Impair both directions when you mean a round trip.
• Repeat each condition and keep every log.
• State your method next to every number you share.

This article was created with the help of AI.