DEV Community

pathvector-dev
pathvector-dev

Posted on • Originally published at blog.pathvector.dev

CUBIC vs BBR: How Your Congestion Control Choice Changes Throughput 7x on a Lossy Path

#tcp #network #containerlab #bbr

Originally published at https://blog.pathvector.dev/protocol-lab-cc-30/ — part of the free Protocol Lab series.

This post is part of Protocol Lab, a free, hands-on series for learning networking protocols by building and breaking them in a container lab. All the lab material — topologies, configs, and scripts — lives in the repo: github.com/pathvector-studio/protocol-lab.

In Lab #08, we watched TCP react to loss and delay. This lab shows that how TCP reacts is a choice — the congestion control algorithm — and that on a lossy, long-RTT path, that choice changes throughput by nearly an order of magnitude.

Reading guide: rfc-notes/tcp-congestion-control.md

Prerequisite: TCP Lab 08: Loss, Retransmission, and the Window

Expected time: 40–55 minutes.

The Goal

You impair one path (100 ms RTT, 2% random loss, 100 Mbit/s cap via tc netem) and run iperf3 twice over it:

  • with CUBIC (the Linux default, loss-based): every drop is read as congestion, so cwnd is held small → throughput collapses far below the cap;
  • with BBR (model-based): it estimates the bottleneck bandwidth and RTT and paces to them, largely ignoring random loss → throughput stays near the cap.

By the end, you should be able to explain this table:

Algorithm signal for "too fast" throughput on 100 ms / 2% loss / 100 Mbit path
CUBIC (loss-based) a packet drop ~12 Mbit/s (collapses)
BBR (model-based) bandwidth/RTT model ~88 Mbit/s (near the cap)

What You Will Learn

  • What a congestion window (cwnd) is, and why throughput ≈ cwnd / RTT.
  • The difference between loss-based (Reno, CUBIC) and model-based (BBR) congestion control.
  • Why random (non-congestive) loss wrecks a loss-based algorithm but not BBR.
  • How to read congestion state from ss -ti (cwnd, ssthresh, pacing_rate, the algorithm's own stats).
  • How to select an algorithm per connection (iperf3 -C) or system-wide (sysctl).

This lab does not cover:

  • Fairness between competing flows (BBR vs CUBIC sharing one bottleneck).
  • BBRv2/v3 specifics, or ECN-based control (DCTCP).
  • Tuning buffers/bufferbloat in depth (touched on in Lab 28).

Where to Read in the RFCs

Reference What to focus on
RFC 5681 The core of TCP congestion control (slow start, congestion avoidance, cwnd/ssthresh)
RFC 8312 CUBIC (loss-based, built for high-BDP paths, the Linux default)
The BBR paper (Cardwell et al.) Model-based control (estimating bottleneck bandwidth × RTT and pacing to it)
RFC 5737 Confirming the addresses used here are local/documentation-only (supplementary)

The Big Picture

Two nodes, a client and a server, with netem impairing the path between them.

client (10.0.0.1) --- eth1/eth1 --- server (10.0.0.2, iperf3 -s)
  netem: 50ms delay,                 netem: 50ms delay (return path)
         2% loss, 100mbit rate cap
  → RTT ~100ms, lossy, capped at 100 Mbit/s
Enter fullscreen mode Exit fullscreen mode

Over the same impaired path, we switch between CUBIC and BBR and measure iperf3 throughput.

flowchart LR
  subgraph path["100 ms RTT · 2% loss · 100 Mbit cap"]
    direction LR
    C["client<br/>iperf3 -c"] -->|data| S["server<br/>iperf3 -s"]
  end
  C -.->|"-C cubic → ~12 Mbit/s<br/>(loss = congestion, cwnd held small)"| note1[ ]
  C -.->|"-C bbr → ~88 Mbit/s<br/>(models bw×RTT, paces to cap)"| note2[ ]
Enter fullscreen mode Exit fullscreen mode

10.0.0.0/24 is a local, closed range.

Note: Everything here uses local address space, so nothing in this lab touches the real internet.

What You Need

Recommended environment:

  • Linux / WSL2 / a Linux VM (a modern kernel with BBR available)
  • Docker
  • containerlab

Images used:

  • nicolaka/netshoot:latest — bundles tc, iperf3, and ss.

Prerequisite: The host kernel must have BBR. Check that sysctl net.ipv4.tcp_available_congestion_control includes bbr (if not, modprobe tcp_bbr). Congestion control runs in the host's TCP stack, so nothing extra is needed inside the containers.

Running the Lab

The quick path, which deploys, verifies, and tears down for you:

./scripts/labctl.sh run cc-30
Enter fullscreen mode Exit fullscreen mode

Or step through it manually:

1. Move into the working directory 

cd protocol-lab/examples/cc-30
Enter fullscreen mode Exit fullscreen mode

2. Deploy and start the iperf3 server 

sudo containerlab deploy -t cc-30.clab.yml
docker exec -d clab-cc-30-server iperf3 -s
Enter fullscreen mode Exit fullscreen mode

3. Impair the path (100 ms RTT, 2% loss, 100 Mbit cap) 

docker exec clab-cc-30-client tc qdisc add dev eth1 root netem delay 50ms loss 2% rate 100mbit
docker exec clab-cc-30-server tc qdisc add dev eth1 root netem delay 50ms
docker exec clab-cc-30-client ping -c3 10.0.0.2   # RTT ~100ms
Enter fullscreen mode Exit fullscreen mode

4. Measure with CUBIC (loss-based) 

docker exec clab-cc-30-client iperf3 -c 10.0.0.2 -C cubic -t 10
# while the transfer runs, peek at cwnd from another terminal:
docker exec clab-cc-30-client sh -c "ss -tin dst 10.0.0.2 | tr ',' '\n' | grep -E 'cubic|cwnd|ssthresh|retrans'"
Enter fullscreen mode Exit fullscreen mode

Throughput lands far below the 100 Mbit cap — loss keeps cwnd suppressed.

5. Measure with BBR (model-based) 

docker exec clab-cc-30-client iperf3 -c 10.0.0.2 -C bbr -t 10
docker exec clab-cc-30-client sh -c "ss -tin dst 10.0.0.2 | tr ',' '\n' | grep -E 'bbr|cwnd|pacing_rate'"
Enter fullscreen mode Exit fullscreen mode

Throughput climbs back to near the cap — BBR estimates the bandwidth and paces to it.

6. Remove the impairment 

docker exec clab-cc-30-client tc qdisc del dev eth1 root
docker exec clab-cc-30-server tc qdisc del dev eth1 root
Enter fullscreen mode Exit fullscreen mode

Expected Output

  • ping shows an RTT of about 100 ms.
  • CUBIC: throughput far below the 100 Mbit cap (about 12 Mbit/s in this environment). ss shows small cwnd/ssthresh.
  • BBR: throughput near the cap (about 88 Mbit/s in this environment). ss shows a large cwnd, a pacing_rate of about 99 Mbit, and bbr:(bw:...) reporting roughly 100 Mbit.

Why It Works

TCP has no dial that says "this link is 100 Mbit/s." The sender has to guess how fast it can go, and it controls how much it sends with the congestion window (cwnd) — the number of bytes allowed in flight at once. Effective speed ≈ cwnd / RTT. What differs between algorithms is what they treat as the signal for "too fast."

  • CUBIC (loss-based). It reads a packet drop as a congestion signal and slashes cwnd every time it sees one (recovery follows a cubic curve to speed things up, but the premise remains loss = congestion). On this path, though, the loss is random — injected by netem, not congestion. CUBIC backs off anyway, so cwnd stays pinned low and cwnd/RTT ends up far below the cap. At 100 ms RTT and 2% loss, it drops to roughly the Mathis approximation MSS/(RTT·√p).
  • BBR (model-based). It doesn't use loss as a congestion signal. Instead it continuously estimates the bottleneck bandwidth and RTprop (the minimum RTT) and paces its sending to match the bandwidth. Random loss is no evidence that the bandwidth shrank, so BBR doesn't slow down. It keeps cwnd large and holds throughput near the cap. BBR still retransmits (you'll see retrans counters) — the key point is that it doesn't collapse cwnd.
  • That's why the same path yields CUBIC at 12 Mbit and BBR at 88 Mbit — a difference of more than 7x. "Slow" isn't always a bandwidth problem; sometimes it's the interaction of loss × RTT × congestion control.

The key insight: the path fixes the link bandwidth and RTT, but how much of it you actually get is decided by the sender's congestion control. How the algorithm interprets loss dramatically changes the outcome on a lossy path.

Common Pitfalls

  • Assuming loss always means congestion. Wireless links, bit errors, and light netem-style loss are not congestion. Loss-based algorithms back off anyway — that's the heart of the CUBIC collapse here.
  • Thinking "BBR ignores loss" means it abandons reliability. It still retransmits (TCP guarantees reliability). The only thing it refuses to cut is cwnd.
  • Thinking speed is determined by bandwidth alone. cwnd/RTT is the constraint. The larger the RTT, the slower you go for the same cwnd.
  • Thinking congestion control is a container-side setting. It runs in the host's TCP stack. If the host doesn't have BBR, you can't select bbr.
  • Measurement variance. netem's random loss makes results jitter. Read them by order of magnitude (is CUBIC a small fraction of the cap? is BBR near it?) rather than exact numbers.
  • Large MSS. The containers use an MTU of 9500 (jumbo frames). ss may show an mss of 9448, but the behavior's essence is the same.

Cleanup 

sudo containerlab destroy -t cc-30.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

If you used labctl.sh run cc-30, the script runs destroy for you at the end.

Check Your Understanding

  1. What is the congestion window (cwnd)? Why is throughput proportional to cwnd/RTT?
  2. What do loss-based (CUBIC) and model-based (BBR) algorithms each treat as the signal for "too fast"?
  3. Why does random (non-congestive) loss badly hurt CUBIC's throughput?
  4. How does BBR handle loss? Why is its throughput high even though it also retransmits?
  5. How do you tell the state of CUBIC vs BBR apart from ss -ti's cwnd / ssthresh / pacing_rate?
  6. When "the line is fast but the transfer is slow," on what kind of path can congestion control be the culprit?

References

Verified Run Log (2026-07-07)

This lab has been confirmed reproducible on real hardware.

Environment:

  • Ubuntu 26.04 LTS (kernel 7.0.0-27-generic, x86_64)
  • Docker 29.1.3
  • containerlab 0.77.0
  • client / server: nicolaka/netshoot:latest (tc, iperf3, ss)
  • host kernel tcp_available_congestion_control: reno cubic bbr

Running PATH="/tmp/pl-shim:$PATH" ./scripts/labctl.sh run cc-30 performed deploy → verify → destroy, and verification.json returned "status": "verified".

CUBIC vs BBR on the same lossy path

Path: RTT ~100 ms, 2% random loss, 100 Mbit/s cap (tc netem).

[protocol-lab][cc-30] impairing the path: 50ms each way (RTT ~100ms), 2% loss, 100mbit cap
rtt min/avg/max/mdev = 100.043/100.058/100.075/0.013 ms
[protocol-lab][cc-30] cubic: 12 Mbit/s
[protocol-lab][cc-30] bbr: 88 Mbit/s
Enter fullscreen mode Exit fullscreen mode

On the identical impaired path, CUBIC collapsed to 12 Mbit/s (about 1/8 of the 100 Mbit cap) while BBR held 88 Mbit/s (near the cap). The algorithm choice alone made a difference of more than 7x.

What ss -ti says about cwnd

ss -tin snapshots during the transfers (key excerpts):

# CUBIC — loss pins cwnd/ssthresh small
cubic ... cwnd:15 ssthresh:9 bytes_retrans:141720 delivery_rate:9601152 ...

# BBR — estimates ~100 Mbit of bandwidth, allows a large cwnd, and paces
bbr ... cwnd:276 ssthresh:142 bbr:(bw:100181184bps cwnd_gain:2)
       pacing_rate:99179368bps delivery_rate:94801776 ...
Enter fullscreen mode Exit fullscreen mode
  • CUBIC sits at cwnd:15 (so throughput ≈ cwnd/RTT is tiny). It keeps treating the random loss as congestion and keeps cutting.
  • BBR estimates bbr:(bw:100181184bps) — roughly 100 Mbit — sends at a pacing_rate of about 99 Mbit, and runs a large cwnd:276.
  • An interesting detail: BBR's absolute retransmission volume (bytes_retrans 992040) is actually higher than CUBIC's (141720). BBR keeps sending through loss (so retransmissions increase), but it doesn't collapse cwnd, so throughput stays high. Its design decision — loss is not a "slow down" signal — shows up directly in the numbers.

Cleanup 

containerlab destroy -t cc-30.clab.yml --cleanup
Enter fullscreen mode Exit fullscreen mode

That's congestion control in one experiment: the path sets the bandwidth and the RTT, but the sender's algorithm decides how much of that path you actually get — and on a lossy path, interpreting loss correctly is worth 7x.

Explore the full Protocol Lab series here: github.com/pathvector-studio/protocol-lab. If these labs are useful to you, please ⭐ star the repo on GitHub — it genuinely helps others find the project.

Next up, Lab #31 flips from "how fast can one path go" to "which server do you even reach": anycast — one IP address announced from many places, with BGP picking the winner.

Top comments (0)