DEV Community

Cover image for Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux
Lyra
Lyra

Posted on

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

#devops #performance #linux #networking

Stop Leaving Throughput on the Table: Practical TCP BBR Congestion Control on Linux

Most Linux hosts still ship CUBIC as the default TCP congestion controller. That is a solid loss-based algorithm — until the path has meaningful RTT, shallow buffers, or a little random loss. Then CUBIC backs off on drops that were never a clean "the pipe is full" signal, and bulk transfers crawl while the link still has headroom.

BBR (Bottleneck Bandwidth and Round-trip propagation time) takes a different approach: it estimates the path's available bandwidth and minimum RTT, then paces sends around that model instead of treating every loss as congestion. Stock mainline Linux has shipped BBR (v1) since kernel 4.9. You do not need a custom kernel to try it.

This post is the operational checklist: load it, pair it with the right qdisc, verify live sockets, A/B test against CUBIC, and understand the fairness and BBRv3 caveats before you make it permanent.

What this is (and is not)

Layer Job Typical tools
Congestion control How fast this TCP sender ramps and reacts cubic, bbr, reno via tcp_congestion_control
Queueing / AQM How the host egress queue shares and drops/marks fq, fq_codel, cake
Packet steering Which CPU handles RX/TX work RSS, RPS, RFS, XPS

BBR is sender congestion control. It is not a replacement for bufferbloat fixes (CAKE / fq_codel on the WAN edge), L4 load balancing (IPVS), or multi-queue IRQ steering. Fix those problems with the right tool; use BBR when the TCP sender is the bottleneck on long or lossy paths.

Prerequisites

  • Linux 4.9+ (ideally a current LTS: 6.1/6.6/6.12-class)
  • Root (or equivalent) for sysctl, modules, and tc
  • iproute2 (ss, tc, ip)
  • Optional: iperf3 for controlled comparisons 
uname -r
sysctl net.ipv4.tcp_congestion_control
sysctl net.ipv4.tcp_available_congestion_control
sysctl net.core.default_qdisc
Enter fullscreen mode Exit fullscreen mode

On many distros the default looks like:

net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_available_congestion_control = reno cubic
net.core.default_qdisc = fq_codel   # or pfifo_fast / cake, depending on distro
Enter fullscreen mode Exit fullscreen mode

tcp_available_congestion_control only lists registered algorithms. If bbr is built as a module and not loaded yet, it may be absent until you load it.

1. Load BBR and set it for new connections 

# Load the module when BBR is not built-in
modprobe tcp_bbr

# Confirm it registered
sysctl net.ipv4.tcp_available_congestion_control
# expect something like: reno cubic bbr

# Switch the default for *new* connections
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl net.ipv4.tcp_congestion_control
# net.ipv4.tcp_congestion_control = bbr
Enter fullscreen mode Exit fullscreen mode

Notes from the kernel IP sysctl docs:

  • tcp_congestion_control applies to new connections.
  • For passive (accepted) connections, the listener's congestion-control choice is inherited.
  • Apps can still override per socket with setsockopt(..., TCP_CONGESTION, ...) when the name is allowed.
  • tcp_allowed_congestion_control restricts which algorithms unprivileged processes may select (default is a small subset including the system default).

Make the module stick across reboots:

# systemd-style module load
printf 'tcp_bbr\n' > /etc/modules-load.d/tcp_bbr.conf
Enter fullscreen mode Exit fullscreen mode

2. Pair BBR with fq (Fair Queue), not only with a wish

BBR paces. The classic pairing is fq (Fair Queue), which does per-flow separation and respects TCP pacing / EDT departure times for locally generated traffic.

# Default qdisc for *new* device setups after this point
sysctl -w net.core.default_qdisc=fq

# Apply fq on the live egress NIC now (replace eth0)
tc qdisc replace dev eth0 root fq
tc -s -d qdisc show dev eth0
Enter fullscreen mode Exit fullscreen mode

From tc-fq(8): FQ is meant mostly for locally generated traffic, separates flows, and honors pacing set by the TCP stack (including EDT after Linux 4.20). That is exactly the host-as-sender case for origin servers, backup nodes, and build hosts.

How this relates to CAKE / fq_codel

Goal Prefer
Host originates bulk TCP and you want BBR pacing to work cleanly fq + BBR on that host
Host is a router/gateway fighting bufferbloat for many flows CAKE or fq_codel on the bottleneck egress (often the WAN uplink)
You already run CAKE on the gateway Keep CAKE there; BBR still helps endpoints that originate long transfers

You can run BBR on servers behind a CAKE-shaped edge. Congestion control and AQM solve different layers. Just do not expect default_qdisc=fq on a pure router to replace proper bottleneck AQM.

systemd-networkd optional qdisc

If you manage interfaces with networkd and want the qdisc declared next to the NIC:

# /etc/systemd/network/20-wan.network
[Match]
Name=eth0

[Network]
DHCP=yes

[CAKE]
# Only if you intentionally want CAKE here instead of fq.
# Bandwidth=300M
Enter fullscreen mode Exit fullscreen mode

For BBR endpoints, prefer an explicit tc oneshot (below) or your distro's documented qdisc hook rather than forcing CAKE onto every origin NIC "because bufferbloat articles said so."

3. Persist with sysctl.d 

cat >/etc/sysctl.d/99-tcp-bbr.conf <<'EOF'
# TCP BBR + pacing-friendly default qdisc
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl --system
# or: sysctl -p /etc/sysctl.d/99-tcp-bbr.conf
Enter fullscreen mode Exit fullscreen mode

Remember: default_qdisc affects qdiscs created after the setting is applied. Existing interfaces may still show the old root qdisc until you tc qdisc replace them or reboot.

Optional: apply fq at boot with a oneshot 

# /etc/systemd/system/fq-wan.service
[Unit]
Description=Install fq qdisc on WAN NIC for BBR pacing
After=network-pre.target
Before=network.target
Wants=network-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
# Adjust interface name
ExecStart=/sbin/tc qdisc replace dev eth0 root fq
ExecReload=/sbin/tc qdisc replace dev eth0 root fq

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
systemctl daemon-reload
systemctl enable --now fq-wan.service
Enter fullscreen mode Exit fullscreen mode

4. Verify on live sockets (not only sysctl)

Sysctl only proves the default. Prove the flows:

# Idle check
sysctl net.ipv4.tcp_congestion_control net.core.default_qdisc

# While a transfer runs (curl, iperf3, restic, apt, etc.)
ss -tin

# Narrow to one peer
ss -tin dst 203.0.113.10
Enter fullscreen mode Exit fullscreen mode

Look for bbr in the TCP info block. On a modern ss, BBR also exposes diagnostics similar to:

bbr wscale:8,7 rto:216 rtt:15.924/4.256 ...
bbr:(bw:2.0Mbps,mrtt:14.451,pacing_gain:2.88672,cwnd_gain:2.88672)
pacing_rate 22.7Mbps delivery_rate 2.0Mbps
Enter fullscreen mode Exit fullscreen mode

Those bbr:(bw:...,mrtt:...) fields are exactly what the BBR FAQ recommends for operational inspection.

Also confirm the qdisc:

tc qdisc show dev eth0
# qdisc fq ... root ...
Enter fullscreen mode Exit fullscreen mode

Existing connections keep their old CC

Changing the sysctl does not rewrite congestion control on sockets that already exist. Restart long-lived proxies, database pools, or VPN daemons if you need them on BBR immediately — or wait for natural reconnect.

5. Fair A/B test: CUBIC vs BBR

Use a path that resembles production (real WAN RTT, not only localhost).

Terminal A (server):

iperf3 -s
Enter fullscreen mode Exit fullscreen mode

Terminal B (client), CUBIC baseline:

sysctl -w net.ipv4.tcp_congestion_control=cubic
# optional multi-stream bulk
iperf3 -c SERVER_IP -t 30 -P 4
# reverse direction (server sends)
iperf3 -c SERVER_IP -t 30 -P 4 -R
Enter fullscreen mode Exit fullscreen mode

Same client, BBR:

modprobe tcp_bbr
sysctl -w net.ipv4.tcp_congestion_control=bbr
sysctl -w net.core.default_qdisc=fq
tc qdisc replace dev eth0 root fq

iperf3 -c SERVER_IP -t 30 -P 4
iperf3 -c SERVER_IP -t 30 -P 4 -R

ss -tin dst SERVER_IP | head
Enter fullscreen mode Exit fullscreen mode

Record for each run:

  • throughput (sender and receiver lines)
  • RTT under load (ping -c 20 in parallel, or ss rtt fields)
  • retransmits (ss -ti retrans, or iperf retransmit counters when available)

Where BBR usually wins

ESnet and others report large gains on paths with higher RTT, shallow buffers, or loss that is not pure congestion. Double-digit speedups on some science/WAN paths are common in published testing; your mileage depends on the bottleneck.

Where CUBIC may be fine (or preferable)

  • Ultra-low-RTT datacenter LAN with deep, well-managed queues
  • Environments that must share fairly with a large population of loss-based senders and cannot tolerate BBRv1 aggressiveness
  • Paths where the real fix is loss elimination (bad optics, duplex mismatch, oversized buffers elsewhere), not a smarter sender

ESnet explicitly notes that BBRv1 can compete unfairly with CUBIC/HTCP on some shared bottlenecks, and that BBR is not a substitute for good network design or reducing loss.

6. Scoped overrides: per-route and per-socket

Per-destination with ip route … congctl

From ip-route(8) (Linux 3.20+):

# Suggest BBR only toward a prefix (apps may still override unless locked)
ip route change default via 192.0.2.1 dev eth0 congctl bbr

# Or lock it so applications cannot override
ip route replace 198.51.100.0/24 via 192.0.2.1 dev eth0 congctl lock bbr

# Prefer CUBIC toward a sensitive internal prefix while global default is BBR
ip route replace 10.0.0.0/8 via 10.0.0.1 dev eth1 congctl lock cubic
Enter fullscreen mode Exit fullscreen mode

Use this when only some destinations benefit (cross-region object storage, backup targets) and you want LAN defaults left alone.

Per-listener with TCP_CONGESTION

Kernel docs point at:

setsockopt(listenfd, SOL_TCP, TCP_CONGESTION, "bbr", 4);
Enter fullscreen mode Exit fullscreen mode

Accepted connections inherit the listener's choice. Handy for a single bulk-transfer service without flipping the whole host.

Unprivileged processes may only choose names listed in tcp_allowed_congestion_control. Expand that list deliberately if apps need to self-select:

sysctl net.ipv4.tcp_allowed_congestion_control
# example expansion (review security/ops policy first):
# sysctl -w net.ipv4.tcp_allowed_congestion_control="reno cubic bbr"
Enter fullscreen mode Exit fullscreen mode

7. BBRv1 vs BBRv2/v3 — be honest about mainline

Variant Where it lives Ops reality
BBRv1 Mainline tcp_bbr since Linux 4.9 What modprobe tcp_bbr gives you on stock distros
BBRv2 Research / older previews Largely superseded by v3 work for new testing
BBRv3 Google’s google/bbr v3 branch Not merged as of mid-2026 mainline; custom kernel + testing required

Google’s v3 README is explicit: clone/build their tree (or patch), reboot into that kernel, then sysctl net.ipv4.tcp_available_congestion_control should still show bbr — but the code behind the name is v3. They also ship iproute2 patches for richer ss diagnostics and an ecn_low per-route feature for L4S-style ECN environments.

Recommendation for production fleets today: run stock BBRv1 + fq, measure, and keep a rollback sysctl. Treat BBRv3 as a lab/custom-kernel project unless you already maintain kernel patches and have path-specific evidence.

8. Emulation pitfalls (so your lab does not lie)

From the BBR FAQ:

  • Do not put netem loss/delay on the sending machine if you want realistic TCP results (interaction with TSQ and related mechanisms). Put netem on an intermediate router namespace/host or on the receiver ingress (IFB pattern).
  • netem loss decisions can apply to whole TSO bursts, which is harsher/burstier than per-MTU loss — disable GRO/LRO when you need finer loss models.
  • For serious CC comparison matrices, consider tools like transperf rather than a single noisy iperf run.

9. Rollback 

sysctl -w net.ipv4.tcp_congestion_control=cubic
# optional: restore previous qdisc policy
sysctl -w net.core.default_qdisc=fq_codel
tc qdisc replace dev eth0 root fq_codel   # or cake / distro default

rm -f /etc/sysctl.d/99-tcp-bbr.conf
rm -f /etc/modules-load.d/tcp_bbr.conf
systemctl disable --now fq-wan.service 2>/dev/null || true
sysctl --system
Enter fullscreen mode Exit fullscreen mode

Long-lived sockets still need recycle to leave BBR.

Practical defaults I use

  1. Origin / backup / artifact hosts that push over WAN: bbr + fq, verified with ss -tin.
  2. Edge routers fighting bufferbloat: keep CAKE/fq_codel on the bottleneck; do not pretend BBR replaces AQM.
  3. Latency-sensitive internal fabrics with excellent loss metrics: stay on CUBIC until measurements say otherwise.
  4. Mixed policy: global CUBIC, congctl lock bbr only toward known high-BDP prefixes — or the inverse.
  5. Never flip production without a timed iperf/restic/apt mirror before/after and a one-line rollback.

Checklist

  • [ ] modprobe tcp_bbr and bbr appears in tcp_available_congestion_control
  • [ ] net.ipv4.tcp_congestion_control=bbr persisted under /etc/sysctl.d/
  • [ ] net.core.default_qdisc=fq persisted
  • [ ] Live NIC root qdisc is fq (tc qdisc show)
  • [ ] Active flows show bbr in ss -tin
  • [ ] A/B iperf (or real workload) recorded vs CUBIC on the real path
  • [ ] Rollback sysctl documented
  • [ ] BBRv3 custom kernels isolated to lab unless intentionally adopted

References

  1. Linux IP sysctl — tcp_congestion_control, tcp_available_congestion_control, tcp_allowed_congestion_control: docs.kernel.org/networking/ip-sysctl.html
  2. tc-fq(8) Fair Queue scheduler: man7.org/linux/man-pages/man8/tc-fq.8.html
  3. ip-route(8) — congctl / congctl lock: man7.org/linux/man-pages/man8/ip-route.8.html
  4. Google BBR FAQ (ss diagnostics, netem caveats, TCP_CC_INFO): github.com/google/bbr/blob/master/Documentation/bbr-faq.md
  5. Google BBRv3 preview README: github.com/google/bbr/blob/v3/README.md
  6. ESnet Fasterdata — BBR TCP notes and fairness cautions: fasterdata.es.net/host-tuning/linux/recent-tcp-enhancements/bbr-tcp/
  7. ACM Queue — BBR: Congestion-Based Congestion Control: queue.acm.org/detail.cfm?id=3022184
  8. Mainline tcp_bbr.c (net-next): git.kernel.org/.../tcp_bbr.c

CUBIC is not "wrong." It is the wrong default for some paths. Measure your RTT, loss, and bulk flows, put BBR where the sender model matches the path, keep AQM on the real bottleneck, and keep the rollback sysctl one file away.

Top comments (0)