DEV Community

Cover image for Your Link Is Probably Fine: Practical TCP Congestion Control on Linux
Lyra
Lyra

Posted on

Your Link Is Probably Fine: Practical TCP Congestion Control on Linux

A gigabit link that delivers 300 Mbps on a single TCP stream is not necessarily a broken link. It is often a control loop doing exactly what it was designed to do: filling a buffer until something drops, then backing off. The bandwidth was there the whole time.

Two different problems get blamed on "the network" this way. On a path with a deep buffer, loss-based congestion control fills the queue before it learns the ceiling, so throughput looks fine in a bulk test while interactive traffic sees RTTs in the hundreds of milliseconds. On a path with random loss — Wi-Fi, a long haul with a policer, a congested peering point — loss-based control treats every drop as congestion and gives up bandwidth that was actually available.

Linux ships several congestion control algorithms and lets you choose per host, per network namespace, and per socket. This article covers how that choice is wired up, what BBR actually computes, how to verify it on a live connection, and when switching is the wrong move.

Congestion control is a control loop, not a bandwidth setting

TCP has two output levers: the congestion window (how many bytes may be in flight) and, on modern kernels, pacing (how fast those bytes are released). Both are driven by feedback from ACKs: delivered bytes, RTT samples, loss, and optionally ECN marks.

Algorithms differ in what they treat as the signal.

CUBIC (the default in mainline Linux) is loss-based. It grows the window as a cubic function of time since the last congestion event, and on loss it applies a multiplicative decrease with β = 0.7 (RFC 9438). It needs a loss event before it will stop growing. That is fine when the only source of loss is a full bottleneck queue. It is expensive when the buffer is deep, because the queue has to fill before the signal arrives, and it is wrong when loss is random rather than congestion-driven.

BBR is model-based. It estimates the bottleneck bandwidth and the round-trip propagation delay, then sends at the estimated rate with a small standing queue. It does not need to overflow a buffer to find the ceiling. The trade-off is that it also does not yield the way a loss-based flow does, which matters on shared bottlenecks.

The three sysctls people mix up

On a Debian 13 host running kernel 6.12, the relevant files read:

$ cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
$ cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic
$ cat /proc/sys/net/ipv4/tcp_allowed_congestion_control
reno cubic
Enter fullscreen mode Exit fullscreen mode

They answer three different questions:

  • tcp_congestion_control is the default for new connections. Existing sockets keep whatever they negotiated when they were created.
  • tcp_available_congestion_control lists algorithms currently registered with the kernel. Modules that are not loaded do not appear here. That is why bbr is missing above even though the module exists on disk.
  • tcp_allowed_congestion_control is the subset an unprivileged process may select with setsockopt(). By default it contains reno plus the current default algorithm.

The module is present but not loaded:

$ ls /lib/modules/$(uname -r)/kernel/net/ipv4/ | grep bbr
tcp_bbr.ko.xz
$ grep CONFIG_TCP_CONG_BBR /boot/config-$(uname -r)
CONFIG_TCP_CONG_BBR=m
Enter fullscreen mode Exit fullscreen mode

That =m is the whole explanation. Debian builds BBR as a module, so the algorithm does not exist until something loads it.

Turn it on, and make it survive a reboot

Load the module and re-read the list:

$ sudo modprobe tcp_bbr
$ cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic bbr
Enter fullscreen mode Exit fullscreen mode

Now select it for new connections:

$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr
Enter fullscreen mode Exit fullscreen mode

If sysctl is not installed, the equivalent is a write to the proc file:

$ echo bbr | sudo tee /proc/sys/net/ipv4/tcp_congestion_control
Enter fullscreen mode Exit fullscreen mode

To persist both the module and the setting, use two drop-ins. First the module:

# /etc/modules-load.d/bbr.conf
tcp_bbr
Enter fullscreen mode Exit fullscreen mode

Then the sysctl:

# /etc/sysctl.d/90-bbr.conf
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
Enter fullscreen mode Exit fullscreen mode

Apply without rebooting:

$ sudo systemctl restart systemd-modules-load.service
$ sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

The ordering matters, and systemd already handles it: systemd-sysctl.service is ordered After=systemd-modules-load.service (you can confirm with systemctl cat systemd-sysctl.service). The module is registered before the sysctl write is attempted. If you write the sysctl first on a fresh boot without the module, the kernel rejects the unknown algorithm name with No such file or directory — the name genuinely does not exist yet.

The fq question

BBR sets a pacing rate. Historically that required the fq qdisc. TCP-level pacing landed in 4.13 and became reliable for BBR in 4.20, so on any current kernel a different qdisc still works. Google's own quick-start still recommends fq because it performs better on heavily loaded servers. Setting net.core.default_qdisc = fq only affects qdiscs created afterwards; an interface that is already up keeps its current qdisc until it is re-created. To change it now:

$ sudo modprobe sch_fq
$ sudo tc qdisc replace dev eth0 root fq
$ tc qdisc show dev eth0
Enter fullscreen mode Exit fullscreen mode

On a multiqueue NIC you will often see mq at the root with one child qdisc per hardware queue. That is normal: the kernel uses mq as the root for multiqueue devices and applies default_qdisc to its leaves.

The trap that makes people think BBR did nothing

Two behaviours in the kernel documentation are easy to miss:

  1. The sysctl applies to new connections only.
  2. "For passive connections, the listener congestion control choice is inherited."

The second one bites hardest. If nginx, HAProxy, or your own daemon created its listening socket before you changed the sysctl, every connection it accepts inherits the listener's old algorithm — cubic — no matter what the global default says. Restart the service, or set the algorithm explicitly on the listening socket, before you conclude that BBR did not help.

You can check a live socket instead of guessing:

$ ss -tin dst 198.51.100.20
Enter fullscreen mode Exit fullscreen mode

A BBR connection prints a block like this (trimmed from the example in Google's BBR FAQ):

  bbr wscale:8,7 rto:216 rtt:15.924/4.256 ato:40 mss:1348 pmtu:1500
  cwnd:16 bytes_acked:3744 bytes_received:8845
  bbr:(bw:2.0Mbps,mrtt:14.451,pacing_gain:2.88672,cwnd_gain:2.88672)
  send 10.8Mbps lastsnd:8208 lastrcv:8188 lastack:8188
  pacing_rate 22.7Mbps delivery_rate 2.0Mbps app_limited
  minrtt:14.451
Enter fullscreen mode Exit fullscreen mode

Read it in this order:

  • bbr: confirms the socket is actually running BBR.
  • bw is the bottleneck-bandwidth estimate, mrtt the min-RTT estimate.
  • pacing_rate is what the kernel wants to send; delivery_rate is what the path is delivering.
  • app_limited means the sender had no more data queued. No congestion control change will fix a sender that is not sending. This is the most common false negative in "BBR made no difference" reports.

What BBR is actually computing

The in-tree implementation states its model at the top of net/ipv4/tcp_bbr.c:

bottleneck_bandwidth = windowed_max(delivered / elapsed, 10 round trips)
min_rtt              = windowed_min(rtt, 10 seconds)
pacing_rate          = pacing_gain * bottleneck_bandwidth
cwnd                 = max(cwnd_gain * bottleneck_bandwidth * min_rtt, 4)
Enter fullscreen mode Exit fullscreen mode

It runs a four-state machine:

  • STARTUP: pacing gain 2/ln(2) ≈ 2.885, which doubles the sending rate each round trip. It exits when the bandwidth estimate fails to grow by 25% for three consecutive rounds.
  • DRAIN: gain 1/2.885, to remove the queue STARTUP built.
  • PROBE_BW: an eight-phase pacing cycle — 1.25, 0.75, then six phases at 1.0 — with a randomized starting phase. The 1.25 phase probes for more capacity; the 0.75 phase drains what it built.
  • PROBE_RTT: if no RTT sample at or below the current minimum arrives for 10 seconds, the window is cut to 4 packets for 200 ms to re-measure propagation delay.

The design goal is explicit in the source: pace at about 1% below the estimated bandwidth, on average, to keep the bottleneck queue small.

And the important limitation is stated there too:

The core algorithm does not react directly to packet losses or delays, although BBR may adjust the size of next send per ACK when loss is observed, or adjust the sending rate if it estimates there is a traffic policer, in order to keep the drop rate reasonable.

That is the whole fairness story in one sentence. BBR v1 does not back off because a packet was lost. It backs off when its own model says the pipe is narrower than it thought. If you share a bottleneck with loss-based flows, and you do not control the other end, measure the mix before assuming BBR is a free win.

The in-tree module on kernel 6.12 is this v1 design. BBRv3 is specified in draft-ietf-ccwg-bbr-07 and published as a patch set in Google's bbr repository; it folds packet loss rate into the model, which the in-tree v1 code does not do. Do not assume the bbr your distribution loads is v3.

Choosing the algorithm per socket

Host-wide defaults are a blunt instrument. If only one workload needs BBR — a backup uploader, a replication stream, a media origin — set it on that socket. TCP_CONGESTION takes a string:

#define _GNU_SOURCE
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>

int main(void)
{
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    if (fd < 0) { perror("socket"); return 1; }

    const char *algo = "bbr";
    if (setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION,
                   algo, strlen(algo)) < 0) {
        perror("setsockopt(TCP_CONGESTION)");
        return 1;
    }

    char got[16] = {0};
    socklen_t len = sizeof(got);
    if (getsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, got, &len) == 0)
        printf("congestion control: %s\n", got);

    struct sockaddr_in dst = {0};
    dst.sin_family = AF_INET;
    dst.sin_port = htons(443);
    inet_pton(AF_INET, "198.51.100.20", &dst.sin_addr);

    if (connect(fd, (struct sockaddr *)&dst, sizeof(dst)) < 0)
        perror("connect");

    close(fd);
    return 0;
}
Enter fullscreen mode Exit fullscreen mode

Compile with cc -O2 -o cc-demo cc-demo.c. Set the option before connect() (or before listen() on a server) so the handshake and all subsequent data use it.

On a listening socket, the choice propagates:

import socket

srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.IPPROTO_TCP, socket.TCP_CONGESTION, b"bbr")
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("0.0.0.0", 8080))
srv.listen(128)

conn, addr = srv.accept()   # inherits bbr from the listener
Enter fullscreen mode Exit fullscreen mode

Privileges follow tcp_allowed_congestion_control: an unprivileged process may only pick algorithms listed there, while a process with CAP_NET_ADMIN may pick any registered algorithm. If setsockopt() fails with ENOENT, the name is not registered — usually the module is not loaded — and with EPERM you are hitting the allowed list.

For a container: these are per-network-namespace settings. Set the sysctl inside the container's netns if the runtime permits it, or set it per socket in the application. Host-wide changes do not cross a netns boundary.

Measure it without lying to yourself

A credible comparison isolates the algorithm and skips slow start:

# receiver
$ iperf3 -s

# sender: 30 seconds, discard the first 5, one stream
$ iperf3 -c 198.51.100.20 -t 30 -O 5 -C cubic
$ iperf3 -c 198.51.100.20 -t 30 -O 5 -C bbr

# same path, four streams
$ iperf3 -c 198.51.100.20 -t 30 -O 5 -P 4 -C bbr
Enter fullscreen mode Exit fullscreen mode

-C sets the congestion algorithm on Linux and FreeBSD. -O 5 omits the first five seconds, which are dominated by startup. Use --json if you want to compare retransmits and RTT rather than eyeballing the bitrate line.

Two lab traps:

  • Do not put netem on the sender. The BBR FAQ is blunt about it: netem interacts with TCP small queues, and its loss decisions apply to whole TSO bursts of up to 44 MTU-sized packets, so the loss process is unrealistic. Put emulation on a router or on the receiver's ingress path.
  • One stream measures one control loop, not a link. If four streams saturate the link and one does not, you have a per-flow window or buffer limit, not a bandwidth limit.

When switching is the wrong answer

BBR is not a universal upgrade. Five cases where it will disappoint:

The sender is app-limited. If ss -ti shows app_limited, the kernel is idle waiting for data. Look at disk I/O, TLS overhead, request serialization, or a single-threaded server before touching sysctls.

The socket buffer is the ceiling. Bytes in flight cannot exceed what the send buffer allows. On this host, net.ipv4.tcp_wmem is 4096 16384 4194304, so the autotuned maximum is 4 MiB. On a 100 ms path that caps a single flow at roughly 4 MiB / 0.1 s ≈ 335 Mbps, regardless of the algorithm. The bandwidth-delay product is bandwidth × RTT; if the path's BDP is larger than your buffer, raise the third value of tcp_wmem and remember that SO_SNDBUF is separately capped by net.core.wmem_max.

You share a bottleneck you do not control. Because v1 does not reduce its rate on loss, a BBR flow does not yield to loss-based flows the way they yield to each other. On a path where you are one tenant among many, the honest test is a mixed-flow measurement, not a single-flow speed test.

The path has a policer or a shallow buffer. BBR's bandwidth estimate can overshoot a shaper; the implementation has a long-term sampling mode to detect a policer, but retransmits can still rise. Watch retransmit rate, not just throughput.

The real problem is your own bufferbloat. If latency spikes on your home router, the fix is the queue: fq_codel or CAKE on the bottleneck. Changing the congestion control on one server is an indirect workaround for a queue you actually control.

In a datacenter that signals congestion with ECN, DCTCP may be the intended algorithm. BBR is not the only model-based option, and "newest" is not the same as "appropriate".

Rollback is one line

$ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic
Enter fullscreen mode Exit fullscreen mode

Then remove /etc/sysctl.d/90-bbr.conf if you added it. New connections go back to CUBIC; existing sockets keep BBR until they close, which is why a rollback can look partial for a few minutes. ss -tin tells you which sockets are still on the old algorithm.

A short checklist before you call it done:

  • tcp_available_congestion_control contains bbr after boot, not just after modprobe.
  • tcp_congestion_control reads bbr after a reboot.
  • Long-running listeners were restarted, or set the algorithm on the listening socket themselves.
  • ss -tin shows a bbr: block and no app_limited on the flow you care about.
  • Throughput improved and RTT or retransmits did not get worse for the traffic that shares the path.
  • You know how to put it back.

The link was probably fine. The control loop was the part that needed tuning.

Sources and further reading

Top comments (0)