Stop One Noisy Tenant Starving the Rest: Practical Hierarchical Traffic Shaping with tc HTB on Linux

CAKE and fq_codel are excellent at fighting bufferbloat — they keep latency low when a link is saturated. TCP BBR helps a sender estimate path capacity. Neither answers a different ops question:

“This uplink is 100 Mbit. Give interactive and API traffic a guaranteed floor, let backups borrow leftover capacity up to a hard ceiling, and stop one bulk flow from eating the whole pipe.”

That is classful hierarchical shaping. On Linux, the workhorse is still HTB (Hierarchy Token Bucket), configured with tc from iproute2.

This post is a practical recipe: root HTB, parent and leaf classes with rate/ceil, filters that classify traffic, fq_codel under each leaf, live verification, and rollback. No custom kernel required.

What this is (and is not)

Tool	Job	Typical use
HTB	Hierarchical bandwidth allocation and borrowing	Tenant/share caps, “API gets 20 Mbit guaranteed”
CAKE / fq_codel	AQM — control queue delay under load	WAN edge bufferbloat control
BBR / CUBIC	Per-connection TCP congestion control	How fast one TCP sender ramps
RSS/RPS/XPS	Which CPU handles packet work	Multi-queue NIC scaling

Use HTB when you need explicit class shares and ceilings. Keep CAKE or fq_codel for latency under congestion (often as the leaf qdisc under HTB, or on a pure edge shaper). Do not treat HTB as a replacement for BBR or packet steering.

Mental model in one page

From tc(8) and tc-htb(8):

1. Shaping is egress. You control how fast the host sends on a device.
2. HTB builds a tree of classes. Each class has:
   • rate — guaranteed bandwidth (tokens replenished at this rate)
   • ceil — hard maximum when borrowing spare capacity from the parent
   • prio — lower number is preferred when classes compete for leftover bandwidth
3. Only leaf classes shape packets. Inner/parent classes define how tokens are shared and borrowed.
4. Filters attached to the HTB qdisc (or classes) decide which leaf gets each packet (u32, fw, flower, …).
5. Unclassified traffic goes to the HTB default minor class id.

Borrowing (from the classic HTB / TLDP model):

• Below rate → class may send (uses its own tokens).
• Between rate and ceil → class may borrow from the parent if the parent has spare capacity.
• At/above ceil → class cannot send more until tokens return (packets queue / delay).

Rule of thumb from long-standing HTB guidance: sum of child rate values should not exceed the parent’s rate (ideally they match), while children may set higher ceil values up to the parent’s ceil so they can borrow leftover capacity.

Prerequisites

• Linux with HTB in the kernel (stock for many years; any current LTS is fine)
• iproute2 (tc)
• Root (or CAP_NET_ADMIN)
• Optional: nftables if you prefer fwmark classification
• Optional: iperf3 for a controlled soak test

uname -r
tc -V
ip -br link
# Pick the egress NIC you will shape (example: eth0 / ens18 / enp1s0)
IFACE=eth0

Shape the real bottleneck. If the host’s NIC is 1 Gbit but the ISP uplink is 100 Mbit, set HTB’s root/ceil to ~95–98% of the true bottleneck (slightly under physical rate so the Linux queue is the controlled one, not a dumb ISP buffer).

Lab topology (example numbers)

Assume uplink eth0 is effectively 100 Mbit outbound. You want:

Class	Purpose	Guaranteed (rate)	Ceiling (ceil)	Priority
1:10	Interactive / low-latency (SSH, DNS, small control)	10 Mbit	100 Mbit	1 (best)
1:20	Default / general web & API	40 Mbit	100 Mbit	2
1:30	Bulk / backups / media	50 Mbit	80 Mbit	3 (worst)

Parent class 1:1 holds the full link budget. Leaf rates sum to 100 Mbit. Bulk is capped at 80 Mbit even when the pipe is idle-ish of interactive traffic — adjust to taste.

Replace eth0 everywhere with your interface name.

1. Install the HTB root and classes

IFACE=eth0

# Clean slate on this NIC (destructive to existing root qdisc)
tc qdisc del dev "$IFACE" root 2>/dev/null || true

# Root HTB. default 20 => unclassified traffic goes to class 1:20
tc qdisc add dev "$IFACE" root handle 1: htb default 20

# Parent: full shaped uplink budget
tc class add dev "$IFACE" parent 1: classid 1:1 htb \
  rate 100mbit ceil 100mbit burst 32k cburst 32k

# Leaves
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb \
  rate 10mbit ceil 100mbit prio 1 burst 15k cburst 15k

tc class add dev "$IFACE" parent 1:1 classid 1:20 htb \
  rate 40mbit ceil 100mbit prio 2 burst 20k cburst 20k

tc class add dev "$IFACE" parent 1:1 classid 1:30 htb \
  rate 50mbit ceil 80mbit prio 3 burst 20k cburst 20k

Why burst / cburst matter

tc-htb(8) notes that timer granularity limits how large a rate you can express without a big enough bucket. If bursts are tiny relative to rate, you get needless throttling and odd latency. Start with the values above (or let tc compute defaults by omitting them once you understand the tree), then raise modestly if high-rate classes under-deliver on small packets.

Attach an AQM leaf under each class

Bare HTB leaves default to a simple FIFO (pfifo). Under load that reintroduces bufferbloat inside each class. Attach fq_codel (or CAKE on a single-class edge) under every leaf:

for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel \
    limit 10240 target 5ms interval 100ms ecn
done

Now each share gets hierarchical bandwidth and flow-fair low-latency queuing inside the share.

2. Classify traffic into classes

You need filters so packets land in 1:10 / 1:20 / 1:30. Two practical patterns:

Pattern A — pure tc u32 (no firewall marks)

Good for simple port/subnet rules kept next to the qdisc:

# Interactive: SSH (22), DNS (53)
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip dport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip sport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip dport 53 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 \
  match ip sport 53 0xffff flowid 1:10

# Bulk example: traffic to a backup subnet 10.20.30.0/24
tc filter add dev "$IFACE" parent 1: protocol ip prio 5 u32 \
  match ip dst 10.20.30.0/24 flowid 1:30

# Everything else falls through to HTB default 1:20

Notes from tc-u32(8):

• match ip dport / sport assume a normal L4 header layout (careful with fragments / unusual encapsulation).
• Lower prio number is consulted earlier.
• flowid / classid send the packet to that HTB class.

Pattern B — nftables mark + tc fw filter (recommended for real policy)

Mark once in Netfilter (where you already express policy), then map marks to classes. From the nftables wiki, packet marks are set with meta mark set …. From tc-fw(8), the fw classifier matches that mark.

# --- nftables: mark bulk vs interactive (example table) ---
nft -f - <<'EOF'
flush table inet qos 2>/dev/null || true
table inet qos {
  chain output {
    type filter hook output priority -150; policy accept;
    # Interactive control plane
    tcp dport { 22 } meta mark set 10
    udp dport { 53 } meta mark set 10
    tcp sport { 22 } meta mark set 10

    # Bulk: backup host or high ports used by your sync tool
    ip daddr 10.20.30.0/24 meta mark set 30
    tcp dport { 873, 2222 } meta mark set 30   # rsync / custom bulk
  }

  chain postrouting {
    type filter hook postrouting priority -150; policy accept;
    # Forwarded traffic (router/gateway use-case)
    tcp dport { 22 } meta mark set 10
    ip daddr 10.20.30.0/24 meta mark set 30
  }
}
EOF

# --- tc: map marks to HTB classes (handle == mark) ---
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 handle 10 fw flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 handle 30 fw flowid 1:30
# mark 0 / unmarked → HTB default 1:20

Why this scales better than a giant u32 forest:

• One place for complex matches (sets, interfaces, conntrack state).
• tc-fw(8) stays a thin mark→class map.
• You can persist marks across related packets with conntrack mark save/restore (ct mark set mark / meta mark set ct mark) when you need whole-flow consistency.

3. Verify the tree

tc -s -d qdisc show dev "$IFACE"
tc -s -d class show dev "$IFACE"
tc -s filter show dev "$IFACE"

What you want to see:

• Root qdisc htb 1: with default 0x20 (hex for minor 20)
• Classes 1:1, 1:10, 1:20, 1:30 with the rates you set
• Leaf qdiscs fq_codel under each leaf
• Filters with non-zero match counts after traffic flows
• Under load, bulk class 1:30 shows sends near its ceil while 1:10 still gets airtime

Watch live counters while generating traffic:

watch -n1 "tc -s class show dev $IFACE | sed -n '1,120p'"

4. Controlled proof with iperf3

On a receiver beyond the shaped path:

# receiver
iperf3 -s

On the shaped host (or a client behind it):

# 1) Bulk-class push (mark 30 path or dst that maps to 1:30)
iperf3 -c RECEIVER -t 30 -P 4

# 2) While bulk runs, start an interactive-class flow (SSH tunnel / small iperf to port classified as 1:10)
# You should still see responsive SSH and class 1:10 counters moving.

# 3) Compare: without HTB, bulk often starves latency; with HTB, bulk sticks near ceil and interactive keeps rate tokens.

Interpret honestly:

• HTB guarantees link shares, not application SLOs end-to-end.
• If the true bottleneck is downstream of this host, shape there or accept that local HTB only protects this egress queue.
• TCP still does congestion control inside each class; fq_codel leaves keep per-flow fairness inside a share.

5. Persist with a systemd oneshot

tc rules are not durable across reboot unless you install them. A small oneshot is explicit and easy to audit:

# /usr/local/sbin/tc-htb-wan.sh
cat >/usr/local/sbin/tc-htb-wan.sh <<'EOF'
#!/bin/bash
set -euo pipefail
IFACE="${IFACE:-eth0}"

tc qdisc del dev "$IFACE" root 2>/dev/null || true

tc qdisc add dev "$IFACE" root handle 1: htb default 20
tc class add dev "$IFACE" parent 1:  classid 1:1  htb rate 100mbit ceil 100mbit burst 32k cburst 32k
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb rate 10mbit  ceil 100mbit prio 1 burst 15k cburst 15k
tc class add dev "$IFACE" parent 1:1 classid 1:20 htb rate 40mbit  ceil 100mbit prio 2 burst 20k cburst 20k
tc class add dev "$IFACE" parent 1:1 classid 1:30 htb rate 50mbit  ceil 80mbit  prio 3 burst 20k cburst 20k

for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel limit 10240 target 5ms interval 100ms ecn
done

# Example u32 bulk + interactive; replace with fw filters if you use nft marks
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip sport 22 0xffff flowid 1:10
tc filter add dev "$IFACE" parent 1: protocol ip prio 5 u32 match ip dst 10.20.30.0/24 flowid 1:30
EOF
chmod 755 /usr/local/sbin/tc-htb-wan.sh

# /etc/systemd/system/tc-htb-wan.service
[Unit]
Description=HTB hierarchical egress shaping on WAN NIC
After=network-pre.target
Before=network.target
Wants=network-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=IFACE=eth0
ExecStart=/usr/local/sbin/tc-htb-wan.sh
ExecStop=/sbin/tc qdisc del dev ${IFACE} root
ExecReload=/usr/local/sbin/tc-htb-wan.sh

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now tc-htb-wan.service
systemctl status tc-htb-wan.service --no-pager
tc -s class show dev eth0

If the interface name is renamed by udev/systemd-networkd, bind the unit with BindsTo=sys-subsystem-net-devices-eth0.device (adjust the netdev unit name) or drive interface selection from a .network drop-in environment file.

6. Operational pitfalls

1. Wrong direction. HTB on egress shapes transmit. Download-heavy home WAN bottlenecks often need shaping on the ISP-facing egress of the router, and sometimes IFB/mirred redirect tricks for ingress — do not expect host-only HTB to fix a dumb downstream buffer you do not own.

2. default 0 trap. HTB’s default default of 0 can send unclassified traffic in ways that bypass your carefully built leaves. Always set default to a real leaf minor id.

3. Leaf rate sum > parent rate. Over-subscribing guarantees means the “guarantee” is fiction under full load. Keep Σ leaf rate ≤ parent rate.

4. Classification misses. If filters never match, everything piles into default. Check tc -s filter hit counts before tuning rates.

5. Offload / TSO surprises. Aggressive NIC offloads rarely break HTB outright, but when rates look “soft,” compare with offloads temporarily simplified for debugging (ethtool -K … gso off tso off on a lab NIC — measure, then restore).

6. Hardware HTB offload. tc-htb(8) documents an offload flag when driver and device support it. Treat it as optional acceleration; verify with the same class counters, and keep a software fallback path.

7. HTB is not multi-tenant security isolation. It allocates bandwidth. It does not replace VRF, netns, firewall policy, or auth.

7. Rollback

# Remove shaping entirely (kernel returns to the previous default qdisc behavior on new setup;
# often fq_codel/cake/pfifo_fast depending on distro defaults)
tc qdisc del dev eth0 root

# If you used the systemd unit:
systemctl disable --now tc-htb-wan.service

# If you added the nft qos table:
nft delete table inet qos

When to prefer something else

Goal	Prefer
Fix bufferbloat on a single WAN uplink with minimal classes	CAKE (bandwidth + nat/docsis/overhead knobs) or fq_codel at the bottleneck
Make this host’s TCP sends cope with long/lossy paths	BBR + fq on the sender
Spread softirq across CPUs	RSS/RPS/RFS/XPS
L4 virtual services	IPVS
Hierarchical multi-share bandwidth with borrow/ceil	HTB (this post)

HTB and CAKE are complementary on many gateways: CAKE (or fq_codel) for latency discipline at the true bottleneck, HTB when you must express policy shares between tenants or traffic classes. Some designs use HTB parents with fq_codel leaves exactly for that split of concerns.

Quick copy-paste checklist

IFACE=eth0
RATE_UP=100mbit   # set to ~95% of real bottleneck

tc qdisc del dev "$IFACE" root 2>/dev/null || true
tc qdisc add dev "$IFACE" root handle 1: htb default 20
tc class add dev "$IFACE" parent 1:  classid 1:1  htb rate $RATE_UP ceil $RATE_UP
tc class add dev "$IFACE" parent 1:1 classid 1:10 htb rate 10mbit ceil $RATE_UP prio 1
tc class add dev "$IFACE" parent 1:1 classid 1:20 htb rate 40mbit ceil $RATE_UP prio 2
tc class add dev "$IFACE" parent 1:1 classid 1:30 htb rate 50mbit ceil 80mbit  prio 3
for id in 10 20 30; do
  tc qdisc add dev "$IFACE" parent 1:$id handle $id: fq_codel
done
tc filter add dev "$IFACE" parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc -s class show dev "$IFACE"

Then add real classification (more u32 matches or nft marks + fw filters), soak-test with bulk + interactive traffic, and only then enable the systemd oneshot.

References

• tc-htb(8) — Hierarchy Token Bucket qdisc and class parameters (rate, ceil, burst, prio, default, offload)
• tc(8) — Traffic control overview: qdiscs, classes, filters; shaping vs policing
• tc-fw(8) — fwmark classifier (handle matches mark)
• tc-u32(8) — Universal 32-bit classifier (match ip …, flowid)
• tc-fq_codel(8) — Fair queuing + CoDel leaf AQM
• TLDP Traffic Control HOWTO — Classful qdiscs / HTB borrowing
• nftables wiki — Setting packet metainformation (meta mark set)
• Martin Devera’s HTB site (historical design notes): http://luxik.cdi.cz/~devik/qos/htb/

---

Hierarchical shaping is old Linux technology — and still the right hammer when one backup job keeps making SSH feel like dial-up. Set the root to the real bottleneck, give each class an honest rate, let them borrow up to ceil, classify on purpose, and put fq_codel under the leaves so shares stay fast and fair.