DEV Community

Cover image for Stop Saturating One CPU Core: Practical RSS, RPS, RFS, and XPS Packet Steering on Linux
Lyra
Lyra

Posted on

Stop Saturating One CPU Core: Practical RSS, RPS, RFS, and XPS Packet Steering on Linux

#devops #performance #linux #networking

Stop Saturating One CPU Core: Practical RSS, RPS, RFS, and XPS Packet Steering on Linux

You upgraded the NIC. Throughput still plateaus. top shows one core stuck near 100% in softirq, while the rest of the machine looks bored.

That is usually not “the kernel is slow.” It is serial receive processing: one hardware queue, one IRQ affinity, one CPU doing most of the protocol work.

Linux already has the tools to spread that work:

Layer What it does Where you configure it
RSS Hardware spreads flows across RX queues driver channels + ethtool RXFH + IRQ affinity
RPS Software spreads protocol processing across CPUs rps_cpus per RX queue
RFS Steers a flow toward the CPU running the app rps_sock_flow_entries + rps_flow_cnt
aRFS Hardware-assisted RFS needs ntuple + driver support
XPS Picks TX queue from CPU (or RX queue) map xps_cpus / xps_rxqs per TX queue

This post is a practical operator path: diagnose the hot core, enable the right steering layer, verify with /proc counters, and make the settings survive reboot. It is not nftables flowtable offload, not IPVS load balancing, and not CAKE/AQM shaping.

What “one core is dying” looks like

1) Softirq imbalance 

# Live view of network softirqs
watch -n1 'grep -E "CPU|NET_RX|NET_TX" /proc/softirqs'
Enter fullscreen mode Exit fullscreen mode

If NET_RX climbs mostly on one CPU, receive work is concentrated.

2) softnet backlog pressure

/proc/net/softnet_stat is one line per CPU. Values are hex. A readable dump:

awk '{
  for (i = 1; i <= NF; i++)
    printf strtonum("0x" $i) (i == NF ? "\n" : " ")
}' /proc/net/softnet_stat | column -t
Enter fullscreen mode Exit fullscreen mode

Useful columns (from kernel/RHEL operational docs):

  1. total frames processed
  2. dropped because the CPU backlog was full
  3. times softirqd could not drain everything in one NAPI poll budget … last: CPU index

Rising column 2 → backlog too small or CPUs too hot.

Rising column 3 → NAPI budget/time may be too tight for the NIC rate.

3) Interrupt vectors stuck on one CPU 

IFACE=eth0   # change me
grep -E "CPU|${IFACE}" /proc/interrupts
Enter fullscreen mode Exit fullscreen mode

Multi-queue NICs expose names like eth0-rx-0, eth0-TxRx-1. If one vector’s counters race while others barely move, RSS/IRQ placement needs work before you chase userspace.

4) How many queues exist right now? 

IFACE=eth0
ethtool -l "$IFACE"          # channel max vs current
ls -d /sys/class/net/$IFACE/queues/rx-*
ls -d /sys/class/net/$IFACE/queues/tx-*
ethtool -i "$IFACE"          # driver / firmware
Enter fullscreen mode Exit fullscreen mode

Mental model (keep this straight) 

NIC RX DMA
   |
   v
 hardware RSS hash  -->  RX queue N  -->  hard IRQ on CPU affinity(N)
                              |
                              v
                     NAPI / softirq bottom half
                              |
              +---------------+----------------+
              |                                |
         plain path                      RPS/RFS path
     (same interrupting CPU)      (enqueue other CPU backlog + IPI)
              |                                |
              v                                v
        protocol stack                  protocol stack
              |                                |
              v                                v
           socket / app                   socket / app
              |
              v
     XPS picks TX queue from CPU map (or RXQ map)
Enter fullscreen mode Exit fullscreen mode
  • RSS chooses the queue (and therefore which CPU takes the hard IRQ).
  • RPS chooses which CPU does stack processing after the IRQ.
  • RFS prefers the CPU where the receiving thread runs.
  • XPS chooses the transmit queue so TX completions stay local and queue locks stay uncontended.

Kernel docs are explicit: RPS is disabled until you write rps_cpus; a zero mask means “process on the interrupting CPU.”

Path A — Fix hardware RSS first (preferred when the NIC can do it)

1) Raise combined/RX/TX channels toward core count 

IFACE=eth0
ethtool -l "$IFACE"
# Example: ask for one combined queue per core (driver-dependent)
sudo ethtool -L "$IFACE" combined 8
ethtool -l "$IFACE"
Enter fullscreen mode Exit fullscreen mode

Not every driver exposes combined. Some want separate rx/tx. Stay within Pre-set maximums.

Kernel guidance worth internalizing:

  • For latency, more queues (up to cores or NIC max) help.
  • For raw high-rate efficiency, the smallest queue count that avoids a saturated CPU can win, because each extra queue can raise interrupt work when coalescing is on.
  • Hyperthreads often do not help IRQ handling; prefer physical cores when sizing queues.

2) Confirm / tune the RSS indirection table 

ethtool -x "$IFACE"                 # show RX flow hash / indir table
# equal spread across N queues (ethtool syntax):
sudo ethtool -X "$IFACE" equal 8
ethtool -x "$IFACE"
Enter fullscreen mode Exit fullscreen mode

Advanced NICs also support n-tuple steering (ethtool --config-ntuple / -N) for “TCP/80 always on queue 2” style rules. That is optional; get plain RSS healthy first.

3) Pin each queue IRQ to a sensible CPU

Find IRQs:

IFACE=eth0
grep "$IFACE" /proc/interrupts
Enter fullscreen mode Exit fullscreen mode

Pin with the list form (easier than huge hex masks):

# Example: IRQ 105 -> CPU 0, IRQ 106 -> CPU 1, ...
echo 0 | sudo tee /proc/irq/105/smp_affinity_list
echo 1 | sudo tee /proc/irq/106/smp_affinity_list
# ...
grep -E "CPU|${IFACE}" /proc/interrupts
Enter fullscreen mode Exit fullscreen mode

/proc/irq/IRQ#/smp_affinity is the bitmask form; smp_affinity_list accepts ranges like 0-3 or 4,5,6,7 (kernel IRQ-affinity docs).

irqbalance note: many distros run irqbalance, which can move affinities out from under you. For a hand-tuned multi-queue NIC:

systemctl status irqbalance 2>/dev/null || true
# Either stop it on dedicated packet engines, or teach it your policy.
# sudo systemctl disable --now irqbalance
Enter fullscreen mode Exit fullscreen mode

RHEL’s performance guide still recommends irqbalance for general-purpose hosts; dedicated gateways often do better with static 1:1 queue↔CPU maps.

4) When is RSS “enough”?

If you already have roughly one RX queue per CPU, IRQ affinities are spread, and softirqs look balanced, RPS is often redundant (kernel scaling doc). Stop here and measure.

Path B — Add RPS when queues < CPUs (or the NIC is single-queue)

RPS is the software cousin of RSS. It hashes the flow and enqueues the packet on another CPU’s backlog, then sends an IPI.

1) CPU bitmaps in sysfs 

IFACE=eth0
# Show current masks (0 = RPS off)
cat /sys/class/net/$IFACE/queues/rx-0/rps_cpus
Enter fullscreen mode Exit fullscreen mode

The file is a hex CPU bitmask, same idea as IRQ affinity masks.

Examples on an 8-CPU host:

Goal Mask Meaning
all CPUs ff CPUs 0–7
CPUs 0–3 0f low half
CPUs 4–7 f0 high half
CPU 2 only 04 bit 2 
# Single-queue NIC: spread protocol work across all CPUs
echo ff | sudo tee /sys/class/net/$IFACE/queues/rx-0/rps_cpus

# Multi-queue with 2 RX queues on an 8-CPU box:
# queue 0 -> CPUs 0-3, queue 1 -> CPUs 4-7 (NUMA/cache-friendly split)
echo 0f | sudo tee /sys/class/net/$IFACE/queues/rx-0/rps_cpus
echo f0 | sudo tee /sys/class/net/$IFACE/queues/rx-1/rps_cpus
Enter fullscreen mode Exit fullscreen mode

Kernel suggestions:

  • Prefer CPUs in the same memory domain as the interrupting CPU.
  • At very high IRQ rates, exclude the interrupting CPU from rps_cpus so it is not double-loaded.
  • If RSS already maps one queue per CPU, leave RPS at 0.

2) Optional: RPS flow limit (many small flows + one elephant)

Flow limit drops packets from oversized flows slightly earlier when a CPU backlog is under pressure, protecting mice flows. Off by default.

# Enable on all CPUs that handle network work (example: 8 CPUs)
echo ff | sudo tee /proc/sys/net/core/flow_limit_cpu_bitmap
# Default table length is 4096 buckets
sysctl net.core.flow_limit_table_len

# Experiments in the kernel doc used backlog 1000 or 10000
# sudo sysctl -w net.core.netdev_max_backlog=10000
Enter fullscreen mode Exit fullscreen mode

It only engages when a CPU’s input queue exceeds half of netdev_max_backlog, and it counts over a recent window (256 packets). Large flows are not hard-killed; they still get through when the queue is healthy.

Path C — RFS when the app CPU matters (cache locality)

RPS balances by hash. RFS steers kernel processing toward the CPU where the userspace thread last called recvmsg/sendmsg (and friends), improving cache hit rates.

Configure tables, then RPS masks 

# Global desired-CPU table (rounded up to power of two)
# Kernel doc: 32768–65536 is a common moderate-server starting point;
# large hosts may want 1048576+.
echo 32768 | sudo tee /proc/sys/net/core/rps_sock_flow_entries

IFACE=eth0
# Per-queue flow table. Single queue: match global.
# Multi-queue: often global / Nqueues
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo 32768 | sudo tee "$q/rps_flow_cnt"
done

# RFS reuses the RPS enqueue path — rps_cpus must be non-zero
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo ff | sudo tee "$q/rps_cpus"
done
Enter fullscreen mode Exit fullscreen mode

On big NUMA boxes, the kernel doc shows interleaving the global table allocation:

# optional on large NUMA hosts
sudo numactl --interleave=all bash -c \
  'echo 1048576 > /proc/sys/net/core/rps_sock_flow_entries'
Enter fullscreen mode Exit fullscreen mode

RFS keeps a second per-queue table so a flow only migrates to a new CPU when the old CPU has no outstanding packets for that flow — that is the out-of-order guard.

Accelerated RFS (aRFS)

If the NIC + driver support it (CONFIG_RFS_ACCEL), enable ntuple and let the stack program hardware flow steering from the RFS tables:

sudo ethtool -K "$IFACE" ntuple on
ethtool -k "$IFACE" | grep -E 'ntuple|receive-hashing'
Enter fullscreen mode Exit fullscreen mode

No extra sysfs knobs beyond healthy RFS + IRQ affinities: the driver builds a CPU→queue reverse map from IRQ affinity.

Path D — XPS so transmit does not undo your work

Multi-queue TX without XPS often means many CPUs fighting one TX ring lock, or TX completions landing far from the sender.

IFACE=eth0
ls /sys/class/net/$IFACE/queues/tx-*/xps_cpus

# 1:1 CPU→TX queue map on an 8-queue device (exclusive pairings)
echo 01 | sudo tee /sys/class/net/$IFACE/queues/tx-0/xps_cpus
echo 02 | sudo tee /sys/class/net/$IFACE/queues/tx-1/xps_cpus
echo 04 | sudo tee /sys/class/net/$IFACE/queues/tx-2/xps_cpus
echo 08 | sudo tee /sys/class/net/$IFACE/queues/tx-3/xps_cpus
echo 10 | sudo tee /sys/class/net/$IFACE/queues/tx-4/xps_cpus
echo 20 | sudo tee /sys/class/net/$IFACE/queues/tx-5/xps_cpus
echo 40 | sudo tee /sys/class/net/$IFACE/queues/tx-6/xps_cpus
echo 80 | sudo tee /sys/class/net/$IFACE/queues/tx-7/xps_cpus
Enter fullscreen mode Exit fullscreen mode

Busy-polling / queue-affine apps sometimes prefer RX-queue → TX-queue maps via xps_rxqs instead of xps_cpus. The common case is still CPU maps with 1:1 pairings when queue count ≈ CPU count.

The stack records the chosen TX queue on the socket so a flow does not bounce queues mid-flight (TCP sets ooo_okay only when it is safe to change).

Support knobs that pair with steering

These do not replace RSS/RPS, but they stop false “steering failed” diagnoses.

# Backlog drops (softnet column 2)
sysctl net.core.netdev_max_backlog
# sudo sysctl -w net.core.netdev_max_backlog=5000

# NAPI poll budget (softnet column 3 pressure)
sysctl net.core.netdev_budget net.core.netdev_budget_usecs
# conservative doubling when time-starved:
# sudo sysctl -w net.core.netdev_budget=600
# sudo sysctl -w net.core.netdev_budget_usecs=4000

# Ring buffers if ethtool -S shows rx_queue_*_drops / discards
ethtool -g "$IFACE"
ethtool -S "$IFACE" | grep -Ei 'drop|discard|fifo|miss' || true
# sudo ethtool -G "$IFACE" rx 4096 tx 4096
Enter fullscreen mode Exit fullscreen mode

Make it survive reboot

Sysfs RPS/RFS/XPS settings are runtime. Persist with a oneshot service (works with systemd-networkd, NetworkManager, or plain ip).

/usr/local/sbin/net-steering-apply (example for eth0, 8 CPUs, 4 combined queues — edit before use):

#!/bin/bash
set -euo pipefail

IFACE="${IFACE:-eth0}"
# hex mask for CPUs 0-7
ALL_CPUS=ff

# Optional: set channels if the driver allows (ignore failures)
ethtool -L "$IFACE" combined 4 2>/dev/null || true

# RFS global table
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

nrx=$(ls -d /sys/class/net/"$IFACE"/queues/rx-* 2>/dev/null | wc -l)
if [[ "$nrx" -lt 1 ]]; then
  echo "no rx queues for $IFACE" >&2
  exit 1
fi

# per-queue flow cnt ≈ global / nrx (power-of-two friendly enough for ops)
per=$(( 32768 / nrx ))
[[ "$per" -lt 1 ]] && per=1

for q in /sys/class/net/"$IFACE"/queues/rx-*; do
  echo "$per" > "$q/rps_flow_cnt"
  echo "$ALL_CPUS" > "$q/rps_cpus"
done

# XPS: simple round-robin exclusive bits when queue count <= 8
i=0
for q in /sys/class/net/"$IFACE"/queues/tx-*; do
  # 1 << i  in hex for i=0..7
  printf '%x\n' $((1 << i)) > "$q/xps_cpus"
  i=$((i + 1))
  [[ "$i" -ge 8 ]] && i=0
done

# Optional flow limit on all CPUs
echo "$ALL_CPUS" > /proc/sys/net/core/flow_limit_cpu_bitmap

# Show result
echo "== rps_cpus =="
grep -H . /sys/class/net/"$IFACE"/queues/rx-*/rps_cpus
echo "== xps_cpus =="
grep -H . /sys/class/net/"$IFACE"/queues/tx-*/xps_cpus 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode 
sudo install -m 0755 /usr/local/sbin/net-steering-apply /usr/local/sbin/net-steering-apply
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/system/net-steering@.service:

[Unit]
Description=Apply RPS/RFS/XPS steering for %i
After=network-pre.target sys-subsystem-net-devices-%i.device
Wants=sys-subsystem-net-devices-%i.device

[Service]
Type=oneshot
Environment=IFACE=%i
ExecStart=/usr/local/sbin/net-steering-apply
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl daemon-reload
sudo systemctl enable --now net-steering@eth0.service
systemctl status net-steering@eth0.service --no-pager
Enter fullscreen mode Exit fullscreen mode

Persist sysctls separately:

sudo tee /etc/sysctl.d/90-net-steering.conf >/dev/null <<'EOF'
net.core.rps_sock_flow_entries = 32768
# net.core.netdev_max_backlog = 5000
# net.core.flow_limit_cpu_bitmap = ff
EOF
sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

IRQ affinity persistence is driver/udev territory; either disable irqbalance on that host or use a small udev/systemd oneshot that rewrites /proc/irq/*/smp_affinity_list after the NIC registers MSI-X vectors.

Verification checklist 

IFACE=eth0

# 1. Queues and RSS table
ethtool -l "$IFACE"
ethtool -x "$IFACE"

# 2. RPS/RFS/XPS sysfs
grep -H . /sys/class/net/$IFACE/queues/rx-*/rps_cpus
grep -H . /sys/class/net/$IFACE/queues/rx-*/rps_flow_cnt
sysctl net.core.rps_sock_flow_entries
grep -H . /sys/class/net/$IFACE/queues/tx-*/xps_cpus

# 3. IRQ spread under load
grep -E "CPU|${IFACE}" /proc/interrupts
watch -n1 'grep -E "CPU|NET_RX" /proc/softirqs'

# 4. Backlog / budget pressure
awk '{for (i=1;i<=NF;i++) printf strtonum("0x"$i) (i==NF?"\n":" ")}' \
  /proc/net/softnet_stat | column -t

# 5. Application-level proof
# Generate many concurrent flows (not one elephant TCP) and compare:
#   - mpstat -P ALL 1
#   - CPU distribution of NET_RX
#   - app p99 latency / throughput
Enter fullscreen mode Exit fullscreen mode

Fair test tip: one fat TCP flow will still pin to one RX queue/CPU by design (flow hashing preserves order). Use many parallel flows (iperf3 -P, real reverse-proxy traffic, etc.) when you judge balance.

Decision tree

  1. softnet drops on one CPU, IRQ counters also one-sided → multi-queue RSS + IRQ affinity first.
  2. NIC is single-queue or queues ≪ cores → RPS masks on those RX queues.
  3. CPU-bound userspace receiver, cache misses high → add RFS tables; enable aRFS if ntuple works.
  4. TX lock contention / TX softirq clump → XPS 1:1 maps.
  5. Many mice + one elephant under load → consider flow limit + sane netdev_max_backlog.
  6. Still dropping with balanced CPUs → rings, NAPI budget, offloads (ethtool -k), or an application/cgroup limit — not more bitmasks.

What this is not

  • nftables flowtables — fast-path established forward flows past classic Netfilter hooks.
  • IPVS/LVS — Layer-4 virtual services across real servers.
  • CAKE/fq_codel — queue discipline / bufferbloat control.
  • XDP/eBPF — programmable early drop/redirect; complementary, different tool.
  • irqbalance alone — helpful default for mixed hosts; not a substitute for understanding RSS vs RPS.

Rollback 

IFACE=eth0
# Disable RPS/RFS path
for q in /sys/class/net/$IFACE/queues/rx-*; do
  echo 0 | sudo tee "$q/rps_cpus"
  echo 0 | sudo tee "$q/rps_flow_cnt"
done
echo 0 | sudo tee /proc/sys/net/core/rps_sock_flow_entries
echo 0 | sudo tee /proc/sys/net/core/flow_limit_cpu_bitmap

# Clear XPS maps (driver may re-init defaults on rebind)
for q in /sys/class/net/$IFACE/queues/tx-*; do
  echo 0 | sudo tee "$q/xps_cpus" 2>/dev/null || true
done

sudo systemctl disable --now net-steering@eth0.service 2>/dev/null || true
# re-enable irqbalance if you stopped it
# sudo systemctl enable --now irqbalance
Enter fullscreen mode Exit fullscreen mode

Closing

High-speed Linux networking is less about a single sysctl silver bullet and more about matching parallelism to hardware:

  1. Let the NIC split flows (RSS).
  2. Let IRQs land on different cores.
  3. If the NIC cannot split enough, let software finish the job (RPS).
  4. Prefer the app’s CPU when locality matters (RFS / aRFS).
  5. Keep transmit on matching queues (XPS).
  6. Prove it with softnet_stat, /proc/interrupts, and multi-flow load — not vibes.

Do that, and the “one core at 100% softirq” box often turns into a boring, evenly busy machine — which is exactly what you want.

References

Top comments (0)