DEV Community

Cover image for Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux
Lyra
Lyra

Posted on

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

#linux #networking #performance #devops

Stop Betting on One Path: Practical Multipath TCP (MPTCP) on Linux

Most Linux TCP sessions still die with the path they started on. Cable unplugged, LTE handoff, or a flaky ISP hop later, the socket is gone — even if another NIC still has a working route to the peer.

Multipath TCP (MPTCP) fixes that at the transport layer. One logical connection can use several TCP subflows over different addresses or interfaces, aggregate bandwidth when paths are healthy, and fail over when one path dies. Upstream Linux implements MPTCPv1 (RFC 8684). This post is a practical operator guide: enable it, configure the in-kernel path manager, wrap legacy services, verify with ss, and avoid the usual traps.

Not covered here: Ethernet bonding/LACP (link aggregation), Device Mapper multipath (storage paths), TCP BBR (congestion control), or tc HTB shaping. Those solve different layers.

What you get (and what you do not)

MPTCP helps when:

  • A host has two or more usable L3 paths (wired + Wi‑Fi, dual ISP, lab dual-homed servers).
  • You want session continuity across path loss (handover / backup path).
  • You want optional aggregation when both paths carry data.

MPTCP does not magically help when:

  • Only one path exists end-to-end.
  • The peer or a middlebox strips MPTCP options (connection falls back to plain TCP).
  • Apps never open MPTCP sockets and nothing wraps them (mptcpize, eBPF helpers, or native IPPROTO_MPTCP).
  • Strict reverse-path filtering drops asymmetric subflow replies.

Kernel docs summarize the model cleanly: path manager (which subflows/addresses exist) + packet scheduler (which subflow sends next). See the kernel MPTCP overview and mptcp.dev.

Requirements

  • Linux 5.6+ for basic MPTCP sockets; multi-subflow usefulness really lands from 5.10+ onward (mainline timeline is documented on the mptcp_net-next wiki).
  • iproute2 with ip mptcp (full iproute2 — BusyBox ip is not enough).
  • Optional: mptcpd package for mptcpize (wrap legacy TCP binaries / systemd units).
  • A peer that speaks MPTCP, or a lab peer you control.

Check whether the stack is present and enabled:

# Kernel feature present?
sysctl net.mptcp.enabled

# Path manager + scheduler (names vary slightly by kernel age)
sysctl net.mptcp.path_manager net.mptcp.scheduler 2>/dev/null || true
sysctl net.mptcp.pm_type 2>/dev/null || true   # deprecated since v6.15; use path_manager

# iproute2 supports MPTCP?
ip mptcp help >/dev/null && echo "ip mptcp OK"
Enter fullscreen mode Exit fullscreen mode

Default on current kernels: net.mptcp.enabled=1 and in-kernel path manager (path_manager=kernel / historically pm_type=0). Sysctl reference: MPTCP Sysfs variables.

Mental model in one diagram 

App socket (IPPROTO_MPTCP)
        │
        ▼
   MPTCP connection  ── scheduler picks subflow(s)
        │
   ┌────┴────┐
   ▼         ▼
 TCP subflow A     TCP subflow B
 (eth0 / ISP1)     (wwan0 / ISP2)
Enter fullscreen mode Exit fullscreen mode
  • Client typically creates extra subflows (subflow endpoints).
  • Server typically announces extra addresses (signal endpoints / ADD_ADDR).
  • Limits cap how many extra subflows and ADD_ADDR acceptances are allowed per connection.

Step 1 — Raise path-manager limits

Defaults are conservative. On many systems add_addr_accepted starts at 0, which means a client will not open subflows toward peer-advertised addresses until you raise it.

# Show current limits
ip mptcp limits

# Allow additional subflows + accept ADD_ADDR from peers
# Values are per MPTCP connection (see ip-mptcp(8))
sudo ip mptcp limits set subflow 2 add_addr_accepted 2

ip mptcp limits
Enter fullscreen mode Exit fullscreen mode

From ip-mptcp(8):

  • subflow — max additional subflows (created or accepted) per connection.
  • add_addr_accepted — max incoming ADD_ADDR options that may trigger new subflows.

Persist with a oneshot unit (example):

sudo tee /etc/systemd/system/mptcp-limits.service >/dev/null <<'EOF'
[Unit]
Description=Set MPTCP path-manager limits
After=network-pre.target
Before=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip mptcp limits set subflow 2 add_addr_accepted 2
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now mptcp-limits.service
Enter fullscreen mode Exit fullscreen mode

Step 2 — Add endpoints (the part people skip)

Endpoints tell the in-kernel path manager which local addresses participate.

Always pass dev <ifname>. Without it, source selection/routing often surprises you (mptcp.dev path manager notes).

Client-style: create subflows from extra NICs 

# Primary path is whatever the initial connect uses.
# Tell MPTCP it may also originate subflows from a second NIC:
sudo ip mptcp endpoint add 192.0.2.20 dev eth1 subflow

# Optional: treat a cellular/USB path as backup only
sudo ip mptcp endpoint add 100.64.1.134 dev wwan0 subflow backup

ip mptcp endpoint
Enter fullscreen mode Exit fullscreen mode

Flags that matter day-to-day (ip-mptcp(8), pm.html):

Flag Typical role
subflow Use this local address to create extra subflows (client-ish).
signal Announce this address to peers via ADD_ADDR (server-ish).
backup Prefer non-backup subflows; use this path when others are unavailable.
fullmesh Pair this source with each known peer address (mesh topology).
laminar Newer kernels: use this source toward peer ADD_ADDR targets, once per connection (see mptcp.dev / recent man pages).

Server-style: advertise an extra address 

# Announce a second server address clients may join
sudo ip mptcp endpoint add 198.51.100.10 dev eth0 signal

ip mptcp endpoint
Enter fullscreen mode Exit fullscreen mode

Flush or delete when testing:

sudo ip mptcp endpoint delete id 1
# or
sudo ip mptcp endpoint flush
Enter fullscreen mode Exit fullscreen mode

Persist endpoints the same way as limits (oneshot ExecStart= lines), or let NetworkManager ≥ 1.40 auto-configure subflow endpoints — and do not fight it with mptcpd at the same time (pm.html automatic configuration).

Step 3 — Open MPTCP sockets (apps are opt-in)

MPTCP is opt-in at the socket API:

int sd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP); /* IPPROTO_MPTCP == 262 */
Enter fullscreen mode Exit fullscreen mode

If MPTCP is disabled or unavailable, you get ENOPROTOOPT / EPROTONOSUPPORT / EINVAL depending on kernel age (kernel docs).

Force legacy TCP programs with mptcpize

From mptcpize(8) (mptcpd package):

# One-shot wrap
mptcpize run curl -sS https://example.com/ >/dev/null

# Debug when a TCP socket is rewritten
mptcpize run -d my-client --flags

# systemd service wrap (updates unit to launch under mptcpize)
sudo mptcpize enable nginx.service
sudo systemctl daemon-reload
sudo systemctl restart nginx.service

# Undo
sudo mptcpize disable nginx.service
Enter fullscreen mode Exit fullscreen mode

Native language support also exists in various stacks (for example Go via GODEBUG=multipathtcp=1 on supported versions — treat as app-specific and verify on your runtime). Kernel docs also mention eBPF-based forcing approaches for advanced setups.

Tiny Python lab listener (native) 

#!/usr/bin/env python3
import socket

IPPROTO_MPTCP = 262
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM, IPPROTO_MPTCP)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("0.0.0.0", 9000))
s.listen(128)
print("MPTCP listen on :9000", flush=True)
while True:
    c, a = s.accept()
    print("accepted", a, flush=True)
    c.sendall(b"hello-mptcp\n")
    c.close()
Enter fullscreen mode Exit fullscreen mode

Pair with mptcpize run nc ... or another MPTCP-capable client from a second path.

Step 4 — Routing and rp_filter (silent killers)

Subflows are still TCP connections with their own source addresses. If reverse-path filtering is strict, return traffic on the “wrong” NIC gets dropped.

From mptcp.dev path manager notes: prefer loose rp_filter when MPTCP is in play:

# Per-interface example — use loose mode (2) instead of strict (1)
sudo sysctl -w net.ipv4.conf.eth1.rp_filter=2
sudo sysctl -w net.ipv4.conf.wwan0.rp_filter=2

# Persist
sudo tee /etc/sysctl.d/70-mptcp-rpfilter.conf >/dev/null <<'EOF'
net.ipv4.conf.eth1.rp_filter = 2
net.ipv4.conf.wwan0.rp_filter = 2
EOF
sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Also ensure each source address has a sensible route out its own interface (policy routing if needed). Dual-default-route hosts without source-based routing will mis-send subflows.

Step 5 — Verify with ss, monitor, and nstat

List MPTCP sockets 

# MPTCP socket table
ss -Mni

# Listening MPTCP sockets
ss -Mln

# TCP sockets including MPTCP ULP/subflow detail on TCP rows
ss -ti | sed -n '1,80p'
Enter fullscreen mode Exit fullscreen mode

ss documents -M, --mptcp for the MPTCP socket table, and TCP info may show tcp-ulp-mptcp ... on subflows (ss(8)). Community docs also use ss -Mai when diagnosing limit counters (pm.html).

Live path-manager events 

# Another terminal while you connect
sudo ip mptcp monitor
Enter fullscreen mode Exit fullscreen mode

You should see connection creation and address/subflow events as endpoints join.

MIB counters 

nstat -az | grep -i mptcp
# or
nstat | grep -i mptcp
Enter fullscreen mode Exit fullscreen mode

Useful when hunting blackholes/fallbacks. Related sysctls include net.mptcp.blackhole_timeout (default 3600s) and net.mptcp.syn_retrans_before_tcp_fallback (sysctl docs).

Failover smoke test (lab)

  1. Start an MPTCP server on dual-homed host A (signal endpoints + limits).
  2. From dual-homed client B, set subflow endpoints + add_addr_accepted, then open a long transfer (mptcpize run iperf3 ... or a large curl).
  3. Confirm multiple subflows via ss / ip mptcp monitor.
  4. Administratively down one client path (ip link set eth1 down) and confirm the transfer continues on the remaining subflow instead of resetting like single-path TCP.

If it immediately falls back to single-path TCP, capture a SYN handshake and check for missing MPTCP options (peer or middlebox).

systemd pattern: durable client wrapper 

# /etc/systemd/system/backup-sync.service
[Unit]
Description=MPTCP-backed sync client
After=network-online.target mptcp-limits.service
Wants=network-online.target

[Service]
Type=simple
# Prefer mptcpize enable on the unit, or wrap ExecStart:
ExecStart=/usr/bin/mptcpize run /usr/local/bin/backup-sync --target dual.example.net
Restart=on-failure

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Keep endpoint programming in mptcp-limits.service (or a dedicated mptcp-endpoints.service) ordered before the app.

Operational pitfalls checklist

  1. Limits left at defaults — especially add_addr_accepted=0 on clients.
  2. No dev on endpoints — broken source routing.
  3. Strict rp_filter=1 — subflow replies blackholed.
  4. Only one side MPTCP-aware — safe fallback to TCP, but no multipath benefit.
  5. Middleboxes — some PEPs/firewalls interfere; blackhole detection may temporarily disable MPTCP on affected sockets (blackhole_timeout).
  6. NM + mptcpd both owning endpoints — pick one automation path.
  7. Confusing layers — bonding aggregates L2 links into one interface; MPTCP spreads one TCP session across multiple L3 paths. You can use both, but they are not substitutes.
  8. Security policy — more subflows mean more allowed 4-tuples; update nftables/conntrack expectations and logging.

Rollback 

# Stop wrapping services
sudo mptcpize disable nginx.service 2>/dev/null || true

# Clear endpoints and tighten limits
sudo ip mptcp endpoint flush
sudo ip mptcp limits set subflow 0 add_addr_accepted 0

# Optional: disable new MPTCP sockets entirely
sudo sysctl -w net.mptcp.enabled=0

# Remove persistence you added
sudo systemctl disable --now mptcp-limits.service 2>/dev/null || true
sudo rm -f /etc/systemd/system/mptcp-limits.service /etc/sysctl.d/70-mptcp-rpfilter.conf
sudo systemctl daemon-reload
Enter fullscreen mode Exit fullscreen mode

Existing plain TCP sockets are unaffected either way.

When MPTCP is the right tool

Choose MPTCP when session continuity or multi-path TCP throughput matters across independent L3 paths, and you can control (or at least test) both ends.

Prefer other tools when:

  • You only need NIC failover on one switch — bonding.
  • You need SAN path redundancy — DM-Multipath.
  • You need fair sharing / AQM — CAKE/fq_codel or classful tc.
  • You need sender congestion behavior on a single path — BBR/CUBIC.

References

Ship dual-homed hosts with endpoints + limits + one wrapped critical client first. Prove failover with a deliberate link down before you chase aggregation benchmarks. Multipath is only real when the second subflow carries traffic while the first is on fire.

Top comments (0)