DEV Community

Cover image for Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux
Lyra
Lyra

Posted on

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

#networking #devops #security #linux

Stop SYN Floods Exhausting Conntrack: Practical nftables SYNPROXY on Linux

A SYN flood is still one of the cheapest ways to hurt a Linux edge host.

Attackers blast SYN packets at a public listener. The kernel allocates half-open state. Conntrack fills up. Real clients start failing long before your application logs anything useful.

Linux already has a last-resort defense: TCP syncookies. That helps the local TCP stack survive backlog pressure. It does not stop Netfilter connection tracking from paying the cost of every unauthenticated handshake attempt.

nftables synproxy moves the three-way handshake into Netfilter itself. Incomplete clients never create full conntrack entries. Only hosts that finish the cookie handshake get a real connection toward your service.

This post is a practical setup for protecting local TCP services (HTTPS, reverse proxies, self-hosted APIs) with stock Linux tooling.

What you are solving

Without SYNPROXY, a flood can burn two scarce resources at once:

  1. Listener backlog / SYN queue on the socket
  2. Conntrack table slots used by half-open or spoofed attempts

net.ipv4.tcp_syncookies=1 is the classic socket-level mitigation. Kernel docs are explicit: syncookies are a fallback when the SYN backlog overflows, not a general-purpose load feature.

SYNPROXY is different. It:

  • intercepts new SYN packets early
  • answers with a cookie SYN/ACK
  • validates the client’s final ACK
  • only then establishes a tracked connection toward the real listener
  • translates sequence numbers so the client and server still speak normal TCP

Patrick McHardy’s original netfilter SYN proxy design describes exactly this split: untracked cookie exchange first, real conntrack entry only after cookie validation.

Prerequisites

You need:

  • Linux with Netfilter conntrack and SYNPROXY support (common on modern kernels)
  • nftables userspace recent enough for the synproxy statement (anonymous objects since nftables 0.9.2; named objects since 0.9.3)
  • root on the host that terminates or forwards the protected TCP ports
  • a way to observe one real SYN/ACK from the backend (tcpdump)

Packages (Debian/Ubuntu-style):

sudo apt-get update
sudo apt-get install -y nftables tcpdump
Enter fullscreen mode Exit fullscreen mode

How SYNPROXY fits the path

A working ruleset has three moving parts:

  1. Mark initial SYNs as untracked in a prerouting/raw chain
  2. Hand untracked + invalid TCP packets to synproxy in input (local service) or forward (backend behind this box)
  3. Disable loose TCP conntrack recovery so the client’s final handshake ACK is seen as INVALID and still reaches SYNPROXY

That third point is not optional. The nftables wiki and nft(8) manpage both require:

sudo sysctl -w net.netfilter.nf_conntrack_tcp_loose=0
Enter fullscreen mode Exit fullscreen mode

Also enable the cookie/timestamp stack SYNPROXY relies on:

sudo sysctl -w net.ipv4.tcp_syncookies=1
sudo sysctl -w net.ipv4.tcp_timestamps=1
Enter fullscreen mode Exit fullscreen mode

Make it durable:

sudo tee /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
# Required for nftables/iptables SYNPROXY cookie handshakes
net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
EOF

sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Why loose tracking must be off

With loose TCP tracking enabled, conntrack may reconstruct state from out-of-flow packets. The final client ACK of the cookie handshake can then create a normal conntrack entry too early and bypass the proxy logic. Turning loose mode off forces that ACK into INVALID, which your SYNPROXY rule deliberately matches.

Step 1 — Measure the backend TCP options

SYNPROXY does not probe your server. You must announce the same MSS, window scale, timestamp, and SACK behavior the real listener uses.

From a client machine (or another host that can reach the service), capture one server SYN/ACK:

# On a host that can see the reply path to the service
sudo tcpdump -pni eth0 -c 1 'tcp[tcpflags] == (tcp-syn|tcp-ack) && port 443' &
curl -vk --max-time 5 https://192.0.2.10/ >/dev/null
Enter fullscreen mode Exit fullscreen mode

Example output shape from nft(8):

Flags [S.], ..., options [mss 1460,sackOK, TS val ..., ecr ..., nop,wscale 9]
Enter fullscreen mode Exit fullscreen mode

Read:

  • mss 1460 → mss 1460
  • wscale 9 → wscale 9
  • sackOK → include sack-perm
  • TS val present → include timestamp

If your listener options change after a kernel/sysctl/NIC MTU change, re-measure. Mismatched MSS/wscale is a classic “SYNPROXY works in theory, clients hang in practice” failure mode.

Step 2 — Protect local listeners with an anonymous synproxy

This ruleset protects TCP/443 on the local host. Adjust ports to match your services.

sudo tee /etc/nftables.d/synproxy-local.nft >/dev/null <<'EOF'
table inet synproxy_local {
  # Early: do not create conntrack entries for bare SYNs to protected ports
  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;

    tcp dport { 80, 443 } tcp flags syn notrack
  }

  chain input_synproxy {
    type filter hook input priority filter; policy accept;

    # Cookie handshake for untracked SYNs and invalid final ACKs
    tcp dport { 80, 443 } ct state invalid,untracked \
      synproxy mss 1460 wscale 9 timestamp sack-perm

    # Anything still invalid after synproxy is junk / failed cookies
    tcp dport { 80, 443 } ct state invalid drop
  }
}
EOF
Enter fullscreen mode Exit fullscreen mode

Load it without wiping your whole firewall:

# Validate first
sudo nft -c -f /etc/nftables.d/synproxy-local.nft

# Apply
sudo nft -f /etc/nftables.d/synproxy-local.nft
sudo nft list table inet synproxy_local
Enter fullscreen mode Exit fullscreen mode

If your distribution already uses a monolithic /etc/nftables.conf, include the file from there instead of loading ad hoc:

# /etc/nftables.conf
#!/usr/sbin/nft -f
flush ruleset

include "/etc/nftables.d/*.nft"
Enter fullscreen mode Exit fullscreen mode

Then:

sudo systemctl enable --now nftables
sudo systemctl reload nftables
Enter fullscreen mode Exit fullscreen mode

What each rule is doing

Stage Match Effect
prerouting / raw tcp flags syn to protected ports notrack — no conntrack allocation yet
input / filter ct state invalid,untracked synproxy ... answers cookies / validates ACKs
input / filter remaining ct state invalid drop failed or out-of-flow junk

Established flows after a successful handshake are ordinary conntrack-managed TCP. Your normal ct state established,related accept rules continue to apply.

Step 3 — Optional named synproxy objects

If several ports or source ranges need different TCP option profiles, use named objects (nftables 0.9.3+):

table ip synproxy_named {
  synproxy https_profile {
    mss 1460
    wscale 9
    timestamp
    sack-perm
  }

  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;
    tcp dport 443 tcp flags syn notrack
  }

  chain input_synproxy {
    type filter hook input priority filter; policy accept;
    tcp dport 443 ct state invalid,untracked synproxy name "https_profile"
    tcp dport 443 ct state invalid drop
  }
}
Enter fullscreen mode Exit fullscreen mode

Named objects are easier to reuse when one box fronts multiple backends with different MSS/wscale values.

Step 4 — Forward-path protection for a backend

If this Linux box is a firewall/load-balancer in front of another server, put SYNPROXY on forward instead of (or in addition to) input:

table inet synproxy_forward {
  chain raw_prerouting {
    type filter hook prerouting priority raw; policy accept;
    iifname "eth0" tcp dport { 80, 443 } tcp flags syn notrack
  }

  chain forward_synproxy {
    type filter hook forward priority filter; policy accept;

    iifname "eth0" tcp dport { 80, 443 } ct state invalid,untracked \
      synproxy mss 1460 wscale 9 timestamp sack-perm

    iifname "eth0" tcp dport { 80, 443 } ct state invalid drop
  }
}
Enter fullscreen mode Exit fullscreen mode

Important: measure MSS/wscale from the real backend’s SYN/ACK, not from an unrelated local socket.

Step 5 — Size conntrack for the traffic you still accept

SYNPROXY reduces garbage half-open pressure, but legitimate concurrent connections still need table headroom:

# Current usage
sudo sysctl net.netfilter.nf_conntrack_count \
             net.netfilter.nf_conntrack_max

# Example bump — pick values from real peaks, not vibes
sudo tee -a /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
net.netfilter.nf_conntrack_max = 524288
EOF

# On many kernels hashsize is a module parameter
# example only — confirm path exists on your host first
if [ -w /sys/module/nf_conntrack/parameters/hashsize ]; then
  echo 131072 | sudo tee /sys/module/nf_conntrack/parameters/hashsize
fi

sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

The nftables wiki explicitly calls out raising nf_conntrack_max and conntrack hash size alongside SYNPROXY.

Verification checklist

1. Sysctls landed 

sysctl net.netfilter.nf_conntrack_tcp_loose \
       net.ipv4.tcp_syncookies \
       net.ipv4.tcp_timestamps
Enter fullscreen mode Exit fullscreen mode

Expected:

net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
Enter fullscreen mode Exit fullscreen mode

2. Rules are attached where you think 

sudo nft list ruleset
sudo nft list hooks 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Confirm:

  • protected ports hit notrack on prerouting/raw
  • synproxy sits on input and/or forward

3. Real clients still complete

From an external client:

curl -I --max-time 10 https://your.example
openssl s_client -connect your.example:443 -servername your.example </dev/null
Enter fullscreen mode Exit fullscreen mode

On the server, a good connection should show up as normal established TCP after the handshake:

ss -Htn state established 'sport = :443' | head
sudo conntrack -L -p tcp --dport 443 2>/dev/null | head
Enter fullscreen mode Exit fullscreen mode

4. Cookie path is active under SYN pressure

Generate controlled SYN-only noise in a lab (not against third-party networks):

# Lab-only example using hping3 if installed
# sudo hping3 -S -p 443 --flood 192.0.2.10
Enter fullscreen mode Exit fullscreen mode

While that runs:

  • protected service should keep answering real completed handshakes
  • nf_conntrack_count should not climb 1:1 with spoofed SYN rate
  • invalid/failed cookies should be dropped by the final ct state invalid drop

If legitimate clients break immediately, re-check MSS/wscale/timestamp/SACK against a fresh tcpdump of the backend SYN/ACK.

Operational pitfalls

  1. Wrong MSS/wscale

    Clients stall after SYN/ACK. Re-capture options after MTU or listener changes.

  2. Leaving nf_conntrack_tcp_loose=1

    Final ACKs may bypass SYNPROXY. Cookie mode becomes inconsistent.

  3. Protecting ports that need exotic TCP options

    If the backend relies on options you did not enable in the synproxy statement, feature mismatch follows. Stick to the measured set.

  4. Applying SYNPROXY globally to every TCP port

    Start with the public listeners that matter (80/443/API). Broad notrack on all SYNs makes debugging harder and can surprise internal health checks.

  5. Confusing this with application rate limits

    SYNPROXY authenticates the TCP handshake. It does not replace HTTP auth, API quotas, Fail2ban-style abuse controls, or upstream DDoS scrubbing.

  6. Expecting miracles under asymmetric routing

    Sequence translation and conntrack both assume the firewall sees both directions of the flow.

Rollback 

# Remove just the synproxy table(s)
sudo nft delete table inet synproxy_local 2>/dev/null || true
sudo nft delete table inet synproxy_forward 2>/dev/null || true
sudo nft delete table ip synproxy_named 2>/dev/null || true

# Or restore your previous full ruleset
# sudo nft -f /etc/nftables.conf.bak

# Restore loose tracking if you intentionally want the old behavior
sudo sysctl -w net.netfilter.nf_conntrack_tcp_loose=1
# and edit/remove /etc/sysctl.d/90-synproxy.conf as needed
Enter fullscreen mode Exit fullscreen mode

Keep a known-good nft list ruleset dump before the change:

sudo nft list ruleset > "/root/nftables-before-synproxy-$(date +%F).nft"
Enter fullscreen mode Exit fullscreen mode

How this differs from nearby defenses

Tool Layer Main job
tcp_syncookies local TCP stack survive SYN backlog overflow on a socket
nftables synproxy Netfilter complete handshake with cookies before spending full conntrack/backend state
Fail2ban + nftables sets auth/log abuse ban sources after application or log signals
nftables flowtables established forward path skip classic Netfilter for already-good flows
tc HTB / CAKE queueing/bandwidth shape or fair-queue traffic, not authenticate SYNs

Use SYNPROXY when the failure mode is “unauthenticated TCP handshakes are cheap for the attacker and expensive for conntrack/backends.”

Minimal production recipe 

# 1) sysctls
sudo tee /etc/sysctl.d/90-synproxy.conf >/dev/null <<'EOF'
net.netfilter.nf_conntrack_tcp_loose = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 1
EOF
sudo sysctl --system

# 2) measure backend options with tcpdump (replace NIC/port/IP)
# sudo tcpdump -pni eth0 -c 1 'tcp[tcpflags] == (tcp-syn|tcp-ack) && port 443'

# 3) load rules with the measured mss/wscale/flags
sudo nft -f /etc/nftables.d/synproxy-local.nft

# 4) verify
sysctl net.netfilter.nf_conntrack_tcp_loose
sudo nft list table inet synproxy_local
curl -I https://127.0.0.1/
Enter fullscreen mode Exit fullscreen mode

Closing

If your public Linux host still treats every bare SYN as worth a full conntrack entry, you are letting the cheapest packets buy the most expensive state.

nftables SYNPROXY is not a full anti-DDoS platform. It is a precise kernel feature with a clear contract:

  • measure the backend TCP options
  • untrack bare SYNs
  • proxy the handshake with cookies
  • drop what still looks invalid
  • keep loose conntrack off

Do that, and SYN floods stop converting directly into conntrack exhaustion on the ports you chose to protect.

Sources and references

Top comments (0)