Stop Dropping Established Sessions on Failover: Practical conntrackd State Sync on Linux

A floating VIP with keepalived is only half of firewall high availability.

You move the address. Clients still send packets to the same IP. Then the new active node drops perfectly good established sessions because its kernel never saw the original handshake — so Netfilter has no conntrack entry, ct state established does not match, and your stateful policy treats the traffic as garbage.

That is the gap conntrackd fills.

This guide walks through a practical active/backup setup:

• why VIP failover alone breaks stateful firewalls
• FTFW sync over a dedicated link
• a clean conntrackd.conf
• keepalived notify hooks that commit state on promote
• verification and a real cutover test
• the boundaries that still bite people

No theory dump. Config you can paste and reason about.

The failure mode (why VRRP is not enough)

Assume two firewalls, one VIP, stateful rules roughly like:

# default deny for forwarded traffic
nft add rule inet filter forward ct state invalid drop
nft add rule inet filter forward ct state established,related accept
nft add rule inet filter forward iifname "lan0" tcp flags syn / syn,rst,ack ct state new accept
# SNAT / MASQUERADE on the WAN path for LAN clients

Or the classic iptables shape from the official conntrack-tools test case:

iptables -P FORWARD DROP
iptables -A FORWARD -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth1 -p tcp --syn -m state --state NEW -j ACCEPT
iptables -A FORWARD -i eth1 -p tcp -m state --state ESTABLISHED -j ACCEPT
iptables -A FORWARD -m state --state INVALID -j LOG
iptables -t nat -A POSTROUTING -s 192.168.0.3 -j SNAT --to-source 192.168.1.100

Now start a long SSH (or HTTPS, or DB) session through the active node.

Kill the active node. keepalived promotes the backup. The VIP moves.

Without conntrack sync:

1. The backup owns the VIP.
2. Mid-flow TCP segments arrive with no local conntrack entry.
3. They are not NEW SYNs, so they miss the allow-new rule.
4. They are not ESTABLISHED locally, so they miss the established rule.
5. With a default-drop forward policy, they die — often logged as INVALID.

With conntrackd, the backup already holds a replica of those flow entries. On promote, it commits them into the kernel table and the same sessions keep flowing.

What you are installing

Package names:

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y conntrackd conntrack keepalived

# Fedora / RHEL-ish
sudo dnf install -y conntrack-tools keepalived

The package gives you two tools:

Tool	Role
conntrack	CLI for the kernel conntrack table (-L, -E, -D, …)
conntrackd	Daemon that replicates flow state between firewalls

Kernel prerequisites (almost always present on modern distro kernels):

• nf_conntrack
• nf_conntrack_netlink / CONFIG_NF_CT_NETLINK
• connection tracking events (CONFIG_NF_CONNTRACK_EVENTS)

Quick sanity:

modinfo nf_conntrack | head
lsmod | grep -E 'nf_conntrack|nfnetlink'
cat /proc/sys/net/netfilter/nf_conntrack_max
conntrack -L | head

Architecture for this guide

Two-node active/backup firewall pair:

Role	Host	LAN iface	WAN iface	Sync iface	Dedicated link IP
Preferred primary	fw1	lan0	wan0	sync0	192.168.100.1/24
Backup	fw2	lan0	wan0	sync0	192.168.100.2/24
Floating VIP (LAN side example)	—	—	—	—	192.168.10.1
Floating VIP (WAN side example)	—	—	—	—	203.0.113.10

Rules of the road from the official manual:

1. Dedicated sync link — do not piggyback state replication on the production LAN/WAN path if you can avoid it. State messages are sensitive and lossy under congestion.
2. Stateful ruleset on both nodes — same policy, same NAT shape.
3. HA manager hooks — keepalived (or equivalent) must call the primary/backup transition script.
4. Prefer FTFW mode (message tracking / recover from loss and reordering) over plain NOTRACK for production.

Step 1 — Size conntrack before you replicate it

Replication multiplies pain if the table is already undersized.

# current usage
conntrack -C
# or
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

Set a realistic max and match conntrackd cache limits to it. The man page guidance: HashLimit should be about double nf_conntrack_max, because the daemon may retain dead entries for retransmission.

Example sysctl drop-in:

# /etc/sysctl.d/99-conntrack.conf
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_buckets = 65536
# optional: be less aggressive about mid-flow recovery edge cases on old kernels
# net.netfilter.nf_conntrack_tcp_be_liberal = 1

sudo sysctl --system

Also bump the Netlink event socket early in conntrackd.conf (shown below). Default ~100 KiB receive buffers are small for busy firewalls and cause event drops / expensive resyncs.

Step 2 — conntrackd.conf (FTFW + UDP dedicated link)

Create /etc/conntrackd/conntrackd.conf on both nodes. Only the local dedicated-link addresses change.

This example uses FTFW over unicast UDP on the sync NIC — a solid default when you have exactly two nodes and multicast is awkward. Multicast works too; the official example ships with multicast 225.0.0.50 / group 3780.

fw1

# /etc/conntrackd/conntrackd.conf  (fw1)
Sync {
    Mode FTFW {
        # ResendQueueSize 131072
        # CommitTimeout 180
        # PurgeTimeout 60
        # DisableExternalCache no
        # StartupResync yes
    }

    UDP {
        IPv4_address 192.168.100.1
        IPv4_Destination_Address 192.168.100.2
        Port 3780
        Interface sync0
        SndSocketBuffer 1249280
        RcvSocketBuffer 1249280
        Checksum on
    }
}

General {
    HashSize 32768
    HashLimit 524288

    LogFile on
    Syslog on
    LockFile /var/lock/conntrack.lock

    UNIX {
        Path /var/run/conntrackd.ctl
        Backlog 20
    }

    NetlinkBufferSize 2097152
    NetlinkBufferSizeMaxGrowth 8388608
    # NetlinkEventsReliable yes   # kernel >= 2.6.31; if yes, consider NetlinkOverrunResync off

    Filter From Userspace {
        Protocol Accept {
            TCP
            UDP
            ICMP
        }

        # Do NOT replicate purely local / VIP / sync-link noise.
        # Only forwarded flows are worth recovering on the peer.
        Address Ignore {
            IPv4_address 127.0.0.1
            IPv4_address 192.168.100.1
            IPv4_address 192.168.100.2
            IPv4_address 192.168.10.1      # LAN VIP
            IPv4_address 203.0.113.10      # WAN VIP
            # add each node's real interface addresses too
            IPv4_address 192.168.10.11
            IPv4_address 192.168.10.12
        }
    }
}

fw2

Same file, swap the UDP addresses:

    UDP {
        IPv4_address 192.168.100.2
        IPv4_Destination_Address 192.168.100.1
        Port 3780
        Interface sync0
        SndSocketBuffer 1249280
        RcvSocketBuffer 1249280
        Checksum on
    }

And flip the local addresses in Address Ignore if you listed node-specific IPs separately.

Why these knobs matter

Setting	Why it is there
Mode FTFW	Reliable-ish replication with ACK / resend queue — recovers from loss and reordering better than pure NOTRACK
UDP on sync0	Dedicated path; unicast is simple for two nodes
Large Snd/RcvSocketBuffer	Avoid overrun on the sync sockets under bursty state churn
Large NetlinkBufferSize*	Avoid dropping kernel→userspace conntrack events
HashLimit ≈ 2 × nf_conntrack_max	Room for live + retransmit cache objects
Address Ignore for locals/VIPs	Official guidance: local traffic is not worth replicating; forwarded flows are
External cache left enabled	Safer first deploy: backup keeps foreign state in userspace until promote commits it

DisableExternalCache yes injects peer state straight into the kernel table on the backup. That skips the commit step and saves userspace memory, but burns kernel conntrack slots and CPU on the idle node. The man page still steers first-time installs toward the fail-over scripts instead.

Step 3 — Transition script (the important part)

Ship the upstream primary-backup.sh logic. On Debian/Ubuntu it is often packaged under /usr/share/doc/conntrackd/examples/sync/ (path varies by release). Install it as /etc/conntrackd/primary-backup.sh:

#!/bin/sh
# Adapted from conntrack-tools doc/sync/primary-backup.sh
# (C) 2006-2011 Pablo Neira Ayuso — GPL-2.0-or-later

CONNTRACKD_BIN=/usr/sbin/conntrackd
CONNTRACKD_LOCK=/var/lock/conntrack.lock
CONNTRACKD_CONFIG=/etc/conntrackd/conntrackd.conf

case "$1" in
  primary)
    # Inject peer-replicated flows into the local kernel table
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -c
    # Flush userspace caches, then rebuild internal cache from kernel
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -f
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -R
    # Tell backups what we now own
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -B
    ;;
  backup)
    # Ensure daemon is alive
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -s
    if [ $? -eq 1 ]; then
        [ -f "$CONNTRACKD_LOCK" ] && rm -f "$CONNTRACKD_LOCK"
        $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -d || exit 1
    fi
    # Shorten timers to age out zombies after demotion
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -t
    # Request resync from the current primary (FTFW/NOTRACK)
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -n
    ;;
  fault)
    $CONNTRACKD_BIN -C "$CONNTRACKD_CONFIG" -t
    ;;
  *)
    echo "Usage: $0 {primary|backup|fault}" >&2
    exit 1
    ;;
esac

exit 0

sudo install -m 0755 primary-backup.sh /etc/conntrackd/primary-backup.sh

What each client flag does

Flag	Meaning
-c	Commit external cache → kernel conntrack table (promote path)
-f	Flush internal/external userspace caches
-R	Resync internal cache from the kernel table
-B	Bulk-send owned state to peers
-t	Reset/shorten in-kernel timers (PurgeTimeout) after demotion
-n	Request resync from the other node
-s	Statistics (also used as a liveness probe)
-i / -e	Dump internal (local) / external (foreign) cache

If promote works, logs should show something like:

[notice] committing external cache
[notice] Committed 1545 new entries

Step 4 — keepalived notify hooks

Minimal keepalived integration (pair with your existing VRRP VIP config):

# /etc/keepalived/keepalived.conf  (sketch — merge with your real instance)
global_defs {
    router_id fw_pair
    script_user root
    enable_script_security
}

vrrp_sync_group G1 {
    group {
        VI_LAN
        VI_WAN
    }
    notify_master "/etc/conntrackd/primary-backup.sh primary"
    notify_backup "/etc/conntrackd/primary-backup.sh backup"
    notify_fault  "/etc/conntrackd/primary-backup.sh fault"
}

vrrp_instance VI_LAN {
    state BACKUP          # both start BACKUP if you use nopreempt patterns
    interface lan0
    virtual_router_id 51
    priority 150          # fw2 uses a lower priority, e.g. 100
    advert_int 1
    # authentication { ... }  # note: VRRPv2 PASS is weak; prefer unicast + ACLs
    virtual_ipaddress {
        192.168.10.1/24
    }
    # track_interface / track_script as needed
}

vrrp_instance VI_WAN {
    state BACKUP
    interface wan0
    virtual_router_id 52
    priority 150
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/32
    }
}

Use a vrrp_sync_group so LAN and WAN VIPs flip together and the conntrack transition runs once for the group. The upstream conntrack-tools example does exactly that.

If you already run keepalived for gateway VIP HA, you are not replacing that design — you are finishing it.

Step 5 — Firewall the sync path (and only the sync path)

Allow state replication on the dedicated link. Examples:

# nftables — dedicated sync NIC only
nft add table inet raw
nft add chain inet raw conntrackd_sync { type filter hook input priority -300 \; }
nft add rule inet raw conntrackd_sync iifname "sync0" udp dport 3780 accept
nft add rule inet raw conntrackd_sync oifname "sync0" udp dport 3780 accept

If you use multicast instead of UDP unicast:

# allow the example group traffic on the sync NIC
iptables -I INPUT  -i sync0 -d 225.0.0.50 -j ACCEPT
iptables -I OUTPUT -o sync0 -d 225.0.0.50 -j ACCEPT

Do not expose sync traffic on untrusted interfaces. Anyone who can inject or observe conntrack replicas learns a lot about your live sessions.

Also make sure your forward policy is truly stateful on both nodes (established/related accept, invalid drop, identical NAT). conntrackd cannot fix a mismatched ruleset.

Step 6 — Enable services

sudo systemctl enable --now conntrackd.service
sudo systemctl enable --now keepalived.service

systemctl status conntrackd --no-pager
systemctl status keepalived --no-pager

Modern conntrackd builds support Type=notify systemd units and watchdog integration (Systemd yes in config when compiled with support). Prefer the distro unit over hand-rolled conntrackd -d in production.

Step 7 — Verify replication before you break anything

On the active node

# generate a long-lived flow through the VIP (SSH, curl --http1.1 keep-alive, iperf3, etc.)
sudo conntrackd -s
sudo conntrackd -i | head
sudo conntrack -L | head

On the backup node

sudo conntrackd -s
sudo conntrackd -e | head

Healthy pair shape:

• Active internal cache count ≈ Backup external cache count
• Backup external dump shows the same 5-tuple / state you care about (ESTABLISHED SSH, etc.)
• conntrackd -s network (or general stats) is not racking up send/receive errors

Example of what you want on the backup external cache (shape from the official test case):

tcp 6 ESTABLISHED src=192.168.0.3 dst=192.168.0.100 sport=51356 dport=22 \
  src=192.168.0.100 dst=192.168.1.3 sport=22 dport=51356 [ASSURED]

Useful ops commands:

sudo conntrackd -s cache
sudo conntrackd -s network
sudo conntrackd -s runtime
sudo journalctl -u conntrackd -e --no-pager
sudo tail -n 100 /var/log/conntrackd.log

Step 8 — Controlled failover test

Do this on purpose once, during a maintenance window.

1. Start a traffic generator through the VIP (SSH session that prints a clock, iperf3 -t 600, a keep-alive API client).
2. Confirm the flow is on the active internal cache and the backup external cache.
3. Force demotion of the active node, for example:

# on current master — pick ONE deliberate failure mode
sudo systemctl stop keepalived
# or: sudo ip link set lan0 down
# or: sudo kill -STOP $(pidof keepalived)   # only in a lab

1. Watch the backup:

# VIP ownership
ip -br addr show lan0
ip -br addr show wan0

# transition should commit external cache
sudo journalctl -u keepalived -u conntrackd -e --no-pager | tail -n 50
sudo grep -i commit /var/log/conntrackd.log | tail
sudo conntrack -L | grep -E 'dport=22|dport=443' | head

1. Confirm the client session survived without a reconnect.
2. Restore the original node and decide whether you want preemption (nopreempt vs priority takeback). Either way, the demoted node should run the backup path (-t + -n).

Pass / fail criteria

Check	Pass
VIP moves	yes
Committed N new entries (N > 0 for live flows)	yes
Long-lived TCP still transfers data	yes
No flood of INVALID drops for that flow	yes
After settle, new primary internal ≈ new backup external	yes

If the VIP moves but sessions die, you almost always have one of: commit script not hooked, external cache empty (sync broken), NAT/filter policy mismatch, or Address Ignore filtering the wrong prefixes.

Optional hardening and tuning

Expectation sync (FTP/SIP helpers)

If you still run helper-dependent protocols:

Sync {
    Mode FTFW {
        # ...
    }
    # ...
    Options {
        ExpectationSync On
        # or a list: ftp, sip, ...
        # TCPWindowTracking Off
    }
}

Needs a modern enough kernel (expectation sync features landed in the 3.x era; check your man page). Many sites are happier reducing helper use than replicating expectations.

Direct kernel injection on backup

Mode FTFW {
    DisableExternalCache yes
}

Faster failover (no -c commit bulk), higher steady-state cost on the backup. Only switch after the scripted path is proven.

Startup catch-up

Mode FTFW {
    StartupResync yes
}

Useful when a node boots while its peer has been carrying production state.

Keep the peer honest

Add a simple timer that alerts if external cache is empty while the node is backup and the peer is up — empty external cache during load means you will fail open into session death.

#!/bin/bash
# /usr/local/sbin/check-conntrack-sync.sh
set -euo pipefail
# only meaningful on backup nodes — detect via VIP absence
if ip -4 addr show lan0 | grep -q '192.168.10.1/'; then
  exit 0  # we are master; internal cache is the source of truth
fi
ext=$(conntrackd -e 2>/dev/null | wc -l || echo 0)
# crude: expect some foreign state during business hours
if [[ "$ext" -lt 1 ]]; then
  logger -t conntrack-sync "WARNING: external cache empty while backup"
  exit 1
fi

Wire it with a systemd timer if you want continuous signal.

What this does not solve

Be explicit so you do not over-promise HA:

• Stateless service HA for apps that bind only to the VIP without shared backend state — different problem (and often keepalived + app-level session affinity).
• LVS/IPVS full load-balancer farms — conntrackd is about Netfilter flow tables, not scheduler persistence tables.
• Asymmetric multi-path active/active where request and reply legs hit different firewalls unpredictably — the official manual warns this fights stateful design; prefer symmetric paths or accept weaker guarantees.
• iptables/nft modules that keep private side state (recent, connbytes, quota, …) — those counters are outside conntrack, so takeover may still mis-handle flows depending on them.
• Broken or divergent NAT/filter policy between nodes — replicated conntrack cannot invent SNAT mappings your backup would never have created the same way if the rules differ.
• Security of the sync channel — treat the dedicated link like cluster interconnect: isolated VLAN/cable, tight input rules, no general routed path.

Minimal operator runbook

# Is the daemon up?
systemctl is-active conntrackd keepalived

# Who is master?
ip -br addr | grep -E '192.168.10.1|203.0.113.10'

# Are we replicating?
sudo conntrackd -s
sudo conntrackd -i | wc -l
sudo conntrackd -e | wc -l

# Force a bulk push after maintenance (on primary)
sudo conntrackd -B

# Force kernel resync into internal cache
sudo conntrackd -R

# Watch live conntrack events (noisy — lab/debug)
sudo conntrack -E

References

• conntrack-tools user manual — state sync modes, caches, keepalived integration: https://conntrack-tools.netfilter.org/manual.html
• Official HA test case (why established sessions die without sync): https://conntrack-tools.netfilter.org/testcase.html
• conntrackd(8) and conntrackd.conf(5) (Debian manpages): https://manpages.debian.org/bookworm/conntrackd/conntrackd.8.en.html, https://manpages.debian.org/bookworm/conntrackd/conntrackd.conf.5.en.html
• Example primary-backup.sh, FTFW conntrackd.conf, and keepalived notify stubs from conntrack-tools doc/sync/
• keepalived project docs for VRRP instances and notify scripts: https://www.keepalived.org

Wrap-up

keepalived moves the address. conntrackd moves the memory of every flow that address was mid-way through.

For a two-node stateful firewall or NAT gateway:

1. Give the pair a dedicated sync link.
2. Run FTFW replication with sane buffer and hash limits.
3. Hook primary-backup.sh into keepalived notify events.
4. Prove it with a live session and a deliberate cutover — not a hope and a ping.

Once that commit path is boring and predictable, VIP failover stops feeling like a soft reboot of every client connection.