DEV Community

Cover image for Stop Sending All Traffic to One Backend: Practical IPVS Load Balancing with keepalived on Linux
Lyra
Lyra

Posted on

Stop Sending All Traffic to One Backend: Practical IPVS Load Balancing with keepalived on Linux

#networking #linux #devops #opensource

Stop Sending All Traffic to One Backend: Practical IPVS Load Balancing with keepalived on Linux

A floating VIP keeps the front door available. That is not the same as spreading work across backends.

If one Linux box owns 203.0.113.10 and every HTTP request still lands on a single app host, you have high availability of an IP — not load balancing of a service. IPVS (IP Virtual Server) is the kernel's Layer-4 load balancer. keepalived is the userspace control plane that programs IPVS, health-checks real servers, and can own the VIP with VRRP when you want director failover too.

This post is the LVS/IPVS half of the stack: virtual services, NAT vs Direct Routing, health checks, ARP-safe real-server VIP binding, and verification with ipvsadm.

What you are building 

 clients
    |
    v
 VIP :80  (director / LinuxDirector)
    |
    +-- RS1 10.0.0.11:80
    +-- RS2 10.0.0.12:80
    +-- RS3 10.0.0.13:80  (weight 0 / inhibited when unhealthy)
Enter fullscreen mode Exit fullscreen mode

IPVS schedules new connections. Established flows stick to the real server chosen for that connection until they expire or the destination is removed. That is why health checks, weights, and optional connection sync matter.

Install the tools

Debian/Ubuntu:

sudo apt-get update
sudo apt-get install -y keepalived ipvsadm iproute2
Enter fullscreen mode Exit fullscreen mode

Fedora/RHEL-family:

sudo dnf install -y keepalived ipvsadm iproute
Enter fullscreen mode Exit fullscreen mode

Load the common scheduler/forwarding modules once so the first service create does not surprise you:

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_wlc
sudo modprobe nf_conntrack
# useful on NAT setups that also use stateful firewall rules:
# echo 1 | sudo tee /proc/sys/net/ipv4/vs/conntrack
Enter fullscreen mode Exit fullscreen mode

Confirm the kernel side is present:

lsmod | grep '^ip_vs'
ipvsadm -Ln || true
Enter fullscreen mode Exit fullscreen mode

Choose a forwarding method first

Method keepalived / ipvsadm Request path Reply path Topology constraint
NAT (masquerading) lvs_method NAT / -m client → director DNAT → RS RS → director SNAT → client RS default route usually via director; private RS nets OK
DR (direct routing / gatewaying) lvs_method DR / -g client → director rewrites L2 dest → RS RS → client directly director + RS on same L2 for the VIP path; VIP must exist on RS without answering ARP
TUN lvs_method TUN / -i director encapsulates to RS RS → client directly RS must decap IPIP/GUE/GRE; more moving parts

Start with NAT if you want the fewest network surprises. Use DR when return bandwidth should leave the director (common for high-throughput HTTP/TCP farms on one LAN).

Lab addressing used below

  • VIP: 203.0.113.10
  • Director real IP: 203.0.113.2 (and optionally a second director 203.0.113.3)
  • Real servers: 10.0.0.11, 10.0.0.12 (NAT example) or 203.0.113.11, 203.0.113.12 (DR example)
  • Service: TCP/80

Replace with your ranges. Do not paste these public documentation addresses into production routing without thinking.

Path A — VS/NAT (simplest correct farm)

1) Director sysctl 

sudo tee /etc/sysctl.d/99-ipvs-nat-director.conf >/dev/null <<'EOF'
net.ipv4.ip_forward = 1
# Only if nftables/iptables stateful rules must see IPVS flows:
# net.ipv4.vs.conntrack = 1
EOF
sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Kernel docs: conntrack under /proc/sys/net/ipv4/vs/* is off by default for performance; enable it when IPVS-handled connections must also match conntrack-based firewall policy.

2) keepalived virtual server (NAT)

/etc/keepalived/keepalived.conf on the director:

global_defs {
    router_id lvs-nat-1
    # Drop stale LVS objects left from earlier experiments
    lvs_flush
    lvs_flush_on_stop
}

# Optional: own the VIP with VRRP (director HA).
# If you already publish 203.0.113.10 another way, omit this block
# and point virtual_server at that address.
vrrp_instance VI_HTTP {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass change-me
    }
    virtual_ipaddress {
        203.0.113.10/24 dev eth0
    }
}

virtual_server 203.0.113.10 80 {
    delay_loop 6
    lvs_sched wlc
    lvs_method NAT
    protocol TCP
    # persistence_timeout 300   # sticky client->RS mapping when needed (TLS/session apps)
    alpha                  # assume RS down until checks pass (avoids false-up at boot)
    omega
    quorum 1
    # sorry_server 10.0.0.99 80
    inhibit_on_failure     # weight 0 on failure instead of deleting the RS (drain-friendly)

    real_server 10.0.0.11 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
        # Or application-aware:
        # HTTP_GET {
        #     url {
        #         path /healthz
        #         status_code 200-299
        #     }
        #     connect_timeout 3
        #     retry 2
        # }
    }

    real_server 10.0.0.12 80 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Notes from keepalived.conf(5):

  • Scheduler and forwarding method are lvs_sched and lvs_method on current keepalived man pages (values such as wlc and NAT/DR).
  • inhibit_on_failure sets weight to 0 on failed checks. IPVS treats weight 0 as quiescent: no new jobs, existing jobs can finish.
  • alpha starts checkers pessimistic so a restart does not briefly advertise dead backends.
  • sorry_server is the overflow/fallback RS when quorum is not met.

Enable and start:

sudo keepalived -t -l    # config test where supported
sudo systemctl enable --now keepalived
sudo systemctl status keepalived --no-pager
Enter fullscreen mode Exit fullscreen mode

3) Real-server side for NAT

  • App listens on 0.0.0.0:80 or the RS IP.
  • Default route points at the director (or at a gateway that returns through the director) so replies are SNATed correctly.
  • No VIP on the real servers.

4) Manual IPVS equivalent (debug only)

keepalived should own the table. For learning, the same NAT service looks like:

sudo ipvsadm -A -t 203.0.113.10:80 -s wlc
sudo ipvsadm -a -t 203.0.113.10:80 -r 10.0.0.11:80 -m -w 1
sudo ipvsadm -a -t 203.0.113.10:80 -r 10.0.0.12:80 -m -w 1
sudo ipvsadm -Ln
Enter fullscreen mode Exit fullscreen mode

-m is masquerading/NAT. Default scheduler without -s is wlc (weighted least-connection).

Path B — VS/DR (director stays thin on the return path)

Direct Routing rewrites the destination MAC toward the chosen real server and leaves the IP VIP untouched. The real server must accept packets for the VIP locally and must not win ARP for that VIP on the LAN.

1) Real-server VIP on lo + modern ARP policy

On each real server:

sudo tee /etc/sysctl.d/99-ipvs-dr-realserver.conf >/dev/null <<'EOF'
# Only answer ARP for addresses configured on the incoming interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
# Prefer announcing source IPs that belong on the egress interface
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
EOF
sudo sysctl --system

# Host route / local VIP on loopback (do not put VIP on eth0)
sudo ip addr add 203.0.113.10/32 dev lo
# persist with your network manager; example systemd-networkd snippet:
# /etc/systemd/network/lo.network.d/vip.conf
# [Address]
# Address=203.0.113.10/32
Enter fullscreen mode Exit fullscreen mode

Why this works (kernel ip-sysctl semantics):

  • arp_ignore = 1 — reply only if the target IP is configured on the incoming interface. VIP on lo therefore does not answer ARP arriving on eth0.
  • arp_announce = 2 — always use the best local address for the target when forming ARP requests, reducing “VIP as ARP source on eth0” surprises.

This is the modern replacement for the old hidden sysctl / patches discussed in classic LVS ARP docs.

App requirement for DR: the daemon must accept connections destined to the VIP (listen on 0.0.0.0 or explicitly on the VIP).

2) Director keepalived for DR 

global_defs {
    router_id lvs-dr-1
    lvs_flush
    lvs_flush_on_stop
}

vrrp_instance VI_HTTP {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/32 dev eth0
    }
}

virtual_server 203.0.113.10 80 {
    delay_loop 5
    lvs_sched wrr
    lvs_method DR
    protocol TCP
    alpha
    inhibit_on_failure

    real_server 203.0.113.11 80 {
        weight 2
        HTTP_GET {
            url {
                path /healthz
                status_code 200-299
            }
            connect_ip 203.0.113.11
            connect_port 80
            connect_timeout 3
            retry 2
            delay_before_retry 2
        }
    }

    real_server 203.0.113.12 80 {
        weight 1
        HTTP_GET {
            url {
                path /healthz
                status_code 200-299
            }
            connect_timeout 3
            retry 2
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

DR constraints that bite people:

  • Director and real servers need a shared L2 path for the VIP delivery method described by LVS DR (director rewrites L2 destination).
  • Port on the real server equals the virtual service port for DR/TUN.
  • Clients on the same LAN as the VIP can behave oddly if ARP policy is incomplete — verify with ip neigh / packet captures before blaming the scheduler.

Manual DR add for debugging:

sudo ipvsadm -A -t 203.0.113.10:80 -s wrr
sudo ipvsadm -a -t 203.0.113.10:80 -r 203.0.113.11:80 -g -w 2
sudo ipvsadm -a -t 203.0.113.10:80 -r 203.0.113.12:80 -g -w 1
Enter fullscreen mode Exit fullscreen mode

-g is gatewaying/DR (default if you omit the method flag).

Scheduler cheat sheet (pick deliberately)

From ipvsadm(8):

Scheduler Good default when…
wlc Mixed connection lengths; default IPVS choice
wrr Roughly equal request cost; simple capacity weights
rr / lc Unweighted variants of the above
sh Source-hash stickiness without full persistence templates
mh Maglev-style consistent hashing; minimal disruption when RS set changes
fo Active/standby by weight, not spreading
ovf Fill highest weight first, then overflow

Persistence (persistence_timeout / ipvsadm -p) pins a client (or masked client net) to one RS for SSL/sessionful apps. Prefer app-level shared session stores when you can; use IPVS persistence when you cannot.

Verify like an operator 

# Table keepalived programmed
sudo ipvsadm -Ln
sudo ipvsadm -Ln --stats
sudo ipvsadm -Ln --rate
sudo ipvsadm -Lnc | head

# Timeouts
sudo ipvsadm -Ln --timeout

# Kernel counters / presence
cat /proc/net/ip_vs
ls /proc/sys/net/ipv4/vs/

# keepalived health / VRRP
journalctl -u keepalived -e --no-pager | tail -n 80
ip -br addr show
Enter fullscreen mode Exit fullscreen mode

Healthy NAT/DR service looks roughly like:

TCP  203.0.113.10:80 wlc
  -> 10.0.0.11:80             Masq   1      0          0
  -> 10.0.0.12:80             Masq   1      0          0
Enter fullscreen mode Exit fullscreen mode

or Route instead of Masq for DR.

Controlled failure test:

  1. curl -sS -o /dev/null -w '%{http_code}\n' http://203.0.113.10/healthz in a loop.
  2. Stop the app or firewall the checker path on RS1.
  3. Watch keepalived log the down transition and ipvsadm -Ln show weight 0 (inhibit) or RS removal.
  4. Confirm new curls still succeed via RS2.
  5. Restore RS1; weight returns after successful checks.

Optional: two directors without dropping L4 state

VRRP moves the VIP. IPVS connection entries are separate. For director pairs:

global_defs {
    router_id lvs-a
    # Bind IPVS sync to VRRP state on the dedicated sync NIC/path
    lvs_sync_daemon eth1 inst VI_HTTP id 51
    lvs_timeouts tcp 900 tcpfin 120 udp 300
}
Enter fullscreen mode Exit fullscreen mode

lvs_sync_daemon starts the kernel IPVS sync daemons and can track a VRRP instance so only the master sends and the backup receives. Details and socket options are in keepalived.conf(5) and ipvsadm --start-daemon. Related knobs live under /proc/sys/net/ipv4/vs/sync_* (see kernel ipvs-sysctl docs).

This is IPVS connection sync, not Netfilter conntrack sync. If your firewall policy depends on conntrack for the same flows, that is a different mechanism.

Also consider:

# On backup-only nodes that should never forward as director while backup (DR/TUN loop guard)
echo 1 | sudo tee /proc/sys/net/ipv4/vs/backup_only
Enter fullscreen mode Exit fullscreen mode

Firewall notes (short, practical)

  • Allow client → VIP service ports on the director.
  • Allow director health checks → RS IPs/ports.
  • For NAT, allow forwarded traffic director ↔ RS and enable the MASQUERADE/SNAT path your design needs.
  • If you filter with conntrack matches against IPVS traffic, set net.ipv4.vs.conntrack=1 (requires CONFIG_IP_VS_NFCT).
  • VRRP (if used) is IP protocol 112 / multicast 224.0.0.18 unless you run unicast peers.

What this is not

  • Not L7 reverse proxying. IPVS does not terminate TLS or route on Host headers. Put HAProxy/nginx/Caddy behind or in front when you need application logic; use IPVS when you want kernel L4 fan-out.
  • Not only VRRP. Floating VIP ownership without virtual_server blocks is gateway HA, not a server farm.
  • Not conntrackd. Session pickup for stateful firewall pairs is a different problem from IPVS sync.
  • Not kube-proxy replacement guidance. Kubernetes may use IPVS mode internally; this article is host/LVS operations with keepalived.

Minimal rollback 

sudo systemctl stop keepalived
sudo ipvsadm -C
# DR real servers:
# sudo ip addr del 203.0.113.10/32 dev lo
# sudo rm /etc/sysctl.d/99-ipvs-dr-realserver.conf && sudo sysctl --system
Enter fullscreen mode Exit fullscreen mode

Operational checklist

  1. Pick NAT or DR from topology, not habit.
  2. Put health checks on something the user needs (/healthz), not only TCP_CHECK to a port that accepts SYNs while the app is wedged.
  3. Prefer inhibit_on_failure + nonzero quorum over silent empty farms.
  4. On DR real servers, VIP on lo + arp_ignore/arp_announce before you open the VIP to clients.
  5. Verify with ipvsadm -Ln, --stats, and a deliberate RS failure — not only a green systemctl status.
  6. If you add a second director, plan VIP failover and IPVS sync (and firewall implications) explicitly.

References

Kernel-space L4 balancing is boring in the best way: small config surface, predictable failure modes, and tools (ipvsadm, keepalived checkers) that show you exactly which backend should get the next SYN. Wire the health checks to reality, keep ARP policy honest on DR, and the farm stops being a single point of overload disguised as a VIP.

Top comments (0)