DEV Community

Cover image for Stop Single-Host Gateway Outages: Practical keepalived VRRP Floating IPs on Linux
Lyra
Lyra

Posted on

Stop Single-Host Gateway Outages: Practical keepalived VRRP Floating IPs on Linux

#opensource #linux #devops #networking

Stop Single-Host Gateway Outages: Practical keepalived VRRP Floating IPs on Linux

A service IP that lives on only one box is a single point of failure.

When that host reboots, freezes, or loses its uplink, every client that hard-coded 192.0.2.10 goes dark — even if an identical standby is sitting next to it with a warm cache and healthy disks.

keepalived implements the Virtual Router Redundancy Protocol (VRRP) on Linux so two or more hosts can share a floating virtual IP (VIP). One node owns the address as MASTER; the others stay BACKUP and take over when advertisements stop or a tracked check fails.

This is host-level IP failover, not NIC bonding on a single machine and not full application clustering. Different failure domain, different tool.

What you get (and what you do not)

You get:

  • A stable service address clients can keep forever
  • Automatic VIP migration when the current master dies or is demoted
  • Optional health coupling (vrrp_script, vrrp_track_process, track_interface) so “host up but nginx dead” still fails over
  • Gratuitous ARP / unsolicited Neighbor Advertisements so L2 neighbors relearn the VIP quickly

This article does not cover:

  • Linux Ethernet bonding / LACP (link aggregation on one host)
  • Device Mapper multipath (storage path HA)
  • Full LVS/IPVS load-balancer farms (keepalived can do that too; different problem)
  • Pacemaker/Corosync multi-resource clusters
  • DNS-based failover (TTLs and client caches behave differently)

If you need two cables on one server, bond first. If you need one IP that survives a whole server dying, keep reading.

How VRRP actually works (short version)

VRRP (RFC 5798 for version 3; RFC 3768 for the older v2 story) elects one Master per virtual router ID (VRID) on a LAN:

  1. Routers periodically send advertisements (default destination for IPv4 multicast is 224.0.0.18, protocol 112).
  2. The highest priority healthy router becomes Master and owns the virtual IPvX address(es).
  3. Backups listen. If advertisements stop for long enough, a backup promotes itself.
  4. On becoming Master, the new owner sends gratuitous ARP (IPv4) or unsolicited NA (IPv6) so switches and hosts update their neighbor caches.

keepalived is the common Linux implementation: it programs VIPs via netlink, runs optional track scripts/processes, and can use a VMAC (use_vmac) so the virtual router MAC stays stable across failovers.

Important keepalived note from the man page: VRRPv2 authentication (auth_type PASS / AH) was removed from the VRRPv2 specification by RFC 3768. PASS sends a cleartext password on the wire. Treat it as a misconfiguration guard, not real security. Prefer network isolation (dedicated VLAN, firewall) for VRRP traffic.

Lab topology

Two Debian/Ubuntu-style hosts on the same L2 segment:

Host Real IP (management) Role intent VRRP priority
gw-a 192.0.2.11/24 Preferred master 150
gw-b 192.0.2.12/24 Standby 100
VIP 192.0.2.10/24 Floating service IP —

Clients (or upstream routes) use 192.0.2.10 only. Never point production traffic at the real host IPs if you want transparent failover.

Replace interface names (eth0 / enp1s0 / bond0) with whatever ip -br link shows. If the uplink is a bond, put VRRP on bond0 — bonding and VRRP compose cleanly and cover different failures.

Install and enable 

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y keepalived

# Fedora/RHEL family
# sudo dnf install -y keepalived

sudo systemctl enable keepalived.service
Enter fullscreen mode Exit fullscreen mode

Config path: /etc/keepalived/keepalived.conf

Logs: journalctl -u keepalived -f

Runtime state: ip -br addr, ip -d link, and keepalived’s own logs on state transitions.

Baseline: two-node VIP with intentional preemption

Prefer the healthier, higher-priority node when it is online. Both nodes share the same virtual_router_id and VIP; only priority (and optional tracks) differ.

Host A (gw-a) — higher priority

/etc/keepalived/keepalived.conf:

global_defs {
    router_id gw-a
    # Prefer a non-root script user when you add track/notify scripts later
    script_user keepalived_script
    enable_script_security

    # GARP tuning after becoming MASTER (modern switches rarely need a storm)
    vrrp_garp_master_refresh 60
    vrrp_garp_master_refresh_repeat 1
}

vrrp_instance VI_GATEWAY {
    # Initial state before advertisements settle. Priority still wins the election.
    state BACKUP
    interface eth0
    virtual_router_id 51          # 1-255, unique per VIP set on this LAN
    priority 150                  # higher = preferred master
    advert_int 1                  # seconds (fractional values allowed)

    # VRRPv3 is the RFC 5798 path; IPv6 instances use v3 anyway
    version 3

    # Optional: keep VIP MAC stable via macvlan VMAC (good with picky switches)
    # use_vmac vrrp51
    # vmac_xmit_base

    authentication {
        auth_type PASS
        auth_pass chg-me-8c        # first 8 chars matter; same on all peers
    }

    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }

    # Demote if the uplink itself disappears
    track_interface {
        eth0
    }
}
Enter fullscreen mode Exit fullscreen mode

Host B (gw-b) — lower priority

Same file with:

global_defs {
    router_id gw-b
    script_user keepalived_script
    enable_script_security
    vrrp_garp_master_refresh 60
    vrrp_garp_master_refresh_repeat 1
}

vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    version 3

    authentication {
        auth_type PASS
        auth_pass chg-me-8c
    }

    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }

    track_interface {
        eth0
    }
}
Enter fullscreen mode Exit fullscreen mode

Create the script user if your package did not:

sudo useradd -r -s /usr/sbin/nologin keepalived_script 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Apply:

sudo keepalived -t -l -f /etc/keepalived/keepalived.conf   # config test
sudo systemctl restart keepalived.service
sudo systemctl --no-pager --full status keepalived.service
Enter fullscreen mode Exit fullscreen mode

Verify ownership

On both hosts:

ip -br addr show eth0
ip addr show eth0 | grep 192.0.2.10 || echo "no VIP here"
journalctl -u keepalived -n 50 --no-pager
Enter fullscreen mode Exit fullscreen mode

Expect:

  • Exactly one host shows 192.0.2.10/24 (secondary address)
  • That host’s journal says transition to MASTER
  • The other stays BACKUP
  • From a third machine: ping -c3 192.0.2.10 succeeds

Quick ownership check you can script:

#!/bin/bash
VIP=192.0.2.10
if ip -4 -o addr show | awk '{print $4}' | grep -q "^${VIP}/"; then
  echo "MASTER holds $VIP on $(hostname -s)"
  exit 0
fi
echo "BACKUP without $VIP on $(hostname -s)"
exit 1
Enter fullscreen mode Exit fullscreen mode

Make failover mean “service healthy,” not just “kernel still boots”

A VIP on a host whose reverse proxy is dead is a polished outage. Couple VRRP priority to real health.

Track a process (simple, low overhead) 

vrrp_track_process track_nginx {
    process nginx          # exact match semantics (not a loose pgrep regex)
    # weight omitted => instance goes to FAULT when process is gone after delay
    delay 1
}

vrrp_instance VI_GATEWAY {
    # ...same as before...
    track_process {
        track_nginx
    }
}
Enter fullscreen mode Exit fullscreen mode

Track a script (port/HTTP check) 

vrrp_script chk_https {
    script "/usr/lib/keepalived/check_https.sh"
    interval 2
    timeout 2
    weight -30          # subtract from priority while failing; 0 weight => FAULT
    fall 2              # failures before down
    rise 2              # successes before up
}

vrrp_instance VI_GATEWAY {
    # ...
    track_script {
        chk_https
    }
}
Enter fullscreen mode Exit fullscreen mode

Example check script (mode 0755, owned by root, not world-writable — enable_script_security cares):

/usr/lib/keepalived/check_https.sh:

#!/bin/bash
# Exit 0 = healthy, non-zero = unhealthy
exec curl -fsS --max-time 1 -o /dev/null "http://127.0.0.1:80/healthz"
Enter fullscreen mode Exit fullscreen mode 
sudo install -o root -g root -m 0755 /tmp/check_https.sh /usr/lib/keepalived/check_https.sh
sudo keepalived -t -l -f /etc/keepalived/keepalived.conf
sudo systemctl reload keepalived.service
Enter fullscreen mode Exit fullscreen mode

Weight semantics (from keepalived.conf(5)):

  • weight 0 (default for scripts): monitoring failure drives the instance to FAULT after fall failures
  • Non-zero weight: adjust effective priority up/down so another node can win without a hard FAULT

Use FAULT when the node must not hold the VIP at all. Use negative weight when you want soft preference (“prefer the node with a warm cache, but either can serve”).

nopreempt: stop flapping when the old master returns

By default, a recovering higher-priority node snatches the VIP back. That is correct for “always prefer gw-a,” and painful for long-lived TCP sessions if gw-a reboots every patch night.

nopreempt keeps the current master until it fails:

vrrp_instance VI_GATEWAY {
    state BACKUP        # REQUIRED: nopreempt is ignored if initial state is MASTER
    nopreempt
    priority 150
    # ...
}
Enter fullscreen mode Exit fullscreen mode

Apply on both nodes (with different priorities). Document the tradeoff:

Mode Behavior Good for
Preempt (default) Highest healthy priority always owns VIP Deterministic “primary DC node”
nopreempt Winner stays until it fails Fewer failbacks, calmer TCP

You can also delay preemption with preempt_delay when you want eventual return to the preferred node without an instant bounce.

Unicast peers when multicast is blocked

Cloud security groups, some Wi-Fi AP isolation modes, and locked-down switches break 224.0.0.18. keepalived can speak VRRP over unicast:

vrrp_instance VI_GATEWAY {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    version 3

    unicast_src_ip 192.0.2.11
    unicast_peer {
        192.0.2.12
    }

    virtual_ipaddress {
        192.0.2.10/24
    }
}
Enter fullscreen mode Exit fullscreen mode

On gw-b, swap unicast_src_ip / peer addresses. Man page warning: unicast mode without peers is invalid — configure real peers explicitly.

If you combine use_vmac with unicast, set vmac_xmit_base as documented so advertisements leave the underlying interface correctly.

Firewall notes

Allow VRRP between peers on the VRRP interface:

# nftables sketch — adjust interface and policy to your base table
sudo nft add rule inet filter input iifname "eth0" ip protocol 112 accept
sudo nft add rule inet filter input iifname "eth0" ip daddr 224.0.0.18 accept
# unicast mode: allow protocol 112 between peer unicast addresses instead
Enter fullscreen mode Exit fullscreen mode

Also permit whatever your track scripts probe (localhost health checks usually need nothing extra).

If you use use_vmac or no_accept, modern keepalived prefers nftables helpers to manage its small firewall table — do not randomly flush all nft tables on those hosts.

Controlled failover test

Do this once before you trust the VIP in DNS or upstream static routes.

  1. From a client: ping -i 0.2 192.0.2.10
  2. Confirm VIP on preferred node: ip addr show eth0 | grep 192.0.2.10
  3. Stop keepalived on the master: sudo systemctl stop keepalived (or pull its cable / kill nginx if you track the process)
  4. Watch ping: a small blip is normal; multi-second black holes are not
  5. On the standby: VIP should appear; journal should show MASTER
  6. Start keepalived on the original node again
  7. Confirm preemption vs nopreempt matches your design

Optional packet view (on systems with tcpdump):

sudo tcpdump -ni eth0 'ip proto 112 or host 224.0.0.18'
Enter fullscreen mode Exit fullscreen mode

You should see regular advertisements from the master and a burst of gratuitous ARP when ownership changes (tcpdump -ni eth0 arp).

Operational pitfalls

Duplicate virtual_router_id for unrelated VIPs on the same LAN.

VRIDs must be unique per virtual router on that broadcast domain. Collisions produce split-brain weirdness.

VIP still configured statically in netplan/networkd/NM.

If both keepalived and your network manager permanently own 192.0.2.10, failover becomes a fight. Let keepalived add/remove the address.

Authentication as security theater.

auth_type PASS is cleartext and non-compliant with modern VRRPv2 expectations. Isolate VRRP; do not rely on the password against a hostile L2 neighbor.

Scripts writable by non-root.

With enable_script_security, keepalived refuses unsafe root scripts. Good. Fix ownership/mode instead of disabling the guard.

Initial state MASTER + nopreempt.

The man page is explicit: for nopreempt to work, initial state must not be MASTER. Use BACKUP on all nodes and let priority decide.

Forgetting track failures are sticky without rise.

Tune fall/rise so a single slow health check does not thrash the VIP.

Assuming VRRP replaces backups.

Failover preserves reachability. It does not replicate disk state. Pair with real data replication for anything stateful.

Minimal systemd-friendly health timer (optional)

If you already alert from node exporters, scrape whether the VIP is local:

/etc/systemd/system/check-vip.service:

[Unit]
Description=Check floating VIP presence
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/check-vip.sh
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/system/check-vip.timer:

[Unit]
Description=Periodic VIP ownership check

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
Unit=check-vip.service

[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

Alert on “VIP missing on all nodes” (bad) separately from “VIP not on preferred node” (info under nopreempt).

Quick chooser

  • Two gateways, prefer A, fail to B, fail back to A: priorities 150/100, default preempt, track uplink + service
  • Two gateways, minimize failback flaps: both state BACKUP, nopreempt, track service hard (weight 0)
  • Multicast blocked: unicast_src_ip + unicast_peer { ... }
  • Picky switch MAC learning: consider use_vmac
  • Need L4 load balancing across many real servers: that is IPVS/virtual_server — adjacent keepalived feature, not this VIP recipe
  • Need two NICs on one box: bonding first, then VRRP on bond0

References

Wrap-up

Floating IPs are one of the highest-leverage HA upgrades you can give a pair of Linux hosts. Keep the recipe boring:

  1. Same virtual_router_id and VIP on every peer
  2. Different priorities (and honest health tracks)
  3. Config test with keepalived -t, then restart/reload
  4. Prove ownership with ip addr + a third-party ping
  5. Fail the master on purpose once before DNS points at the VIP

Do that, and a single host outage stops being “the gateway is down” and becomes a short, tested cutover your clients barely notice.

Top comments (0)