Your Load Balancer Is a Single Point of Failure. Fix It With VRRP

You built the system the way everyone tells you to.

Three API servers instead of one, so a crash doesn't take you down. A load balancer in front of them spreading traffic around. Your domain points at the load balancer, the load balancer health-checks the backends, and when one server dies the other two pick up the slack. Textbook.

Then at 2 AM the load balancer itself dies.

Not a backend. The balancer. Maybe the box lost power, maybe the kernel panicked, maybe HAProxy hit an out-of-memory kill. It doesn't matter which. Your three API servers are sitting there completely healthy, idle, ready to serve, and every single request is timing out. Because nothing can reach them anymore.

You removed the single point of failure from your application tier and quietly moved it to the thing in front.

Adding a second load balancer doesn't fix it

The instinct is right: run two.

The problem is what clients are actually dialling. Your DNS points api.example.com at 203.0.113.10, and that address belongs to load balancer 1. Load balancer 2 sits there with 203.0.113.11, perfectly healthy, and no traffic at all. Nobody is asking for it.

So you update DNS to point at the second one. Now you wait. TTLs are cached by resolvers, by operating systems, by browsers, and some of them ignore your TTL entirely. You're looking at minutes of downtime at best, and a long tail of clients still hammering the dead address. That's not failover, that's a slow-motion recovery.

What you actually need is for the address to move. Clients keep dialling 203.0.113.10, and a different machine starts answering for it. No DNS change, nothing to wait for.

That's what a virtual IP is, and VRRP is how two machines agree on who's currently answering.

What is a virtual IP address?

A virtual IP (VIP) is an address that no machine owns permanently.

Clients connect to it, DNS points at it, firewall rules reference it. But it can move from one server to another without anything on the client side changing. The address stays fixed while the hardware behind it is replaced, scaled, or fails.

You're already using one whether you call it that or not. api.example.com resolving to 203.0.113.10 with three backends behind it on private addresses is a VIP in front of a pool. Clients never learn a backend address, so you can add or remove servers freely.

Virtual IP address load balancing: the front door

A client opens a TCP connection to 203.0.113.10. The load balancer accepts it and picks a backend, say 10.0.1.6. How the packet gets there depends on the mode:

• Proxy mode (ALB, nginx, HAProxy): the balancer terminates the client connection and opens a separate one to the backend. Two TCP connections. The backend sees the balancer's IP as the source, which is why X-Forwarded-For exists.
• NAT or direct server return (LVS, classic L4 balancers): the balancer rewrites the destination and forwards the packet. The backend replies straight to the client. Faster, but the backend has to be configured to accept traffic addressed to the VIP.

Either way, the client only ever talks to the VIP. Now we need that VIP to survive the balancer holding it.

What is VRRP?

VRRP stands for Virtual Router Redundancy Protocol. Defined in RFC 5798, it's the protocol that lets several machines agree on who currently owns a virtual IP, and elect a replacement when that owner disappears.

It was designed for redundant routers, but it's used for anything that needs a floating address: load balancers, firewalls, database proxies, NFS heads.

How does VRRP work?

Two or more nodes are configured with the same virtual IP and a shared group ID (the VRID). Each node gets a priority number.

Election. The highest-priority node becomes the master. It is the only node that claims the VIP and answers for it. Everyone else is a backup and stays completely silent. Two machines answering for one address would be a disaster, so silence is the default state.

Heartbeat. The master multicasts small advertisement packets to 224.0.0.18, by default once per second. This is the "I'm alive" signal. VRRP runs directly over IP as protocol number 112, not TCP or UDP, so there's no port to open — your firewall rule has to allow the protocol itself.

Failover. Backups listen for those advertisements. If roughly three intervals pass with silence, they conclude the master is gone, hold an election by priority, and the winner claims the VIP. Typical failover lands in the 1–3 second range.

VRRP priority: higher or lower?

Higher wins. The range is 1–254, the default is 100, and 255 is reserved for the node that owns the IP as a real interface address. Give your intended master 100 and the standby 90.

Preempt controls what happens when a recovered node with higher priority comes back. With preemption on (the default), it takes the VIP back immediately. Turning it off is often wiser: it avoids a second disruption, and it stops a flapping node from bouncing traffic back and forth every time it recovers.

The VRRP MAC address

VRRP doesn't just move an IP. The group also gets a virtual MAC address, formatted 00:00:5E:00:01:XX, where XX is the VRID.

This matters more than it sounds. Because the MAC moves with the IP, some switches don't need to relearn anything at all during a failover — the address pair is identical, only the physical port changes.

Where ARP fits in

VRRP decides who owns the VIP. ARP is how the rest of the network finds out.

ARP (Address Resolution Protocol) is the lookup that maps an IP address to a hardware MAC address on the local network. When a machine wants to send to 203.0.113.10, it broadcasts "who has this address?" and caches the answer for a few minutes.

That cache is the problem during failover. Neighbours are still holding a mapping that points at a dead machine, and waiting for it to expire would mean minutes of downtime.

So the new master sends a gratuitous ARP: an unsolicited broadcast announcing "203.0.113.10 is at my MAC." Nobody asked. Every device on the segment overwrites its cached entry immediately, and switches update their MAC tables. That single broadcast is what makes VIP failover take seconds instead of minutes.

This is also why VIP failover beats DNS failover. No TTL to wait out, no client-side DNS caching to fight.

Setting it up with keepalived

On Linux, keepalived is the standard VRRP implementation. A minimal config:

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51      # same VRID on both nodes
    priority 100              # 90 on the standby
    advert_int 1              # heartbeat interval, seconds
    authentication {
        auth_type PASS
        auth_pass changeme
    }
    virtual_ipaddress {
        203.0.113.10
    }
}

That config only fails over when the whole box dies. If HAProxy crashes but the server stays up, the master keeps the VIP and serves nothing. Add a health check so the node drops its own priority when the service is unhealthy:

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight -20               # drops priority below the standby
}

Then reference it with track_script { check_haproxy } inside the instance block.

Split-brain: the failure mode to plan for

If the heartbeat path breaks but both nodes are alive and healthy, each concludes the other is dead. Both claim the VIP. The ARP table flaps between two MACs, and traffic splits unpredictably between two machines that both think they're in charge.

Mitigations: run the heartbeat over a dedicated link or more than one path, use VRRP authentication so stray packets can't interfere, and add an external check so a node can tell "the peer is down" apart from "I am isolated."

VRRP vs HSRP

Both solve the same problem. The differences that matter:

	VRRP	HSRP
Standard	Open, RFC 5798	Cisco proprietary
Terminology	Master / backup	Active / standby
Default timer	1 second	3 seconds
Virtual IP	Can be a real interface IP	Must be separate
Multicast	224.0.0.18	224.0.0.2

In a mixed-vendor environment, VRRP is the only real option. In an all-Cisco shop it's mostly preference, though VRRP's faster default timers give it an edge on failover speed.

What this looks like in the cloud

If you're on AWS or GCP, you probably won't configure VRRP by hand. An ALB or NLB hands you a DNS name and runs multiple nodes across availability zones, handling all of this internally. That's why the docs insist you point at the DNS name and never hardcode the resolved IP.

Kubernetes does the same trick at a different layer: a ClusterIP Service is a virtual address that exists only in iptables or IPVS rules, with kube-proxy rewriting the destination to a real pod IP.

VRRP still shows up in self-managed clusters, on-prem load balancers, bare-metal Kubernetes with kube-vip or MetalLB, and anywhere you run HAProxy or nginx yourself.

One thing to handle in your application

Failover is fast, but it isn't transparent. Every TCP connection through the old master is gone, including your database connection pools.

Your app will hold sockets that look open and are actually dead. So validate connections on checkout, set aggressive TCP keepalives, and retry idempotent operations once. And keep session state in Redis rather than in the load balancer's memory, because sticky sessions held on the failed node don't come back.

The short version

A virtual IP is an address that isn't tied to a machine. VRRP is how machines agree on who currently holds it. Gratuitous ARP is how the rest of the network finds out within seconds.

Between them, the address your clients depend on outlives any single server that ever answers for it.