Highly Available HAProxy and keepalived Load Balancers for OpenShift UPI: A Lab Repo

If you install OpenShift with UPI, or agent-based on platform none, you provide the load balancer. Most guides show a single HAProxy VM and stop there. That VM then becomes the one thing that can take down your API, node ignition and every route at once.

I wanted a lab that treats the load balancer as a real HA component. That means two nodes, a floating VIP, proper health checks, and a way to prove failover works instead of assuming it does. The result is ocp-haproxy-ha-lab. It covers prep, load balancers, ingress placement, failover testing and monitoring, in that order.

The topology

The lab targets a cluster with 3 masters and 2 infra nodes. Two RHEL 9 VMs (lb1, lb2) run identical HAProxy configs and share one virtual IP through keepalived (VRRP). DNS for api, api-int and *.apps all points at the VIP, never at a single LB.

Port	Frontend	Backends	Health check
6443	Kubernetes API	3 masters (+ bootstrap at install)	HTTPS GET /readyz, expects 200
22623	Machine Config Server	3 masters (+ bootstrap)	TCP connect
443 / 80	Ingress	2 infra nodes	GET /healthz/ready on router port 1936
9000	Stats page	local	/stats (basic auth), /healthz for keepalived
8405	Prometheus metrics	local	/metrics

Every address in the repo is a placeholder on one lab subnet, and prep/inventory.env is the single list of them. Change it once and the prep scripts follow.

Design choices that matter

Most of the value is in the details, and each one is commented in the config:

• /readyz over TLS for the API, not a TCP check. The kube-apiserver starts failing /readyz before it shuts down, so HAProxy drains a master that is rebooting gracefully. A plain TCP check keeps sending requests to a dying API server.
• Router health on port 1936. A TCP check on 443 stays green when the node is up but the router pod is gone.
• keepalived checks HAProxy, not just the process. check_haproxy.sh curls the local /healthz monitor URI, which catches a hung HAProxy as well as a dead one. With interval 2 and fall 2, the VIP moves in about 4 seconds.
• Unicast VRRP for networks that filter multicast, and periodic gratuitous ARP so switches don’t keep a stale MAC for the VIP.
• TLS passthrough with source affinity on ingress, and 1-hour tunnel timeouts so oc watches and websockets are not cut.
• SELinux stays enforcing. The prep scripts label ports and scripts rather than disabling it.

What is in the repo

Directory	What it does
prep/	Inventory, BIND zone templates for api/api-int/*.apps with A and PTR records, firewalld rules, sysctl (ip_nonlocal_bind), SELinux settings, and DNS and host verification scripts
haproxy/	haproxy.cfg, keepalived configs for lb1 (MASTER) and lb2 (BACKUP), check and notify scripts, offline validation
ingress/	Infra MachineConfigPool, IngressController placement patch, setup, verify and rollback scripts, plus an install-time manifest
failover/	Failover test runbook and scripts, LB metrics, alert rules, and two Prometheus setups

Before copying anything to /etc, you can syntax-check every config from a workstation:

git clone https://github.com/Willey2003/ocp-haproxy-ha-lab.git
cd ocp-haproxy-ha-lab
./haproxy/validate.sh    # haproxy -c plus keepalived -t on both LB configs

The ingress setup script follows a safe order. The infra taint includes NoExecute, so the routers are moved onto the infra nodes and rolled out with tolerations before the taint is applied. Run it with DRY_RUN=true first to preview every command.

Proving failover: ten tests

From a support background, my view is simple: an HA design you haven’t broken on purpose is just a diagram. failover/RUNBOOK.md defines ten tests (T0 to T9). Each one breaks a single thing, measures what a client sees through the VIP, and puts it back:

• T0 baseline, zero failed requests for 60 seconds
• T1 to T3 HAProxy stopped, keepalived stopped, and a hard crash of the VIP holder
• T4 VRRP blocked between the LBs to force split brain
• T5 to T8 master and infra node loss, graceful and hard
• T9 router pod deleted

scripts/failover-test.sh runs each test end to end. It prechecks the cluster and refuses to run on a degraded one, starts a probe pinned to the VIP with curl --resolve, breaks something, restores it, and grades the longest outage per endpoint against configurable limits (10 seconds by default). Results are saved as a raw probe.csv plus a summary.

cp failover/lab.env.example failover/lab.env   # edit for your lab
failover/scripts/failover-test.sh baseline
failover/scripts/failover-test.sh lb-haproxy-stop

The runbook lists the outage I expect from each setting, for example about 4 to 6 seconds for T1 and 0 seconds for a graceful master reboot. These are design targets. The results table in the runbook is a blank template for you to fill in from your own runs, and the repo does not publish measured numbers.

Monitoring and alerts

failover/monitoring/ adds node_exporter and a VRRP-state hook on each LB, then defines alerts including KeepalivedNoMaster, KeepalivedSplitBrain, HAProxyBackendDown, LBServiceNotActive and VIPEndpointDown. A “live MASTER” only counts while keepalived.service is actually active, so a stale state file from a killed keepalived cannot hide a missing VIP holder.

There are two Prometheus setups on purpose. One uses OpenShift user workload monitoring, so LB alerts show up in the console. But it sits behind the VIP it is watching. The other runs Prometheus, Alertmanager and blackbox_exporter in podman on a bastion, probing from outside, so it keeps alerting when the cluster is unreachable. For a lab, one is enough. For production, the repo recommends both.

Limits to know

• There are no dedicated worker nodes, so with infra nodes tainted, application pods land on the masters. Fine for a lab. For real workloads, add workers or set TAINT=false.
• Replace every placeholder before deploying, including the VRRP password and stats credentials, which are marked CHANGE ME.
• It assumes OpenShift 4.x with a user-managed load balancer and DNS you control.

Get it

Repo: github.com/Willey2003/ocp-haproxy-ha-lab

Start with prep/inventory.env, then work through prep/, haproxy/, ingress/ and failover/ in order. If you run the failover tests in your own lab, I would like to see your numbers. Open an issue with your results table.