Most HAProxy examples for Galera or Percona XtraDB Cluster show a working configuration. Very few explain why each parameter exists, how it interacts with Galera's architecture, or how it affects application failover. This article explores those design decisions.
Introduction
When deploying a Galera-based cluster such as Percona XtraDB Cluster (PXC 8.4), the first instinct is often to copy the HAProxy configuration from the documentation.
The official documentation provides an excellent starting point, but production environments usually raise additional questions:
- Why use
clustercheckinstead of simply checking TCP port 3306? - Should we use
balance first,roundrobin, orleastconn? - What does
backupreally do? - What happens to existing client sessions when a node fails?
- Is
on-marked-down shutdown-sessionsuseful for database workloads? - How should HAProxy timing relate to Galera membership detection?
This article explores the reasoning behind these questions.
Understanding the architecture
Application
│
│
HAProxy + Keepalived
│
┌────────────┼────────────┐
│ │ │
Node-01 Node-02 NOde-03
HAProxy is not responsible for Galera failover.
Galera already maintains a synchronous multi-primary cluster.
HAProxy is responsible for:
- Selecting a healthy backend
- Detecting unhealthy nodes
- Closing dead sessions
- Routing new client connections
Why checking port 3306 is insufficient
A common configuration is
server db01 10.0.0.1:3306 check
This only verifies:
- mysqld is running
- TCP port is accepting connections
It does not verify whether the node is:
- JOINING
- DONOR
- DESYNCED
- DISCONNECTED
Percona therefore recommends using clustercheck with HTTP health checks instead of relying only on TCP port availability. ([Percona Documentation][1])
Example:
option httpchk GET /
http-check expect string Percona\ XtraDB\ Cluster\ Node\ is\ synced
default-server port 9200
Choosing the load-balancing algorithm
Many examples use
balance roundrobin
because Galera is multi-primary.
However, application architecture matters.
Option 1 – roundrobin
Conn1 → Node1
Conn2 → Node2
Conn3 → Node3
Advantages
- Even distribution
- Good for true active-active workloads
Disadvantages
- Higher probability of certification conflicts
- More difficult debugging
- Applications may read/write from different nodes
Option 2 – leastconn
Chooses the server with the fewest active connections.
Useful for stateless workloads.
Less useful when intentionally preferring a single database writer.
Option 3 – balance first
Node1
↓
Node2 (backup)
↓
Node3 (backup)
Advantages
- Predictable routing
- Simplified troubleshooting
- Reduced certification conflicts
- Easier operational model
This is often a good choice when the application itself only exposes one database endpoint and a preferred node is desired.
Understanding backup
Example
server Node01 ... check
server Node02 ... check backup
server Node03 ... check backup
This means:
Normal operation
Application
↓
Node-01
If Node-01 becomes unhealthy
Application
↓
Node-02
The third node remains unused until required.
Existing sessions versus new sessions
One important point:
HAProxy never migrates an existing MySQL session.
Suppose
Application
↓
Node-01
If Node-01 crashes:
- Existing TCP sessions fail.
- Transactions in progress are rolled back by MySQL.
- HAProxy cannot move that transaction to another node.
- New TCP connections are routed to the next healthy backend.
Why on-marked-down shutdown-sessions deserves attention
One option that deserves more discussion is
default-server inter 5s rise 2 fall 3 on-marked-down shutdown-sessions
When a backend is declared DOWN:
Without this option
- Existing TCP sessions may remain open until timeout.
- Applications may spend longer waiting on dead sockets.
With this option
- HAProxy actively closes sessions to the failed backend.
- Applications immediately receive a connection failure.
- Clients capable of reconnecting can establish a fresh connection to another healthy node.
Whether this improves recovery depends on the application's reconnect behavior.
Relationship with Galera failover
One misconception is that HAProxy performs failover.
It does not.
The sequence is:
Node failure
↓
Galera detects membership change
↓
clustercheck reports unhealthy
↓
HAProxy removes backend
↓
Application reconnects
↓
HAProxy routes incoming traffic to another node
This means HAProxy health check timing should complement Galera's state changes rather than attempting to "outrun" them.
Recommended validation
Configuration review alone is not sufficient.
Recommended tests include:
- Graceful MySQL shutdown
- Hard node power-off
- Network partition
- Node recovery
- Long-running transaction interruption
- Application reconnect validation
Measure:
- Galera membership convergence
- clustercheck state change
- HAProxy backend transition
- Application recovery time
A practical baseline
backend myqsl-cluster-write
mode tcp
balance first
option httpchk GET /
http-check expect string Percona\ XtraDB\ Cluster\ Node\ is\ synced
default-server port 9200 inter 5s rise 2 fall 3 on-marked-down shutdown-sessions
server Node01 10.0.0.11:3306 check
server Node02 10.0.0.12:3306 check backup
server Node03 10.0.0.13:3306 check backup
This is not the only valid configuration, but it is a reasonable starting point for applications that use a single database endpoint and prefer one active backend with automatic failover.
Key takeaways
- HAProxy does not perform Galera failover.
-
clustercheckis significantly more informative than checking TCP port 3306 alone. -
balance firstis a valid design choice for applications that intentionally use a preferred database node. -
backupservers remain idle until the preferred node becomes unavailable. - Existing MySQL sessions cannot be migrated; only new connections can be redirected.
-
on-marked-down shutdown-sessionscan reduce recovery time for applications that reconnect automatically, but its impact should be validated in your environment. - HAProxy timing should be tuned based on measured Galera and application behavior rather than arbitrary values.
Top comments (0)