How to Build a High-Availability (HA) Cluster on Bare Metal

#devops #architecture #linux #database

When deploying mission-critical applications, a Single Point of Failure (SPOF) is a disaster waiting to happen. High Availability (HA) is essential for achieving uptime targets like 99.99%.

In this tutorial, we outline a production-grade, 7-node High-Availability cluster built on bare-metal servers using a Private VLAN for maximum performance, security, and hardware control.

🏗️ 3-Tier Architecture Overview

Traffic flows through three distinct, isolated tiers:

Floating VIP (203.0.113.100): Single public IP entry point.
Tier 1 (Load Balancers - LB-01 & LB-02): Active/Passive HAProxy + Keepalived setup for automated 1–3 second VIP failover.
Tier 2 (Web Tier - WEB-01 & WEB-02): Nginx nodes isolated from public access.
Tier 3 (Database Tier - DB-01, DB-02, DB-03): MariaDB Galera Cluster with synchronous, certification-based replication.

📋 Server Topology & IP Scheme

+----------+-----------------+---------------+--------------------+
| Hostname | Role            | Public IP     | Private IP (VLAN)  |
+----------+-----------------+---------------+--------------------+
| VIP      | Floating IP     | 203.0.113.100 | -                  |
| LB-01    | Load Balancer 1 | 203.0.113.101 | 10.0.0.10          |
| LB-02    | Load Balancer 2 | 203.0.113.102 | 10.0.0.11          |
| WEB-01   | Web Node 1      | -             | 10.0.0.20          |
| WEB-02   | Web Node 2      | -             | 10.0.0.21          |
| DB-01    | DB Node 1       | -             | 10.0.0.30          |
| DB-02    | DB Node 2       | -             | 10.0.0.31          |
| DB-03    | DB Node 3       | -             | 10.0.0.32          |
+----------+-----------------+---------------+--------------------+

🔒 Security & Sysctl Tuning

Allow incoming traffic strictly from your Private VLAN (10.0.0.0/24) using ufw:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh

# Allow internal Web Traffic
sudo ufw allow from 10.0.0.0/24 to any port 80
sudo ufw allow from 10.0.0.0/24 to any port 443

# Allow internal Galera & MySQL Traffic
sudo ufw allow from 10.0.0.0/24 to any port 3306  # MySQL
sudo ufw allow from 10.0.0.0/24 to any port 4444  # Galera SST
sudo ufw allow from 10.0.0.0/24 to any port 4567  # Galera Cluster
sudo ufw allow from 10.0.0.0/24 to any port 4568  # Galera IST

sudo ufw enable

Apply kernel tweaks on all nodes to enable non-local binding and scale network queues:

1. MariaDB Galera Configuration

Edit the Galera configuration file:

/etc/mysql/mariadb.conf.d/60-galera.cnf

[galera]
bind-address = 0.0.0.0

binlog_format = row
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2

wsrep_on = ON
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name = ha_production_cluster
wsrep_cluster_address = "gcomm://10.0.0.30,10.0.0.31,10.0.0.32"

wsrep_sst_method = mariabackup

Kernel Networking (Sysctl)

Create the sysctl configuration file:

/etc/sysctl.d/99-ha-cluster.conf

net.core.somaxconn = 65535
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_max_syn_backlog = 65535

Apply the changes:

sudo sysctl --system

💡 Note

Bootstrap the Galera cluster only on DB-01:
sudo galera_new_cluster
After the cluster is initialized, start MariaDB normally on DB-02 and DB-03:
sudo systemctl start mariadb

2. Keepalived Unicast VRRP

Modern bare-metal servers and many cloud providers block multicast traffic. To ensure reliable VIP failover, configure Unicast VRRP.

Edit the Keepalived configuration:

/etc/keepalived/keepalived.conf

LB-01 (MASTER)

vrrp_instance VI_1 {
    state MASTER
    interface ens18
    virtual_router_id 51
    priority 100

    unicast_src_ip 10.0.0.10

    unicast_peer {
        10.0.0.11
    }

    virtual_ipaddress {
        203.0.113.100/32
    }
}

LB-02 (BACKUP)

vrrp_instance VI_1 {
    state BACKUP
    interface ens18
    virtual_router_id 51
    priority 90

    unicast_src_ip 10.0.0.11

    unicast_peer {
        10.0.0.10
    }

    virtual_ipaddress {
        203.0.113.100/32
    }
}