Hub-and-Spoke Kubernetes: Building a Multi-Cluster RKE2 Platform with HAProxy, Keepalived & Centralized Operations

Syed Muhammad Jamal Uddin Gilani — Wed, 02 Sep 2026 07:05:10 +0000

The Hub-and-Spoke Model

Instead of duplicating monitoring stacks, container registries, secret managers, and GitOps controllers in every cluster, we run them once in a central hub cluster and connect all spoke clusters back to it.

Why Hub-and-Spoke?

Without it: Every cluster runs its own Prometheus, Grafana, Loki, ArgoCD, Harbor, and Vault. That's 4× the infrastructure, 4× the maintenance, 4× the dashboards to check, and zero correlation between clusters.

With it:

Centralized monitoring — One Grafana, one set of dashboards. Spoke clusters ship metrics (via remote-write Prometheus) and logs (via Promtail → central Loki) to the hub. You see all clusters in one place.
Single container registry — Harbor runs in the hub. All spoke clusters pull images from one place. One vulnerability scan covers everything.
Unified secrets management — Vault in the hub, with agents or sidecar injectors in each spoke. Rotate a secret once, it propagates everywhere.
GitOps from one control plane — ArgoCD in the hub manages deployments across all spoke clusters. One PR, one review, deploys to the right cluster.

Cluster Inventory

Cluster	Role	Masters	Workers	Shared Services
Hub (Internal Tools)	Central operations	3 × (8C/16G/300G)	1	Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault
Spoke A	Application workloads	3 × (8C/16G/300G)	8	Promtail, Prometheus (remote-write), Vault agent
Spoke B	Application workloads	3 × (8C/16G/300G)	8	Same as Spoke A
Spoke C	Application workloads	3 × (8C/16G/300G)	8	Same as Spoke A

Each cluster also has 2 HAProxy nodes (1C/2G/30G each) for load balancing.

Architecture of a Single Cluster

Every cluster — hub or spoke — follows the same HA skeleton: a floating VIP in front of two HAProxy nodes, three masters running the control plane, and a data-plane tier of workers. The two diagrams below are the actual topology diagrams I drew while planning this, with every project/cluster name scrubbed out.

Hub Cluster Layout

The hub only needs to run shared tooling for one "tenant" (itself), so it gets a single worker node. Masters pick up the slack, once the NoSchedule taint is lifted (more on that later).

Spoke Cluster Layout

Spokes carry real application traffic, so they get more workers and a MetalLB address pool for LoadBalancer-type Services — something the hub doesn't need since its own services are exposed through Ingress. That extra range pushes the LB tier and masters a bit further up the subnet compared to the hub.

There are 3 spoke clusters in this platform, and all three are byte-for-byte identical in topology — they only differ in the subnet octet (192.168.X.0/24) and hostnames. I'm showing one generic template instead of three near-duplicate diagrams:

Spoke Subnet (anonymized) Masters Workers

Spoke A 192.168.11.0/24 3 8

Spoke B 192.168.12.0/24 3 8

Spoke C 192.168.13.0/24 3 8

Spoke	Subnet (anonymized)	Masters	Workers
Spoke A	`192.168.11.0/24`	3	8
Spoke B	`192.168.12.0/24`	3	8
Spoke C	`192.168.13.0/24`	3	8

How HA works:

Master failure: If one master dies, the other two still have etcd quorum. The API server stays up.
Load balancer failure: If haproxy-01 dies, Keepalived moves the VIP to haproxy-02 within ~4 seconds.
kubectl never breaks: It always points at the VIP, which always points at a healthy load balancer, which always routes to a healthy master.

IP Allocation Scheme

The hub and the spokes use slightly different offsets, because spokes reserve a block for MetalLB and the hub doesn't. Only the subnet octet (x / X) changes between clusters of the same type.

Hub cluster (192.168.x.0/24):

Range	Purpose
`.1 – .39`	Network infrastructure (reserved)
`.40 – .49`	HAProxy load balancers
`.50 – .59`	Master nodes
`.60 – .99`	Worker nodes
`.100`	Floating VIP

Node	IP	Specs
haproxy-01	`.40`	1C / 2G / 30G
haproxy-02	`.41`	1C / 2G / 30G
master-01	`.50`	8C / 16G / 300G
master-02	`.51`	8C / 16G / 300G
master-03	`.52`	8C / 16G / 300G
worker-01	`.60`	As needed
VIP	`.100`	Floating

Spoke cluster (192.168.X.0/24, X = per-spoke octet):

Range	Purpose
`.1 – .50`	Network infrastructure (reserved)
`.51 – .59`	MetalLB address pool
`.60 – .69`	HAProxy load balancers
`.70 – .79`	Master nodes
`.80 – .99`	Worker nodes
`.100`	Floating VIP

Node	IP	Specs
haproxy-01	`.60`	1C / 2G / 30G
haproxy-02	`.61`	1C / 2G / 30G
master-01	`.70`	8C / 16G / 300G
master-02	`.71`	8C / 16G / 300G
master-03	`.72`	8C / 16G / 300G
worker-01 … worker-08	`.80` – `.87`	As needed
VIP	`.100`	Floating

The hub cluster uses only worker-01 (.60). Spoke clusters use workers 01–08 (.80–.87) plus the MetalLB pool for LoadBalancer services.

Prerequisites

For each cluster, generate a unique token:

openssl rand -hex 32

Each cluster gets its own token. Don't reuse tokens across clusters.

Phase 1: Preparing the Nodes

Run on: Every master and worker node (NOT the haproxy nodes).

1.1 — System Update

sudo apt update && sudo apt dist-upgrade -y

1.2 — Host Resolution

sudo tee -a /etc/hosts << 'EOF'
# RKE2 Cluster Nodes
192.168.x.50   master-01
192.168.x.51   master-02
192.168.x.52   master-03
192.168.x.60   worker-01
# Add remaining workers as needed
192.168.x.40   haproxy-01
192.168.x.41   haproxy-02
EOF

1.3 — Disable Swap

sudo swapoff -a
sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab

kubelet won't start with swap enabled. Kubernetes does its own memory management — swap adds unpredictable latency.

1.4 — Kernel Modules

sudo tee /etc/modules-load.d/k8s.conf << 'EOF'
br_netfilter
overlay
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

overlay is required by containerd. br_netfilter makes bridged traffic visible to iptables for pod networking.

1.5 — Network Parameters

sudo tee /etc/sysctl.d/99-rke2.conf << 'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

Without ip_forward, packets between pods on different nodes stop at the node boundary.

1.6 — Disable UFW

sudo systemctl stop ufw
sudo systemctl disable ufw

1.7 — Required Packages

sudo apt install -y curl wget tar nfs-common open-iscsi lsscsi sg3-utils

Package	Purpose
`nfs-common`	Mount NFS persistent volumes
`open-iscsi`, `lsscsi`, `sg3-utils`	Block storage (Longhorn)

1.8 — Enable iSCSI

sudo systemctl enable iscsid
sudo systemctl start iscsid

1.9 — Reboot

sudo reboot

Verify after reboot:

lsmod | grep br_netfilter        # ✓ loaded
sysctl net.ipv4.ip_forward       # ✓ = 1
free -h                          # ✓ swap = 0
systemctl is-active iscsid       # ✓ active

✅ Complete on ALL master + worker nodes before Phase 2.

Phase 2: HAProxy & Keepalived

Run on: haproxy-01 (.40) first, then haproxy-02 (.41).

2.1 — Install

sudo apt update
sudo apt install -y haproxy keepalived curl

2.2 — Sysctl Tuning

sudo tee /etc/sysctl.d/99-lb-tuning.conf << 'EOF'
net.ipv4.ip_nonlocal_bind = 1
net.core.somaxconn = 65535
EOF

sudo sysctl --system

ip_nonlocal_bind lets HAProxy bind to the VIP before Keepalived assigns it. Without this, HAProxy crashes on startup when the VIP is on the other node.

2.3 — File Descriptor Limits

sudo mkdir -p /etc/systemd/system/haproxy.service.d

sudo tee /etc/systemd/system/haproxy.service.d/limits.conf << 'EOF'
[Service]
LimitNOFILE=100000
EOF

sudo systemctl daemon-reload

2.4 — HAProxy Configuration

Find your interface:

ip -4 addr show | grep "192.168.x.40"

Deploy the config:

sudo tee /etc/haproxy/haproxy.cfg << 'EOF'
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    maxconn 30000

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    option  tcpka
    timeout connect 5s
    timeout client  60s
    timeout server  60s
    retries 3

listen stats
    bind 127.0.0.1:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s

frontend rke2_registration_9345
    bind 192.168.x.100:9345
    maxconn 2500
    default_backend rke2_servers_9345

backend rke2_servers_9345
    balance roundrobin
    option tcp-check
    default-server inter 2s fall 2 rise 3 slowstart 10s maxconn 1000
    server master1 192.168.x.50:9345 check
    server master2 192.168.x.51:9345 check
    server master3 192.168.x.52:9345 check

frontend kubernetes_api_6443
    bind 192.168.x.100:6443
    maxconn 26000
    default_backend kube_apiservers_6443

backend kube_apiservers_6443
    balance roundrobin
    option tcp-check
    default-server inter 2s fall 2 rise 3 slowstart 15s maxconn 10000
    server master1 192.168.x.50:6443 check
    server master2 192.168.x.51:6443 check
    server master3 192.168.x.52:6443 check
EOF

Two frontends bind to the VIP: port 9345 (node registration) and port 6443 (Kubernetes API). Both round-robin across the 3 masters with health checks every 2 seconds.

2.5 — Start HAProxy

sudo systemctl enable haproxy
sudo systemctl start haproxy

2.6 — Health Check Script

sudo tee /etc/keepalived/check_haproxy.sh << 'SCRIPT'
#!/bin/bash
systemctl is-active --quiet haproxy || exit 1
curl -sf http://127.0.0.1:8404/stats >/dev/null || exit 1
exit 0
SCRIPT

sudo chmod 755 /etc/keepalived/check_haproxy.sh

Keepalived runs this every 2 seconds. Non-zero exit = HAProxy is dead = VIP moves to backup.

2.7 — Keepalived Configuration

On haproxy-01 (PRIMARY):

INTERFACE="eth0"   # Replace with your actual interface

sudo tee /etc/keepalived/keepalived.conf << EOF
global_defs {
    router_id HAPROXY_01
    enable_script_security
    script_user root
}

vrrp_script chk_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    fall 2
    rise 2
    timeout 2
}

vrrp_instance VI_RKE2 {
    state MASTER
    interface ${INTERFACE}
    virtual_router_id 51
    priority 120
    advert_int 1

    unicast_src_ip 192.168.x.40
    unicast_peer {
        192.168.x.41
    }

    authentication {
        auth_type PASS
        auth_pass <your-vrrp-password>
    }

    virtual_ipaddress {
        192.168.x.100/24
    }

    track_script {
        chk_haproxy
    }
}
EOF

How VRRP failover works: Both nodes send heartbeats. The higher-priority node (120) holds the VIP. If the primary stops advertising — crash, network failure, failed health check — the backup (priority 110) takes ownership of the VIP by adding it to its own interface. Recovery is automatic when the primary comes back.

Multi-cluster note: Each cluster needs a unique virtual_router_id. If all clusters share the same L2 network, use 51 for the hub, 52 for Spoke A, 53 for Spoke B, 54 for Spoke C.

2.8 — Start Keepalived

sudo systemctl enable keepalived
sudo systemctl start keepalived

2.9 — Repeat on haproxy-02

Same steps, but in the Keepalived config:

router_id HAPROXY_02
state BACKUP
priority 110
unicast_src_ip 192.168.x.41
unicast_peer {
    192.168.x.40
}

2.10 — Verify

sudo systemctl is-active haproxy
sudo systemctl is-active keepalived
curl -s http://127.0.0.1:8404/stats | head -5
ip addr show | grep 192.168.x.100   # VIP visible on primary
ping -c 2 192.168.x.100             # Reachable from the network

✅ Load balancer tier ready.

Phase 3: Bootstrapping RKE2

Strictly sequential. master-01 must be fully running before touching master-02.

3.1 — First Master (.50)

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="server" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

First Master Config

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
token: "<your-cluster-token>"

tls-san:
  - "192.168.x.50"
  - "192.168.x.51"
  - "192.168.x.52"
  - "192.168.x.100"
cni: cilium
disable-cloud-controller: true

disable: rke2-ingress-nginx

etcd-expose-metrics: true
etcd-snapshot-schedule-cron: "0 2 * * *"
etcd-snapshot-retention: 7
etcd-snapshot-compress: true
etcd-s3: true
etcd-s3-config-secret: rke2-etcd-snapshot-s3-config

etcd-arg:
  - "heartbeat-interval=200"
  - "election-timeout=2000"

kube-apiserver-arg:
  - "audit-log-maxage=30"
  - "audit-log-maxbackup=10"
  - "audit-log-maxsize=100"
  - "audit-log-mode=blocking"
  - "request-timeout=120s"
  - "default-not-ready-toleration-seconds=120"
  - "default-unreachable-toleration-seconds=120"
  - "event-ttl=2h"

kube-controller-manager-arg:
  - "terminated-pod-gc-threshold=200"
  - "bind-address=0.0.0.0"

kube-scheduler-arg:
  - "bind-address=0.0.0.0"

kube-proxy-arg:
  - "metrics-bind-address=0.0.0.0"

node-taint:
  - "node-role.kubernetes.io/control-plane:NoSchedule"
EOF

Key settings explained:

Setting	Why
`token`	Shared secret for cluster join authentication
`tls-san`	All master IPs + VIP in the API server's TLS cert
`disable: rke2-ingress-nginx`	Install your own ingress controller instead
`etcd-snapshot-*`	Automated daily backups, compressed, keep 7
`node-taint: NoSchedule`	Keep workloads off masters
`bind-address=0.0.0.0`	Expose metrics for Prometheus scraping

CNI is left as default (Canal). Add cni: calico or cni: cilium based on your needs.

Start and Wait

sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service

# Watch logs (takes 2-5 min)
sudo journalctl -u rke2-server -f

# Wait for registration port
while ! sudo ss -tlnp | grep -q ':9345'; do
  echo "Waiting for port 9345..."
  sleep 10
done
echo "Ready for additional masters"

Verify and Set Up kubectl

sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes

# Set up shortcuts
cat >> ~/.bashrc << 'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
EOF

mkdir -p ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
chmod 600 ~/.kube/config
source ~/.bashrc

3.2 — Additional Masters (.51, .52)

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="server" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

The config is identical to the first master plus one line at the top:

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"

tls-san:
  - "192.168.x.50"
  - "192.168.x.51"
  - "192.168.x.52"
  - "192.168.x.100"

# ... rest identical to first master config ...

node-taint:
  - "node-role.kubernetes.io/control-plane:NoSchedule"
EOF

The server: line is the only difference — it tells this node to join the existing cluster through the VIP instead of bootstrapping a new one.

sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
sudo journalctl -u rke2-server -f

Wait for Ready, then repeat for master-03. After all three:

NAME        STATUS   ROLES                       AGE
master-01   Ready    control-plane,etcd,master   10m
master-02   Ready    control-plane,etcd,master   5m
master-03   Ready    control-plane,etcd,master   2m

3.3 — Worker Nodes

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="agent" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

Note: agent not server.

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"

disable:
  - rke2-ingress-nginx

kube-proxy-arg:
  - "metrics-bind-address=0.0.0.0"
EOF

sudo systemctl enable rke2-agent.service
sudo systemctl start rke2-agent.service

Repeat for all workers. Spoke clusters get 8 workers (.60–.67), the hub gets 1 worker (.60).

Connecting the Spokes to the Hub

Once all clusters are running, the hub-and-spoke connectivity looks like this:

How Each Service Connects

Prometheus (Metrics):
Each spoke cluster runs its own Prometheus instance, but instead of storing metrics locally, it uses remote_write to push them to the hub's central Prometheus. The hub's Grafana queries the central Prometheus for dashboards across all clusters.

Loki (Logs):
Each spoke cluster runs Promtail as a DaemonSet. Promtail ships container logs to the hub's Loki endpoint. In Grafana, you can filter logs by cluster, namespace, and pod — all from one place.

ArgoCD (GitOps):
ArgoCD runs in the hub and registers each spoke cluster as a target. Application manifests live in Git. A PR merge triggers ArgoCD to sync the changes to the correct spoke cluster. One ArgoCD instance, one Git source of truth, deploys everywhere.

Harbor (Container Registry):
Harbor runs in the hub cluster. All spoke clusters have their containerd configured to pull images from the hub's Harbor instance. One scan, one set of vulnerability reports, one place to manage image lifecycle.

Vault (Secrets):
HashiCorp Vault runs in the hub. Spoke clusters run the Vault Agent Injector, which authenticates using Kubernetes service accounts and injects secrets into pods at runtime. Rotate a database password in Vault once — all clusters get it.

Hub Cluster: Special Considerations

The hub cluster has only 1 worker node, but it runs critical shared services. A few things to consider:

Removing the NoSchedule Taint

With only 1 worker, you might need the masters to share the workload:

# Allow pods on masters (only for the hub cluster)
kubectl taint nodes master-01 \
  node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-02 \
  node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-03 \
  node-role.kubernetes.io/control-plane:NoSchedule-

The 3 masters have 24 CPUs and 48 GB RAM combined — plenty for Grafana, ArgoCD, Harbor, and Vault alongside the control plane.

Storage Planning

Hub services are storage-heavy:

Service	Storage Needs
Prometheus	Time-series data from all clusters. Plan 50–100 GB+
Loki	Log chunks from all clusters. Plan 100 GB+ (use object storage)
Harbor	Container images. Plan 200 GB+
Vault	Minimal (secrets are small), but HA storage backend needed

This is why the masters have 300 GB each. Consider using Longhorn or a dedicated NFS server for persistent volumes.

Execution Cheat Sheet

FOR EACH CLUSTER (hub first, then spokes):

PHASE 1 — All master + worker nodes (parallel):
  ├── apt update & upgrade
  ├── /etc/hosts
  ├── disable swap
  ├── kernel modules (overlay, br_netfilter)
  ├── sysctl (iptables bridge, ip_forward)
  ├── disable UFW
  ├── install packages
  ├── enable iscsid
  └── reboot

PHASE 2 — HAProxy nodes (primary first):
  ├── install haproxy + keepalived
  ├── sysctl (nonlocal_bind, somaxconn)
  ├── systemd NOFILE limit
  ├── haproxy.cfg
  ├── start haproxy
  ├── check_haproxy.sh
  ├── keepalived.conf
  ├── start keepalived
  └── verify VIP

PHASE 3 — RKE2 (sequential):
  ├── master-01: first-master config, start, wait for :9345
  ├── master-02: additional-master config (server: line), start
  ├── master-03: same
  └── workers: agent install, start

POST-CLUSTER:
  Hub:  Install Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault
  Spokes: Install Promtail, Prometheus (remote-write), Vault agent
          Register as ArgoCD target
          Configure containerd to pull from Hub's Harbor

Troubleshooting

Master won't start:

sudo journalctl -u rke2-server -f --no-pager | tail -50

Node stuck in NotReady:

kubectl describe node <node-name>

Connection refused on port 9345:

sudo ss -tlnp | grep 9345
curl -k https://192.168.x.100:9345

VIP not failing over:

sudo journalctl -u keepalived -f
sudo tcpdump -i eth0 vrrp

Spoke metrics not appearing in Hub Grafana:
Check the spoke's Prometheus remote-write config — the endpoint should point to the hub's Prometheus push endpoint, and network policies should allow cross-cluster traffic.

Spoke logs missing in Hub Loki:
Check Promtail's config on the spoke — clients.url should point to http://<hub-loki-gateway>:3100/loki/api/v1/push. Verify the hub's Loki ingester isn't rejecting streams (check for "rate limit" or "stream limit" errors).

Ports Reference

Port	Protocol	Purpose
6443	TCP	Kubernetes API server
9345	TCP	RKE2 node registration
2379–2380	TCP	etcd client/peer
8472	UDP	VXLAN overlay (Canal/Flannel)
10250	TCP	kubelet API
8404	TCP	HAProxy stats (localhost)
112	VRRP	Keepalived heartbeat

What I Learned

Building clusters manually takes 10× longer than running Ansible. But now when something breaks at 2 AM — and in a hub-and-spoke setup, the hub going down means all clusters lose their operational visibility — I know exactly which component failed and why.

The hub-and-spoke model is worth the added complexity. One Grafana to check instead of four. One ArgoCD to manage instead of four. One Harbor to scan instead of four. The operational overhead drops dramatically once the initial setup is done.

If you're managing more than two clusters, centralize your tooling. Your future self — the one debugging a production incident across three clusters at midnight — will thank you.

This article is based on building a hub-and-spoke platform with 4 production RKE2 clusters. All project names, secrets, credentials, and identifying details have been anonymized.

Questions? Hit the comments — I've probably run into the same issue.

DEV Community: Syed Muhammad Jamal Uddin Gilani