The Hub-and-Spoke Model
Instead of duplicating monitoring stacks, container registries, secret managers, and GitOps controllers in every cluster, we run them once in a central hub cluster and connect all spoke clusters back to it.
Why Hub-and-Spoke?
Without it: Every cluster runs its own Prometheus, Grafana, Loki, ArgoCD, Harbor, and Vault. That's 4× the infrastructure, 4× the maintenance, 4× the dashboards to check, and zero correlation between clusters.
With it:
- Centralized monitoring — One Grafana, one set of dashboards. Spoke clusters ship metrics (via remote-write Prometheus) and logs (via Promtail → central Loki) to the hub. You see all clusters in one place.
- Single container registry — Harbor runs in the hub. All spoke clusters pull images from one place. One vulnerability scan covers everything.
- Unified secrets management — Vault in the hub, with agents or sidecar injectors in each spoke. Rotate a secret once, it propagates everywhere.
- GitOps from one control plane — ArgoCD in the hub manages deployments across all spoke clusters. One PR, one review, deploys to the right cluster.
Cluster Inventory
| Cluster | Role | Masters | Workers | Shared Services |
|---|---|---|---|---|
| Hub (Internal Tools) | Central operations | 3 × (8C/16G/300G) | 1 | Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault |
| Spoke A | Application workloads | 3 × (8C/16G/300G) | 8 | Promtail, Prometheus (remote-write), Vault agent |
| Spoke B | Application workloads | 3 × (8C/16G/300G) | 8 | Same as Spoke A |
| Spoke C | Application workloads | 3 × (8C/16G/300G) | 8 | Same as Spoke A |
Each cluster also has 2 HAProxy nodes (1C/2G/30G each) for load balancing.
Architecture of a Single Cluster
Every cluster — hub or spoke — follows the same HA skeleton: a floating VIP in front of two HAProxy nodes, three masters running the control plane, and a data-plane tier of workers. The two diagrams below are the actual topology diagrams I drew while planning this, with every project/cluster name scrubbed out.
Hub Cluster Layout
The hub only needs to run shared tooling for one "tenant" (itself), so it gets a single worker node. Masters pick up the slack, once the NoSchedule taint is lifted (more on that later).
Spoke Cluster Layout
Spokes carry real application traffic, so they get more workers and a MetalLB address pool for LoadBalancer-type Services — something the hub doesn't need since its own services are exposed through Ingress. That extra range pushes the LB tier and masters a bit further up the subnet compared to the hub.
There are 3 spoke clusters in this platform, and all three are byte-for-byte identical in topology — they only differ in the subnet octet (
192.168.X.0/24) and hostnames. I'm showing one generic template instead of three near-duplicate diagrams:
Spoke Subnet (anonymized) Masters Workers Spoke A 192.168.11.0/243 8 Spoke B 192.168.12.0/243 8 Spoke C 192.168.13.0/243 8
How HA works:
- Master failure: If one master dies, the other two still have etcd quorum. The API server stays up.
- Load balancer failure: If haproxy-01 dies, Keepalived moves the VIP to haproxy-02 within ~4 seconds.
- kubectl never breaks: It always points at the VIP, which always points at a healthy load balancer, which always routes to a healthy master.
IP Allocation Scheme
The hub and the spokes use slightly different offsets, because spokes reserve a block for MetalLB and the hub doesn't. Only the subnet octet (x / X) changes between clusters of the same type.
Hub cluster (192.168.x.0/24):
| Range | Purpose |
|---|---|
.1 – .39 |
Network infrastructure (reserved) |
.40 – .49 |
HAProxy load balancers |
.50 – .59 |
Master nodes |
.60 – .99 |
Worker nodes |
.100 |
Floating VIP |
| Node | IP | Specs |
|---|---|---|
| haproxy-01 | .40 |
1C / 2G / 30G |
| haproxy-02 | .41 |
1C / 2G / 30G |
| master-01 | .50 |
8C / 16G / 300G |
| master-02 | .51 |
8C / 16G / 300G |
| master-03 | .52 |
8C / 16G / 300G |
| worker-01 | .60 |
As needed |
| VIP | .100 |
Floating |
Spoke cluster (192.168.X.0/24, X = per-spoke octet):
| Range | Purpose |
|---|---|
.1 – .50 |
Network infrastructure (reserved) |
.51 – .59 |
MetalLB address pool |
.60 – .69 |
HAProxy load balancers |
.70 – .79 |
Master nodes |
.80 – .99 |
Worker nodes |
.100 |
Floating VIP |
| Node | IP | Specs |
|---|---|---|
| haproxy-01 | .60 |
1C / 2G / 30G |
| haproxy-02 | .61 |
1C / 2G / 30G |
| master-01 | .70 |
8C / 16G / 300G |
| master-02 | .71 |
8C / 16G / 300G |
| master-03 | .72 |
8C / 16G / 300G |
| worker-01 … worker-08 |
.80 – .87
|
As needed |
| VIP | .100 |
Floating |
The hub cluster uses only worker-01 (
.60). Spoke clusters use workers 01–08 (.80–.87) plus the MetalLB pool forLoadBalancerservices.
Prerequisites
For each cluster, generate a unique token:
openssl rand -hex 32
Each cluster gets its own token. Don't reuse tokens across clusters.
Phase 1: Preparing the Nodes
Run on: Every master and worker node (NOT the haproxy nodes).
1.1 — System Update
sudo apt update && sudo apt dist-upgrade -y
1.2 — Host Resolution
sudo tee -a /etc/hosts << 'EOF'
# RKE2 Cluster Nodes
192.168.x.50 master-01
192.168.x.51 master-02
192.168.x.52 master-03
192.168.x.60 worker-01
# Add remaining workers as needed
192.168.x.40 haproxy-01
192.168.x.41 haproxy-02
EOF
1.3 — Disable Swap
sudo swapoff -a
sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab
kubelet won't start with swap enabled. Kubernetes does its own memory management — swap adds unpredictable latency.
1.4 — Kernel Modules
sudo tee /etc/modules-load.d/k8s.conf << 'EOF'
br_netfilter
overlay
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
overlay is required by containerd. br_netfilter makes bridged traffic visible to iptables for pod networking.
1.5 — Network Parameters
sudo tee /etc/sysctl.d/99-rke2.conf << 'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
Without ip_forward, packets between pods on different nodes stop at the node boundary.
1.6 — Disable UFW
sudo systemctl stop ufw
sudo systemctl disable ufw
1.7 — Required Packages
sudo apt install -y curl wget tar nfs-common open-iscsi lsscsi sg3-utils
| Package | Purpose |
|---|---|
nfs-common |
Mount NFS persistent volumes |
open-iscsi, lsscsi, sg3-utils
|
Block storage (Longhorn) |
1.8 — Enable iSCSI
sudo systemctl enable iscsid
sudo systemctl start iscsid
1.9 — Reboot
sudo reboot
Verify after reboot:
lsmod | grep br_netfilter # ✓ loaded
sysctl net.ipv4.ip_forward # ✓ = 1
free -h # ✓ swap = 0
systemctl is-active iscsid # ✓ active
✅ Complete on ALL master + worker nodes before Phase 2.
Phase 2: HAProxy & Keepalived
Run on: haproxy-01 (.40) first, then haproxy-02 (.41).
2.1 — Install
sudo apt update
sudo apt install -y haproxy keepalived curl
2.2 — Sysctl Tuning
sudo tee /etc/sysctl.d/99-lb-tuning.conf << 'EOF'
net.ipv4.ip_nonlocal_bind = 1
net.core.somaxconn = 65535
EOF
sudo sysctl --system
ip_nonlocal_bind lets HAProxy bind to the VIP before Keepalived assigns it. Without this, HAProxy crashes on startup when the VIP is on the other node.
2.3 — File Descriptor Limits
sudo mkdir -p /etc/systemd/system/haproxy.service.d
sudo tee /etc/systemd/system/haproxy.service.d/limits.conf << 'EOF'
[Service]
LimitNOFILE=100000
EOF
sudo systemctl daemon-reload
2.4 — HAProxy Configuration
Find your interface:
ip -4 addr show | grep "192.168.x.40"
Deploy the config:
sudo tee /etc/haproxy/haproxy.cfg << 'EOF'
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
maxconn 30000
defaults
log global
mode tcp
option tcplog
option dontlognull
option tcpka
timeout connect 5s
timeout client 60s
timeout server 60s
retries 3
listen stats
bind 127.0.0.1:8404
mode http
stats enable
stats uri /stats
stats refresh 10s
frontend rke2_registration_9345
bind 192.168.x.100:9345
maxconn 2500
default_backend rke2_servers_9345
backend rke2_servers_9345
balance roundrobin
option tcp-check
default-server inter 2s fall 2 rise 3 slowstart 10s maxconn 1000
server master1 192.168.x.50:9345 check
server master2 192.168.x.51:9345 check
server master3 192.168.x.52:9345 check
frontend kubernetes_api_6443
bind 192.168.x.100:6443
maxconn 26000
default_backend kube_apiservers_6443
backend kube_apiservers_6443
balance roundrobin
option tcp-check
default-server inter 2s fall 2 rise 3 slowstart 15s maxconn 10000
server master1 192.168.x.50:6443 check
server master2 192.168.x.51:6443 check
server master3 192.168.x.52:6443 check
EOF
Two frontends bind to the VIP: port 9345 (node registration) and port 6443 (Kubernetes API). Both round-robin across the 3 masters with health checks every 2 seconds.
2.5 — Start HAProxy
sudo systemctl enable haproxy
sudo systemctl start haproxy
2.6 — Health Check Script
sudo tee /etc/keepalived/check_haproxy.sh << 'SCRIPT'
#!/bin/bash
systemctl is-active --quiet haproxy || exit 1
curl -sf http://127.0.0.1:8404/stats >/dev/null || exit 1
exit 0
SCRIPT
sudo chmod 755 /etc/keepalived/check_haproxy.sh
Keepalived runs this every 2 seconds. Non-zero exit = HAProxy is dead = VIP moves to backup.
2.7 — Keepalived Configuration
On haproxy-01 (PRIMARY):
INTERFACE="eth0" # Replace with your actual interface
sudo tee /etc/keepalived/keepalived.conf << EOF
global_defs {
router_id HAPROXY_01
enable_script_security
script_user root
}
vrrp_script chk_haproxy {
script "/etc/keepalived/check_haproxy.sh"
interval 2
fall 2
rise 2
timeout 2
}
vrrp_instance VI_RKE2 {
state MASTER
interface ${INTERFACE}
virtual_router_id 51
priority 120
advert_int 1
unicast_src_ip 192.168.x.40
unicast_peer {
192.168.x.41
}
authentication {
auth_type PASS
auth_pass <your-vrrp-password>
}
virtual_ipaddress {
192.168.x.100/24
}
track_script {
chk_haproxy
}
}
EOF
How VRRP failover works: Both nodes send heartbeats. The higher-priority node (120) holds the VIP. If the primary stops advertising — crash, network failure, failed health check — the backup (priority 110) takes ownership of the VIP by adding it to its own interface. Recovery is automatic when the primary comes back.
Multi-cluster note: Each cluster needs a unique
virtual_router_id. If all clusters share the same L2 network, use 51 for the hub, 52 for Spoke A, 53 for Spoke B, 54 for Spoke C.
2.8 — Start Keepalived
sudo systemctl enable keepalived
sudo systemctl start keepalived
2.9 — Repeat on haproxy-02
Same steps, but in the Keepalived config:
router_id HAPROXY_02
state BACKUP
priority 110
unicast_src_ip 192.168.x.41
unicast_peer {
192.168.x.40
}
2.10 — Verify
sudo systemctl is-active haproxy
sudo systemctl is-active keepalived
curl -s http://127.0.0.1:8404/stats | head -5
ip addr show | grep 192.168.x.100 # VIP visible on primary
ping -c 2 192.168.x.100 # Reachable from the network
✅ Load balancer tier ready.
Phase 3: Bootstrapping RKE2
Strictly sequential. master-01 must be fully running before touching master-02.
3.1 — First Master (.50)
sudo mkdir -p /etc/rancher/rke2
curl -sfL https://get.rke2.io | \
sudo INSTALL_RKE2_TYPE="server" \
INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -
First Master Config
sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
token: "<your-cluster-token>"
tls-san:
- "192.168.x.50"
- "192.168.x.51"
- "192.168.x.52"
- "192.168.x.100"
cni: cilium
disable-cloud-controller: true
disable: rke2-ingress-nginx
etcd-expose-metrics: true
etcd-snapshot-schedule-cron: "0 2 * * *"
etcd-snapshot-retention: 7
etcd-snapshot-compress: true
etcd-s3: true
etcd-s3-config-secret: rke2-etcd-snapshot-s3-config
etcd-arg:
- "heartbeat-interval=200"
- "election-timeout=2000"
kube-apiserver-arg:
- "audit-log-maxage=30"
- "audit-log-maxbackup=10"
- "audit-log-maxsize=100"
- "audit-log-mode=blocking"
- "request-timeout=120s"
- "default-not-ready-toleration-seconds=120"
- "default-unreachable-toleration-seconds=120"
- "event-ttl=2h"
kube-controller-manager-arg:
- "terminated-pod-gc-threshold=200"
- "bind-address=0.0.0.0"
kube-scheduler-arg:
- "bind-address=0.0.0.0"
kube-proxy-arg:
- "metrics-bind-address=0.0.0.0"
node-taint:
- "node-role.kubernetes.io/control-plane:NoSchedule"
EOF
Key settings explained:
| Setting | Why |
|---|---|
token |
Shared secret for cluster join authentication |
tls-san |
All master IPs + VIP in the API server's TLS cert |
disable: rke2-ingress-nginx |
Install your own ingress controller instead |
etcd-snapshot-* |
Automated daily backups, compressed, keep 7 |
node-taint: NoSchedule |
Keep workloads off masters |
bind-address=0.0.0.0 |
Expose metrics for Prometheus scraping |
CNI is left as default (Canal). Add
cni: calicoorcni: ciliumbased on your needs.
Start and Wait
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
# Watch logs (takes 2-5 min)
sudo journalctl -u rke2-server -f
# Wait for registration port
while ! sudo ss -tlnp | grep -q ':9345'; do
echo "Waiting for port 9345..."
sleep 10
done
echo "Ready for additional masters"
Verify and Set Up kubectl
sudo /var/lib/rancher/rke2/bin/kubectl \
--kubeconfig /etc/rancher/rke2/rke2.yaml get nodes
# Set up shortcuts
cat >> ~/.bashrc << 'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
EOF
mkdir -p ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
chmod 600 ~/.kube/config
source ~/.bashrc
3.2 — Additional Masters (.51, .52)
sudo mkdir -p /etc/rancher/rke2
curl -sfL https://get.rke2.io | \
sudo INSTALL_RKE2_TYPE="server" \
INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -
The config is identical to the first master plus one line at the top:
sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"
tls-san:
- "192.168.x.50"
- "192.168.x.51"
- "192.168.x.52"
- "192.168.x.100"
# ... rest identical to first master config ...
node-taint:
- "node-role.kubernetes.io/control-plane:NoSchedule"
EOF
The server: line is the only difference — it tells this node to join the existing cluster through the VIP instead of bootstrapping a new one.
sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
sudo journalctl -u rke2-server -f
Wait for Ready, then repeat for master-03. After all three:
NAME STATUS ROLES AGE
master-01 Ready control-plane,etcd,master 10m
master-02 Ready control-plane,etcd,master 5m
master-03 Ready control-plane,etcd,master 2m
3.3 — Worker Nodes
sudo mkdir -p /etc/rancher/rke2
curl -sfL https://get.rke2.io | \
sudo INSTALL_RKE2_TYPE="agent" \
INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -
Note: agent not server.
sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"
disable:
- rke2-ingress-nginx
kube-proxy-arg:
- "metrics-bind-address=0.0.0.0"
EOF
sudo systemctl enable rke2-agent.service
sudo systemctl start rke2-agent.service
Repeat for all workers. Spoke clusters get 8 workers (.60–.67), the hub gets 1 worker (.60).
Connecting the Spokes to the Hub
Once all clusters are running, the hub-and-spoke connectivity looks like this:
How Each Service Connects
Prometheus (Metrics):
Each spoke cluster runs its own Prometheus instance, but instead of storing metrics locally, it uses remote_write to push them to the hub's central Prometheus. The hub's Grafana queries the central Prometheus for dashboards across all clusters.
Loki (Logs):
Each spoke cluster runs Promtail as a DaemonSet. Promtail ships container logs to the hub's Loki endpoint. In Grafana, you can filter logs by cluster, namespace, and pod — all from one place.
ArgoCD (GitOps):
ArgoCD runs in the hub and registers each spoke cluster as a target. Application manifests live in Git. A PR merge triggers ArgoCD to sync the changes to the correct spoke cluster. One ArgoCD instance, one Git source of truth, deploys everywhere.
Harbor (Container Registry):
Harbor runs in the hub cluster. All spoke clusters have their containerd configured to pull images from the hub's Harbor instance. One scan, one set of vulnerability reports, one place to manage image lifecycle.
Vault (Secrets):
HashiCorp Vault runs in the hub. Spoke clusters run the Vault Agent Injector, which authenticates using Kubernetes service accounts and injects secrets into pods at runtime. Rotate a database password in Vault once — all clusters get it.
Hub Cluster: Special Considerations
The hub cluster has only 1 worker node, but it runs critical shared services. A few things to consider:
Removing the NoSchedule Taint
With only 1 worker, you might need the masters to share the workload:
# Allow pods on masters (only for the hub cluster)
kubectl taint nodes master-01 \
node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-02 \
node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-03 \
node-role.kubernetes.io/control-plane:NoSchedule-
The 3 masters have 24 CPUs and 48 GB RAM combined — plenty for Grafana, ArgoCD, Harbor, and Vault alongside the control plane.
Storage Planning
Hub services are storage-heavy:
| Service | Storage Needs |
|---|---|
| Prometheus | Time-series data from all clusters. Plan 50–100 GB+ |
| Loki | Log chunks from all clusters. Plan 100 GB+ (use object storage) |
| Harbor | Container images. Plan 200 GB+ |
| Vault | Minimal (secrets are small), but HA storage backend needed |
This is why the masters have 300 GB each. Consider using Longhorn or a dedicated NFS server for persistent volumes.
Execution Cheat Sheet
FOR EACH CLUSTER (hub first, then spokes):
PHASE 1 — All master + worker nodes (parallel):
├── apt update & upgrade
├── /etc/hosts
├── disable swap
├── kernel modules (overlay, br_netfilter)
├── sysctl (iptables bridge, ip_forward)
├── disable UFW
├── install packages
├── enable iscsid
└── reboot
PHASE 2 — HAProxy nodes (primary first):
├── install haproxy + keepalived
├── sysctl (nonlocal_bind, somaxconn)
├── systemd NOFILE limit
├── haproxy.cfg
├── start haproxy
├── check_haproxy.sh
├── keepalived.conf
├── start keepalived
└── verify VIP
PHASE 3 — RKE2 (sequential):
├── master-01: first-master config, start, wait for :9345
├── master-02: additional-master config (server: line), start
├── master-03: same
└── workers: agent install, start
POST-CLUSTER:
Hub: Install Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault
Spokes: Install Promtail, Prometheus (remote-write), Vault agent
Register as ArgoCD target
Configure containerd to pull from Hub's Harbor
Troubleshooting
Master won't start:
sudo journalctl -u rke2-server -f --no-pager | tail -50
Node stuck in NotReady:
kubectl describe node <node-name>
Connection refused on port 9345:
sudo ss -tlnp | grep 9345
curl -k https://192.168.x.100:9345
VIP not failing over:
sudo journalctl -u keepalived -f
sudo tcpdump -i eth0 vrrp
Spoke metrics not appearing in Hub Grafana:
Check the spoke's Prometheus remote-write config — the endpoint should point to the hub's Prometheus push endpoint, and network policies should allow cross-cluster traffic.
Spoke logs missing in Hub Loki:
Check Promtail's config on the spoke — clients.url should point to http://<hub-loki-gateway>:3100/loki/api/v1/push. Verify the hub's Loki ingester isn't rejecting streams (check for "rate limit" or "stream limit" errors).
Ports Reference
| Port | Protocol | Purpose |
|---|---|---|
| 6443 | TCP | Kubernetes API server |
| 9345 | TCP | RKE2 node registration |
| 2379–2380 | TCP | etcd client/peer |
| 8472 | UDP | VXLAN overlay (Canal/Flannel) |
| 10250 | TCP | kubelet API |
| 8404 | TCP | HAProxy stats (localhost) |
| 112 | VRRP | Keepalived heartbeat |
What I Learned
Building clusters manually takes 10× longer than running Ansible. But now when something breaks at 2 AM — and in a hub-and-spoke setup, the hub going down means all clusters lose their operational visibility — I know exactly which component failed and why.
The hub-and-spoke model is worth the added complexity. One Grafana to check instead of four. One ArgoCD to manage instead of four. One Harbor to scan instead of four. The operational overhead drops dramatically once the initial setup is done.
If you're managing more than two clusters, centralize your tooling. Your future self — the one debugging a production incident across three clusters at midnight — will thank you.
This article is based on building a hub-and-spoke platform with 4 production RKE2 clusters. All project names, secrets, credentials, and identifying details have been anonymized.
Questions? Hit the comments — I've probably run into the same issue.




Top comments (0)