DEV Community: Syed Muhammad Jamal Uddin Gilani

Running a Live Media Monitoring Platform on Kubernetes

Syed Muhammad Jamal Uddin Gilani — Sat, 05 Sep 2026 06:11:39 +0000

Media monitoring sounds simple from the outside: watch TV channels, pull out mentions, show sentiment on a dashboard. Underneath, it's a real-time media pipeline — live UDP video feeds coming in continuously, AI models transcribing and analyzing them, and a production system that isn't allowed to just "restart and try again" when something breaks, because the broadcast doesn't pause for you.

That's the infrastructure challenge behind BetaMedia, Beta Codes' electronic media monitoring platform, and it's the project where I moved from managing container networking to owning a full Kubernetes-based microservices deployment end to end.

What BetaMedia Does

BetaMedia is built to watch media so people don't have to — pulling in live TV broadcasts and digital media, then running them through AI to make them searchable and analyzable. On the product side that means:

Live TV streaming and playback, with clipping and archiving of past broadcasts
Sentiment and emotion analysis across mentions, in real time
Speech-to-text for turning broadcast audio into searchable transcripts
OCR, ticker extraction, and "flasher" detection for on-screen text, scrolling news tickers, and breaking-news banners
Facial recognition for tracking public figures across clips
Customizable dashboards built around KPIs like sentiment shifts and keyword frequency

It's used by broadcasters, government and policy units, regulators, and security teams who need to know what's being said about a brand, a policy, or an event, as it's being said — not the next morning.

None of that AI output means anything if the pipeline feeding it drops frames, falls behind, or goes down. That's the part I own.

The Core Challenge: Continuous Streams, Zero Room for Downtime

Media monitoring is unlike a typical web app in one important way: the input never stops. A broadcast doesn't wait for your deployment window. That shapes almost every infrastructure decision:

UDP video streams have to be captured and handed off to processing continuously, not batched
Deployments and updates need to happen without interrupting streams already in flight
A single-service app isn't enough — ingestion, transcoding, and each AI module need to scale and fail independently
Storage has to keep up with continuously growing video, transcripts, and clips

Kubernetes as the Backbone

BetaMedia runs as a set of microservices on Kubernetes — ingestion, transcoding, and each AI module (speech-to-text, OCR, facial recognition, sentiment analysis) deployed and scaled independently rather than bundled into one monolith. That separation matters here specifically because these services have very different resource profiles: transcoding is CPU/GPU-heavy and constant, while something like ticker extraction only needs to run against new segments as they arrive.

Managing the cluster means keeping that mix of workloads healthy together — right-sizing resources per service, keeping pods scheduled sensibly across nodes, and making sure a spike in one AI module doesn't starve the ingestion pipeline feeding everything else.

Serving UDP Streams Into the Processing Pipeline

Live TV feeds come in as UDP streams, which then have to be transcoded (FFmpeg) and routed into the application layer for AI processing. Getting this right in Kubernetes means building the networking and workflow so a stream goes from "raw broadcast feed" to "frames the AI services can consume" reliably and continuously — with the pipeline treated as a first-class, always-on system rather than a batch job that happens to run often.

Deploying to Production Without Breaking the Stream

"Streamline the deployment process without failure" is a simple sentence that hides a lot of constraints once you're running live infrastructure. It meant building workflows where:

Releases roll out predictably, with a clear path to roll back if something's wrong
Deployments don't interrupt streams that are actively being ingested or processed
The path from a code change to a running production service is a defined workflow, not a manual sequence of steps someone has to remember correctly every time

Working closely with the production side of this — rather than handing off a build and hoping — is what turned deployments from a risk event into a routine one.

Monitoring: Watching the Watchers

A media monitoring platform that silently stops ingesting a channel is its own kind of failure — the dashboards look calm, but the data behind them just stopped. So monitoring here has to answer more than "is the pod up":

Are streams actively being ingested, per channel/source?
Are the AI processing microservices keeping up, or falling behind?
Is storage growth on track, or about to become a problem?

Dashboards built around these questions turn silent pipeline stalls into visible, actionable alerts instead of a client noticing a gap in coverage first.

Data: Databases, S3, and a Lot of Microservices

Behind the AI modules sits the less visible work: managing the databases behind transcripts, mentions, and metadata, and the S3-compatible object storage handling the growing archive of video and clips. With this many independent microservices, keeping data infrastructure consistent, backed up, and correctly connected to the right service is as much of the job as the Kubernetes layer itself.

What This Project Taught Me

BetaMedia taught me something different: how to run many different services, with very different resource needs and failure modes, as one coherent, always-on system — where "always-on" isn't a nice-to-have; it's the actual product requirement.

The two problems look similar from a distance — Kubernetes, monitoring, microservices — but the constraints that actually shape the architecture are completely different once you look closely. That's been one of the more useful lessons of doing DevOps across both.

I'm a DevOps Engineer at Beta Codes.pk, working on Kubernetes, GitOps, and infrastructure for AI platforms including BetaMedia and BetaVision. Find me on LinkedIn or check out my portfolio.

Scaling AI Surveillance to 400+ Sites: The DevOps Behind BetaVision

Syed Muhammad Jamal Uddin Gilani — Fri, 04 Sep 2026 15:40:07 +0000

Most DevOps write-ups talk about deploying an application to a cluster. This one is about deploying the same AI platform to hundreds of independent, physically separate sites each with its own hardware, its own network, and its own failure modes and still being able to see all of them from one dashboard.

That's the problem I work on as a DevOps Engineer at Beta Codes.pk, building the infrastructure behind BetaVision, our AI-powered smart surveillance platform for urban and organizational security.

What BetaVision Does

BetaVision is an AI-driven surveillance platform built around continuous, automated monitoring rather than passive CCTV footage. On the product side, it combines:

24/7 AI-driven monitoring across critical infrastructure, public spaces, and high-risk areas
Video analytics for spotting suspicious activity and unusual patterns automatically
License plate recognition (LPR) for traffic control, parking, and law-enforcement use cases
Facial recognition for identifying individuals of interest and managing access
Real-time alerts that notify security personnel the moment something needs attention
A centralized command-and-control dashboard giving a single, unified view across every connected feed
A mobile app for remote monitoring

It's aimed at city governments, law enforcement agencies, corporate security teams, event organizers, retailers, and smart-city / property developments — anywhere a lot of camera feeds need to become actionable intelligence instead of just recorded video. The platform is deployed on GPU-enabled client and government infrastructure, running object detection, facial recognition, and ANPR/LPR on live feeds, with results streamed back to a central production dashboard.

The AI/ML side is only half the story. The harder, less glamorous half is: how do you take a platform like this and reliably run it on 400 sites, each behind different networks, different hardware, different connectivity — and still ship updates, catch failures, and keep every stream visible in real time?

That's the part I own.

The Core Challenge: One Codebase, Hundreds of Independent Deployments

Every site runs its own instance of BetaVision — backend, frontend, and supporting services — on Docker containers, on-premise, on the client's own GPU hardware. That means:

400 separate deployment targets, not one cluster to scale horizontally
No guarantee of stable public IPs or open inbound ports on client networks
Updates that need to reach every site without 400 manual SSH sessions
A central command-and-control dashboard that needs live visibility into sites it can't directly reach

Here's how each of those gets solved.

Making 400 Private Networks Publicly Reachable

Client sites are, by design, locked-down networks. We don't control their firewalls, and we can't always get inbound access. But the whole point of BetaVision is that streams and results need to be visible centrally, 24/7.

The solution is FRP (Fast Reverse Proxy), running as a client on each site and a server on our side. Every site's containers only need to make an outbound connection — no inbound firewall rules required on the client's end. FRP tunnels the stream back through, and we wrap every tunnel in TLS so nothing is exposed in plaintext across the public internet.

This one decision is what turns "400 isolated on-prem boxes" into "400 endpoints visible from a single pane of glass."

Docker + Networking at Site Level

Each site runs BetaVision's backend and frontend as Docker containers, with internal networks configured so services can talk to each other — the AI inference services, the API layer, the database layer, and the streaming layer all need to communicate reliably on-box before anything goes external through FRP.

Multiply the usual "make sure your containers can talk to each other" problem by 400, across hardware you don't physically control, and debugging becomes as much about reproducibility as it is about the fix itself. A problem on site #217 needs to be diagnosable and fixable using the same playbook as site #12.

Automating 400 Pipelines Without Losing Control

Manually deploying to 400 sites isn't a scaling problem, it's a guaranteed-to-fail problem. So the deployment pipeline is built on GitHub Actions, with dedicated pipelines per site for backend and frontend releases.

A few things matter more here than in a typical single-cluster CI/CD setup:

Release history has to be real, not aspirational. With this many deployment targets, "what version is site X running right now" needs to be answerable instantly, not reconstructed from memory.
One-click releases replace multi-step manual processes. What used to be a manual, error-prone rollout became a single pipeline trigger — this matters even more at 400-site scale than it did rolling this pattern out internally at 10-microservice scale.
Failures need to be isolated. A bad deploy to one site should never cascade or block releases to the other 399.

Monitoring: Seeing Problems Before Clients Report Them

None of this matters if you find out about outages from a client phone call. Monitoring across this many independent deployments has to answer two different questions:

Is each individual site healthy?
Is the aggregate platform healthy?

We run a Prometheus, Grafana, and Loki stack for visibility and alerting, so incidents surface through dashboards and automated alerts instead of user reports. At single-cluster scale this is standard practice; at 400-site scale, it's the difference between catching a dead stream in minutes versus finding out days later.

Data: MinIO S3 and Databases, Per Site

Each deployment needs its own storage and database layer — provisioned, backed up, and secured independently. We use MinIO S3 buckets for object storage and manage the database infrastructure per deployment, with backup and recovery built in from day one rather than bolted on after a data loss incident.

Security runs through all of this, not around it — from access control down to how data moves off each site, aligned with our ISO 9001 and ISO 27001 certified information security standards.

From POC to Production, Repeatedly

Before a site goes live, it usually starts as a proof of concept — deployed on a client's own GPU server, demoed live, sometimes directly to CIOs, under real production conditions. That means the deployment tooling has to work identically whether it's POC #1 or production site #380. There's no separate "demo mode" — if the automation only works for the polished cases, it doesn't scale to 400 real ones.

What This Actually Teaches You About DevOps

Working on BetaVision reframed how I think about scale. It's not just "more replicas of the same thing." It's:

Reliability without central control — most of your infrastructure lives on hardware and networks you don't own.
Observability as a first-class deliverable, not an afterthought — you cannot manually check 400 sites.
Automation as risk reduction, not just convenience — every manual step is a failure point multiplied by 400.

If you're building infrastructure for a platform that needs to run in many independent, disconnected environments rather than one big cluster, I'd genuinely love to compare notes — feel free to reach out.

I'm a DevOps Engineer at Beta Codes.pk, working on Kubernetes, GitOps, and infrastructure for AI platforms deployed across hundreds of sites. Find me on LinkedIn or check out my portfolio.

Hub-and-Spoke Kubernetes: Building a Multi-Cluster RKE2 Platform with HAProxy, Keepalived & Centralized Operations

Syed Muhammad Jamal Uddin Gilani — Wed, 02 Sep 2026 07:05:10 +0000

The Hub-and-Spoke Model

Instead of duplicating monitoring stacks, container registries, secret managers, and GitOps controllers in every cluster, we run them once in a central hub cluster and connect all spoke clusters back to it.

Why Hub-and-Spoke?

Without it: Every cluster runs its own Prometheus, Grafana, Loki, ArgoCD, Harbor, and Vault. That's 4× the infrastructure, 4× the maintenance, 4× the dashboards to check, and zero correlation between clusters.

With it:

Centralized monitoring — One Grafana, one set of dashboards. Spoke clusters ship metrics (via remote-write Prometheus) and logs (via Promtail → central Loki) to the hub. You see all clusters in one place.
Single container registry — Harbor runs in the hub. All spoke clusters pull images from one place. One vulnerability scan covers everything.
Unified secrets management — Vault in the hub, with agents or sidecar injectors in each spoke. Rotate a secret once, it propagates everywhere.
GitOps from one control plane — ArgoCD in the hub manages deployments across all spoke clusters. One PR, one review, deploys to the right cluster.

Cluster Inventory

Cluster	Role	Masters	Workers	Shared Services
Hub (Internal Tools)	Central operations	3 × (8C/16G/300G)	1	Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault
Spoke A	Application workloads	3 × (8C/16G/300G)	8	Promtail, Prometheus (remote-write), Vault agent
Spoke B	Application workloads	3 × (8C/16G/300G)	8	Same as Spoke A
Spoke C	Application workloads	3 × (8C/16G/300G)	8	Same as Spoke A

Each cluster also has 2 HAProxy nodes (1C/2G/30G each) for load balancing.

Architecture of a Single Cluster

Every cluster — hub or spoke — follows the same HA skeleton: a floating VIP in front of two HAProxy nodes, three masters running the control plane, and a data-plane tier of workers. The two diagrams below are the actual topology diagrams I drew while planning this, with every project/cluster name scrubbed out.

Hub Cluster Layout

The hub only needs to run shared tooling for one "tenant" (itself), so it gets a single worker node. Masters pick up the slack, once the NoSchedule taint is lifted (more on that later).

Spoke Cluster Layout

Spokes carry real application traffic, so they get more workers and a MetalLB address pool for LoadBalancer-type Services — something the hub doesn't need since its own services are exposed through Ingress. That extra range pushes the LB tier and masters a bit further up the subnet compared to the hub.

There are 3 spoke clusters in this platform, and all three are byte-for-byte identical in topology — they only differ in the subnet octet (192.168.X.0/24) and hostnames. I'm showing one generic template instead of three near-duplicate diagrams:

Spoke Subnet (anonymized) Masters Workers

Spoke A 192.168.11.0/24 3 8

Spoke B 192.168.12.0/24 3 8

Spoke C 192.168.13.0/24 3 8

Spoke	Subnet (anonymized)	Masters	Workers
Spoke A	`192.168.11.0/24`	3	8
Spoke B	`192.168.12.0/24`	3	8
Spoke C	`192.168.13.0/24`	3	8

How HA works:

Master failure: If one master dies, the other two still have etcd quorum. The API server stays up.
Load balancer failure: If haproxy-01 dies, Keepalived moves the VIP to haproxy-02 within ~4 seconds.
kubectl never breaks: It always points at the VIP, which always points at a healthy load balancer, which always routes to a healthy master.

IP Allocation Scheme

The hub and the spokes use slightly different offsets, because spokes reserve a block for MetalLB and the hub doesn't. Only the subnet octet (x / X) changes between clusters of the same type.

Hub cluster (192.168.x.0/24):

Range	Purpose
`.1 – .39`	Network infrastructure (reserved)
`.40 – .49`	HAProxy load balancers
`.50 – .59`	Master nodes
`.60 – .99`	Worker nodes
`.100`	Floating VIP

Node	IP	Specs
haproxy-01	`.40`	1C / 2G / 30G
haproxy-02	`.41`	1C / 2G / 30G
master-01	`.50`	8C / 16G / 300G
master-02	`.51`	8C / 16G / 300G
master-03	`.52`	8C / 16G / 300G
worker-01	`.60`	As needed
VIP	`.100`	Floating

Spoke cluster (192.168.X.0/24, X = per-spoke octet):

Range	Purpose
`.1 – .50`	Network infrastructure (reserved)
`.51 – .59`	MetalLB address pool
`.60 – .69`	HAProxy load balancers
`.70 – .79`	Master nodes
`.80 – .99`	Worker nodes
`.100`	Floating VIP

Node	IP	Specs
haproxy-01	`.60`	1C / 2G / 30G
haproxy-02	`.61`	1C / 2G / 30G
master-01	`.70`	8C / 16G / 300G
master-02	`.71`	8C / 16G / 300G
master-03	`.72`	8C / 16G / 300G
worker-01 … worker-08	`.80` – `.87`	As needed
VIP	`.100`	Floating

The hub cluster uses only worker-01 (.60). Spoke clusters use workers 01–08 (.80–.87) plus the MetalLB pool for LoadBalancer services.

Prerequisites

For each cluster, generate a unique token:

openssl rand -hex 32

Each cluster gets its own token. Don't reuse tokens across clusters.

Phase 1: Preparing the Nodes

Run on: Every master and worker node (NOT the haproxy nodes).

1.1 — System Update

sudo apt update && sudo apt dist-upgrade -y

1.2 — Host Resolution

sudo tee -a /etc/hosts << 'EOF'
# RKE2 Cluster Nodes
192.168.x.50   master-01
192.168.x.51   master-02
192.168.x.52   master-03
192.168.x.60   worker-01
# Add remaining workers as needed
192.168.x.40   haproxy-01
192.168.x.41   haproxy-02
EOF

1.3 — Disable Swap

sudo swapoff -a
sudo sed -i '/\sswap\s/ s/^/#/' /etc/fstab

kubelet won't start with swap enabled. Kubernetes does its own memory management — swap adds unpredictable latency.

1.4 — Kernel Modules

sudo tee /etc/modules-load.d/k8s.conf << 'EOF'
br_netfilter
overlay
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

overlay is required by containerd. br_netfilter makes bridged traffic visible to iptables for pod networking.

1.5 — Network Parameters

sudo tee /etc/sysctl.d/99-rke2.conf << 'EOF'
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

Without ip_forward, packets between pods on different nodes stop at the node boundary.

1.6 — Disable UFW

sudo systemctl stop ufw
sudo systemctl disable ufw

1.7 — Required Packages

sudo apt install -y curl wget tar nfs-common open-iscsi lsscsi sg3-utils

Package	Purpose
`nfs-common`	Mount NFS persistent volumes
`open-iscsi`, `lsscsi`, `sg3-utils`	Block storage (Longhorn)

1.8 — Enable iSCSI

sudo systemctl enable iscsid
sudo systemctl start iscsid

1.9 — Reboot

sudo reboot

Verify after reboot:

lsmod | grep br_netfilter        # ✓ loaded
sysctl net.ipv4.ip_forward       # ✓ = 1
free -h                          # ✓ swap = 0
systemctl is-active iscsid       # ✓ active

✅ Complete on ALL master + worker nodes before Phase 2.

Phase 2: HAProxy & Keepalived

Run on: haproxy-01 (.40) first, then haproxy-02 (.41).

2.1 — Install

sudo apt update
sudo apt install -y haproxy keepalived curl

2.2 — Sysctl Tuning

sudo tee /etc/sysctl.d/99-lb-tuning.conf << 'EOF'
net.ipv4.ip_nonlocal_bind = 1
net.core.somaxconn = 65535
EOF

sudo sysctl --system

ip_nonlocal_bind lets HAProxy bind to the VIP before Keepalived assigns it. Without this, HAProxy crashes on startup when the VIP is on the other node.

2.3 — File Descriptor Limits

sudo mkdir -p /etc/systemd/system/haproxy.service.d

sudo tee /etc/systemd/system/haproxy.service.d/limits.conf << 'EOF'
[Service]
LimitNOFILE=100000
EOF

sudo systemctl daemon-reload

2.4 — HAProxy Configuration

Find your interface:

ip -4 addr show | grep "192.168.x.40"

Deploy the config:

sudo tee /etc/haproxy/haproxy.cfg << 'EOF'
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon
    maxconn 30000

defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    option  tcpka
    timeout connect 5s
    timeout client  60s
    timeout server  60s
    retries 3

listen stats
    bind 127.0.0.1:8404
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s

frontend rke2_registration_9345
    bind 192.168.x.100:9345
    maxconn 2500
    default_backend rke2_servers_9345

backend rke2_servers_9345
    balance roundrobin
    option tcp-check
    default-server inter 2s fall 2 rise 3 slowstart 10s maxconn 1000
    server master1 192.168.x.50:9345 check
    server master2 192.168.x.51:9345 check
    server master3 192.168.x.52:9345 check

frontend kubernetes_api_6443
    bind 192.168.x.100:6443
    maxconn 26000
    default_backend kube_apiservers_6443

backend kube_apiservers_6443
    balance roundrobin
    option tcp-check
    default-server inter 2s fall 2 rise 3 slowstart 15s maxconn 10000
    server master1 192.168.x.50:6443 check
    server master2 192.168.x.51:6443 check
    server master3 192.168.x.52:6443 check
EOF

Two frontends bind to the VIP: port 9345 (node registration) and port 6443 (Kubernetes API). Both round-robin across the 3 masters with health checks every 2 seconds.

2.5 — Start HAProxy

sudo systemctl enable haproxy
sudo systemctl start haproxy

2.6 — Health Check Script

sudo tee /etc/keepalived/check_haproxy.sh << 'SCRIPT'
#!/bin/bash
systemctl is-active --quiet haproxy || exit 1
curl -sf http://127.0.0.1:8404/stats >/dev/null || exit 1
exit 0
SCRIPT

sudo chmod 755 /etc/keepalived/check_haproxy.sh

Keepalived runs this every 2 seconds. Non-zero exit = HAProxy is dead = VIP moves to backup.

2.7 — Keepalived Configuration

On haproxy-01 (PRIMARY):

INTERFACE="eth0"   # Replace with your actual interface

sudo tee /etc/keepalived/keepalived.conf << EOF
global_defs {
    router_id HAPROXY_01
    enable_script_security
    script_user root
}

vrrp_script chk_haproxy {
    script "/etc/keepalived/check_haproxy.sh"
    interval 2
    fall 2
    rise 2
    timeout 2
}

vrrp_instance VI_RKE2 {
    state MASTER
    interface ${INTERFACE}
    virtual_router_id 51
    priority 120
    advert_int 1

    unicast_src_ip 192.168.x.40
    unicast_peer {
        192.168.x.41
    }

    authentication {
        auth_type PASS
        auth_pass <your-vrrp-password>
    }

    virtual_ipaddress {
        192.168.x.100/24
    }

    track_script {
        chk_haproxy
    }
}
EOF

How VRRP failover works: Both nodes send heartbeats. The higher-priority node (120) holds the VIP. If the primary stops advertising — crash, network failure, failed health check — the backup (priority 110) takes ownership of the VIP by adding it to its own interface. Recovery is automatic when the primary comes back.

Multi-cluster note: Each cluster needs a unique virtual_router_id. If all clusters share the same L2 network, use 51 for the hub, 52 for Spoke A, 53 for Spoke B, 54 for Spoke C.

2.8 — Start Keepalived

sudo systemctl enable keepalived
sudo systemctl start keepalived

2.9 — Repeat on haproxy-02

Same steps, but in the Keepalived config:

router_id HAPROXY_02
state BACKUP
priority 110
unicast_src_ip 192.168.x.41
unicast_peer {
    192.168.x.40
}

2.10 — Verify

sudo systemctl is-active haproxy
sudo systemctl is-active keepalived
curl -s http://127.0.0.1:8404/stats | head -5
ip addr show | grep 192.168.x.100   # VIP visible on primary
ping -c 2 192.168.x.100             # Reachable from the network

✅ Load balancer tier ready.

Phase 3: Bootstrapping RKE2

Strictly sequential. master-01 must be fully running before touching master-02.

3.1 — First Master (.50)

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="server" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

First Master Config

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
token: "<your-cluster-token>"

tls-san:
  - "192.168.x.50"
  - "192.168.x.51"
  - "192.168.x.52"
  - "192.168.x.100"
cni: cilium
disable-cloud-controller: true

disable: rke2-ingress-nginx

etcd-expose-metrics: true
etcd-snapshot-schedule-cron: "0 2 * * *"
etcd-snapshot-retention: 7
etcd-snapshot-compress: true
etcd-s3: true
etcd-s3-config-secret: rke2-etcd-snapshot-s3-config

etcd-arg:
  - "heartbeat-interval=200"
  - "election-timeout=2000"

kube-apiserver-arg:
  - "audit-log-maxage=30"
  - "audit-log-maxbackup=10"
  - "audit-log-maxsize=100"
  - "audit-log-mode=blocking"
  - "request-timeout=120s"
  - "default-not-ready-toleration-seconds=120"
  - "default-unreachable-toleration-seconds=120"
  - "event-ttl=2h"

kube-controller-manager-arg:
  - "terminated-pod-gc-threshold=200"
  - "bind-address=0.0.0.0"

kube-scheduler-arg:
  - "bind-address=0.0.0.0"

kube-proxy-arg:
  - "metrics-bind-address=0.0.0.0"

node-taint:
  - "node-role.kubernetes.io/control-plane:NoSchedule"
EOF

Key settings explained:

Setting	Why
`token`	Shared secret for cluster join authentication
`tls-san`	All master IPs + VIP in the API server's TLS cert
`disable: rke2-ingress-nginx`	Install your own ingress controller instead
`etcd-snapshot-*`	Automated daily backups, compressed, keep 7
`node-taint: NoSchedule`	Keep workloads off masters
`bind-address=0.0.0.0`	Expose metrics for Prometheus scraping

CNI is left as default (Canal). Add cni: calico or cni: cilium based on your needs.

Start and Wait

sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service

# Watch logs (takes 2-5 min)
sudo journalctl -u rke2-server -f

# Wait for registration port
while ! sudo ss -tlnp | grep -q ':9345'; do
  echo "Waiting for port 9345..."
  sleep 10
done
echo "Ready for additional masters"

Verify and Set Up kubectl

sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes

# Set up shortcuts
cat >> ~/.bashrc << 'EOF'
export PATH=$PATH:/var/lib/rancher/rke2/bin
source <(kubectl completion bash)
alias k=kubectl
complete -F __start_kubectl k
EOF

mkdir -p ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
chmod 600 ~/.kube/config
source ~/.bashrc

3.2 — Additional Masters (.51, .52)

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="server" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

The config is identical to the first master plus one line at the top:

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"

tls-san:
  - "192.168.x.50"
  - "192.168.x.51"
  - "192.168.x.52"
  - "192.168.x.100"

# ... rest identical to first master config ...

node-taint:
  - "node-role.kubernetes.io/control-plane:NoSchedule"
EOF

The server: line is the only difference — it tells this node to join the existing cluster through the VIP instead of bootstrapping a new one.

sudo systemctl enable rke2-server.service
sudo systemctl start rke2-server.service
sudo journalctl -u rke2-server -f

Wait for Ready, then repeat for master-03. After all three:

NAME        STATUS   ROLES                       AGE
master-01   Ready    control-plane,etcd,master   10m
master-02   Ready    control-plane,etcd,master   5m
master-03   Ready    control-plane,etcd,master   2m

3.3 — Worker Nodes

sudo mkdir -p /etc/rancher/rke2

curl -sfL https://get.rke2.io | \
  sudo INSTALL_RKE2_TYPE="agent" \
  INSTALL_RKE2_VERSION="v1.35.2+rke2r1" sh -

Note: agent not server.

sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
server: "https://192.168.x.100:9345"
token: "<your-cluster-token>"

disable:
  - rke2-ingress-nginx

kube-proxy-arg:
  - "metrics-bind-address=0.0.0.0"
EOF

sudo systemctl enable rke2-agent.service
sudo systemctl start rke2-agent.service

Repeat for all workers. Spoke clusters get 8 workers (.60–.67), the hub gets 1 worker (.60).

Connecting the Spokes to the Hub

Once all clusters are running, the hub-and-spoke connectivity looks like this:

How Each Service Connects

Prometheus (Metrics):
Each spoke cluster runs its own Prometheus instance, but instead of storing metrics locally, it uses remote_write to push them to the hub's central Prometheus. The hub's Grafana queries the central Prometheus for dashboards across all clusters.

Loki (Logs):
Each spoke cluster runs Promtail as a DaemonSet. Promtail ships container logs to the hub's Loki endpoint. In Grafana, you can filter logs by cluster, namespace, and pod — all from one place.

ArgoCD (GitOps):
ArgoCD runs in the hub and registers each spoke cluster as a target. Application manifests live in Git. A PR merge triggers ArgoCD to sync the changes to the correct spoke cluster. One ArgoCD instance, one Git source of truth, deploys everywhere.

Harbor (Container Registry):
Harbor runs in the hub cluster. All spoke clusters have their containerd configured to pull images from the hub's Harbor instance. One scan, one set of vulnerability reports, one place to manage image lifecycle.

Vault (Secrets):
HashiCorp Vault runs in the hub. Spoke clusters run the Vault Agent Injector, which authenticates using Kubernetes service accounts and injects secrets into pods at runtime. Rotate a database password in Vault once — all clusters get it.

Hub Cluster: Special Considerations

The hub cluster has only 1 worker node, but it runs critical shared services. A few things to consider:

Removing the NoSchedule Taint

With only 1 worker, you might need the masters to share the workload:

# Allow pods on masters (only for the hub cluster)
kubectl taint nodes master-01 \
  node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-02 \
  node-role.kubernetes.io/control-plane:NoSchedule-
kubectl taint nodes master-03 \
  node-role.kubernetes.io/control-plane:NoSchedule-

The 3 masters have 24 CPUs and 48 GB RAM combined — plenty for Grafana, ArgoCD, Harbor, and Vault alongside the control plane.

Storage Planning

Hub services are storage-heavy:

Service	Storage Needs
Prometheus	Time-series data from all clusters. Plan 50–100 GB+
Loki	Log chunks from all clusters. Plan 100 GB+ (use object storage)
Harbor	Container images. Plan 200 GB+
Vault	Minimal (secrets are small), but HA storage backend needed

This is why the masters have 300 GB each. Consider using Longhorn or a dedicated NFS server for persistent volumes.

Execution Cheat Sheet

FOR EACH CLUSTER (hub first, then spokes):

PHASE 1 — All master + worker nodes (parallel):
  ├── apt update & upgrade
  ├── /etc/hosts
  ├── disable swap
  ├── kernel modules (overlay, br_netfilter)
  ├── sysctl (iptables bridge, ip_forward)
  ├── disable UFW
  ├── install packages
  ├── enable iscsid
  └── reboot

PHASE 2 — HAProxy nodes (primary first):
  ├── install haproxy + keepalived
  ├── sysctl (nonlocal_bind, somaxconn)
  ├── systemd NOFILE limit
  ├── haproxy.cfg
  ├── start haproxy
  ├── check_haproxy.sh
  ├── keepalived.conf
  ├── start keepalived
  └── verify VIP

PHASE 3 — RKE2 (sequential):
  ├── master-01: first-master config, start, wait for :9345
  ├── master-02: additional-master config (server: line), start
  ├── master-03: same
  └── workers: agent install, start

POST-CLUSTER:
  Hub:  Install Grafana, Prometheus, Loki, ArgoCD, Harbor, Vault
  Spokes: Install Promtail, Prometheus (remote-write), Vault agent
          Register as ArgoCD target
          Configure containerd to pull from Hub's Harbor

Troubleshooting

Master won't start:

sudo journalctl -u rke2-server -f --no-pager | tail -50

Node stuck in NotReady:

kubectl describe node <node-name>

Connection refused on port 9345:

sudo ss -tlnp | grep 9345
curl -k https://192.168.x.100:9345

VIP not failing over:

sudo journalctl -u keepalived -f
sudo tcpdump -i eth0 vrrp

Spoke metrics not appearing in Hub Grafana:
Check the spoke's Prometheus remote-write config — the endpoint should point to the hub's Prometheus push endpoint, and network policies should allow cross-cluster traffic.

Spoke logs missing in Hub Loki:
Check Promtail's config on the spoke — clients.url should point to http://<hub-loki-gateway>:3100/loki/api/v1/push. Verify the hub's Loki ingester isn't rejecting streams (check for "rate limit" or "stream limit" errors).

Ports Reference

Port	Protocol	Purpose
6443	TCP	Kubernetes API server
9345	TCP	RKE2 node registration
2379–2380	TCP	etcd client/peer
8472	UDP	VXLAN overlay (Canal/Flannel)
10250	TCP	kubelet API
8404	TCP	HAProxy stats (localhost)
112	VRRP	Keepalived heartbeat

What I Learned

Building clusters manually takes 10× longer than running Ansible. But now when something breaks at 2 AM — and in a hub-and-spoke setup, the hub going down means all clusters lose their operational visibility — I know exactly which component failed and why.

The hub-and-spoke model is worth the added complexity. One Grafana to check instead of four. One ArgoCD to manage instead of four. One Harbor to scan instead of four. The operational overhead drops dramatically once the initial setup is done.

If you're managing more than two clusters, centralize your tooling. Your future self — the one debugging a production incident across three clusters at midnight — will thank you.

This article is based on building a hub-and-spoke platform with 4 production RKE2 clusters. All project names, secrets, credentials, and identifying details have been anonymized.

Questions? Hit the comments — I've probably run into the same issue.