Originally published at woitzik.dev
Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.
The idea was elegant: three Raspberry Pi 5 nodes running k3s control-plane + embedded etcd, replacing the three Proxmox VMs. Lower power consumption, physical separation from the main host, and a genuine HA cluster spread across real hardware.
It didn't work. The failure wasn't dramatic — no kernel panic, no cluster death. It was a slow accumulation of fragility that made the cluster less reliable than the single-node setup it replaced.
View the complete homelab infrastructure source on GitHub 🐙
The Attempt
The plan (ADR-014, Option B): run k3s embedded etcd on three Raspberry Pi 5 (8GB)* nodes. The Pis already handle DNS (AdGuard + Unbound) and Keepalived VIP. Adding k3s control-plane seemed like a natural extension.
The k3s cluster spec:
- 3x Raspberry Pi 5 (8GB)
- 256GB microSD cards (A2 rated)
- Gigabit Ethernet via USB 3.0 adapter (Pi 5's native Ethernet is limited)
- k3s v1.31 with embedded etcd
Why It Failed: SD Card Fragility
etcd is write-heavy. Every Kubernetes API operation — pod scheduling, configmap updates, secret rotations — generates etcd writes. The k3s embedded etcd writes continuously to the local filesystem.
SD cards have limited write endurance. The A2-rated cards I used are rated for ~150 MB/s sequential write, but the random 4K write IOPS that etcd generates are a different story. Under sustained write load, the SD card's write amplification factor increases, the garbage collection cycle can't keep up, and write latency spikes.
The symptom: etcd response times would occasionally jump from 10ms to 500ms+ for no apparent reason. No CPU load, no network congestion, no memory pressure. Just slow disk writes. On a NVMe-backed VM, etcd writes complete in microseconds. On an SD card, they're orders of magnitude slower.
etcd has a built-in leader election timeout (default 5s). When etcd write latency exceeds the election timeout, the leader steps down and a new election starts. If the new leader is also on an SD card with the same write latency problem, the election can fail too. The result: brief periods where the Kubernetes API server is unavailable, even though all nodes are "healthy."
Why It Failed: Network Latency
The three Pis communicate over Gigabit Ethernet via USB 3.0 adapters. The adapter adds ~0.5ms of latency per hop compared to native Gigabit. For etcd consensus, which requires a majority of nodes to acknowledge each write, the added latency compounds:
- Write to leader: 0.5ms
- Leader replicates to 2 followers: 0.5ms × 2 = 1ms
- Followers acknowledge: 0.5ms × 2 = 1ms
- Total round-trip: ~2.5ms
Compare to three VMs on the same NVMe-backed host: ~0.1ms round-trip. The 25x latency increase doesn't matter for normal API operations, but it matters during high-write periods or leader elections, exactly when low latency is most critical.
Why It Failed: Resource Contention
The Pis were already running AdGuard + Unbound for network DNS, Keepalived for the VIP, and node_exporter for monitoring. Adding k3s control-plane + etcd meant four services competing for the same CPU, memory, and — critically — the same SD card.
When AdGuard's DNS cache expired and refreshed simultaneously with an etcd compaction cycle, both hit the SD card at once. The resulting I/O contention produced the same pattern as the Proxmox host freeze: etcd write latency spikes → leader election → temporary API unavailability.
The Deliberate Reversion
After three weeks of monitoring, the k3s-on-Pi cluster was less reliable than the single-node setup it replaced. The decision (ADR-014): revert to a single k3s control-plane on the Proxmox VM (vm-srv-k3s-11), with two agent-only worker nodes.
# vm-srv-k3s-11: control-plane + etcd (sole server)
# vm-srv-k3s-12: agent only
# vm-srv-k3s-13: agent only
The single-server setup has a known limitation: if k3s-11 goes down, the entire cluster is down. No failover, no HA. But "down" in a homelab context means "inaccessible for a few minutes while I restart the VM" — not "data center outage affecting thousands of users."
The trade-off: reliability (single NVMe-backed VM) over availability (three SD-card-backed Pis). For a homelab, reliability is the right priority.
What the Pis Do Instead
The Raspberry Pis remain in the rack, running DNS and Keepalived — exactly what they're good at:
- AdGuard Home: DNS filtering for the entire network, low write volume, well within SD card endurance
- Unbound: Recursive DNS resolver, almost entirely read operations after cache warm-up
- Keepalived: Active/passive VIP failover, heartbeat-only, negligible I/O
- node_exporter: System metrics, read-only
These workloads are low-write, low-latency-sensitive, and can tolerate brief interruptions without affecting the cluster. The Keepalived VIP failing over to the other Pi is a 1-second blip. The k3s API server going down for 30 seconds during an etcd election breaks every kubectl command and ArgoCD sync.
The General Lesson
etcd's requirements are specific: low-latency, high-endurance storage with consistent write performance. NVMe provides this. SATA SSDs provide this (mostly). SD cards and USB-attached storage do not, because their write latency is unpredictable under sustained load.
If you're running k3s on Raspberry Pis, the stable configuration is: Pis as agent-only workers, control-plane on a VM or dedicated x86 box with proper storage. The Pis handle DNS, monitoring, and lightweight workloads — things where an SD card's write endurance is adequate and a brief interruption doesn't cascade into a cluster-wide event.
The same logic applies to etcd on cloud VMs: don't put etcd on Standard HDD. The IOPS and latency guarantees of Premium SSD or Ultra Disk exist specifically because etcd's consensus protocol requires consistent, low-latency writes. The Pi's SD card is the cloud equivalent of a Standard HDD — it works until you need consistent performance under load.
K3s on ARM is production-ready for worker nodes. The control-plane decision is about storage performance and write consistency, not CPU architecture. Azure's AKS control plane runs on the same principle: the control-plane nodes use Premium SSD managed disks precisely because etcd needs consistent, low-latency writes. The underlying hardware doesn't matter — the I/O guarantees do.
Top comments (0)