Three machines sit quietly in a corner of my house. Together, they run a platform that rarely stays quiet.
The physical cluster is simple:
k3s-cpk3s-w1k3s-w2
All three nodes participate in the K3s control plane and embedded etcd.
That detail matters. I did not want a lab that only looked like production from a distance. I wanted something small enough to own completely, but real enough to expose the failure modes that architecture diagrams often hide.
The shape of the platform
The cluster uses kube-vip for the control-plane virtual IP and MetalLB for local LoadBalancer services.
Argo CD is the canonical reconciler. Git is the desired state, and the cluster is expected to converge back to it.
Forgejo and self-hosted runners handle CI.
Longhorn provides distributed persistent storage. Velero is part of the backup and recovery path.
Vault holds secrets, and External Secrets delivers them into Kubernetes at runtime so secret values do not need to live in Git.
Kyverno provides policy enforcement.
Prometheus, Grafana, and Loki give me metrics, dashboards, and logs when something goes wrong.
Cloudflare Tunnel provides controlled external access where I need it.
None of those tools are especially interesting in isolation.
The interesting part is operating them together.
What the lab is actually for
I use this environment to test the things that are easy to talk about and harder to understand until they fail:
- losing a node
- maintaining etcd quorum
- reconciling drift
- recovering persistent workloads
- restoring from backup
- debugging a bad deployment
- validating secret delivery
- testing policy failures
- understanding what observability looks like during an actual incident
A successful deployment teaches you that the happy path works.
A broken deployment teaches you where ownership really sits.
A restore drill teaches you whether your backup story was ever real.
A quorum problem teaches you very quickly that "three nodes" is not the same thing as "high availability" unless you understand how the control plane behaves.
That is the value of the lab for me.
Why keep this at home?
Because failure is cheap enough to study here.
I can break the control plane without opening an incident bridge. I can rebuild from Git without negotiating a maintenance window. I can test recovery assumptions against hardware I physically own.
That freedom creates a useful engineering loop:
build, break, observe, recover, document, repeat.
The goal is not to collect self-hosted services.
The goal is to create evidence that I can operate systems, not just describe them.
Small machines. Real systems.
I am also teaching the intersection of AI, automation, DevOps, platform engineering, and modern engineering workflows during AI Leverage Week, October 5 to 9 at 7 PM Eastern.
Registration: https://tayoca.com/aiweek
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support