DEV Community

sanskar arora
sanskar arora

Posted on

🔒 HomeLab Chronicles: Episode 7 - Locked Out of My Own House

#kubernetes #rbac #homelab #microk8s

Hey all 👋

Short episode. Painful episode. The kind where the call is coming from inside the house, the house is the control plane, and nobody inside the house is allowed to answer the phone.

Fresh rebuild, Flux repo ready, and I did the responsible thing: enabled RBAC while the cluster was empty. Every best-practices doc agrees — turn on authorization early, before workloads exist, while it's painless.

Reader, it was not painless.

🚨 Symptom One: Flux Bootstrap Times Out 

✗ timeout waiting for: [CustomResourceDefinition/kustomizations... status: 'InProgress', ...]
Enter fullscreen mode Exit fullscreen mode

Eleven CRDs, all stuck. But committed and pushed fine — the repo half worked, the cluster half hung. And when I inspected a stuck CRD:

status:
  acceptedNames: {kind: "", plural: ""}
  conditions: null
Enter fullscreen mode Exit fullscreen mode

conditions: null. Not "failing." Not "pending." Never touched. On a healthy cluster the apiserver stamps a CRD as Established within milliseconds. These sat blank, like forms nobody would ever process.

🚨 Symptom Two: A Deployment Frozen in Time 

$ kubectl -n kube-system get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
coredns    0/1     0            0           5h7m
Enter fullscreen mode Exit fullscreen mode

Five hours old. UP-TO-DATE: 0. That column being zero means no ReplicaSet was ever created — which means the deployment controller never even looked at it. And with no DNS pod, everything downstream was queued up to fail alphabetically.

🔍 The Logs Confess Immediately 

E... leaderelection.go: "Error retrieving lease lock"
  err="leases... is forbidden: User \"system:kube-controller-manager\"
  cannot get resource \"leases\" in namespace \"kube-system\""

E... "Failed to watch" err="csidrivers... is forbidden:
  User \"system:kube-scheduler\" cannot list resource \"csidrivers\""
Enter fullscreen mode Exit fullscreen mode

Sit with that. The controller-manager — the component whose entire job is running every controller in Kubernetes — was forbidden from acquiring its own leader-election lease. No lease, no leadership; no leadership, no controllers; no controllers, no ReplicaSets, no pods, no anything. The scheduler was equally banned from reading the objects it schedules around.

I enabled authorization, and the first citizens it deported were the government.

🧩 The Part That Made It Weird

The obvious theory: missing ClusterRoleBindings. Checked. They existed. Correct subjects (User: system:kube-controller-manager), correct roles, roles containing exactly the permissions being denied. Checked the client certificates on disk:

subject=CN=system:kube-controller-manager
subject=CN=system:kube-scheduler
Enter fullscreen mode Exit fullscreen mode

Perfect CNs. Perfect bindings. Perfect roles. Still forbidden.

Then the tell. I tried impersonation to test permissions, and even that failed:

selfsubjectaccessreviews is forbidden: User "system:kube-controller-manager"
cannot create resource "selfsubjectaccessreviews"
Enter fullscreen mode Exit fullscreen mode

That permission comes from system:basic-user, bound to system:authenticated — the most universal binding in all of Kubernetes. Literally everyone gets it. When that denies, the story isn't "a binding is missing." The story is the RBAC authorizer isn't loading policy at all. Every rule in the cluster: correct, present, and completely unread.

Meanwhile kubectl get --raw='/readyz?verbose' said readyz check passed, because my admin cert (hello, system:masters) bypasses authorization entirely. The apiserver was healthy. The apiserver was also the only one allowed in the building. Health checks measure the doorman, not whether anyone can get past him.

🧯 The Retreat 

sudo microk8s disable rbac
sudo microk8s stop && sudo microk8s start
$ grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver
--authorization-mode=AlwaysAllow
Enter fullscreen mode Exit fullscreen mode

Forbidden errors: gone. Lease: held and renewing. And coredns... still 0/1, UP-TO-DATE 0. Because five hours of objects written into a paralyzed control plane don't spring to life on forgiveness — they'd been created into a void, and some of them (plus a growing chorus of node "msi" not found) were simply inert. At that point the cluster was 24 hours old and held nothing I loved, so I invoked Episode 6's superpower: reset, rejoin, bootstrap. Twenty minutes. This is what the git repo was for.

The rebuilt version skips enable rbac, and adds the check I'll never skip again: coredns must reach 1/1 before anything else happens. It's the cheapest possible proof that the datastore persists writes and the controllers act on them. One deployment, one honest answer.

🧠 What This Taught Me

  • readyz measures the apiserver, not the cluster. A green apiserver can preside over a completely dead controller plane, especially when your admin cert skips the auth queue.
  • UP-TO-DATE: 0 on an old deployment is a five-alarm signal. It doesn't mean "unhealthy pods." It means "no controller has ever processed this."
  • When even system:basic-user is denied, stop auditing bindings. Universal denial means the authorizer itself, not the policy.
  • A homelab without RBAC is a defensible trade. It governs in-cluster identities, not network exposure; the API server is still cert-authenticated. MicroK8s ships it off by default, and I now understand that as a mercy, not an oversight.
  • "Enable it while it's painless" assumes it works. Best practice on paper, lockout in my kitchen.

📋 Quick Reference (For Skimmers)

Purpose Command
Is the controller plane actually alive kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.renewTime}' — must be seconds old
The canary kubectl -n kube-system get deploy coredns → wants 1/1
CRD ever processed? -o jsonpath='{.status.acceptedNames.kind}' — empty = never
Current auth mode grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver
Component identities decode client-certificate-data from /var/snap/microk8s/current/credentials/*.config → openssl x509 -noout -subject

🚀 What's Next

Cluster rebuilt, Flux green, Airflow installing... and then Helm taught me its favorite trick: accepting configuration it has no intention of using, silently, with a straight face. Next episode is about every value I set that did absolutely nothing.

💬 Final Thoughts

I gave my cluster a security system. The security system's first act was arresting the janitor, the electrician, and itself. The house stood perfectly secure and perfectly dark.

Popcorn 🍿, coffee ☕, UPS 🔌, and a laminated card that says check coredns first.

Top comments (0)