DEV Community

sanskar arora
sanskar arora

Posted on

🔦 HomeLab Chronicles: Episode 5 - Blackout

#kubernetes #homelab #dqlite #devops

Hey all 👋

Last episode ended on a high. Airflow UI on my phone, logs working, DAGs syncing, and a solemn promise: Dell out of the drawer, Pi out of the other drawer, Cilium out of my dreams and into the cluster.

Here's the scoreboard. The Dell came out of the drawer — actually joined the cluster, same night I published. The Pi remains in its drawer, thriving, unbothered. Cilium remains a concept I respect from a distance. One out of three. In this economy, that's a win.

And then at 05:22 the next morning, the power went out. And when it came back, my cluster had amnesia.

$ kubectl get deployments -A
NAMESPACE     NAME                      READY
kube-system   calico-kube-controllers   1/1
Enter fullscreen mode Exit fullscreen mode

That's it. That's the whole cluster. Airflow: gone. Longhorn: gone. My will to live: buffering.

🕵️ The Crime Scene

First instinct: the datastore got corrupted by the power cut. Reasonable. Dramatic. Wrong — but let's earn that.

MicroK8s doesn't use etcd; it uses dqlite — Raft-replicated SQLite, living at /var/snap/microk8s/current/var/kubernetes/backend/. So that's where I went digging:

$ sudo ls -la /var/snap/microk8s/current/var/kubernetes/backend/
-rw-rw-r-- cluster.crt        Aug 28 21:42
-rw-rw---- cluster.key        Aug 28 21:42
-rw-rw---- metadata1          Aug 28 21:42
-rw-rw---- 0000000000074753-0000000000075264   ...
Enter fullscreen mode Exit fullscreen mode

Read those timestamps again. The database certificates were born at 21:42 the night before. The raft index was in the seventy-thousands — a day of idle heartbeats, not weeks of a running cluster. And:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   16h
kube-node-lease   Active   16h
kube-public       Active   16h
kube-system       Active   16h
Enter fullscreen mode Exit fullscreen mode

Four namespaces. The four a cluster is born with. All sixteen hours old.

The plot twist, delivered by my own filesystem: nothing was lost in the power cut, because nothing was there. When I joined the Dell the previous night, the datastore got reinitialized underneath me. The database I spent the morning mourning was an empty sixteen-hour-old husk. The power cut didn't rob the house. The power cut just knocked on the door of a house that was already empty.

Grief has five stages. ls -la has one.

📡 So What Did the Power Cut Do?

Something sneakier. Look at the dqlite logs from 05:22:

dial: Failed to connect to HTTP endpoint: dial tcp 192.168.68.210:19001:
connect: network is unreachable
Enter fullscreen mode Exit fullscreen mode

Repeated. For a while. Now, the tempting read is "boot race — Wi-Fi wasn't up yet when dqlite started." I almost bought it. But the dqlite process was PID 1919211. Kubelite was PID 2.3 million. You don't get seven-digit PIDs on a fresh boot. That machine had been up for eight hours when this happened.

So it wasn't a boot race. The network vanished mid-run. And here's the geometry of my particular disaster:

My nodes are laptops. My router is not.

When the power cuts, the MSI and the Dell shrug and switch to battery. The router — a small plastic box with the survival instincts of a houseplant — dies instantly. Wi-Fi drops, the interface loses its address, and dqlite, which is bound to 192.168.68.210:19001, suddenly can't reach itself.

And why is it bound to the LAN address at all? Because of the Dell. dqlite binds to 127.0.0.1 by default — a fresh single-node MicroK8s is structurally immune to network outages. The moment a second node joins, dqlite moves to the LAN IP, because the other node has to reach it. Joining the Dell put my datastore on Wi-Fi. The drawer had a curse on it.

🔋 The Villain Nobody Suspects

But wait — a frozen datastore recovers when the network comes back. Frozen isn't corrupted. Where does actual damage come from?

Follow the battery. Power cut → laptops keep running → outage outlasts the battery → laptop hits 0% → hard poweroff. No filesystem sync. No dqlite checkpoint. Just lights out, mid-write. That is the dirty shutdown that corrupts a SQLite-family datastore, and it doesn't happen during the outage. It happens hours later, quietly, while you're asleep and feeling safe.

The fix has absolutely nothing to do with Kubernetes. It's laptop power management:

# /etc/UPower/UPower.conf
UsePercentageForPolicy=true
PercentageLow=25
PercentageCritical=22
PercentageAction=20
CriticalPowerAction=PowerOff
Enter fullscreen mode Exit fullscreen mode

Clean shutdown at 20% instead of a crash at 0%. (UPower insists Low > Critical > Action, so mind the ordering.) Plus telling logind that a closed lid on a server is decor, not a command:

# /etc/systemd/logind.conf.d/99-homelab.conf
[Login]
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
IdleAction=ignore
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
Enter fullscreen mode Exit fullscreen mode

And a longer shutdown grace so dqlite gets to checkpoint instead of eating SIGKILL at the 90-second default:

# /etc/systemd/system.conf.d/99-shutdown-timeout.conf
[Manager]
DefaultTimeoutStopSec=120s
Enter fullscreen mode Exit fullscreen mode

🧠 The Mental Model I Was Missing

The reason "my deployments are gone" felt so confusing is that three completely different things all get called "the cluster":

Layer Lives in Survives a reboot?
Cluster state — Deployments, Services, routes dqlite, on disk Yes, if dqlite comes up clean
Data — Postgres, volumes Longhorn, on disk Yes
Pods Nowhere. Vapor. No — and that's correct

Pods are supposed to vanish and be recreated from Deployments. That's not fragility, that's the design. The only question that ever matters after an outage is: did layer one come back intact? Everything else follows from it.

And ranked by what actually threatens layer one:

  1. Battery draining to a dirty poweroff ← the real one
  2. Network loss while running (freezes it; recovers)
  3. Clean reboot (total non-event)

I'd been emotionally preparing for #3, mildly worried about #2, and completely blind to #1.

💾 And Because Paranoia Is Now a Feature

dqlite has a backup tool nobody talks about:

microk8s dbctl backup /var/backups/microk8s/dqlite-$(date +%Y%m%d-%H%M%S)
Enter fullscreen mode Exit fullscreen mode

Mine now runs nightly from a systemd timer, and building that had its own comedy: /snap/bin isn't on systemd's default PATH, so the naïve version dies at 3am with command not found, silently, forever — a backup job that never runs but that you believe exists is strictly worse than no backup job. Absolute paths, artifact-size verification, a free-space guard, 7-day retention with a keep-minimum floor, and chmod 700 on the directory because a full cluster dump contains every Secret you own.

Oh — and copy them off the node. A backup on the disk that died is a souvenir.

🧠 What This Taught Me

  • Check timestamps before you grieve. ls -la on the datastore told me in ten seconds what I'd theorized about for an hour.
  • Joining a second node moves dqlite from loopback to the LAN. Single-node clusters are accidentally immune to network chaos. Multi-node clusters opt into it. Nobody sends you a consent form.
  • Laptops-as-servers invert the failure mode. The nodes survive the outage; the router is the single point of failure, and the battery turns a power cut into a delayed-action dirty shutdown.
  • Pods vanishing is fine. The question is always the datastore.
  • A silent backup job is worse than none. Verify the artifact exists, verify it has bytes, and scream loudly if not.

📋 Quick Reference (For Skimmers)

Purpose Command
Where dqlite lives /var/snap/microk8s/current/var/kubernetes/backend/
Who dqlite binds to sudo cat .../backend/cluster.yaml
Datastore's actual age sudo ls -la .../backend/ — read cluster.crt's date
Snapshot the datastore microk8s dbctl backup <path>
Restore it microk8s stop && microk8s dbctl restore <path> && microk8s start
Stop battery murder UPower PercentageAction=20, CriticalPowerAction=PowerOff
Lid ≠ off switch logind HandleLidSwitch=ignore + mask sleep targets

🚀 What's Next

If the cluster can be vaporized by a plastic router and a tired battery, then the cluster can't be the source of truth anymore. Next episode: I put the entire thing in git and make "everything is gone" a twenty-minute inconvenience instead of a personality crisis.

💬 Final Thoughts

The power cut was innocent. The router was negligent. The battery was armed. And the database was already empty before any of them showed up.

Stay tuned. Popcorn 🍿, coffee ☕, and now a UPS 🔌 — the shopping list grows with the wisdom.

Top comments (0)