DEV Community: sanskar arora

🐈 HomeLab Chronicles: Episode 11 - schrodingers cluster

sanskar arora — Mon, 02 Nov 2026 22:28:27 +0000

Hey all 👋

Last episode was an RCA with seven root causes and a port number. This one is an RCA with one root cause, five wrong theories, and a cluster that reported itself healthy for thirty-eight hours while doing absolutely nothing.

State of the lab going in: two nodes, Flux in charge, Airflow on a public domain through a Cloudflare tunnel, nightly backups, DHCP reservations finally ticked. Episode 10 ended with "the router remains the weakest member of the team." I would like to amend that statement. The router has competition.

Here's what happened, in the order I found out, which is not the order it happened.

🚨 Symptom: The Tunnel Is Down

Sunday night, 10 PM. Cloudflare's Zero Trust dashboard, which has shown two happy connectors since Episode 9, says the tunnel is down. No connectors. Nothing.

First look at the cluster:

$ kubectl get nodes
NAME   STATUS     ROLES    AGE   VERSION
dell   NotReady   <none>   64d   v1.35.6
msi    Ready      <none>   64d   v1.35.6

Dell is down. Has been for two and a half days, it turns out. And the pods tell the failover story exactly the way the textbook says it should go:

$ kubectl -n cloudflared get pods -o wide
NAME                           READY   STATUS        RESTARTS   AGE     NODE
cloudflared-6cf54796d6-42rg9   1/1     Running       0          2d16h   msi
cloudflared-6cf54796d6-7g425   1/1     Terminating   0          5d9h    dell
cloudflared-6cf54796d6-jkt44   1/1     Running       0          2d16h   msi
cloudflared-6cf54796d6-t567p   1/1     Terminating   0          5d9h    dell

Dell died, the old pods went Terminating, two fresh replicas came up on msi 2d16h ago. Running. 1/1. Zero restarts. Kubernetes did its job.

Small aside before we continue. Episode 9 proudly said: two replicas with pod anti-affinity so the tunnel survives a node dying. Look at the ages. Both old replicas were on dell. Anti-affinity on a two-node cluster is a preference, and the scheduler preferred dell twice. The tunnel survived the node dying anyway, because failover worked. It just didn't survive what came next.

So the question was simple. Two healthy connectors on msi, dashboard says zero. Why?

🔬 Hypothesis 1: The Failover Path Was Never Tested

Both replicas had always lived on dell. This was the first time cloudflared had ever run from msi. Maybe msi's egress was different: UDP 7844 blocked, QUIC failing, DNS for argotunnel.com not resolving from msi's pods.

The test: read the pod's log. The fix if true: TUNNEL_TRANSPORT_PROTOCOL=http2 to force TCP, or fix whatever was eating UDP.

What happened instead:

$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9
Error from server: Get "https://192.168.0.201:10250/containerLogs/...":
tls: failed to verify certificate: x509: certificate is valid for 192.168.68.210, not 192.168.0.201

Episode 10, Sidequest 2, verbatim, except on the other node. When I moved subnets, I fixed dell's kubelet certificate because that's where the pod I wanted to read was. msi's kubelet had the same stale certificate the entire time, and nobody noticed because nobody asked msi for logs. kubectl get doesn't touch the kubelet. kubectl logs does.

Fine. The host has the logs on disk. Over ssh:

sudo tail -n 80 /var/log/pods/cloudflared_cloudflared-6cf54796d6-42rg9_*/cloudflared/0.log

And the log was a complete short story:

2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=1 ... location=blr03 protocol=quic
2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=2 ...
2026-10-31T01:11:59Z INF Registered tunnel connection connIndex=3 ...
2026-10-31T01:12:03Z INF |  SUMMARY: Environment is healthy. cloudflared will use 'quic' as primary protocol.  |
...
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=0
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=3
2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=2
2026-11-01T07:24:54Z ERR Serve tunnel error error="... timeout: no recent network activity" connIndex=1
...
2026-11-01T07:25:19Z INF Initiating graceful shutdown due to signal terminated ...
2026-11-01T07:25:22Z ERR no more connections active and exiting
2026-11-01T07:25:22Z INF Tunnel server stopped

Hypothesis 1, dead. Failover worked perfectly. The msi replica registered four connections on Friday morning and every pre-check passed. Then on Saturday at 12:54 PM (the logs are UTC; I'm not), all four connections timed out in the same second, a SIGTERM arrived twenty-five seconds later, and the process exited cleanly.

And then nothing. No new container. The log file ends there. For thirty-eight hours, the API said Running, 0 restarts about a process that had logged its own obituary.

That changed the question. It was no longer "why is the tunnel down." It was "why did the kubelet not restart a dead container, and why is it lying about it?"

📡 Interlude: Who Killed It

Four connections dying in the same second is not Cloudflare. That's the floor falling out from under the network. msi's journal for that minute:

Nov 01 07:24:49 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-BEACON-LOSS
Nov 01 07:24:50 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-DISCONNECTED bssid=... reason=4 locally_generated=1
Nov 01 07:25:32 msi microk8s.daemon-kubelite[1122435]: ...   <- kubelite starting
Nov 01 07:25:43 msi microk8s.daemon-kubelite[1122802]: ...   <- and again
Nov 01 07:25:53 msi microk8s.daemon-kubelite[1123187]: ...   <- and again
Nov 01 07:25:54 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-CONNECTED
Nov 01 07:26:55 msi microk8s.daemon-kubelite[1123923]: ...   <- fourth time's the charm
Nov 01 07:28:19 msi kernel: wlo1: deauthenticated from ... (Reason: 7=CLASS3_FRAME_FROM_NONASSOC_STA)
Nov 01 07:28:19 msi kernel: iwlwifi 0000:00:14.3: 0x004D9C74 | branchlink2

Wi-Fi power save. The laptop's radio napped through too many beacons, decided the router was gone, and disconnected itself (reason=4 locally_generated=1, which is the card saying "this was my idea"). When the address vanished, so did the thing dqlite is bound to. Episode 5 explained this exact mechanism: a single-node MicroK8s keeps its datastore on loopback and is immune to this, and the day the Dell joined, dqlite moved to the LAN address and became a hostage of the Wi-Fi card. kubelite crash-looped four times in two minutes. Then the router kicked msi off twice more (reason=7, the router's idea this time), with an Intel firmware crash dump for garnish.

Somewhere in that mess, cloudflared got its SIGTERM. I never pinned down who sent it; my best guess is a liveness probe doing its job during the chaos. It doesn't matter much. What matters is that the kubelet came back after the fourth restart and then did nothing, for a day and a half.

The Episode 5 fix for the symptom, applied on the spot:

sudo iw dev wlo1 set power_save off

The actual fix is a cable. We'll get there.

🪪 Blocker: The Certificate, Again

Before any more theories I needed kubectl logs and exec working on msi, because a debugging session without them is archaeology. Episode 10's trick was --rotate-server-certificates=true: let the kubelet request a fresh cert, approve the CSR, done. On msi it produced a different error:

Error from server: Get "https://192.168.0.201:10250/containerLogs/...": remote error: tls: internal error

$ kubectl get csr
No resources found

tls: internal error is the kubelet saying "rotation is on, I have nothing to serve yet, and I'm not going to serve the old file either." And it never filed a CSR. No error about why. Just a kubelet politely waiting for a certificate it never asked for. Hold that thought, because in hindsight it's the same bug wearing a different hat.

I stopped debugging rotation and signed the certificate by hand with the cluster CA. Same private key, new SAN list:

cd /var/snap/microk8s/current/certs
sudo cp -p kubelet.crt kubelet.crt.bak

printf '%s\n' \
  'basicConstraints=CA:FALSE' \
  'keyUsage=critical,digitalSignature,keyEncipherment' \
  'extendedKeyUsage=serverAuth' \
  'subjectAltName=DNS:msi,IP:192.168.0.201,IP:127.0.0.1' > /tmp/kubelet-ext.cnf

sudo openssl req -new -key kubelet.key -subj "/CN=msi" -out /tmp/kubelet.csr
sudo openssl x509 -req -in /tmp/kubelet.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -CAserial /tmp/kubelet-ca.srl -days 3650 -sha256 \
  -extfile /tmp/kubelet-ext.cnf -out /tmp/kubelet.crt

sudo install -m 660 -o root -g microk8s /tmp/kubelet.crt kubelet.crt
sudo sed -i '/^--rotate-server-certificates=true$/d' /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite

$ echo | openssl s_client -connect 192.168.0.201:10250 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:msi, IP Address:192.168.0.201, IP Address:127.0.0.1

Certificate fixed. kubectl logs again:

$ kubectl -n kube-system logs deploy/coredns --tail=3
Error from server (NotFound): the server could not find the requested resource ( pods/log coredns-78894c95f4-l8n8j)

A 404. From the kubelet. The TLS handshake succeeded, the API server asked msi's kubelet for coredns's logs, and the kubelet said "never heard of it." About a pod the API says has been running on msi for 64 days.

That's not a certificate problem. That's the kubelet not knowing what it's running.

🔬 Hypothesis 2: The Kubelet Has Lost Its Pods

The test: ask the kubelet directly for its pod list, through the API server's node proxy, and compare with what the API thinks.

sudo microk8s kubectl get pods -A --field-selector spec.nodeName=msi -o name | wc -l
# 34
sudo microk8s kubectl get --raw /api/v1/nodes/msi/proxy/pods
# {"kind":"PodList", ..., "items": null}

Thirty-four pods in the API. Zero in the kubelet. Not "some missing." Null. And the kubelet's journal, every ten seconds, since a very specific moment:

Nov 01 07:26:57 msi kubelite: E1101 07:26:57 kubelet_node_status.go:392]
  "Error getting the current node from lister" err="node \"msi\" not found"

07:26:57. Two seconds after the fourth kubelite restart on Saturday. The kubelet can't find its own node, and it's been saying so 15,000 times while kubectl get nodes says msi Ready.

Confirmed, then: the kubelet's view of the cluster is empty. Everything downstream follows. It won't restart a container belonging to a pod it doesn't know. It won't write status for it, so the API keeps showing the last status it ever wrote (the frozen Running, 0 restarts). It answers log requests with 404. And meanwhile containerd, which doesn't care about any of this, keeps running whatever was already running:

$ sudo microk8s ctr tasks ls | awk 'NR>1{print $3}' | sort | uniq -c
     59 RUNNING

Airflow was fine. Postgres was fine. Fifty-nine containers happily running with no supervisor. The only things missing were the ones that had died, which nobody was around to restart.

But "the kubelet's view is empty" is a symptom. The question is why.

🔬 Hypothesis 3: The API Server's Watch Cache Is Empty

Quick mechanism, because the rest of the post depends on it. The API server has two ways to answer a read. A plain GET goes to the datastore. But every component that watches the cluster (the kubelet, the controller-manager, the scheduler, Flux) reads through the watch cache: an in-memory copy of the datastore that the API server keeps current by watching the datastore itself. Thousands of watchers, one datastore watch. That's what lets Kubernetes scale. You can also read from it on purpose, by asking for resourceVersion=0.

If that cache were empty, you'd get exactly this: kubectl sees a healthy cluster (datastore), every component sees nothing (cache).

The test: the same question, both ways. The fix if true: restart kubelite, since the cache is rebuilt on every API server start.

$ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi' | ...
34
$ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ...
34

Dead. The cache had all 34 pods. (I had also restarted kubelite twice by then, so an empty cache would have refilled anyway. Keep that in mind; it comes back.)

🔬 Hypothesis 4: The Kubelet's Credentials Are Broken

The API server answers me fine. Maybe it doesn't answer the kubelet. Episode 7 was an entire episode about the control plane being forbidden from reading its own objects, so this had form.

The test: replay the kubelet's exact startup queries using the kubelet's own kubeconfig. The fix if true: whatever the error says, probably certificates or RBAC.

$ K="sudo /snap/microk8s/current/kubectl --kubeconfig /var/snap/microk8s/current/credentials/kubelet.config"
$ $K auth whoami
Username   system:node:msi
Groups     [system:nodes system:authenticated]
$ $K get --raw '/api/v1/nodes?fieldSelector=metadata.name%3Dmsi&resourceVersion=0' | ...
1 ['msi']
$ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ...
34
$ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&watch=1&resourceVersion=0' | head -c 200
{"type":"ADDED","object":{"kind":"Pod", ...

Dead. The kubelet's identity gets the node, gets all 34 pods, and can open a watch. Every request the kubelet could make works when I make it.

So the kubelet isn't making them.

🔬 Hypothesis 5: The Kubelet's Informers Are Failing

Each component keeps its own local cache of the objects it cares about (an informer): one LIST to fill it, then a WATCH to keep it current. The kubelet won't touch pods until its informers have loaded once. If the LIST or WATCH were failing, the kubelet would log it and retry. Loudly.

The test: grep for reflector errors. The fix if true: depends on the error, but at least there'd be one.

$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" \
    | grep -E 'reflector|failed to list|Failed to watch|forbidden|deadline exceeded'
(nothing)

Dead, in a way that was more informative than an error would have been. Nothing was failing. And the controller-manager, which runs in the same kubelite process, told me what "nothing" looked like from the inside:

$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" | grep -E 'Waiting for caches to sync|Caches are synced'
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync"
... forty of these, all from the same second ...

Fifty minutes after startup. Forty controllers waiting for caches to sync. Zero Caches are synced. Every informer in the entire control plane was stuck in the waiting room, not erroring, just waiting. Which, incidentally, is why kubectl rollout restart deployment/cloudflared an hour earlier had done precisely nothing: the deployment controller was in the same waiting room.

This also explains the certificate rotation that never filed a CSR. The kubelet's cert manager needs the node informer to have synced before it can build the request. Same wait, different symptom.

Four hypotheses down. Every test says the API server is fine, the data is fine, the credentials are fine, nothing is failing, and nothing is happening.

🧊 Hypothesis 6: The Cache Isn't Empty. It's Frozen.

Here's the hole in Hypothesis 3. When an informer starts, it doesn't ask for "whatever the cache has." It asks for the current state, and the API server, before answering, waits for its watch cache to catch up to the datastore's latest revision. A fresh snapshot, guaranteed. My resourceVersion=0 test never asked for that guarantee. It asked for the stale copy and got it.

A frozen cache, one that took a snapshot and then stopped receiving updates, would pass my test and hang every informer forever. And my test couldn't tell frozen from live because nothing in the cluster was changing, because everything that changes things was stuck waiting on the cache. Circular. Beautiful. Infuriating.

To catch it I needed an object that was still being written. The kubelet renews its node lease every ten seconds, straight to the datastore, informers or no informers.

The test: read the lease from the datastore, then from the cache.

$ date -u
Mon Nov  2 21:19:24 UTC 2026
$ ... leases/msi               -> rv 3815025  renewTime 2026-11-02T21:19:19Z
$ ... leases/msi?resourceVersion=0 -> rv 3810400  renewTime 2026-11-02T20:23:53Z

Confirmed. The datastore's copy was five seconds old. The cache's copy was fifty-five minutes old, from 20:23:53, which is three seconds before I had restarted kubelite for the certificate. The cache took one snapshot at startup and then received nothing. Four thousand six hundred revisions behind and counting.

Every informer in kubelite was waiting for a cache to catch up to a datastore it had stopped listening to.

And why did restarting kubelite not fix it, twice? Because the cache is fed by a watch stream from k8s-dqlite, which is its own snap service:

$ systemctl show snap.microk8s.daemon-k8s-dqlite -p ActiveEnterTimestamp
ActiveEnterTimestamp=Wed 2026-10-28 07:07:22 UTC

Running since the Episode 10 fix. The one component in the whole chain that had never restarted. And its log from Saturday, during the Wi-Fi hole:

Nov 01 07:32:33 k8s-dqlite: [go-dqlite] attempt 10: server 192.168.0.201:19001: dial: connect: network is unreachable
Nov 01 07:32:33 k8s-dqlite: error in txn: update transaction failed for key /registry/longhorn.io/nodes/longhorn-system/msi: context canceled
Nov 01 07:32:42 k8s-dqlite: error while range on /registry/serviceaccounts/: query (try: 0): context canceled

The datastore, unable to reach its own address, cancelling its own queries. Episode 10's dial ... i/o timeout to your own old IP, except this time the IP was correct and simply didn't exist for ninety seconds.

k8s-dqlite is really kine, a shim that makes dqlite speak etcd's API. kine finds new writes by polling its table and pushes them to anyone watching, including the API server's cache. Somewhere in that context canceled storm, the push stopped, while reads and writes carried on working. Every kubelite restart after that got a fresh cache with a perfect snapshot and a dead feed. I never found the exact line of kine that wedged. I found the fix, which was good enough at 3 AM.

🔧 The Fix

Restart the datastore, with the API server stopped so it isn't holding connections mid-stream, then bring the API server back to build a cache on a live feed. Containers keep running throughout; containerd doesn't care about any of this.

sudo microk8s dbctl backup /var/backups/microk8s/pre-restart-$(date +%F)   # paranoia is a feature
sudo snap stop microk8s.daemon-kubelite
sudo snap restart microk8s.daemon-k8s-dqlite
sleep 15
sudo snap start microk8s.daemon-kubelite

Ninety seconds later:

$ ... leases/msi               -> rv 3817920  renewTime 2026-11-02T21:50:45Z
$ ... leases/msi?resourceVersion=0 -> rv 3817920  renewTime 2026-11-02T21:50:45Z
$ journalctl ... | grep -c 'Caches are synced'
58
$ journalctl ... | grep 'Watching apiserver'
Nov 02 21:49:13 msi kubelite: I1102 21:49:13 apiserver.go:52] "Watching apiserver"
$ ... /api/v1/nodes/msi/proxy/pods -> 34

Same revision from both read paths. Fifty-eight controllers synced. The kubelet watching the API server again. Thirty-four pods known. And:

$ kubectl -n cloudflared get pods
NAME                           READY   STATUS    RESTARTS      AGE
cloudflared-6cf54796d6-42rg9   1/1     Running   1 (38h ago)   2d20h
cloudflared-6cf54796d6-jkt44   1/1     Running   1 (38h ago)   2d20h

RESTARTS 1 (38h ago). The kubelet woke up, found two dead containers, restarted them, and for the first time in a day and a half wrote a status that was true. The "38h ago" is when the previous container died: Saturday, 12:55 PM, exactly where the log file ended.

$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9 --tail=5
INF Registered tunnel connection connIndex=0 ...
INF Registered tunnel connection connIndex=1 ...
INF Registered tunnel connection connIndex=2 ...
INF Registered tunnel connection connIndex=3 ...

Dashboard: Healthy. Two connectors. Airflow back on its domain.

🧠 What This Taught Me

A cluster has two read paths, and they can disagree for days. kubectl get reads the datastore. Every controller reads the watch cache. When the cache freezes, you get a cluster where every human check passes and every automated action silently stops. Nothing errors, because waiting isn't an error.
"Running, 0 restarts" is a claim the kubelet made once. Pod status is written by the kubelet. If the kubelet stops writing, the API keeps serving the last thing it wrote, indefinitely. Status is a snapshot, not a heartbeat.
Every canary I had lied. Episode 7's readyz was green. Episode 7's coredns was 1/1 (it was frozen there). The controller-manager lease was even renewing (leader election doesn't need informers). The one thing that told the truth was a renewTime read two ways. New laminated card: kubectl get lease -n kube-node-lease <node> with and without resourceVersion=0. They must match.
Restarting the thing that's stuck doesn't help when the stuck thing is downstream. I restarted kubelite twice. The feed it depended on lived in a process I hadn't touched since October. Find the oldest PID in the chain.
Hypotheses die by test, not by argument. Five theories, each plausible, each killed by one command. The frozen cache only became visible when I stopped testing content and started testing freshness, with an object that was still moving.
Anti-affinity on two nodes is a suggestion. Both tunnel replicas lived on the same node for five days. The fix is a topologySpreadConstraint, which is a rule, not a wish.
Joining the second node moved the datastore onto Wi-Fi, and Wi-Fi power save is a slow-motion power cut. Episode 5 said this about the router dying. Turns out the laptop can do it to itself, from the inside, whenever the radio feels sleepy. power_save off is on now. The real fix is a cable into msi, and I've stopped pretending otherwise.
The certificate was never the problem, and I'd have been lost without fixing it. The stale kubelet cert blocked logs and exec, and the rotation trick from Episode 10 failed because of the same frozen cache I hadn't found yet. Hand-signing it with the cluster CA took five minutes and unblocked everything after.

🚀 What's Next

Dell is still NotReady, with its ghost pods stuck in Terminating and a triggerer StatefulSet that won't schedule a replacement until the old pod is confirmed dead. That's its own evening and its own post. After that: a spread constraint and a /ready liveness probe on the tunnel pods, so a dead connector gets restarted instead of memorialized. And a monitoring question I don't have a good answer to yet: what do you alert on when the cluster's own health signals are the thing that froze?

💬 Final Thoughts

For thirty-eight hours my cluster was Schrödinger's: alive if you asked the datastore, dead if you asked anything that actually does work, and perfectly consistent with itself the whole time. The Wi-Fi card took a nap, the datastore cancelled its own queries, the cache stopped listening, and every controller in the building sat down to wait for an update that was never coming. Nobody errored. Nobody complained. Fifty-nine containers kept running on autopilot with no one at the controls, which is either reassuring or terrifying depending on how recently you've slept.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated card 🪪, chart-source bookmark 🔖, a domain 🌐, a DHCP reservation 📌, and now a second laminated card that just says compare the renewTimes.

❓ HomeLab Chronicles: Episode 10 - new router who dis

sanskar arora — Wed, 28 Oct 2026 08:49:28 +0000

Hey all 👋

Last episode ended with a victory lap: a two-node, git-defined, nightly-backed-up, tunnel-fronted platform, and a sentence I stand by: the router remains the weakest member of the team and has been architected around accordingly.

Reader, I changed the router.

The old one had been established, across several episodes, as this saga's least reliable character. The new one arrived, got plugged in, and did what every new router does on its first day: it handed out addresses from a different subnet. My whole network moved from 192.168.68.x to 192.168.0.x:

Machine	Before	After
msi (control plane)	`192.168.68.210`	`192.168.0.201`
dell (worker)	`192.168.68.111`	`192.168.0.202`

A note on format: this is my first RCA-style post on this blog. Real RCAs have an impact statement, a clean timeline, and one tidy root cause, written by someone calm. Mine has an impact statement ("kubectl said EOF"), a timeline ("evening, then more evening"), and a root cause that kept growing a plus-one every time I fixed the previous one. I am calling it an RCA anyway, because "Seven Root Causes and a Port Number" did not fit in the title bar.

Here is what I learned about my own architecture. It had been carefully built around the router's flaws. It had never once been built around the router's address range. Episode 3 me even wrote this down, in the lessons list, in plain text: reserve the address at the router level, not just on the box. Episode 3 me was talking to a wall, and the wall was me.

By the end of the night I had found seven places where my cluster had the old address memorized, and none of them had been told about the move. Here they are, in the order I found them.

🚨 Symptom One: `kubectl` Says "Unexpected EOF"

$ kubectl get nodes -o wide
Unable to connect to the server: unexpected EOF

My first theory was certificates, because it is everyone's first theory. I added the new IP to csr.conf and csr.conf.template on msi, ran microk8s config, and pasted the output over my workstation's kubeconfig. Same error, delivered with the same calm.

That should have told me something. A certificate mismatch is not shy: it says x509: certificate is valid for X, not Y, out loud, in a full sentence. An EOF is the sound of a door closing. So I started ruling things out, from the outside in:

The kubeconfig server: line already pointed at https://192.168.0.201:16443. No KUBECONFIG override, no proxy variables.
ip -4 addr on msi showed 192.168.0.201.
nc -vz 192.168.0.201 16443: open. kubelite was listening on it.
And the one that confused me most:

$ curl -vk https://192.168.0.201:16443/version
... TLS handshake completes ...
< HTTP/2 401
{ "kind": "Status", "reason": "Unauthorized", "code": 401 }

A clean TLS handshake and a polite 401. The API server was awake, reachable, and well-mannered. This is Episode 7's trap with the sign reversed: a healthy-looking API server proves the doorman is on shift, not that anyone is home.

The decisive test was boring. I ran the same command on msi itself:

$ sudo microk8s kubectl get nodes
^C

It hung. A command that hangs identically on the machine itself has stopped being a networking problem and become a cluster problem. So: logs. kubelite first:

awaiting required bookmark event for initial events stream, no events received for 30s

The API server was waiting for events from its datastore, and the datastore was not sending any. So the datastore's own logs:

[go-dqlite] attempt 10386: server 192.168.68.210:19001: dial: dial tcp 192.168.68.210:19001: i/o timeout

Attempt ten thousand, three hundred and eighty-six. dqlite was trying to reach itself at the old address, patiently, thousands of times. Both of its state files agreed with the logs:

# /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml
- ID: 3297041220608546238
  Address: 192.168.68.210:19001
  Role: 0

info.yaml said the same thing. I should have seen this coming, because Episode 5 explains exactly why: the moment the Dell joined, dqlite stopped listening on loopback and bound itself to the LAN address so the other node could reach it. A single-node MicroK8s is structurally immune to network changes. Mine lost its immunity the day it made a friend. With its only member unreachable, dqlite couldn't elect a leader, and an API server without a datastore is a very confident receptionist for an empty building.

The fix is the single-node version of the documented dqlite recovery: stop MicroK8s, back up the backend directory (the one step here that can't be redone), swap the address in the state files, reconfigure dqlite with the bundled tool, start.

sudo microk8s stop
sudo tar -cvf ~/microk8s-backend-backup.tar /var/snap/microk8s/current/var/kubernetes/backend

cd /var/snap/microk8s/current/var/kubernetes/backend
sudo sed -i "s/192.168.68.210/192.168.0.201/g" info.yaml cluster.yaml
[ -f localnode.yaml ] && sudo sed -i "s/192.168.68.210/192.168.0.201/g" localnode.yaml

sudo /snap/microk8s/current/bin/dqlite \
  -s 127.0.0.1:19001 \
  -c /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt \
  -k /var/snap/microk8s/current/var/kubernetes/backend/cluster.key \
  k8s ".reconfigure /var/snap/microk8s/current/var/kubernetes/backend/ /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml"

sudo microk8s start
sudo microk8s kubectl get nodes

msi: Ready. Nobody clapped, because the Dell was still very quiet.

🧟 Symptom Two: The Dell Stops Answering the Phone

NAME   STATUS     ROLES    AGE   VERSION
dell   NotReady   <none>   59d   v1.35.6
msi    Ready      <none>   59d   v1.35.6

The Dell is a worker, which means it isn't part of dqlite at all. It talks to the control plane through a local apiserver-proxy service, and the list of control plane endpoints that proxy forwards to lives in args/traefik/provider.yaml. Mine had exactly one entry:

services:
  kube-apiserver:
    loadBalancer:
      servers:
      - address: 192.168.68.210:16443

One endpoint, and it was the old address of msi. The proxy normally refreshes that list from the cluster, but it can't ask the cluster anything when its only way of asking has moved house. A worker keeps its own copy of where the control plane lives, and nobody tells it when that changes.

sudo sed -i 's/192\.168\.68\.210/192.168.0.201/g' /var/snap/microk8s/current/args/traefik/provider.yaml
sudo microk8s stop
sudo microk8s start

While grepping the Dell's config for the old subnet I found one more tenant: can-reach=192.168.68.210 in args/cni-network/cni.yaml. That is Calico's hint for which network interface to use for pod traffic, pointed at an address that no longer exists. The live value sits on the calico-node DaemonSet, so that is the place to check:

kubectl -n kube-system get ds calico-node -o yaml | grep -B1 can-reach

🔐 Sidequest 1: The Pod That Remembered the Old Address

The Airflow API server pod, which gets scheduled on the Dell, hit the same species of problem at startup: a certificate that only knew the old addresses. The fix was the usual ritual. Add the new IPs to the CSR template on the Dell, regenerate the certificates from msi, and the pod came up healthy.

Notice how many fixes in this episode are "find the file where the old address lives and tell it about the move." That is the episode.

🪪 Sidequest 2: `kubectl logs` and the Other Certificate

With the nodes healthy, I tried to read the logs of a cloudflared pod running on the Dell:

Error from server: Get "https://192.168.0.202:10250/containerLogs/cloudflared/cloudflared-.../cloudflared":
tls: failed to verify certificate: x509: certificate is valid for 192.168.68.111, not 192.168.0.202

This is not about cloudflared at all. kubectl logs, exec and port-forward take a different road from every other command:

kubectl -> API server (msi) -> kubelet on dell :10250 -> container

The API server connects to the kubelet's HTTPS port on the node, and the address it uses must appear in the kubelet's own serving certificate. Every node has one, and it is a different certificate from the API server's, which I had already fixed and was feeling smug about. The Dell's had been issued when it joined the cluster:

subject=CN=system:node:dell, O=system:nodes
issuer=CN=10.152.183.1
X509v3 Subject Alternative Name:
    DNS:dell, IP Address:192.168.68.111

Instead of re-joining the node, I asked the kubelet to request a fresh serving certificate from the cluster CA, which MicroK8s doesn't turn on by default. On the Dell:

echo "--rotate-server-certificates=true" | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap stop microk8s && sudo snap start microk8s

The kubelet filed a certificate signing request. Serving CSRs are not auto-approved, which is a sensible default and a mild inconvenience:

$ kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR          CONDITION
csr-bgrrf   19s   kubernetes.io/kubelet-serving   system:node:dell   Pending

$ kubectl certificate approve csr-bgrrf
certificatesigningrequest.certificates.k8s.io/csr-bgrrf approved

(Approve only a CSR you expect, from the node you expect.) Then I asked the kubelet what it was actually serving:

$ echo | openssl s_client -connect 192.168.0.202:10250 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    DNS:dell, IP Address:192.168.0.202

The old kubelet.crt on disk still lists the old IP, but the kubelet no longer serves it, so I left the file alone like a souvenir. kubectl logs worked again.

The fine print: with rotation on, the next IP change will produce a new CSR, and logs and exec will stay broken until I approve it. A manual step reintroduced, written on the checklist, where future-me will find it and sigh.

🚇 Sidequest 3: The Tunnel Route

Episode 9 had some fine print, and I am quoting my past self: remotely managed means the hostname routing lives in Cloudflare's dashboard, not in my repo, which makes it the one thing a cluster rebuild can't restore. This episode, that fine print sent me an invoice. The route in the dashboard still pointed at an old node address.

The tunnel itself was fine. The cloudflared pods run inside the cluster, so they didn't care about the router, and the logs showed four registered connections. The only broken thing was where the tunnel was sending traffic.

The robust fix is to stop sending it to a number at all. cloudflared runs in the cluster, which means it can resolve in-cluster Service names, and every Service gets <service>.<namespace>.svc.cluster.local. So I pointed the route at the Airflow API server Service by name, and kept the port from the old route:

http://airflow-api-server.airflow.svc.cluster.local:30080

The tunnel logs objected immediately:

ERR error="Unable to reach the origin service ... dial tcp 10.152.183.41:30080: i/o timeout"

Look at that address again. 10.152.183.41 is the Service's ClusterIP, so the name resolved fine and DNS was innocent. The port was the problem. 30080 is the NodePort I pinned for the Envoy gateway back in Episode 4, and a NodePort exists on the nodes' own IPs, nowhere else. The ClusterIP listens on the Service's regular port, which for the Airflow API server is 8080. I had combined a perfectly good name with a port that only exists on a different kind of address, which is like mailing a letter to the right house with the apartment number from the old one.

You connect to	Use this port
A node IP, like `192.168.0.201`	The NodePort (`30080`)
The ClusterIP or the Service DNS name	The Service port (`8080`)

I changed 30080 to 8080 in the dashboard. Instant. No restart. The tunnel now goes straight to the Airflow API server Service, which also means the Envoy gateway from Episode 4 is no longer in the tunnel's path. And because the route is a name, the next router can do whatever it likes.

🗺️ The Census: Where My Cluster Kept the Old Address

#	Where it lived	What it held	What broke
1	dqlite `cluster.yaml` and `info.yaml`	the datastore's own address	API server hung, `kubectl` got EOF
2	Dell's `args/traefik/provider.yaml`	control plane endpoint	worker went `NotReady`
3	Calico `can-reach` (`cni.yaml`, `calico-node` DaemonSet)	interface detection hint	nothing I noticed, which is the scary kind
4	API server and pod certificates (`csr.conf.template`)	names and IPs the cert is valid for	x509 errors at pod startup
5	Dell's kubelet serving certificate	the node's IP	`kubectl logs` and `exec`
6	Workstation kubeconfig	API server address	`kubectl` pointed at nowhere
7	Tunnel route in the dashboard	origin address	errors on the public URL

🧠 What This Taught Me

A cluster's address is not one fact. Mine lived in seven places, in seven dialects: datastore state files, a worker's proxy list, a CNI hint, certificates, a kubeconfig, a tunnel route. Changing the IP is a census, not an edit.
unexpected EOF is not a certificate error. Certificates complain in full sentences. EOF means the door closed. Curl the endpoint, then run the same command on the node itself, to split client problems from cluster problems.
A polite 401 proves the API server is awake, not that the control plane is. Episode 7, with the sign reversed.
When the API server hangs, read the datastore logs. dial ... i/o timeout to your own old IP is a very specific kind of confession.
Workers keep their own copy of where the control plane lives. provider.yaml does not update itself when its only entry stops existing.
logs, exec and port-forward go through the kubelet's certificate. A different certificate, with its own list of addresses, fixed by its own CSR.
NodePort and Service port are not interchangeable. Node IP plus NodePort, or Service name plus Service port. Never one from each column.
Names beat numbers, and reservations beat both. Route to Service DNS, and tell the router who gets which address. Episode 3 said so, and I intend to start reading Episode 3.
The one thing git can't restore is the thing living in the dashboard. Episode 9's fine print came due right on schedule.

📋 Quick Reference (For Skimmers)

Purpose	Command / Place
Is dqlite reaching itself?	`sudo journalctl -u snap.microk8s.daemon-k8s-dqlite -n 50 --no-pager`
dqlite's idea of its own address	`sudo cat /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml`
The same command, from the node itself	`sudo microk8s kubectl get nodes` (hangs = cluster problem, not client problem)
Where a worker thinks the control plane is	`/var/snap/microk8s/current/args/traefik/provider.yaml`
Calico's interface hint	`kubectl -n kube-system get ds calico-node -o yaml \
Pending kubelet cert requests	{% raw %}`kubectl get csr`, then `kubectl certificate approve <name>`
What the kubelet is actually serving	`echo \
Service port vs NodePort	{% raw %}`kubectl -n airflow get svc` (a ClusterIP service shows only `<port>/TCP`)
Tunnel healthy?	`kubectl -n cloudflared logs -l app=cloudflared` -> `Registered tunnel connection`

🚀 What's Next

The permanent fix for all of this is one checkbox on the new router: a DHCP reservation per laptop, so the addresses stop being a suggestion. That is the next ten minutes of my life, and I am doing it before anything interesting. After that, the third-voter-shaped hole in this cluster's HA story is still there, and the Pi is still in its drawer. The drawer has heard promises before.

💬 Final Thoughts

New router, same cluster. Seven places remembered the old address, one dashboard had to be told in person, and one port number turned out to belong to a different kind of address than I thought. The router remains the weakest member of the team. It now has a newer face and a different subnet, and I have stopped being surprised.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated card 🪪, chart-source bookmark 🔖, a domain 🌐, and now a DHCP reservation 📌.

🧭 Neovim Thinks in One Directory

sanskar arora — Wed, 28 Oct 2026 03:37:32 +0000

Hey all 👋

Here is an assumption so deep in Neovim that you stop seeing it: there is one project, and it is wherever :pwd points. Telescope greps the cwd. nvim-tree roots itself at the cwd. Neogit opens the repo that contains the cwd. git-worktree.nvim is a cwd changer with a picker on top. Harpoon keys its marks by cwd. Every one of these plugins is excellent, and every one of them believes the same small lie.

My work does not look like that. On a normal day it is a service, the shared library it imports, and the infra repo that deploys it - three repositories, one problem. Or it is three git worktrees of the same repo, side by side, because a review and a hotfix and a feature branch all want to be open at once. VS Code has had an answer since forever: the multi-root workspace, a .code-workspace file with N folders in it, and every search, tree and git view spanning all of them.

Neovim's project plugins mostly answer a different question - "switch me between projects" - which is one root at a time with better bookkeeping. I wanted the VS Code shape without giving up any of the plugins above, and without a plugin that replaces them. That is what workspaces.nvim is: a name, an ordered list of folders, a session, and a set of careful conversations with plugins that only understand one directory.

This post is the design: the decisions, why they went that way, and the three or four places where the mechanism was more interesting than the feature. Code is at the end.

🧨 Why "just cd somewhere" does not work

The naive answer to multi-root is :cd to the common parent and let everything fall out. Three things go wrong at once, and they have different fixes.

1. The common parent is usually garbage. ~/github/service, ~/github/shared-lib and ~/infra/deploy have ~ in common. A tree rooted at your home directory and a grep across it are not a workspace, they are a punishment.

2. The plugins disagree about what "where" means. Telescope will happily take a list of directories (search_dirs). Neogit takes a cwd option per call. nvim-tree takes exactly one root and will :lcd its own window to get there. git-worktree resolves the repository from the process cwd, synchronously, at the top of every function. There is no single knob that moves all four.

3. Sessions and folders are different things. :mksession remembers windows, buffers, tabs and the cwd. It knows nothing about "these three folders belong together", and if you let it be the only persistence you get a session that reopens your buffers in the wrong place with no memory of why.

So the design has to carry its own notion of folders, hand it to each plugin in the dialect that plugin speaks, and keep its hands off anything that would make a small workspace worse than no plugin at all.

That last clause became the rule everything else answers to:

The fallback rule. With zero folders, every feature behaves exactly like stock Neovim. An empty workspace is never worse than not having the plugin.

It sounds like a nicety. It is actually the thing that made the design converge, because every integration got the same first question: what do you do when there is nothing to do? The answer is always "what you did before".

🧭 Decision 1: what is a workspace?

A workspace is three things, stored as two files:

Thing	Where it lives	Why
A name	the filenames	validated: 1-200 chars, no `/` or `\`, no control characters, no leading `.`
An ordered folder list	`<name>.json`	plain data, read with `vim.json.decode`, never executed
A session	`<name>.vim`	the unmodified output of `:mksession`, restored with `:source`

Two choices in that table are worth defending.

Reuse :mksession instead of inventing a session format. Neovim's own session machinery already handles windows, splits, tabs, folds, the argument list and cwd, and it is maintained by people who are not me. A plugin-specific layout format would be a worse version of it within a month. The cost is that a session file is Vimscript that gets executed on load, which has a security consequence I will come back to.

The folder list is data, not code. An early version stored the folders as a Lua file and loaded it with dofile. That is one line and it works, and it also means anyone who can write to your session directory can run arbitrary Lua in your editor the next time you open a workspace. The current version writes JSON. Old .vars.lua files are migrated on first read through load(src, "=workspace-folders", "t", {}) - text-only chunk, empty environment, so a hostile file cannot reach os, io or vim. The test for this was a legacy file containing os.execute("touch PWNED"). Nothing ran; the folder list came back empty; the file was converted and removed.

LLD lesson worth generalising: if a file is persistence, it is data. The moment you load persistence with a code loader because it is convenient, you have an execution path you did not design, and you will not remember it is there.

Storage is one directory, stdpath("data")/workspaces/, created 0700. Switching is: save the current workspace (:wall, then mksession!), wipe every buffer, :source the target session, read its folder list, then tell every integration the folders changed. Every one of those steps is under pcall with a message, because the failure modes are mundane - a read-only directory, a session whose saved cwd was deleted last week - and none of them should abort a switch halfway. :source runs with silent! for the same reason: one dead cd line inside a session should not stop the twenty buffer lines after it from loading.

🔍 Decision 2: search is the easy part, so make it nice

Telescope already speaks multi-root; it just calls it search_dirs. find_files, live_grep, grep_string and oldfiles all get the workspace folder list, or nothing when the list is empty (the fallback rule, applied for the first time).

What was worth building on top is a ripgrep prompt with a small grammar:

TODO                              every file in every folder
TODO |:| *.lua                    only Lua files
TODO |:| *.lua, !**/tests/**      Lua files, but not tests

Everything after |:| becomes -g globs. The command that runs is rg -e <pattern> -g <glob>... --color=never --no-heading --with-filename --line-number --column --smart-case -- <folders>. The sorter is empty on purpose: ripgrep already did the filtering, so Telescope sorting the results again is CPU spent on nothing.

Then there is the picky version. <leader>fs greps every folder. <leader>fns first asks where: a picker listing every workspace folder and every subdirectory under them, multi-select with <Tab>, then the same grep over exactly the marked paths. Three decisions make it fast enough to open without thinking:

Scan once, stream in. fd --type d --hidden --no-ignore . <folders> runs as an async job and entries appear as they arrive. The picker is usable before the scan finishes, and the process is killed when the picker closes. On a six-core laptop, 1,060 directories under ~/github take 26 ms; 45 plugin repos with 1,287 directories take 30 ms.
Ignore .gitignore, keep the exclude list. --no-ignore is deliberate, so a build directory or a venv can be grepped on purpose. .git, node_modules, target and dist are still skipped, with an in-picker toggle to bring them back that keeps your marks.
Remember the last selection in memory only. The marked paths are pre-marked next time, survive a workspace switch, and are never written to disk. Between uses, the whole feature holds one short list of strings and one boolean.

There is a trap in the scanning that is worth its own sentence. fd --max-results is the obvious way to cap a scan on a slow machine. It does not exist on fd 7.4, which is what Ubuntu 20.04 ships, and an unknown flag makes fd exit with an error and an empty picker. The cap moved into the finder: count entries as they stream, close the job at N. Works for fd, works for the find fallback, works on the old box.

🌳 Decision 3: nvim-tree has one root, and that is not negotiable

This is the compromise at the heart of the plugin, and I would rather say so than pretend. nvim-tree shows exactly one root. VS Code's explorer shows N. There is no setting to change that, and forking nvim-tree was never on the table.

The workaround has two halves.

Root the tree at the deepest common ancestor of the folders. For ~/github/service and ~/github/shared-lib, that is ~/github. For folders that share nothing, it is /. Yes, really / - the alternative was special-casing, and a tree rooted at / with the filter below applied is actually fine.

Filter everything that is not on the way to a folder. nvim-tree reads a filters.custom function from its own setup(), and the plugin exports one:

filters = {
  custom = function(path)
    local ok, ws = pcall(require, "workspaces")
    return ok and ws.nvim_tree_filter(path) or false
  end,
},

With two or more folders, a path is shown only if it is a workspace folder, is inside one, or is an ancestor of one. Everything else is hidden. The result is surprisingly close to VS Code: your folders, nested under whatever they share, siblings gone.

Then two details that took longer than the feature.

The tree must never move your cwd. nvim-tree's change_root runs :lcd on its own window by default (actions.change_dir), so after re-rooting, the tree window has a different working directory from everything else. Harmless until you press a key in the tree that opens a picker, and the picker scans somewhere you did not expect - and until :mksession saves that :lcd into the session. The fix is plain Vim: right after the plugin's own change_root, every tree window with a local directory gets noautocmd cd <global cwd>, and :cd inside a window clears its window-local directory. No nvim-tree internals touched, and manual navigation in the tree behaves exactly as before.

The tree must never be in the session. :mksession with the tree open records a dead NvimTree_N buffer, which comes back on load as an empty window with a filetype and no explorer behind it. So every save closes the tree first, runs mksession!, then reopens it and puts focus back where it was. Switching closes it once and opens it once; a first version reopened, wiped, and reopened again, which was three flickers for the price of one.

LLD lesson worth generalising: when you integrate with a plugin, enumerate the side effects it has on shared state - cwd, buffers, focus - not just its API. The API was fine. The :lcd and the stray buffer were the actual work.

📍 Decision 4: who owns the cwd?

The first real-use surprise: switch workspace, run :!pwd, same directory as before. Nothing in the plugin had ever :cd'd. The only cwd change on a switch came from :source restoring whatever cwd the session was saved with, and every session had been saved from the same launch directory.

The rule that replaced the accident: the cwd follows the common root of the folders, on every switch and on every folder add or remove. One folder, the cwd is that folder. Two related folders, their shared parent. Folders that share nothing, / - and it does cd / anyway, because the searches use the folder list and are unaffected, and because a rule with an exception is two rules. Zero folders, the cwd is left alone (fallback rule again).

It is a plain :cd, so DirChanged fires and every plugin that keys on it gets the news. The add-folder picker now starts at the common root. Sessions save the common root. The tree's :lcd is still undone. It is one option (cwd_follows_root) and it is on by default, because a workspace whose :pwd lies to you is not finished.

🔀 Decision 5: plugins that take a cwd versus plugins that are the cwd

This is the distinction I would hand anyone integrating with the Neovim git ecosystem.

Neogit takes a cwd option. That makes routing a pure function of the current buffer. If you are in a real file (buftype == "", named, readable) inside a workspace folder, Neogit opens in that file's repository, found with vim.fs.root(file, ".git") - stat calls, no git process, no lag. From anything else - the tree, a Telescope prompt, a [No Name] buffer, a file outside the workspace - you get a repo picker, always, even with one repo, because guessing wrong about which repo you meant is worse than one extra keypress.

The repo picker itself earned some care because everything uses it. Entries look like work/app/ (main, 3 changed). Discovery is pure Lua (vim.fs.root per folder, deduplicated, so two folders in one repo collapse to one entry and linked worktrees count separately). The picker opens instantly with (…) placeholders and fills in as one async git status --porcelain=v1 -b --untracked-files=no per repo returns; the jobs are killed if you close it early. Tracked changes only, by choice - untracked counts are noise on any repo with a build directory.

git-worktree.nvim is the cwd. Its model is: create a worktree, cd into it, swap your buffers. Resolving the repository happens synchronously from the process cwd at the top of each function. In a multi-root workspace that model is simply wrong: a worktree is a folder, and the thing you want is to add it.

So, inside a workspace, three functions on the git-worktree module table are wrapped - create_worktree, switch_worktree, delete_worktree. Its Telescope extension calls them through the module table, so no fork is needed. The wrappers change the meaning of each verb:

git-worktree verb	Upstream	Inside a workspace
create	create, then `cd` into it	create, then add it as a folder; no `cd`
switch (picker `<CR>`)	`cd` + buffer swap	add it as a folder (warn if already present)
delete (picker `<C-d>`)	`git worktree remove` in cwd	`git -C <repo> worktree remove [--force] -- <path>`, then drop the folder (and any folder nested inside it)

The one wrinkle: because git-worktree reads the repo from the cwd synchronously at the start, and every job it spawns afterwards carries an explicit cwd, a momentary noautocmd cd <repo> → call → cd back is enough and is invisible to everything else. A small context object { kind, repo } is set when the plugin's own worktree picker opens, consumed by the first wrapped call it applies to, and reset on every worktree key press, so a half-finished flow can never hijack a later real one. Outside a workspace none of this happens and git-worktree is byte-for-byte upstream.

There is also a cross-repo changed-files picker (<leader>gs): one git status --porcelain=v1 -z per workspace repo, two-letter XY status plus folder label plus path, git diff HEAD -- <file> as the preview. The preview kills the previous diff job before starting the next, because holding j in a picker should not fork thirty git processes.

LLD lesson worth generalising: sort your integrations by what they assume, not by what they do. "Takes a cwd" is a parameter. "Is the cwd" is a worldview, and you integrate with a worldview by wrapping the three functions where it touches reality.

🔌 Everything else rides on hooks the plugins already offer

The remaining integrations needed no cleverness, only using what was there:

LSP. Every running server that advertises workspace.workspaceFolders.supported gets a workspace/didChangeWorkspaceFolders notification with the workspace folders. Only folders the plugin added are ever removed again; a server's own root is never touched. Workspace symbols and references now span repos. It adapts to the client API change between Neovim 0.10 (client.notify) and 0.11+ (client:notify), because the dot form logs a deprecation on one and the colon form crashes on the other.
Harpoon 2. Its settings.key hook decides which mark list is active. require("workspaces").harpoon_key() returns workspace:<name> inside a workspace and the cwd otherwise, so marks become per-workspace and Harpoon's own behaviour returns the moment no workspace is active.
Statusline. statusline() returns <name> (<folder count>) or an empty string. Drop it in lualine as a component; styling is yours.

And then the piece that turned "optional" from a comment into a contract. Every external integration has a switch, on by default:

integrations = { nvim_tree = true, neogit = true, git_worktree = true, harpoon = true },
lsp = { enabled = true, servers = nil },

Off means the plugin never even requires that plugin: its default keymaps are not registered, calling its API warns, and the hooks your own config calls (harpoon_key, nvim_tree_filter) fall back to stock behaviour so your snippets can stay in place. pcall(require, ...) handles "not installed". A switch handles "installed, and I do not want you touching it", which is a different thing.

🔒 Hardening, because some of these machines are bastion hosts

Some of the hosts this runs on are small, slow, and not entirely mine. That forced a round of reading every line with three questions: can this run something it should not, can this crash the editor, and can this be cheaper?

Execution surface. Nothing in the plugin runs a shell; every external command is an argv list, with paths after -- where the tool supports it. The folder list is JSON (above). Workspace names are validated everywhere a name arrives, which closes the :WorkspaceDelete ../something shape of path traversal. The session directory is created 0700.

And the one that cannot be fixed, stated plainly: <name>.vim is Vimscript and :source executes it. That is inherent to :mksession. Whoever can write to your session directory can run code in your editor. The directory defaults to stdpath("data"), is private by default, and should never be a synced or shared folder.

Crash surface. The things that used to throw now report: mksession into a read-only directory, :source of a session whose cwd is gone, :cd to nowhere, %bd! on a stubborn buffer. Missing Neogit, missing git-worktree, missing its Telescope extension, missing rg, missing Telescope itself all warn instead of erroring. The startup picker only opens on a clean interactive launch - a UI attached, no file arguments, no -S, nothing piped in - because the first version opened Telescope inside nvim --headless, which is a wonderful way to hang a scripted run on a server.

Three bugs from that round were the kind that pass every test and lie to you:

vim.fn.expand on a typed path. ~ and $VAR expand, which is what you want. % and # also expand, to the current and alternate buffer names, which is not. A directory with a % in its name became a path to nowhere. vim.fs.normalize does the first job without the second.
find -path "*/.*" -prune as the fallback for machines without fd. It prunes any path containing a dot-directory - including the scan base itself when the cwd is under ~/.config. Result: an empty picker, only on machines without fd, only from inside a dotfiles directory. Replaced with -mindepth 1 and -name prunes.
fd --max-results as described above: present on the laptop, absent on the server.

Footprint. One mkdir per session instead of one per storage call. readdir with a filter instead of glob. The fd/rg lookups cached after the first PATH scan. Recent-files filtered by path prefix before any stat. One tree close and one open per switch. The previous diff job killed before the next preview. Auto-saves silent, because a notification after every folder change produces a hit-enter prompt on a 24-line terminal.

None of these are clever. All of them are the difference between a plugin that is fine on a Ryzen and one that is fine on a two-core VM you reach through a jump host.

🚧 Honest gaps

nvim-tree is an approximation. Common ancestor plus filter looks like multi-root. It is not. Unrelated folders live under /.
Switching loses unnamed buffers. :wall cannot save [No Name], and %bd! then discards it, along with terminals and Neogit buffers.
Two pickers lean on Telescope internals - async_oneshot_finder for seeding results, and the _multi option for sharing a multi-select across a refresh. A Telescope refactor could break them.
A deleted cwd breaks Telescope previews until you :cd elsewhere. That is Neovim behaviour (vim.fs.abspath asserting inside the buffer previewer), not plugin code, but you will meet it if you remove a workspace's only folder from disk while it is open.
The name. natecraddock/workspaces.nvim already answers to require("workspaces"). The two cannot be installed together as things stand.

🧠 What I would tell the next person

Write the fallback rule before the first feature. "Zero folders means stock Neovim" decided a dozen edge cases before they were edge cases.
Reuse the host's persistence; own only your data. :mksession for layout, a JSON list for the thing Neovim does not know about. Inventing a session format is a month of bugs for no feature.
Persistence is data. Load it as data. dofile on a config file is an execution path you did not design.
Integrate with a plugin's side effects, not just its API. The :lcd, the stray buffer, the focus steal - that was the work. The API calls were two lines.
Sort plugins by what they assume about the world. "Takes a cwd" is a parameter. "Is the cwd" means wrapping the three functions where that belief meets reality.
A switch is not a pcall. "Not installed" and "installed, hands off" are different requirements, and a plugin that is going to live in other people's configs needs both.
Test on the machine that has the old tools. fd 7.4, no fd at all, a dot-directory as cwd, nvim --headless. The laptop passes everything; the bastion host is where the real test suite lives.

📋 Quick Reference (For Skimmers)

Action	Key	Command
New workspace (name popup)	`<leader>wn`	`:WorkspaceNew`
Open / switch (picker, `<C-d>` deletes)	`<leader>wo`	`:WorkspaceOpen`
Delete (picker, or by name)	`<leader>wD`	`:WorkspaceDelete [name]`
Save now	-	`:WorkspaceSave`
Add folder (picker: `<C-o>` browse mode, `<C-y>` typed path)	`<leader>wa`	`:WorkspaceAddFolder [dir]`
Remove folders (multi-select, `<C-s>` applies)	`<leader>wd`	`:WorkspaceRemoveFolder`
Find files / recent / live grep / word	`<leader>ff` `fr` `feus` `grep`	-
Grep with `pattern \	:\	globs`, all folders
Same, but pick folders first	`<leader>fns`	`:WorkspaceGrepSelective`
Neogit (file's repo, else picker)	`<leader>gt`	-
Changed files across repos	`<leader>gs`	-
Worktrees: list → add as folder / create → add as folder	`<leader>gwt` / `<leader>gwa`	-

Install with lazy.nvim:

{ "snskArora/workspaces.nvim", dependencies = { "nvim-telescope/telescope.nvim" }, opts = {} }

Needs Neovim 0.10+ and rg; fd is recommended and find is the fallback. The plugin is at snskArora/workspaces.nvim, and the config it lives in - lualine pill, nvim-tree filter, Harpoon hook, the lot - is snskArora/nvim_configs.

💬 Final Thoughts

Neovim still thinks in one directory. It just has a plugin now that speaks to each of its friends in the dialect they understand, and keeps a list of the directories it should have been thinking about all along.

One window, several repos, and :pwd finally telling the truth.

📊 One File, Twenty-Five Images, Part 2/2 - The Numbers

sanskar arora — Sun, 27 Sep 2026 18:17:00 +0000

Hey all 👋

Part 1 worked out how to move twenty-five container images across an air gap as one file: pack them into a single OCI layout so shared layers are stored once, and pick between two modes - pristine, where layers travel exactly as the registry stores them and every digest survives, and repack, where layers travel uncompressed and the whole bundle is compressed as one zstd stream.

Nice theory. This half is the part where the theory meets a laptop: how I measured it, the several ways my measurements were wrong before they were right, what four reviewers found when they went looking for holes, and the numbers that came out - including the one that overruled a decision I had just finished defending.

🔬 The process: build it small, measure it honestly

I wrote the prototype in bash, on purpose. The final implementation is planned to be in Go (a static binary you can drop on the far side), but the question M0 had to answer was "how big is the file, and what does it cost", not "is my Go code nice". Bash plus skopeo, zstd, jq and GNU tar reaches that answer in a day or two, and every step stays visible.

One rule was meant to keep it honest: anything that decides a digest lives in Go, in the pinned cz1 helper, with bash owning only the outer compression (digest-irrelevant) and the glue. I half-kept it. Layer bytes are Go; the manifest rewrite is jq - and the manifest digest is the image digest, so jq is deciding identity too. That shortcut is exactly where a reviewer later found a hole, and it is why the manifest rewrite rules have to be frozen alongside the encoder. (I also learned the hard way that you cannot reproduce klauspost's output with the zstd CLI. Different implementations, different bytes, same settings.)

Measuring, and the several ways I got it wrong first

The first version of my measurement rig was wrong in ways that flattered the results:

CPU was the change in the shell's cutime, which swept up the tar, tee and hashing processes alongside the compressor.
Peak memory was sampled from /proc/<pid>/status every 100 ms. On short runs that read 24-38% low, and sometimes sampled before exec.
The report said zstd used "12 threads". zstd -T0 uses one worker per physical core: 6 on this machine.

The fix was a 58-line Go helper that runs one command and reports what the kernel says via wait4: exact user+sys CPU and exact ru_maxrss. No sampling, no guessing.

Then, because a laptop is not a lab, every run also records how much CPU the rest of the machine used during it (system-wide busy time minus our own) and how many pages swapped. Rows where outside load exceeded 10% get flagged in the report, and cached measurements taken under load are automatically re-timed. An idle desktop here sits at 2-5%, so the threshold had to be calibrated rather than guessed - at my first attempt (3%) every single row got flagged, which is the same as flagging none.

There is also a guard that estimates zstd's peak RAM from level, window and worker count, and skips a variant the machine cannot hold. It puts a 2 GiB window at level 19 at about 4.8 GiB: roughly 3.8 GiB of estimated zstd footprint - it actually peaked at 3.90 - plus a deliberate 1 GiB margin for everything else on the machine. It still swapped 68 MiB on that run, which is why the timing carries a flag.

Four reviewers and a skeptic

Before trusting any of it, I had the prototype reviewed along four independent lenses (AI Agents 🫣) - OCI correctness, bash robustness, measurement validity, architecture fidelity - with a separate pass trying to refute each finding. 29 findings, 22 confirmed. Then the same treatment for the report itself: fact-checking every number against the raw data, technical claims against the sources, and a read-through as the person receiving it. That pass produced about 40 corrections, including a percentage stated backwards and a claim that three of four bundles were round-tripped when I had written "all".

The bash bugs are worth listing, because they are the kind that pass tests and lie to you:

$(...) strips trailing newlines. I hashed a manifest through a command substitution; two images in my test set have manifests ending in \n; those two "landed with the wrong digest". The registry was right, my check was wrong.
[[ $a == $(cmd) ]] treats the right side as a glob pattern. JSON is full of [ and ], so a comparison that looked fine silently never matched, and the code path it guarded was never exercised.
set -e + pipefail + [[ cond ]] && cmd as the last line of a function exits the whole script silently when the condition is false, because the function returns 1. Hunting that one is how I learned to add an ERR trap that prints the failing command and line.
A backgrounded command gets /dev/null as stdin unless it has its own redirect, so my measured compressor was cheerfully compressing nothing.
FIFO deadlock: a reader blocked on open() waits forever if the writer never starts. Always check the producer's status before waiting on consumers.

📊 Results

The size and time tables below are from a clean-slate run: every generated artifact deleted, all 25 images re-pulled, measured on a 6-core/12-thread laptop. The window/level sweep and the layer-order result after them come from the earlier full measurement matrix - same machine, same images, a week earlier - because the clean-slate re-run only rebuilt the settings I would recommend. The dataset is 190 layer references collapsing to 123 unique layers, 7.00 GiB uncompressed.

Size

Packaging	gzip -9 †	zstd -6	zstd -9
One `podman save` archive per image (today)	3.43 GiB	3.44 GiB	3.43 GiB
Registry blobs per image, no dedup	3.30 GiB	3.30 GiB	3.30 GiB
Deduplicated, registry gzip layers (pristine, before ingest)	2.30 GiB	2.30 GiB	2.29 GiB
Deduplicated, `cz1` zstd layers (pristine, after ingest) ‡			1.82 GiB
Deduplicated, uncompressed layers, one zstd stream (repack)	2.26 GiB	1.93 GiB	1.84 GiB
… same, with a 1 GiB window			1.48 GiB

† Every gzip -9 figure here was produced with pigz -9 - same format, deflate spread across all twelve threads, so gzip gets the same cores zstd does. Plain gzip -9 produces the same bytes and spends the whole CPU figure on the clock.
‡ From the spike matrix rather than the clean-slate re-run: this is the mode where each layer is compressed individually with cz1, so there is no single-stream column for it.

(The zstd -9 row of the first line lands on exactly the size of my original images.tar.zst - 3,681,162,191 bytes - so the baseline is anchored to a re-measurement, not remembered from a week ago.)

The savings stack in four steps: deduplication ~30%, plus 3% for not letting podman re-compress (3.43 → 2.30 GiB), then re-encoding each layer with cz1 instead of gzip to reach 47% (2.30 → 1.82 GiB), then compressing the whole bundle as one stream with a big window to 57% (1.82 → 1.48 GiB).

Time

Packaging	Codec	Compress (wall / CPU)	Decompress
One archive per image (today)	gzip -9 (via `pigz`)	11.7 s / 112.9 s	5.8 s
One archive per image (today)	zstd -9	6.2 s / 42.1 s	1.3 s
Deduplicated, registry gzip layers (pristine)	zstd -9	4.0 s / 25.6 s	0.8 s
Deduplicated, uncompressed layers (repack)	gzip -9 (via `pigz`)	84.0 s / 898.5 s	15.4 s
Deduplicated, uncompressed layers (repack)	zstd -6	20.3 s / 124.7 s	5.2 s
Deduplicated, uncompressed layers (repack)	zstd -9	34.4 s / 203.1 s	5.1 s
Deduplicated, uncompressed layers (repack)	zstd -9, 1 GiB window	32.9 s / 188.7 s	4.8 s

gzip loses on every axis that matters here: on uncompressed layers it burns 899 CPU-seconds to produce a larger file (2.26 GiB) than zstd -9 does in 203 (1.84 GiB), and it decodes 3× slower - and that is with parallel deflate; single-threaded gzip -9 would take fifteen minutes of wall clock for the privilege.

There is no budget at which gzip is the right answer for the bundle itself. In registries it survives for compatibility, and that part is not nothing: both modes land tar+zstd layers, so every registry on the path has to accept OCI 1.1 media types and every puller has to be containerd ≥ 1.5 or Docker ≥ 23. That is a precondition to check before adopting any of this, not a footnote.

The finding that overruled my design

I had specified a 128 MiB long-distance window as the repack default. The data says that is the wrong knob setting, and it is not close:

Setting	Bundle	Compress
level 6, 128 MiB window	1.78 GiB	24.7 s
level 9, 128 MiB window	1.70 GiB	36.8 s
level 9, 1 GiB window	1.48 GiB	35.3 s
level 9, 2 GiB window	1.43 GiB	141 s §
level 19, 128 MiB window	1.46 GiB	625 s §
level 19, 2 GiB window	1.21 GiB	869 s §

Those three runs hit the 15 GiB laptop's memory ceiling rather than measuring zstd cleanly: they swapped 18, 538 and 68 MiB, and 19%, 5% and 10% of the machine's CPU went somewhere other than the compressor. The sizes are deterministic and unaffected; the wall times are inflated by an unknown amount. A repeat of the 2 GiB/level 9 run in digest order took 132 s, so the roughly 4× gap to the 1 GiB window is real, not an artifact.

Moving through levels 6→9 buys 4%. Moving the window from 128 MiB to 1 GiB buys 13%, at the same speed. Level 19 gets you to roughly the same place as the bigger window, 17× slower.

The window is not free at the far end, though, and this is worth budgeting before adopting it: peak decode memory goes from 133 MiB to 1.01 GiB, the zstd CLI refuses a frame needing more than a 128 MiB window unless you pass --long=31, and klauspost's Go decoder rejects anything above 512 MiB until you call WithDecoderMaxWindow - which has to happen before decoding starts, so the importer cannot read the setting out of the bundle it has not opened yet.

The mechanism is visible in the data. Remember those six near-identical revisions of the Debian base - same 74.3 MiB of files, different digests, so deduplication cannot touch them. In first-use order they sit at offsets 142, 216, 403, 549, 5367 and 5563 MiB in a 7 GiB stream. A 128 MiB window reaching back from 403 MiB sees nothing but itself; a 1 GiB window covers the whole early cluster at once, and separately pairs the two stragglers at 5367 and 5563. That is where the 13% comes from: not from any single image compressing better, but from the compressor finally being able to look far enough back to notice it has seen this Debian base several times already.

(On a 5-image test set the whole tar is 181 MiB, everything related is already within reach, and larger windows add exactly nothing - which is how a small test set talks you out of the right answer.)

One more small thing with a mechanism behind it: writing layers in first-use order beat digest order by 2.8% at a 2 GiB window, and made no difference at 128 MiB. Related images' layers end up near each other only if you don't scatter them by hash.

🧪 The clean-slate test

Numbers in a table are a claim. This was the test:

Delete every generated artifact and re-pull all 25 images.
Export the 1.48 GiB bundle, verify it offline.
podman rmi --all - 28 images gone, images left: 0. That includes the skopeo and registry:3 images the import tooling itself runs, which the import then re-pulls. So this is not a test of the air gap; it is a test that nothing about the twenty-five payload images was left cached anywhere on the machine.
Import from that one file into a fresh registry.
Pull all 25 back and compare every image ID against the original config digest.
Run containers.

image-ID mismatches: 0

alpine:3.24                  Alpine Linux v3.24 x86_64 ok
debian:bookworm-slim         12.15
python:3.13-slim-bookworm    3.13.15 2689367b205c
node:22-bookworm-slim        v22.23.3 x64
redis:7-alpine               Redis server v=7.4.11 ...
postgres:16-bookworm         postgres (PostgreSQL) 16.15
busybox:1.36                 busybox ok 406

25 of 25 image IDs matched, podman verified every layer digest on pull, and the containers ran. 3.43 GiB of per-image tarballs became a 1.48 GiB file that fully reconstructs the fleet.

One honest gap, since it is the premise of the whole post: the destination here was a local registry:3, not the Artifactory this is ultimately for. Whether Artifactory accepts OCI 1.1 zstd manifests, exposes the referrers API and allows cross-repo mounts is the first thing on the list, and until that run happens these numbers describe a design that has been proven against a reference registry, not against production.

🧭 What I would tell the next person

Find the decision that forks the design, and make the fork explicit. Here it was digest fidelity. Everything else - bundle format, compression, import logic - followed from it. If I had started with "which compression level?", I would have optimised the wrong axis for a week.
Make the expensive property a property of the format, not of the code. Dedup came from content addressing. Streaming import came from entry ordering. Reproducibility came from tar flags. No algorithms required.
If a hash of your output is an identity, freeze everything that can change a byte - library version, API call, serialisation, key order - and add a test that fails when it moves.
Measure before you tune. My design said a 128 MiB window; the data said 1 GiB, for 13% at the same speed. My instinct said single-stream compression was the big win; the data said deduplication was (3.43 → 2.30 GiB), that re-encoding each layer once with cz1 was the next step (2.30 → 1.82 GiB), and that compressing the bundle as one stream added 18.5% on top (1.82 → 1.48 GiB) only with a window large enough to see across it.
Then measure your measurements. Sampled memory read a third low; CPU accounting swept in helper processes; a "quiet" laptop is 2-5% busy. If a number is going to justify a decision, know how it was produced.
Adversarial review earns its keep. Reviewing the code, then the numbers, then the write-up, each with someone trying to refute the findings, caught a reversed percentage, an encoder that was not as frozen as I claimed, and a signature claim I could not support.

The code, the architecture document, the delta design and the full measurement report - including the bits that did not flatter me - are on GitHub: snskArora/oci_export. The bash prototype is complete end to end (fetch → transform → bundle → verify → import) and the Go implementation is next.

Until then: one file, twenty-five images, and a very quiet air gap.

📦 One File, Twenty-Five Images, Part 1/2 - The Design

sanskar arora — Sun, 27 Sep 2026 17:59:34 +0000

Hey all 👋

Here is a problem that sounds boring until you try it: move twenty-five container images from one world to another, where the two worlds share no network. Source registry on one side, Artifactory on the other, and between them a link, a laptop, or in the worst case a person carrying a disk.

The obvious script exists, everybody writes it, and it was the first thing I wrote too:

for image in "${images[@]}"; do
  podman pull "$image"
  podman save --format oci-archive --output "images/${safe_name}.tar" "$image"
done
tar --use-compress-program="zstd -T0 -9" -cf images.tar.zst images

Twenty-five images in, 3.43 GiB out. It works. Ship it, go home.

Except it bothered me, because I knew what was inside that file. Nine of those images start from the exact same debian:bookworm-slim base layer - same digest, 74 MiB each - and every one carries its own private copy. Four Alpine images share a second base, the three Jupyter notebooks a third. And five more each start from a slightly different revision of that same Debian base: five near-identical 74 MiB layers with five different digests, which content addressing cannot collapse at all. The tarball is a museum of duplication, and then we compress the duplicates separately, so the compressor cannot even notice they are duplicates.

So I did what the situation deserved: designed it properly, built a prototype, measured everything, and let the numbers overrule my instincts twice. Two answers came out, not one. 1.82 GiB (47% smaller) keeps every image digest unconditionally and imports with a plain push - that is the default I would ship. 1.48 GiB (57% smaller) goes further by compressing the whole bundle as one stream, at the cost of re-encoding every layer on import, about 690 CPU-seconds for this set. Either way it is one file, and that file reconstructs all twenty-five working images after you delete every image on the machine.

This post is the whole path - every design decision, why it went that way, and what the measurements said when I finally stopped guessing. Code is at the end.

🧨 Why the naive approach is expensive

Three separate costs stack up in that one-liner, and it is worth separating them because they have different fixes.

1. No deduplication. A container image is a manifest pointing at a config blob and a list of layer blobs, each addressed by the SHA-256 of its bytes. podman save writes one self-contained archive per image, so a layer shared by nine images is written nine times. Content addressing already gives you deduplication for free - if you put all the images in one content-addressed store instead of twenty-five separate ones.

2. Layers are gzip, and gzip is old. Registries store layers gzipped, mostly for compatibility reasons dating to Docker's first manifest format. zstd at a comparable CPU budget is roughly a fifth smaller.

3. Compressing already-compressed bytes. Wrapping gzipped layers in zstd -9 buys almost nothing - it is re-compressing entropy. In my measurements the outer level changed a pristine bundle's size by less than 0.2% between zstd -3 and zstd -9. All that CPU for nothing.

There is also a fourth cost that only showed up when I measured: podman save re-compresses layers as it writes them. Podman stores layers unpacked and re-gzips on the way out, and its output is about 3% larger than the bytes the registry served. You pay a few percent for a round trip through local storage.

🧭 HLD, decision 1: what is the artifact?

First real decision. The bundle format could be anything - a custom container format with an index, a squashfs, a big tarball with a manifest at the front.

I picked: a valid OCI image layout inside a tar, compressed with zstd.

bundle.tar.zst
└── (zstd)
    └── (tar)
        ├── oci-layout
        ├── bundle-meta.json      <- my metadata: mode, images, inventory
        ├── index.json            <- the OCI index: one entry per image
        └── blobs/sha256/
            ├── <manifest digests>
            ├── <config digests>
            └── <layer digests>

Why this and not something bespoke:

Deduplication falls out of the format. blobs/sha256/<digest> is content-addressed by definition. Two images sharing a layer write the same path. I did not implement dedup; I picked a layout where dedup is the only possible behaviour.
It is not a dead end. If my tool is unavailable on the far side - and on an air-gapped host, that is a real scenario - skopeo copy oci:unpacked:<ref> docker://registry/... pushes images straight out of an unpacked pristine bundle. That is the case I actually tried, and it wants skopeo ≥ 1.22, because older versions refuse the Docker schema-2 manifests that three of these twenty-five images still carry. A repack bundle is spec-valid too, so the same command should work, but it would push the uncompressed layers as-is. The format degrades to "standard tools work", with an asterisk.
The ecosystem already agreed on it. Manifests, configs, indexes, media types, annotations: all specified, all understood by every registry client. Inventing a format means inventing all of that badly.

The interesting design work is not the container, it is what goes inside it.

⚖️ HLD, decision 2: the fork that surprised me

Here is the thing I did not see coming when I sketched the first version. My plan was: store layers uncompressed inside the bundle, compress the whole bundle with zstd, recompress layers at import. A bigger compression window - zstd's window is how far back in the stream it may look for a repeat, 4 MiB at level 9 by default and up to 2 GiB with long-distance matching enabled - would finally let the compressor see that those nine Debian bases are the same bytes. Cross-image redundancy, everybody wins.

It does win. It also changes every image digest, and that took me a while to accept.

A manifest references each layer by the digest of its compressed bytes:

{ "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
  "digest": "sha256:e2de96513ba9...", "size": 3849738 }

Decompress a layer for transport and you cannot reproduce those exact bytes later. gzip output depends on the implementation, the level, the window, the version. So the importer must recompress, which yields different layer digests, which changes every manifest, and the manifest digest is the image digest. Everything pinned as image@sha256:... breaks. Every cosign signature, which signs the manifest digest, stops verifying.

What survives is more subtle and worth knowing: the image config lists rootfs.diff_ids, the digests of the uncompressed layers. Those are preserved exactly, and runtimes verify them on pull. So integrity is fully intact and tags work perfectly; it is identity that changes.

I considered reconstructing the original compressed bytes (the trick pristine-tar uses for Debian source packages) and rejected it: it means capturing enough of the gzip encoder's state to replay it bit-exactly across versions, which is fragile machinery in exchange for an edge case.

One term first, because everything after this leans on it. Ingest is a one-time step that runs before an image is signed or pinned: pull it, re-encode each layer with the canonical profile described in the next section, push it back. From then on the registry itself holds canonical zstd layers, and that is the version tags, pins and signatures refer to.

The resolution I landed on is the part of the design I would defend hardest: pair the transport mode with the trust anchor.

	pristine	repack
What is signed	each image, individually	the bundle, once
Layers travel	exactly as stored	uncompressed, deduplicated by `diff_id`
Compression happens	once per layer, at ingest	over the whole bundle, at export
Digests	preserved byte for byte	rewritten (but see the twist below)

If images are individually signed, their bytes may not be touched, so the bundle carries them verbatim and the outer zstd does almost nothing (the layers are already compressed). If only the bundle is signed, layers can travel raw and the outer compressor can do real work across image boundaries. One flag, two coherent worlds, and no mode where a signature quietly stops verifying.

This is the kind of decision I now look for in designs generally: not "which is better", but "what do these two options actually depend on, and can I make the dependency explicit in the interface?"

🧊 HLD, decision 3: a frozen encoder, and why determinism is the whole game

If images are going to be re-encoded (once at ingest for pristine, at every import for repack), the encoder's output must be reproducible byte for byte, forever. If it is not:

the same image ingested twice gets two different digests;
the importer cannot tell "already present" from "different";
deduplication against the destination stops working;
and repack's digests drift every time a library is upgraded.

So cz1 (canonical zstd, profile 1) is a pinned set of settings, not "zstd -19":

enc, err := zstd.NewWriter(out,
    zstd.WithEncoderLevel(zstd.SpeedBestCompression),
    zstd.WithWindowSize(8<<20),
    zstd.WithEncoderConcurrency(1),   // concurrency changes block boundaries
    zstd.WithEncoderCRC(true),
    zstd.WithAllLitEntropyCompression(true),
    zstd.WithZeroFrames(true),
    zstd.WithLowerEncoderMem(false),
)

The library version is pinned in go.mod. Upgrading it is not a dependency bump, it is a profile change (cz2) and a deliberate, scheduled re-canonicalisation.

Then a hole is uncovered, and it is a good one. With identical options, klauspost/compress writes different bytes depending on how you drive it:

NewWriter + streaming, for input larger than one block → frame with no Frame_Content_Size;
ResetContentSize → content size in the header;
EncodeAll → content size, plus the single-segment flag when the input fits the window;
and NewWriter + streaming for input smaller than one block → the library takes the EncodeAll path internally and emits that header shape anyway.

Three different digests for the same layer and the same settings. And a Go implementation written from my architecture document would plausibly reach for EncodeAll, because tar headers hand you the size up front. My own doc would have produced a tool that disagreed with my prototype.

Fix, in two parts. The frame shape became part of the profile definition: one frame, driven through ReadFrom, with whatever header the library writes for that input size. Naming an API is not enough, because the library crosses between them on its own - so the profile also got golden vectors, checked on every run and in CI:

# sha256 of 'seq 1 N | cz1' - frozen outputs of the cz1 profile
8adb12de4536b3c50b4fed4c43fa4411088d61374243b51ae0b8237807d33d92 10000
719f1575295ca6deeb551131294ebc3c20d7c186ea76ad94e9e82726ffdcc878 500000
4ed7f465a721e86af9d211023cc8e473cf1f8787550a5f58d9db5a31504eaaae 3000000

Three sizes on purpose. N is the argument to seq, so the inputs are 48 KB, 3.4 MB and 22.9 MB: one that fits in a single 128 KiB block, one below the 8 MiB window, one spanning several windows. They exercise different frame-header paths, and you can see it in the first byte after the magic: the smallest vector gets 0x64 (content size present, single segment), the larger two 0x04 (neither). That is the library switching paths on input size - precisely the behaviour a hand-written implementation would get wrong.

If a library upgrade changes a single byte of output, CI fails with "cz1 drift" instead of silently giving every future image a new identity.

LLD lesson worth generalising: when a hash of your output is part of your system's identity, your dependency's default behaviour is not a detail you can leave to the library. Pin it, and prove the pin with a test that would fail if it moved.

🧱 LLD: the bundle, in detail

Entry order is a design decision

The tar is written in a fixed order:

oci-layout → bundle-meta.json → index.json → manifests → configs → layers

Because a tar is a stream. An importer reading it sequentially meets the metadata first: it can plan, check the format version, verify sizes, and decide what to skip before gigabytes of layers arrive. Put the index last and streaming import becomes impossible - you would have to buffer everything to disk first. Ordering is free at write time and decides what the reader is capable of.

The tar must be deterministic

Same images in, byte-identical file out. That makes bundles cacheable, resumable, comparable, and auditable:

tar --create --file=- --format=posix \
    --pax-option='exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime' \
    --mtime='@0' --owner=0 --group=0 --numeric-owner --mode='a=r,u+w,a+X' \
    --no-recursion --verbatim-files-from -C "$layout" --files-from="$list"

Every flag there is load-bearing, and one guards against a trap: the pax extended-header name. POSIX specifies %d/PaxHeaders.%p/%f, and %p is the process ID - so on GNU tar 1.32 and older, or on any version with POSIXLY_CORRECT set, a "deterministic" archive quietly embeds the PID of the process that made it. Two runs, two different files, identical content. GNU tar 1.35 (what I measured with) already defaults to the PID-free form, which means this is exactly the kind of bug that works on your laptop and breaks on a build box.

(Bonus finding: zstd's compressed output is byte-identical at 1, 2, 4 and 12 threads. So the compressed bundle is reproducible too, not just the tar.)

`bundle-meta.json` carries the inventory

{ "formatVersion": 1, "mode": "repack", "profile": "cz1",
  "platform": "linux/amd64",
  "outer": { "codec": "zstd", "level": 9, "windowLog": 30 },
  "images": [ { "ref": "...", "digest": "sha256:...", "sourceDigest": "sha256:..." } ],
  "inventory": [ { "digest": "sha256:...", "kind": "layer", "size": 41234567, "included": true } ],
  "exceptions": [ { "digest": "...", "mediaType": "...", "reason": "unrecognized layer media type" } ] }

Two pieces of foresight in there:

inventory with included: true - today everything is included. But it is the hook that makes delta bundles possible later: ship only the blobs the destination lacks, and mark the rest included: false for the importer to find locally. Designing the field in now costs nothing; adding it later is a format break. (The full delta design is a separate document in the repo, deliberately not built yet.)
exceptions - the safety rule. Only recognised image layers are ever transformed. Helm charts, ORAS artifacts, foreign/nondistributable layers, anything with a media type I don't know: carried verbatim and listed here. A converter that touches things it doesn't understand is a data-corruption bug waiting for a slow afternoon.

Verify while you are already touching the bytes

Every layer is decompressed exactly once during a repack export, and its diff_id is checked in the same pass, not in a second read:

mkfifo "$tmp.fifo"
sha256sum <"$tmp.fifo" >"$tmp.sha" &
shapid=$!
gzip -dc <"$src" | tee "$tmp.fifo" | "${encode[@]}" >"$tmp"
wait "$shapid"
[[ "sha256:$(cut -d' ' -f1 "$tmp.sha")" == "$diff_id" ]] || die "layer does not match its diff_id"

Verification that costs a second pass over 7 GB gets switched off "temporarily" by someone in a hurry. Verification that rides along a pass you are already making is free, so it stays on. Design your integrity checks to be cheap enough that nobody has an argument for disabling them.

🔁 LLD: the import side, and a twist I did not plan

Import is the mirror, with one ordering rule that comes from the registry protocol: blobs before manifests. A registry rejects a manifest whose blobs are not uploaded yet, which conveniently means a half-finished import is invisible - unreferenced blobs, no tags, and re-running is idempotent.

For repack, the importer re-encodes every layer with cz1 and rewrites the manifests. And here is where the measurement produced something better than the design:

Because cz1 is deterministic, a repack import lands on exactly the digests that the ingest step would have produced. Both modes converge on one canonical digest per image. I tested this at 25 images: 25/25 identical.

Which leads to the twist. If the images were already ingested - canonical cz1 layers - then the importer's re-encode reproduces the original layer blobs bit for bit. So the importer can put the original manifest back, byte for byte, and the image keeps its digest through repack transport. The prototype does this: it carries the pre-repack manifest in the bundle, and restores it when the re-encode reproduces every blob it references. 25/25 restored.

The consequence is a change to the architecture I wrote: repack no longer has to break signatures. For already-ingested images it preserves digests, so a per-image signature made at ingest would still verify - if the bundle also carried the signature artifact, which my repack design deliberately doesn't, and which the prototype never tested, and as long as the importer runs the same cz1 profile the ingest did. My trust table was more pessimistic than reality; it is still the honest version that says "digests preserved", not "signatures shipped".

But a second reviewer caught me overstating it, and the correction is the honest version: digest preservation here depends on the manifest rewrite rules as much as on the encoder. Annotation set, key order, JSON serialisation - jq preserves insertion order; Go's encoding/json sorts map keys and HTML-escapes by default. Two implementations following my architecture doc would produce different manifest bytes and therefore different digests. So the rewrite rules have to be frozen alongside the encoder, with golden manifests in the test suite. The digest is a hash of the whole manifest; every byte of it is part of your public interface.

🔜 Next

That is the design: two transport modes tied to their trust anchors, a frozen encoder, a bundle whose format does the deduplication for you, and an importer that turned out to preserve more than I designed it to.

Designs are cheap. Part 2 is where I stop reasoning and start measuring - the bash prototype, the several ways my own measurements lied to me, four reviewers trying to break it, and the numbers. Including the one that overruled a decision I had just finished making.

🕳️ HomeLab Chronicles: Episode 9 - Tunnel Vision

sanskar arora — Sun, 30 Aug 2026 10:04:13 +0000

Hey all 👋

State of the lab: two nodes, Flux in charge, Airflow alive, backups nightly, RBAC banished to therapy. The UI works from anywhere in the house. But "in the house" is a service radius measured in Wi-Fi bars, and I wanted my Airflow at airflow.sanskararora.in, from anywhere, like a person with a real platform and not two laptops fighting a router for their lives.

The classic move is port forwarding: open a hole in the router, point it at the NodePort, pray. Every part of that sentence should upset you. My router has already been established, across two episodes, as this saga's least reliable character — and now I'd make it my front door too? Also: home IP published to the world, DDoS surface, dynamic-IP DNS jank. No.

Enter Cloudflare Tunnel. The trick that makes it homelab-perfect: a small daemon (cloudflared) runs inside the cluster and dials out to Cloudflare's edge. Traffic to my domain rides that already-open outbound connection back in. Zero inbound ports. Zero public IP. The router is demoted from front door to hallway, which is frankly all it ever deserved.

🚇 Two Tunnels Diverged in a Yellow Dashboard

First fork in the road, and I took both prongs like a professional: Cloudflare tunnels come in two flavors.

Locally-managed: cloudflared tunnel create on your machine, credentials JSON, routing rules in a config file → which for me means in git. GitOps-pure. Chef's kiss.

Remotely-managed: click "create tunnel" in the Zero Trust dashboard, get a token, routing rules live in the UI.

I wrote the manifests for flavor one, then created the tunnel in the dashboard, which is flavor two. The dashboard then helpfully offers you install commands — sudo cloudflared service install eyJ... — which are for running it on a host, i.e., not what a cluster deployment wants at all. The only thing you need from that whole screen is the token. The long eyJhIjoi... string. That's the tunnel's entire identity; treat it like a password, because it is one.

The in-cluster version is refreshingly small — a Deployment, two replicas with pod anti-affinity so the tunnel survives a node dying, cloudflared tunnel run reading TUNNEL_TOKEN from a Secret:

env:
  - name: TUNNEL_TOKEN
    valueFrom:
      secretKeyRef: {name: tunnel-token, key: token}

The token secret is created by hand with stringData: (paste as-is, the API server base64s it — hand-encoding a token that's already opaque gibberish is how you double-encode and then debug "auth errors" for an hour). The manifest holding it never touches git. .gitignore first, file second — that order, specifically, because git add -A has ended better tokens than mine.

Then in the dashboard: Public Hostname → airflow.sanskararora.in → HTTP → the Envoy gateway service's in-cluster FQDN on 8080. DNS record appears automatically. And because my HTTPRoute matches any Host header (Episode 4's hard-won lesson about hostnames), the request flows: Cloudflare edge → tunnel → Envoy → route → Airflow. Login page. On my phone. On mobile data. In a parking lot. I looked deranged and I did not care.

$ kubectl -n cloudflared logs -l app=cloudflared --tail=5
INF Registered tunnel connection ...
INF Registered tunnel connection ...

Four registered connections per replica. The dashboard's sad little "No connectors installed" flips to two healthy connectors. Lovely.

🚪 The Part You Don't Get to Skip

Now the uncomfortable paragraph. What I just built is: my Airflow admin login, on the public internet, guarded by one password, terminating inside my house. Airflow is a genuinely juicy target — it runs arbitrary code on schedule, that's its job — and it has the CVE history of software that important.

And if you're thinking "nobody knows my subdomain exists" — the TLS certificate for it landed in public Certificate Transparency logs the moment it was issued. Scanners watch those logs like RSS feeds. Your "secret" subdomain gets its first uninvited visitors in minutes, not months. Obscurity isn't a moat; it's a countdown. But in order for me to access it freely i am skipping it for now.

🧾 The Fine Print I'm Choosing to Live With

Remotely-managed means the hostname→service routing lives in Cloudflare's dashboard, not in my repo. Which dents the Episode 6 doctrine: a full cluster rebuild now restores everything except the tunnel's routing table. That's one manual step reintroduced into a system I spent three episodes making manual-step-free. It's written in the README, on the rebuild checklist, where future-me will find it and sigh.

The pure path exists — locally-managed tunnel, config in git, SOPS-encrypted token committed alongside it, flux bootstrap --decryption-provider=sops — and it's genuinely the endgame. But the dashboard tunnel was already made, it works, and I have learned (Episodes 5 through 8, inclusive) to stop refactoring things that are currently succeeding.

🧠 What This Taught Me

Outbound-only beats port forwarding in every dimension that matters. No inbound holes, no public IP, no router in the trust path.
Dashboard tunnel = token + rules in UI. CLI tunnel = credentials file + rules in git. Pick one before writing manifests, unlike some people.
Two replicas + anti-affinity makes the tunnel survive exactly the node-death this cluster has proven it can produce.
Certificate Transparency means your subdomain was never secret. Assume discovery is immediate, because it approximately is.
Access in front of everything. Authentication at the edge, before your network — not a nice-to-have, the whole point of fronting with Cloudflare.
stringData for token secrets. Let the API server do the base64. You will mangle it; it won't.

📋 Quick Reference (For Skimmers)

Purpose	Command / Place
Token into cluster	`kubectl -n cloudflared create secret generic tunnel-token --from-literal=token='eyJ...'`
Tunnel healthy?	`kubectl -n cloudflared logs -l app=cloudflared` → `Registered tunnel connection`
Route a hostname	Dashboard → Tunnels → Public Hostname → `HTTP` → `<envoy-svc>.envoy-gateway-system.svc.cluster.local:8080`
Find the Envoy svc name	`kubectl -n envoy-gateway-system get svc`
The moat	Zero Trust → Access → Applications → Self-hosted → Allow: your email
Who already knows your subdomain	crt.sh — search your domain, feel observed

🚀 What's Next

The Pi is still in the drawer. Cilium is still a rumor. But there's now a third-voter-shaped hole in this cluster's HA story, and dqlite quorum math says three is the first number that actually means anything. So: next episode, the drawer opens. Probably. The drawer has heard promises before.

💬 Final Thoughts

Nine episodes ago this was one laptop that couldn't survive a reboot. It's now a two-node, git-defined, nightly-backed-up, tunnel-fronted, edge-authenticated platform that survives power cuts out of spite. The router remains the weakest member of the team and has been architected around accordingly.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated card 🪪, chart-source bookmark 🔖 — and now a domain. The lab is public. Send help, or DAGs.

🤫 HomeLab Chronicles: Episode 8 - The Silent Treatment

sanskar arora — Sun, 30 Aug 2026 09:58:28 +0000

Hey all 👋

Quick recap: power cut (Ep5), GitOps (Ep6), locked myself out of my own control plane (Ep7), rebuilt in twenty minutes because GitOps (Ep6, smugly). Now: getting Airflow back up. Which should be easy, since I've installed it before — Episode 4, eight sidequests, badge earned.

This episode is about a different kind of enemy. Not the loud failure — the silent one. The config that gets accepted, nodded at, and thrown away. Because here's Helm's dirtiest little behavior, in one sentence:

Helm does not validate your values. An unknown key isn't an error. It's a shrug.

Typo a field, use a field from an older chart, invent a field that sounds plausible — Helm renders the templates, your key matches nothing, and the install proceeds looking exactly like success. No warning. No error. Just vibes.

🪦 The Graveyard of Values I Was Setting

Instead of trusting tutorials (or my own Episode 4, ahem), I finally did the thing: fetched the actual chart source — apache/airflow at tag helm-chart/1.22.0, whose appVersion is exactly my Airflow 3.2.2 — and checked every field I was setting against values.yaml and the templates. The census results:

Exhibit A: webserver: — valid key, wrong decade. Chart 1.22.0 ships both webserver: (Airflow 2) and apiServer: (Airflow 3). My replicas and resource limits sat under webserver:, where Airflow 3 never looks. Every limit I thought I'd set on the UI? Decorative. Episode 4 established the service was renamed; it didn't occur to me the values tree split too.

Exhibit B: logs.persistence.accessMode — does not exist. This one's beautiful. The chart's log-PVC template hardcodes accessModes: ["ReadWriteMany"]. Which means my careful Episode 4 setting of ReadWriteOnce — with its confident little comment about single-node clusters — was ignored the entire time. I got RWX whether I asked or not. The chart was protecting me from a config I was proud of.

Exhibit C: webserverSecretKeySecretName — Airflow <3 only. The comment in the chart source says it plainly: that's the Flask secret for the old webserver. Airflow 3 signs JWTs, wants jwtSecretName, and the secret's key must be exactly jwt-secret — that name comes from a template helper, not from you. Get this wrong and the chart generates a fresh key on every render, and your components restart "for no reason." A warning my very first install printed, months of installs ago, that I finally understand.

Exhibit D: knownHosts — a string, not a secret. I'd stuffed a known_hosts file into the SSH secret next to the deploy key, feeling organized. The chart wanted it as a plain values string under dags.gitSync.knownHosts. My copy sat in the secret, unread, while git-sync remained one bad default away from host-key failure. (Also, wait: got superseded by period:. Also-also, the SSH secret's key must be named gitSshKey verbatim — that one Episode 4 got right. Growth.)

Four fields. Four different flavors of silently doing nothing. helm lint caught zero of them, because none of them are syntax errors. They're fluent, grammatical nonsense.

🎩 Sidequest: The Job That Vanished Like a Magician

With values corrected, install. Pods appear! Pods crash-loop. Init container says:

TimeoutError: There are still unapplied migrations after 60 seconds.
MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'1d6611b6ab7c'}

set() — empty. Postgres reachable, schema blank. The migration job never ran. And when I went looking for it:

$ kubectl get jobs -A
No resources found

Nothing. Anywhere. Because by default the chart runs migrations as a Helm hook — it lives inside the install ceremony and gets cleaned up after, success or failure. My install had stalled (see below), Helm aborted early, the hook evaporated, and I was left with pods waiting for a schema that a ghost was supposed to write. Under Flux this deadlocks beautifully: pods can't start without migrations, Helm won't rerun migrations because pods failed.

The chart's own answer for GitOps land:

migrateDatabaseJob:
  enabled: true
  useHelmHooks: false
createUserJob:
  useHelmHooks: false

Now the jobs are ordinary resources: applied like everything else, visible in kubectl get jobs, logs readable when they fail. Debuggable beats ceremonial.

🥷 Sidequest: The Disk That Was "Already In Use"

One more silent saboteur, this time below Kubernetes entirely. Fresh Postgres volume, brand new, zero history — and:

MountVolume.MountDevice failed ... mke2fs 1.47.0
/dev/longhorn/pvc-... is apparently in use by the system;
will not make a filesystem here!

In use? It was born thirty seconds ago. I deleted every PV, PVC, and Longhorn volume in the cluster and got the identical error on the replacement — which is the tell that the problem isn't in the cluster at all.

It's multipathd. Ubuntu ships it enabled, and its hobby is grabbing new block devices the instant they appear, in case they're part of a multipath SAN. I do not have a SAN. I have two laptops. But multipathd claimed each fresh Longhorn device anyway, held it open, and mke2fs politely declined to format a held disk.

sudo systemctl disable --now multipathd multipathd.socket
kubectl -n airflow delete pod airflow-postgresql-0

Postgres: Running. It's a documented Longhorn prerequisite that approximately nobody's tutorial mentions, including — checks notes — my own host-prep list from Episode 4.

🧷 Bonus Round: The Password That Broke YAML

The admin password flows in through a generated values snippet. First version built that YAML with shell string concatenation, which works perfectly until someone's password contains a " — at which point the YAML is invalid and the failure surfaces much later as an inscrutable HelmRelease error, with the password (correctly!) absent from every log. The fix is to stop hand-assembling YAML like it's 1997: serialize with json.dumps — JSON is valid YAML, it's stdlib, and it escapes quotes, backslashes, colons and leading #s without opinion. Tested against the most hostile passwords I could invent. All survived.

🧠 What This Taught Me

Helm ignores unknown values silently. All of them. Forever. The only defense is reading the chart's actual values.yaml — at the tag you're installing — not tutorials, not memory, not last episode's notes.
helm show values <chart> | grep <field> before setting anything load-bearing. Thirty seconds. Cheaper than a weekend.
A valid key from the wrong era is worse than a typo. webserver: parsed fine, rendered fine, and did nothing. Typos at least look wrong.
Helm hooks and GitOps are a bad marriage. useHelmHooks: false turns invisible ceremonies into inspectable resources.
MigrationHead(s) in DB: set() means "empty schema," not "slow schema." The waiter isn't slow; the kitchen never got the order.
multipathd eats block devices on stock Ubuntu. If mke2fs claims a newborn disk is busy, that's your poltergeist.
Never build YAML with string concatenation when a serializer is right there.

📋 Quick Reference (For Skimmers)

Purpose	Command
The ground truth for chart fields	`helm show values apache-airflow/airflow > /tmp/v.yaml` then grep
Even grounder truth	read the chart repo at the release tag — templates don't lie
Airflow 3 UI settings live under	`apiServer:` (not `webserver:`)
JWT secret	`jwtSecretName:` → secret key must be `jwt-secret`
Jobs you can actually see	`migrateDatabaseJob.useHelmHooks: false`
Disk claimed at birth	`systemctl disable --now multipathd multipathd.socket`
Schema actually migrated?	look for `MigrationHead(s) in DB:` ≠ `set()`

🚀 What's Next

Airflow is up, migrations ran, and I can see it from my couch. But couch-radius availability is so Episode 4. Next time: a Cloudflare Tunnel puts it on my actual domain, from anywhere, without opening a single port on the plastic router of doom.

💬 Final Thoughts

Loud failures cost hours. Silent ones cost weekends — and this episode had four values, one hook, one daemon, and one quotation mark, all failing without making a sound.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated coredns card 🪪, and now a bookmark to the chart source. The toolkit is mostly talismans at this point.

🔒 HomeLab Chronicles: Episode 7 - Locked Out of My Own House

sanskar arora — Sun, 30 Aug 2026 09:50:03 +0000

Hey all 👋

Short episode. Painful episode. The kind where the call is coming from inside the house, the house is the control plane, and nobody inside the house is allowed to answer the phone.

Fresh rebuild, Flux repo ready, and I did the responsible thing: enabled RBAC while the cluster was empty. Every best-practices doc agrees — turn on authorization early, before workloads exist, while it's painless.

Reader, it was not painless.

🚨 Symptom One: Flux Bootstrap Times Out

✗ timeout waiting for: [CustomResourceDefinition/kustomizations... status: 'InProgress', ...]

Eleven CRDs, all stuck. But committed and pushed fine — the repo half worked, the cluster half hung. And when I inspected a stuck CRD:

status:
  acceptedNames: {kind: "", plural: ""}
  conditions: null

conditions: null. Not "failing." Not "pending." Never touched. On a healthy cluster the apiserver stamps a CRD as Established within milliseconds. These sat blank, like forms nobody would ever process.

🚨 Symptom Two: A Deployment Frozen in Time

$ kubectl -n kube-system get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
coredns    0/1     0            0           5h7m

Five hours old. UP-TO-DATE: 0. That column being zero means no ReplicaSet was ever created — which means the deployment controller never even looked at it. And with no DNS pod, everything downstream was queued up to fail alphabetically.

🔍 The Logs Confess Immediately

E... leaderelection.go: "Error retrieving lease lock"
  err="leases... is forbidden: User \"system:kube-controller-manager\"
  cannot get resource \"leases\" in namespace \"kube-system\""

E... "Failed to watch" err="csidrivers... is forbidden:
  User \"system:kube-scheduler\" cannot list resource \"csidrivers\""

Sit with that. The controller-manager — the component whose entire job is running every controller in Kubernetes — was forbidden from acquiring its own leader-election lease. No lease, no leadership; no leadership, no controllers; no controllers, no ReplicaSets, no pods, no anything. The scheduler was equally banned from reading the objects it schedules around.

I enabled authorization, and the first citizens it deported were the government.

🧩 The Part That Made It Weird

The obvious theory: missing ClusterRoleBindings. Checked. They existed. Correct subjects (User: system:kube-controller-manager), correct roles, roles containing exactly the permissions being denied. Checked the client certificates on disk:

subject=CN=system:kube-controller-manager
subject=CN=system:kube-scheduler

Perfect CNs. Perfect bindings. Perfect roles. Still forbidden.

Then the tell. I tried impersonation to test permissions, and even that failed:

selfsubjectaccessreviews is forbidden: User "system:kube-controller-manager"
cannot create resource "selfsubjectaccessreviews"

That permission comes from system:basic-user, bound to system:authenticated — the most universal binding in all of Kubernetes. Literally everyone gets it. When that denies, the story isn't "a binding is missing." The story is the RBAC authorizer isn't loading policy at all. Every rule in the cluster: correct, present, and completely unread.

Meanwhile kubectl get --raw='/readyz?verbose' said readyz check passed, because my admin cert (hello, system:masters) bypasses authorization entirely. The apiserver was healthy. The apiserver was also the only one allowed in the building. Health checks measure the doorman, not whether anyone can get past him.

🧯 The Retreat

sudo microk8s disable rbac
sudo microk8s stop && sudo microk8s start
$ grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver
--authorization-mode=AlwaysAllow

Forbidden errors: gone. Lease: held and renewing. And coredns... still 0/1, UP-TO-DATE 0. Because five hours of objects written into a paralyzed control plane don't spring to life on forgiveness — they'd been created into a void, and some of them (plus a growing chorus of node "msi" not found) were simply inert. At that point the cluster was 24 hours old and held nothing I loved, so I invoked Episode 6's superpower: reset, rejoin, bootstrap. Twenty minutes. This is what the git repo was for.

The rebuilt version skips enable rbac, and adds the check I'll never skip again: coredns must reach 1/1 before anything else happens. It's the cheapest possible proof that the datastore persists writes and the controllers act on them. One deployment, one honest answer.

🧠 What This Taught Me

readyz measures the apiserver, not the cluster. A green apiserver can preside over a completely dead controller plane, especially when your admin cert skips the auth queue.
UP-TO-DATE: 0 on an old deployment is a five-alarm signal. It doesn't mean "unhealthy pods." It means "no controller has ever processed this."
When even system:basic-user is denied, stop auditing bindings. Universal denial means the authorizer itself, not the policy.
A homelab without RBAC is a defensible trade. It governs in-cluster identities, not network exposure; the API server is still cert-authenticated. MicroK8s ships it off by default, and I now understand that as a mercy, not an oversight.
"Enable it while it's painless" assumes it works. Best practice on paper, lockout in my kitchen.

📋 Quick Reference (For Skimmers)

Purpose	Command
Is the controller plane actually alive	`kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.renewTime}'` — must be seconds old
The canary	`kubectl -n kube-system get deploy coredns` → wants `1/1`
CRD ever processed?	`-o jsonpath='{.status.acceptedNames.kind}'` — empty = never
Current auth mode	`grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver`
Component identities	decode `client-certificate-data` from `/var/snap/microk8s/current/credentials/*.config` → `openssl x509 -noout -subject`

🚀 What's Next

Cluster rebuilt, Flux green, Airflow installing... and then Helm taught me its favorite trick: accepting configuration it has no intention of using, silently, with a straight face. Next episode is about every value I set that did absolutely nothing.

💬 Final Thoughts

I gave my cluster a security system. The security system's first act was arresting the janitor, the electrician, and itself. The house stood perfectly secure and perfectly dark.

Popcorn 🍿, coffee ☕, UPS 🔌, and a laminated card that says check coredns first.

📜 HomeLab Chronicles: Episode 6 - Source of Truth

sanskar arora — Sun, 30 Aug 2026 09:45:04 +0000

Hey all 👋

Last episode a power cut exposed an uncomfortable fact: my cluster's entire memory lived in one SQLite-flavored database, on one laptop, bound to one Wi-Fi address, guarded by one aging battery. Four single points of failure in a trench coat.

The fix isn't making that database unkillable. The fix is making it unimportant. If every manifest lives in git and something reconciles the cluster against git continuously, then "the datastore died" stops being a tragedy and becomes a reboot with extra steps.

So: Flux. Here's the setup, and the four ways I face-planted installing it.

🗂️ The Shape of the Repo

clusters/homelab/
  flux-system/          <- Flux writes this at bootstrap; hands off
  infrastructure.yaml   <- points at infrastructure/
  apps.yaml             <- points at apps/
infrastructure/
  controllers/          <- Longhorn + Envoy Gateway HelmReleases
  configs/              <- GatewayClass, Gateway, StorageClasses
apps/homelab/
  airflow/              <- the actual point of all this

Three Flux Kustomizations, chained: infra-controllers → infra-configs → apps, via dependsOn. That chaining is not decoration. My GatewayClass can't exist until Envoy Gateway's CRDs exist, and the CRDs arrive with the controller's Helm chart. Without dependsOn, Flux sprints ahead, tries to create a GatewayClass into a cluster that's never heard of GatewayClasses, and fails with the enthusiasm of a golden retriever running into a glass door.

dependsOn plus wait: true turns that into: install controllers, wait until healthy, then configs, then apps. Boring. Sequential. Correct. The three great virtues.

🔑 Sidequest 1: The Token Bureaucracy

flux bootstrap github needs a GitHub token, and the docs-diving summary is:

Classic PAT: repo scope. Needed if Flux should create the repo.
Fine-grained PAT (pre-created repo): Contents read/write, Metadata read, and — the one everyone misses — Administration read/write, because Flux installs an SSH deploy key on the repo, and deploy keys are an admin operation.

Here's the nice part: the deploy key is what Flux uses forever after. The PAT is scaffolding. Bootstrap, confirm flux get kustomizations goes green, then revoke the token. Give it a 7-day expiry at creation and let it die of natural causes. Also: I renamed the repo mid-project and exactly zero manifests cared — the repo URL lives only in the bootstrap-generated gotk-sync.yaml. Rename after bootstrapping though, and Flux keeps pulling the old URL. Ask me how I know I almost knew that.

💥 Sidequest 2: The Secret That Must Come First

Some things cannot live in a public-ish git repo: the git deploy key for DAG sync, the Airflow admin password, the JWT signing key. Those get created by hand (well — by script), out-of-band.

And order matters more than I expected. My Airflow HelmRelease pulls its password via valuesFrom with optional: false — no secret, no install. I bootstrapped first and created secrets second, like a person with hope in their heart. Flux tried to install Airflow, found no secret, retried three times, exhausted its retry budget, and parked the release in a failed state.

Then I ran the secrets script and hit this beauty:

✗ Failed to perform remediation: missing target release for rollback:
  cannot remediate failed release

Translation: the first ever install failed, so there's no previous good release to roll back to, so Flux's remediation has nothing to grab. A plain flux reconcile won't retry it. You need --force, or a suspend/resume, or — the actual lesson — create the secrets before you bootstrap. The cluster exists after MicroK8s comes up; Flux doesn't need to be watching yet for you to kubectl create secret.

📌 Sidequest 3: Pin Your Charts Like You Mean It

First draft, every HelmRelease said version: "1.x". Feels flexible. Is flexible. Flexible means "a major chart upgrade can land at 3am while a reconcile loop runs unattended," and for a storage system that's not a version policy, that's a séance.

So I went and found the real versions — including the fun discovery that Airflow's chart 1.22.0 has appVersion: 3.2.2, exactly the Airflow I was running. Now the repo says:

airflow:      version: "1.22.0"
longhorn:     version: "1.12.1"
gateway-helm: version: "v1.9.1"

Upgrades still happen. They just happen when I commit them, on purpose, with coffee.

🫥 Sidequest 4: The Dot of Shame

One evening a reconcile hung with context deadline exceeded and zero pods appearing. Twenty minutes of cluster spelunking later, I looked at my own shell prompt:

╭─sunny@sunny ~/github/lab ‹main●›

That little dot. Uncommitted changes. I'd committed the kustomization entry for a new directory but not the directory. Flux was faithfully applying exactly what I'd pushed, which was a reference to a folder that existed only on my laptop. Flux cannot read your working tree. Flux cannot read your intentions. Flux reads origin/main, and origin/main reads whatever you actually pushed.

The dot was there the whole time. Judging me.

⚖️ What Flux Is Not

Worth saying plainly, because I briefly believed otherwise: Flux is not a backup system. It restores what's declared — Deployments, HelmReleases, routes. It knows nothing about what accumulated at runtime: your Postgres data, DAG run history, Longhorn volume contents, the secrets you created by hand.

So the full recovery story is a two-piece set: Flux rebuilds the skeleton from git, and last episode's nightly dbctl snapshots plus (eventually) Longhorn backup targets restore the organs. Either one alone is half a cluster.

The payoff, though, is real: the rebuild I did mid-way through all this — full microk8s reset, rejoin the Dell, bootstrap, secrets script — took about twenty minutes, most of it image pulls. Episode 5 me would have wept. Episode 6 me made tea.

🧠 What This Taught Me

dependsOn is load-bearing. CRDs before custom resources, controllers before configs, or Flux runs face-first into glass.
Secrets before bootstrap. An optional: false reference plus a missing secret burns the retry budget and needs --force to resurrect.
A first-install failure has no rollback target. "Cannot remediate failed release" means delete-and-retry, not reconcile-harder.
Version ranges are unattended upgrades wearing a disguise. Pin, then upgrade on purpose.
Flux reads the remote, not your working tree. The ● in your prompt is a Flux error message you're rendering locally.
GitOps restores manifests, backups restore data. You need both, and neither is impressed by the other.

📋 Quick Reference (For Skimmers)

Purpose	Command
Bootstrap	`flux bootstrap github --owner=X --repository=Y --branch=main --path=clusters/homelab --personal`
Fine-grained PAT needs	Contents RW, Metadata R, Administration RW
What's reconciling	`flux get kustomizations`
Chart status + real errors	`flux get helmreleases -A`
Push a change now	`flux reconcile kustomization apps --with-source`
Resurrect a dead first install	`flux reconcile helmrelease X -n ns --force`
Why is it stuck	`flux logs --kind=HelmRelease --name=X -n ns`

🚀 What's Next

With the repo in charge, I did the responsible thing and enabled RBAC on the fresh cluster — you know, best practice — and the control plane immediately locked itself out of its own house. Next episode is short, painful, and has a body count of one coredns.

💬 Final Thoughts

Git is now the cluster. The cluster is now a cache. The dot in my prompt is now a trauma response.

Popcorn 🍿, coffee ☕, UPS 🔌, and a pre-push git status — the ritual expands.

🔦 HomeLab Chronicles: Episode 5 - Blackout

sanskar arora — Sun, 30 Aug 2026 09:40:23 +0000

Hey all 👋

Last episode ended on a high. Airflow UI on my phone, logs working, DAGs syncing, and a solemn promise: Dell out of the drawer, Pi out of the other drawer, Cilium out of my dreams and into the cluster.

Here's the scoreboard. The Dell came out of the drawer — actually joined the cluster, same night I published. The Pi remains in its drawer, thriving, unbothered. Cilium remains a concept I respect from a distance. One out of three. In this economy, that's a win.

And then at 05:22 the next morning, the power went out. And when it came back, my cluster had amnesia.

$ kubectl get deployments -A
NAMESPACE     NAME                      READY
kube-system   calico-kube-controllers   1/1

That's it. That's the whole cluster. Airflow: gone. Longhorn: gone. My will to live: buffering.

🕵️ The Crime Scene

First instinct: the datastore got corrupted by the power cut. Reasonable. Dramatic. Wrong — but let's earn that.

MicroK8s doesn't use etcd; it uses dqlite — Raft-replicated SQLite, living at /var/snap/microk8s/current/var/kubernetes/backend/. So that's where I went digging:

$ sudo ls -la /var/snap/microk8s/current/var/kubernetes/backend/
-rw-rw-r-- cluster.crt        Aug 28 21:42
-rw-rw---- cluster.key        Aug 28 21:42
-rw-rw---- metadata1          Aug 28 21:42
-rw-rw---- 0000000000074753-0000000000075264   ...

Read those timestamps again. The database certificates were born at 21:42 the night before. The raft index was in the seventy-thousands — a day of idle heartbeats, not weeks of a running cluster. And:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   16h
kube-node-lease   Active   16h
kube-public       Active   16h
kube-system       Active   16h

Four namespaces. The four a cluster is born with. All sixteen hours old.

The plot twist, delivered by my own filesystem: nothing was lost in the power cut, because nothing was there. When I joined the Dell the previous night, the datastore got reinitialized underneath me. The database I spent the morning mourning was an empty sixteen-hour-old husk. The power cut didn't rob the house. The power cut just knocked on the door of a house that was already empty.

Grief has five stages. ls -la has one.

📡 So What Did the Power Cut Do?

Something sneakier. Look at the dqlite logs from 05:22:

dial: Failed to connect to HTTP endpoint: dial tcp 192.168.68.210:19001:
connect: network is unreachable

Repeated. For a while. Now, the tempting read is "boot race — Wi-Fi wasn't up yet when dqlite started." I almost bought it. But the dqlite process was PID 1919211. Kubelite was PID 2.3 million. You don't get seven-digit PIDs on a fresh boot. That machine had been up for eight hours when this happened.

So it wasn't a boot race. The network vanished mid-run. And here's the geometry of my particular disaster:

My nodes are laptops. My router is not.

When the power cuts, the MSI and the Dell shrug and switch to battery. The router — a small plastic box with the survival instincts of a houseplant — dies instantly. Wi-Fi drops, the interface loses its address, and dqlite, which is bound to 192.168.68.210:19001, suddenly can't reach itself.

And why is it bound to the LAN address at all? Because of the Dell. dqlite binds to 127.0.0.1 by default — a fresh single-node MicroK8s is structurally immune to network outages. The moment a second node joins, dqlite moves to the LAN IP, because the other node has to reach it. Joining the Dell put my datastore on Wi-Fi. The drawer had a curse on it.

🔋 The Villain Nobody Suspects

But wait — a frozen datastore recovers when the network comes back. Frozen isn't corrupted. Where does actual damage come from?

Follow the battery. Power cut → laptops keep running → outage outlasts the battery → laptop hits 0% → hard poweroff. No filesystem sync. No dqlite checkpoint. Just lights out, mid-write. That is the dirty shutdown that corrupts a SQLite-family datastore, and it doesn't happen during the outage. It happens hours later, quietly, while you're asleep and feeling safe.

The fix has absolutely nothing to do with Kubernetes. It's laptop power management:

# /etc/UPower/UPower.conf
UsePercentageForPolicy=true
PercentageLow=25
PercentageCritical=22
PercentageAction=20
CriticalPowerAction=PowerOff

Clean shutdown at 20% instead of a crash at 0%. (UPower insists Low > Critical > Action, so mind the ordering.) Plus telling logind that a closed lid on a server is decor, not a command:

# /etc/systemd/logind.conf.d/99-homelab.conf
[Login]
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
IdleAction=ignore

sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

And a longer shutdown grace so dqlite gets to checkpoint instead of eating SIGKILL at the 90-second default:

# /etc/systemd/system.conf.d/99-shutdown-timeout.conf
[Manager]
DefaultTimeoutStopSec=120s

🧠 The Mental Model I Was Missing

The reason "my deployments are gone" felt so confusing is that three completely different things all get called "the cluster":

Layer	Lives in	Survives a reboot?
Cluster state — Deployments, Services, routes	dqlite, on disk	Yes, if dqlite comes up clean
Data — Postgres, volumes	Longhorn, on disk	Yes
Pods	Nowhere. Vapor.	No — and that's correct

Pods are supposed to vanish and be recreated from Deployments. That's not fragility, that's the design. The only question that ever matters after an outage is: did layer one come back intact? Everything else follows from it.

And ranked by what actually threatens layer one:

Battery draining to a dirty poweroff ← the real one
Network loss while running (freezes it; recovers)
Clean reboot (total non-event)

I'd been emotionally preparing for #3, mildly worried about #2, and completely blind to #1.

💾 And Because Paranoia Is Now a Feature

dqlite has a backup tool nobody talks about:

microk8s dbctl backup /var/backups/microk8s/dqlite-$(date +%Y%m%d-%H%M%S)

Mine now runs nightly from a systemd timer, and building that had its own comedy: /snap/bin isn't on systemd's default PATH, so the naïve version dies at 3am with command not found, silently, forever — a backup job that never runs but that you believe exists is strictly worse than no backup job. Absolute paths, artifact-size verification, a free-space guard, 7-day retention with a keep-minimum floor, and chmod 700 on the directory because a full cluster dump contains every Secret you own.

Oh — and copy them off the node. A backup on the disk that died is a souvenir.

🧠 What This Taught Me

Check timestamps before you grieve. ls -la on the datastore told me in ten seconds what I'd theorized about for an hour.
Joining a second node moves dqlite from loopback to the LAN. Single-node clusters are accidentally immune to network chaos. Multi-node clusters opt into it. Nobody sends you a consent form.
Laptops-as-servers invert the failure mode. The nodes survive the outage; the router is the single point of failure, and the battery turns a power cut into a delayed-action dirty shutdown.
Pods vanishing is fine. The question is always the datastore.
A silent backup job is worse than none. Verify the artifact exists, verify it has bytes, and scream loudly if not.

📋 Quick Reference (For Skimmers)

Purpose	Command
Where dqlite lives	`/var/snap/microk8s/current/var/kubernetes/backend/`
Who dqlite binds to	`sudo cat .../backend/cluster.yaml`
Datastore's actual age	`sudo ls -la .../backend/` — read `cluster.crt`'s date
Snapshot the datastore	`microk8s dbctl backup <path>`
Restore it	`microk8s stop && microk8s dbctl restore <path> && microk8s start`
Stop battery murder	UPower `PercentageAction=20`, `CriticalPowerAction=PowerOff`
Lid ≠ off switch	logind `HandleLidSwitch=ignore` + mask sleep targets

🚀 What's Next

If the cluster can be vaporized by a plastic router and a tired battery, then the cluster can't be the source of truth anymore. Next episode: I put the entire thing in git and make "everything is gone" a twenty-minute inconvenience instead of a personality crisis.

💬 Final Thoughts

The power cut was innocent. The router was negligent. The battery was armed. And the database was already empty before any of them showed up.

Stay tuned. Popcorn 🍿, coffee ☕, and now a UPS 🔌 — the shopping list grows with the wisdom.

🌪️ HomeLab Chronicles: Episode 4 - Turbulence

sanskar arora — Fri, 28 Aug 2026 14:47:14 +0000

Hey all 👋
Sit tight! Because this is a little lengthy one.

Last episode I closed by announcing that I would be adding two more nodes and bringing in Cilium. Confident. Specific. A real roadmap.

The Dell is still in the drawer. The Raspberry Pi is still in a different drawer. Cilium remains a concept I respect from a distance.

What I did instead was install Apache Airflow on the one node I already had, which I estimated at "an evening." Reader, it was not an evening. It was eight distinct sidequests, two of which were caused by me, three by Airflow 3 quietly renaming things, and one by a YAML field that Kubernetes refuses to let you edit after the fact.

Here's all of it.

🎯 Why Kubernetes At All

The honest alternative was Docker Compose on bare Ubuntu, and for a single-box homelab that's a completely defensible choice -- lower overhead, one file, done in twenty minutes.

I went with MicroK8s anyway, for three reasons that I still think hold up:

Task isolation. With KubernetesExecutor, every task run gets its own ephemeral pod with its own dependencies and its own CPU/memory bounds. A runaway DAG starves itself, not the box.
No idle workers. Pods spawn on demand and die when they're done. No permanent Celery workers sitting there eating 1–2GB of RAM to do nothing at 3am.
Production parity. This is roughly how Airflow runs on EKS/GKE/Composer. If I'm going to fight something, I'd rather fight the thing people actually run.

The tuning that follows from that: KubernetesExecutor, Postgres with persistent storage, and statsd / redis / flower all switched off, since with KubernetesExecutor the Kubernetes API is the queue and those three are just RAM tax.

🕹️ Driving From the Couch

First useful realization: Helm is a client-side tool. It talks to the cluster over the Kubernetes API (port 16443 on MicroK8s) using nothing but a kubeconfig. There is no reason to SSH into the server to run it.

So the server exports its config:

sudo microk8s config

And my laptop takes it:

mkdir -p ~/.kube
ssh msi@192.168.68.210 "sudo microk8s config" > ~/.kube/config-microk8s
export KUBECONFIG=~/.kube/config-microk8s
chmod 600 ~/.kube/config-microk8s

kubectl get nodes

One gotcha: if the generated config says 127.0.0.1:16443, swap it for the server's actual LAN IP or you'll spend a confusing minute deploying Airflow to your own laptop.

From here on, everything in this post runs from my laptop. The server just quietly does as it's told.

💥 Sidequest 1: The Migration Job That Wasn't the Problem

First install. Confident. Straight into a wall:

Error: INSTALLATION FAILED: failed post-install: resource Job/airflow/airflow-run-airflow-migrations
not ready. status: InProgress, message: Job in progress
context deadline exceeded

The database migration job timed out. Fine, let's read its logs:

$ kubectl logs airflow-run-airflow-migrations -n airflow
error: error from server (NotFound): pods "airflow-run-airflow-migrations" not found in namespace "airflow"

That's me being sloppy -- that's a Job name, not a Pod name. kubectl logs wants a pod. And by the time I asked, the job controller had blown past its backoff limit and deleted the pod anyway. Use the selector instead:

kubectl logs -n airflow -l component=run-airflow-migrations --tail=100

Which returned nothing, because there was nothing left to return. So I went and looked at what everything else was doing:

$ kubectl get pods -n airflow -o wide
NAME                                     READY   STATUS                  RESTARTS      AGE
airflow-api-server-669b5665d7-5h87d      0/1     Init:CrashLoopBackOff   6 (98s ago)   9m14s
airflow-dag-processor-6d6ffd87cc-lr9n9   0/2     Pending                 0             9m14s
airflow-postgresql-0                     0/1     Pending                 0             9m14s
airflow-scheduler-7d9648844f-g248q       0/2     Init:CrashLoopBackOff   6 (90s ago)   9m14s
airflow-triggerer-0                      0/2     Pending                 0             9m14s

Three pods Pending, two crash-looping on init. The crash-loopers are just waiting for the database, so they're symptoms. Postgres is the interesting one, and describe gave it up immediately:

Warning  FailedScheduling  4m10s  default-scheduler
  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.

There it is. MicroK8s had no default StorageClass. No provisioner meant no PVC could ever bind, which meant Postgres could never schedule, which meant the migration job had nothing to migrate against, which meant Helm reported a migration timeout.

The error I got was four layers downstream of the actual problem. The install didn't fail because of migrations. It failed because I never turned on storage.

sudo microk8s enable hostpath-storage
kubectl get storageclass   # microk8s-hostpath (default)

Lesson filed: when a Helm install fails on a post-install hook, the hook is almost never the thing that's broken. Go find whatever's Pending.

🐄 Sidequest 2: Longhorn, and the Snap Path Nobody Mentions

hostpath-storage unblocked me, and I could have stopped there. I did not stop there, because "hostpath" means "a directory on the node" and I wanted an actual CSI layer with snapshots and a UI before I started putting data I cared about into this thing.

Longhorn needs host-level prep first -- it's iSCSI-based, and that has to exist on the machine, not in the cluster:

sudo apt update
sudo apt install -y open-iscsi nfs-common util-linux
sudo systemctl enable --now iscsid
sudo modprobe iscsi_tcp

Then the values file, which has two things in it that matter enormously and are easy to miss:

persistence:
  defaultClass: true
  defaultClassReplicaCount: 1
  reclaimPolicy: Delete

defaultSettings:
  defaultReplicaCount: 1
  replicaSoftAntiAffinity: true
  storageOverProvisioningPercentage: 200
  storageMinimalAvailablePercentage: 15

csi:
  kubeletRootDir: "/var/snap/microk8s/common/var/lib/kubelet"

Replica count 1, because Longhorn defaults to 3 and on a one-node cluster every single volume will sit there forever politely waiting for two more nodes that are, as established, in drawers.

kubeletRootDir, because MicroK8s is a snap, and snaps don't put kubelet where every Longhorn tutorial on the internet assumes it is. Get this wrong and the CSI plugin comes up looking healthy and then mounts nothing.

And then Helm couldn't even download the chart:

Error: INSTALLATION FAILED: Get "https://release-assets.githubusercontent.com/...longhorn-1.12.1.tgz":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Not a cluster problem at all -- my laptop's connection to GitHub's release CDN was too slow for Helm's built-in client timeout. The fix is to stop asking Helm to be your downloader:

helm pull longhorn/longhorn
helm install longhorn ./longhorn-*.tgz \
  --namespace longhorn-system \
  --create-namespace \
  --values longhorn-values.yaml \
  --timeout 15m

Fetch the tarball with a tool whose entire job is fetching things, then install from disk. Works offline too, which is a nice side effect.

🚪 Sidequest 3: Two Owners, One CRD

Airflow was up. Now I wanted to reach it from my phone without kubectl port-forward, which meant an ingress, and I decided on Envoy Gateway -- the CNCF one, driven by the Gateway API instead of a pile of NGINX annotations.

Step one: install the Gateway API CRDs. Step two: install Envoy Gateway. Both from the docs, both correct in isolation:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.4 --namespace envoy-gateway-system --create-namespace

Error: INSTALLATION FAILED: failed to install CRD crds/gatewayapi-crds.yaml: conflict occurred while
applying object /gatewayclasses.gateway.networking.k8s.io: Apply failed with 3 conflicts:
conflicts with "kubectl-client-side-apply" using apiextensions.k8s.io/v1:
- .metadata.annotations.gateway.networking.k8s.io/bundle-version
- .metadata.annotations.gateway.networking.k8s.io/channel
- .spec.versions

This one's genuinely interesting. The Envoy Gateway chart ships the Gateway API CRDs itself. So I installed them twice: once via kubectl apply (which registers a field manager called kubectl-client-side-apply), then again via Helm's server-side apply. Kubernetes tracks who owns which field, saw two different managers claiming the same annotations, and refused to guess.

Two ways out. Let Helm own them:

helm uninstall eg -n envoy-gateway-system 2>/dev/null
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml
# reinstall -- the chart brings its own

Or keep mine and tell the chart to stay out of it:

helm install eg oci://docker.io/envoyproxy/gateway-helm --skip-crds ...

I went with letting Helm own them, so version compatibility stays somebody else's problem on future upgrades. Worth noting the delete nukes any existing Gateway/HTTPRoute objects -- harmless on a fresh cluster, less harmless later.

🧭 The Manifests, And Two Things The Obvious Version Gets Wrong

The GatewayClass registers Envoy's controller. The Gateway defines a listener, and creating it makes Envoy Gateway provision an actual Envoy data-plane Deployment and Service. Standard:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-homelab-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
      allowedRoutes:
        namespaces:
          from: All

allowedRoutes.namespaces.from: All is what lets the HTTPRoute live over in the airflow namespace and still attach here.

Now the two traps.

Trap one: you cannot put an IP address in hostnames. The Gateway API Hostname type explicitly excludes numeric IPs. The CRD's regex is loose enough that 192.168.68.210 slips past admission -- each octet looks like a valid DNS label -- so nothing errors and you assume it worked. It is not supported and the matching behaviour isn't guaranteed. (This one bit me later. See sidequest 5.)

Trap two: don't kubectl patch the Envoy Service. Every guide tells you to find the generated service and patch it to NodePort:

kubectl patch svc $ENVOY_SVC -n envoy-gateway-system -p '{"spec": {"type": "NodePort", ...}}'

That works, right up until it doesn't. That Service is controller-owned -- Envoy Gateway recomputes it from the Gateway spec on controller restarts, leader-election events, and Gateway updates. Nothing in the cluster knows you wanted NodePort, so the next reconcile quietly resets it to ClusterIP. It'll break at the worst possible moment and look like a networking problem.

The durable way is an EnvoyProxy resource wired into the GatewayClass via parametersRef, which also lets you pin the nodePort declaratively:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
                - name: http-8080
                  port: 8080
                  protocol: TCP
                  targetPort: 8080
                  nodePort: 30080

Then add the parametersRef to the GatewayClass pointing at it. Now NodePort 30080 is part of the desired state, and reconciliation enforces it instead of undoing it.

🕵️ Sidequest 4: A 500 From A Service That Doesn't Exist

Gateway green. Route applied. Browser: nothing.

Start at the top and work down. Gateway first:

$ kubectl get gateway envoy-homelab-gateway -n envoy-gateway-system -o yaml
status:
  addresses:
  - value: 192.168.68.210
  conditions:
  - reason: Accepted    status: "True"
  - reason: Programmed  status: "True"   message: Address assigned, 1/1 envoy replicas available

Perfect. Service is NodePort on 30080, Envoy pod 2/2 Running. So the whole ingress layer is fine. Then the route:

$ kubectl get httproute airflow-route -n airflow -o yaml
status:
  parents:
  - conditions:
    - reason: Accepted           status: "True"    message: Route is accepted
    - reason: BackendNotFound    status: "False"   message: Service airflow/airflow-webserver not found

Service airflow/airflow-webserver not found. And sure enough:

$ kubectl get svc -n airflow
NAME                    TYPE        CLUSTER-IP       PORT(S)
airflow-api-server      ClusterIP   10.152.183.153   8080/TCP
airflow-postgresql      ClusterIP   10.152.183.194   5432/TCP
airflow-postgresql-hl   ClusterIP   None             5432/TCP
airflow-triggerer       ClusterIP   None             8794/TCP

There is no airflow-webserver. Airflow 3.x deleted the webserver. The UI got folded into the API server, so the chart now creates <release>-api-server. Every tutorial written for Airflow 2 points at a service that no longer exists.

This is also why the local curl returned a 500 rather than a connection error -- Envoy was alive and listening, it just had no valid backend cluster to route to, so it answered every request with an empty 500. The proxy was working perfectly. It had simply been told to forward traffic to a ghost.

One word changed in backendRefs, and:

$ curl -v -H "Host: airflow.local" http://localhost:30080/
< HTTP/1.1 200 OK
< server: uvicorn
<!doctype html>
<title>Airflow</title>

🌐 Sidequest 5: curl Works, The Browser 404s

Victory lap over to the browser on my phone. http://192.168.68.210:30080/:

Looks like there's a problem with this site Error code: 404 Not Found

Two problems stacked, and I'd caused both.

First, I'd been typing 192.168.68.21 into the browser. The node is 192.168.68.210. I stared at that for longer than I want to admit.

Second, and more instructive: my curl was lying to me. Look at what I'd been testing with:

curl -H "Host: airflow.local" http://localhost:30080/

I was manually forcing the Host header to the one value my HTTPRoute matched. The browser sends Host: 192.168.68.210:30080, my route only had hostnames: ["airflow.local"], so Envoy had no matching route and correctly 404'd. Note the failure mode here -- a 404 from the browser is Chrome rendering a real HTTP response from Envoy, not a connection failure. The packet arrived. The routing table just didn't want it.

Since the Gateway API won't let me legitimately match an IP anyway (trap one, from earlier), the fix is to stop matching hostnames entirely. An HTTPRoute with no hostnames field matches any Host header:

spec:
  parentRefs:
    - name: envoy-homelab-gateway
      namespace: envoy-gateway-system
      sectionName: http
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: airflow-api-server
          port: 8080

Airflow UI, on my phone, on the couch. Roughly six hours after "this should take an evening."

📂 Sidequest 6: Where Do DAGs Even Go?

Next: actually put a DAG in it. I went looking for the DAGs folder on the scheduler:

$ kubectl describe pod -n airflow -l component=scheduler
Volumes:
  config:  ConfigMap
  logs:    EmptyDir
  kube-api-access-7zh2t: Projected

No DAGs volume. At all. Not empty -- absent.

Because Airflow 3 also split DAG parsing out of the scheduler into its own dag-processor component. The scheduler genuinely doesn't need DAG files anymore. The volume was on the pod I hadn't looked at:

$ kubectl describe pod airflow-dag-processor-... -n airflow
Mounts:
  /opt/airflow/dags from dags (rw)
Volumes:
  dags:
    Type:       PersistentVolumeClaim
    ClaimName:  airflow-dags

That's two components Airflow 3 renamed or relocated out from under every guide I was reading. Worth internalizing if you're following Airflow 2 material: webserver → api-server, and DAG parsing → dag-processor.

I could have kubectl cp'd files into that PVC. I wanted git-sync instead, from a private repo, which needs a read-only deploy key:

ssh-keygen -t ed25519 -C "airflow-gitsync" -f ./airflow-gitsync-key -N ""

kubectl create secret generic airflow-git-ssh-key-secret \
  -n airflow --from-file=gitSshKey=./airflow-gitsync-key

The secret key must be named gitSshKey -- that's the exact name the chart looks for. Public half goes on the repo as a deploy key with write access unchecked; git-sync only ever pulls.

Then in values:

dags:
  persistence:
    enabled: true      # keep this ON -- git-sync syncs *into* the PVC
    size: 5Gi
    accessMode: ReadWriteOnce
  gitSync:
    enabled: true
    repo: "git@github.com:<user>/<private-repo>.git"
    branch: "main"
    subPath: "dags"
    sshKeySecret: "airflow-git-ssh-key-secret"
    wait: 10

Git-sync runs as a sidecar on the dag-processor, worker, and triggerer pods -- not the scheduler, for the reason above.

📜 Sidequest 7: Logs That Point At Ghosts

Triggered a DAG. It ran. Clicked the logs tab:

Could not read served logs: HTTPConnectionPool(host='test-branch-and-bash-option-a-2njq669w', port=8793):
Max retries exceeded ... Failed to resolve 'test-branch-and-bash-option-a-2njq669w'
([Errno -2] Name or service not known)

Two episodes ago I spent an evening on a ghost that answered pings. This time it's a hostname that used to be a pod.

Here's the mechanism: by default the API server fetches task logs by making an HTTP request directly to the pod that ran the task, on port 8793. Under KubernetesExecutor, task pods are deleted the instant they finish. So the API server is trying to resolve the DNS name of something that stopped existing several seconds ago -- and a bare pod name isn't independently resolvable anyway without a governing Service.

The fix is to stop making it a network problem. Give everything a shared volume and let the API server read the file off disk:

logs:
  persistence:
    enabled: true
    size: 5Gi

The usual caveat is that this wants ReadWriteMany and most simple provisioners don't do RWX. On a single-node cluster it doesn't matter -- RWO means "mountable by one node", not one pod, and every pod here lands on the same node regardless. One-node homelabs get to skip a whole class of storage problem.

🧊 Sidequest 8: The StatefulSet That Wouldn't Budge

Applied that. Helm said no:

Error: UPGRADE FAILED: server-side apply failed for object airflow/airflow-triggerer
apps/v1, Kind=StatefulSet: StatefulSet.apps "airflow-triggerer" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template',
'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

Not a chart bug, not a values mistake -- a hard Kubernetes rule. volumeClaimTemplates on a StatefulSet is immutable after creation. The triggerer had been created with its own per-pod log volume. Turning on shared log persistence removes that block entirely. Kubernetes doesn't allow that field to change on a live object, full stop.

The only move is to delete the object and let Helm rebuild it:

kubectl delete statefulset airflow-triggerer -n airflow
helm upgrade airflow apache-airflow/airflow -n airflow --values airflow-values.yaml

The pod blips for a few seconds, deferred-task processing pauses, nothing is lost. StatefulSets don't garbage-collect their PVCs, so the old per-pod log volume survives the delete as an orphan -- worth cleaning up once you've confirmed the new setup is healthy:

kubectl get pvc -n airflow          # look for logs-airflow-triggerer-0
kubectl delete pvc logs-airflow-triggerer-0 -n airflow

🚧 And Then, A Cliff to jump off of

or at-least it looked like one

Logs working. DAGs syncing. UI reachable from every device in the house. I typed in the admin password, hit enter, and:

Bad Request The CSRF session token is missing.

Created tens of theories hundreds of stories but at a end one patient moment solved it. When i tried longing in after a few seconds it worked, my working hypothesis as of is that the pod to pod auth/authz was not initialised and once it got properly synced i was error free for a few mins.

🧠 What This Taught Me

A failing Helm hook is usually a symptom, not a cause. The migration job "failed" because Postgres couldn't schedule because there was no StorageClass. Go find whatever's Pending before you read a single hook log.
kubectl logs <job-name> doesn't work. Jobs aren't pods. Use -l component=<label>, and do it before the controller cleans up the evidence.
Installing the same CRDs twice is a real failure mode. Field managers collide, and "the docs told me to" doesn't help. Check whether your chart already ships them.
If a controller generates a resource, don't kubectl patch it. Your change lives until the next reconcile and then vanishes without a trace. Put it in the spec the controller reads.
Curl with a forced -H "Host:" is not the same test as a browser. I proved my stack worked for a request no real client would ever send.
Airflow 3 renamed the furniture. webserver → api-server, DAG parsing → dag-processor, [scheduler] dag_dir_list_interval → [dag_processor]. Most tutorials out there are Airflow 2 and will point you at things that no longer exist.
volumeClaimTemplates is immutable. Some Helm upgrades simply require deleting the StatefulSet. That's not a workaround, that's the procedure.
Single-node clusters make RWO vs RWX a non-problem. One of the very few advantages of having exactly one node.
Read the status.conditions block. BackendNotFound, ResolvedRefs: False, FailedScheduling -- the Gateway API and Kubernetes both told me exactly what was wrong, in plain English, every single time. I just had to look.

📋 Quick Reference (For Skimmers)

Purpose	Command
Pull kubeconfig off the server	`ssh user@host "sudo microk8s config" > ~/.kube/config-microk8s`
Enable storage before anything else	`sudo microk8s enable hostpath-storage`
Find what's actually broken	`kubectl get pods -n airflow -o wide` then `describe` the `Pending` one
Logs from a Job's pod	`kubectl logs -n airflow -l component=<label> --tail=100`
Check a Gateway is live	`kubectl get gateway <name> -n <ns> -o yaml` → `Accepted` + `Programmed`
Check a route resolved its backend	`kubectl get httproute <name> -n <ns> -o yaml` → `ResolvedRefs`
Find the generated Envoy service	`kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=<gw>`
Confirm the real service names	`kubectl get svc -n airflow` (it's `api-server`, not `webserver`)
Create the git-sync key secret	`kubectl create secret generic <name> -n airflow --from-file=gitSshKey=<keyfile>`
Watch git-sync do its thing	`kubectl logs -n airflow -l component=dag-processor -c git-sync`
Escape an immutable StatefulSet	`kubectl delete statefulset <name> -n airflow`, then `helm upgrade`

🚀 What's Next

Genuinely, this time -- the Dell and the Pi come out of the drawers. Three nodes. Cilium. The thing I promised you last episode and will keep promising until it becomes true or I run out of drawers.

💬 Final Thoughts

Eight sidequests. Two self-inflicted, three courtesy of Airflow 3's renaming spree, one Kubernetes immutability rule, one CRD ownership fight, and one CDN that was too slow for Helm's patience.

The actual Airflow install -- the part I'd budgeted the evening for -- took about four minutes and worked on the first try.

Stay tuned. Popcorn 🍿 this time optional, but tea is now mandatory 🫖.

🏠 HomeLab Chronicles: Episode 3 - Ghost in the Mesh

sanskar arora — Sun, 23 Aug 2026 22:21:33 +0000

Hey all 👋

Last episode I said the network was going to need luck instead of the laptop. I'd like to formally apologize to the network. It didn't need luck. It needed an exorcism.

The task itself was supposed to be the single most boring item on the entire mission list: Step 2, remote SSH access. Generate a key, copy it over, done, never touch a keyboard directly on this machine again. Ten minutes, tops. I've done it a hundred times on a hundred boxes.

This time it took an evening, a diagram, and a newfound distrust of my own WiFi.

🔑 The Part That Was Supposed to Be Boring

Standard stuff, nothing to see here:

ssh-keygen -t ed25519 -C "msi@homelab"
ssh-copy-id msi@192.168.68.110
ssh msi@192.168.68.110

Key on the client, key copied to the server, connect, no password, go live my life. That's the whole plan. It fell apart before I even got through it.

🚫 Connection Refused, Apparently

$ ssh msi@192.168.68.110
ssh: connect to host 192.168.68.110 port 22: Connection refused

Refused. Not timed out — refused, like the server had looked at me personally and said no. Which felt extra unfair, because when I actually checked the server, it had done nothing wrong:

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Active: active (running)

$ sudo ss -tlnp | grep :22
LISTEN  0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",...))

sshd up, listening on every interface, no firewall even configured yet to blame. By every metric the server had access to about itself, it was innocent. And yet.

📡 Clue One: A Ping With Main Character Energy

Before spiraling into SSH configs, I did the normal thing and pinged it.

$ ping 192.168.68.110
64 bytes from 192.168.68.110: icmp_seq=1 ttl=64 time=312 ms
64 bytes from 192.168.68.110: icmp_seq=2 ttl=64 time=298 ms

It answered! Great, host's up, must be an SSH-level problem — except 300ms on a LAN isn't a number, it's a cry for help. Same-subnet pings on wired gear should be sub-millisecond. 300ms is what you get pinging a different continent, not a laptop three feet away on the same switch. I almost let that slide. I am glad I didn't.

🕵️ Clue Two: The ARP Table Doesn't Lie (Unlike Ping, Apparently)

Here's the part nobody warns you about: a ping reply only proves something answered at that IP. It doesn't prove it was the thing you meant. On a LAN, IP-to-hardware resolution happens over ARP, and your machine caches whatever answer it got first. Wrong cache, wrong destination — for every protocol, ping included.

$ ip neigh show 192.168.68.110
192.168.68.110 dev eth0 lladdr aa:bb:cc:11:22:33 REACHABLE

Then, straight from the server's own mouth:

$ ip link show enp0s3
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
    link/ether aa:bb:cc:44:55:66 brd ff:ff:ff:ff:ff:ff

aa:bb:cc:11:22:33 versus aa:bb:cc:44:55:66. Two different pieces of silicon, one IP address, and my laptop had confidently bet on the wrong one.

👻 So Who's the Impostor?

Somewhere on my network, something else was also answering to .110. And once I actually thought about my network topology instead of just interrogating my server, I had a solid suspect.

Here's the setup: fiber comes into a modem, the modem feeds a TP-Link mesh router, and that mesh router has an extender hanging off it to cover the dead zone at the far end of the apartment.

Internet → Modem → TP-Link Mesh (main node)
                         │
                         ├── extender ── tp-link device, same IP as my server
                         │
                         └── my server (when the mesh feels like it)

Mesh systems are supposed to make all of this invisible — one big happy network, roam anywhere, nobody notices the seams. In practice, an extender bolted onto a mesh is still, underneath, its own little hop with its own opinions about who's connected where. My working theory: something on the extender's side ended up holding .110 at the same time my server did, and depending on which hop answered the ARP broadcast first, my laptop's cache latched onto the extender's guest instead of my actual box.

Ping worked because whatever was squatting on .110 answered ICMP just fine — a real, alive device, just not mine.
SSH got refused, not timed out, because a refusal means the SYN reached a live host that slammed the door on port 22 — nothing was listening there. A timeout would've meant something dropping packets silently. A refusal meant I'd reached a machine. Just not the machine.
The 300ms latency fits the theory too — an extra wireless hop through an extender is exactly the kind of detour that turns sub-millisecond LAN pings into "did this packet go on vacation" pings.

Nothing here was actually broken. It was all working exactly as designed. The design just never accounted for me.

🛠️ The Fix: Stop Negotiating, Just Move

I could've gone spelunking through the mesh app's client list hunting for the impostor. I did not have that kind of patience left in me. I gave the server a new, boring, unclaimed IP instead.

sudo nano /etc/netplan/99-static.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: false
      dhcp6: false
      addresses:
        - 192.168.68.230/24
      routes:
        - to: default
          via: 192.168.68.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 8.8.8.8

sudo netplan try

netplan try deserves a fan club. It applies the change immediately and auto-reverts after 120 seconds unless you confirm — extremely relevant when you're one YAML indent away from locking yourself out over the very connection you're using to make the change. Confirmed the box was still reachable, then made it permanent:

sudo netplan apply

✅ Receipts

$ ping 192.168.68.210
64 bytes from 192.168.68.210: icmp_seq=1 ttl=64 time=0.91 ms

$ ip neigh show 192.168.68.210
192.168.68.210 dev eth0 lladdr aa:bb:cc:44:55:66 REACHABLE

Sub-millisecond, and the MAC finally matches the server's actual NIC. Proof, at last, that I was talking to my own hardware.

🔑 Back to the Boring Part, Finally

ssh-copy-id msi@192.168.68.210
ssh msi@192.168.68.210

Straight in. No password. The ten-minute task, delivered roughly one evening and one network diagram late:

sudo nano /etc/ssh/sshd_config
# PasswordAuthentication no

sudo systemctl restart ssh

🧠 What This Taught Me

Ping proves a device answered. Not which device. Don't let it clear a network problem it never actually investigated.
"Refused" and "timed out" are different diagnoses wearing the same "it didn't work" costume — refused means a live host said no, timeout means something's silently eating your packets. Learn to tell them apart fast.
Weird LAN latency is a symptom, not background noise. Sub-millisecond is the bar; three hundred milliseconds is a red flag wearing a ping reply as a disguise.
ip neigh plus ip link settles an identity crisis in about ten seconds.
Mesh WiFi plus a bolted-on extender is a surprisingly good recipe for exactly this kind of ghost — worth remembering before blaming the server for anything again.
Reserve the address at the router level (MAC-based DHCP reservation), not just on the box. A static IP the router doesn't know about is one collision away from becoming somebody else's problem — namely mine, again.

📋 Quick Reference (For Skimmers)

Purpose	Command
Confirm sshd is listening	`sudo ss -tlnp \
Check the MAC your client has cached	{% raw %}`ip neigh show <ip>`
Check a device's real MAC	`ip link show <interface>`
Clear a stale ARP entry	`sudo ip neigh flush dev <interface>` or `sudo arp -d <ip>`
Apply a network change with a safety net	`sudo netplan try`
Copy your public key to a server	`ssh-copy-id user@host`

🚀 What's Next

Given how well "one server, one IP" just went, the obvious next move is clearly to add more servers and more IPs. So: the Dell Inspiron isn't staying benched after all — it's getting reflashed with the same minimal Ubuntu Server + LXD/LXC combo that worked the first time, except this time with a job description it can actually survive, alongside an old Raspberry Pi I dug out of a drawer. Three nodes, one fledgling cluster.

And since apparently I haven't suffered enough at the hands of networking this episode, the plan is to bring in Cilium — eBPF-based pod networking, or as I now think of it, round two against the concept of "IP address" as a stable idea — to actually handle traffic across all of it instead of hoping defaults figure it out on their own.

Given this episode's track record, I would like the record to reflect I have concerns.

💬 Final Thoughts

Ghost: identified, evicted. SSH: finally, actually working. Trust in my own home WiFi: still recovering.

Stay tuned. Popcorn 🍿 mandatory this time, not optional.

DEV Community: sanskar arora

🐈 HomeLab Chronicles: Episode 11 - schrodingers cluster

🚨 Symptom: The Tunnel Is Down

🔬 Hypothesis 1: The Failover Path Was Never Tested

📡 Interlude: Who Killed It

🪪 Blocker: The Certificate, Again

🔬 Hypothesis 2: The Kubelet Has Lost Its Pods

🔬 Hypothesis 3: The API Server's Watch Cache Is Empty

🔬 Hypothesis 4: The Kubelet's Credentials Are Broken

🔬 Hypothesis 5: The Kubelet's Informers Are Failing

🧊 Hypothesis 6: The Cache Isn't Empty. It's Frozen.

🔧 The Fix

🧠 What This Taught Me

🚀 What's Next

💬 Final Thoughts

❓ HomeLab Chronicles: Episode 10 - new router who dis

🚨 Symptom One: kubectl Says "Unexpected EOF"

🧟 Symptom Two: The Dell Stops Answering the Phone

🔐 Sidequest 1: The Pod That Remembered the Old Address

🪪 Sidequest 2: kubectl logs and the Other Certificate

🚇 Sidequest 3: The Tunnel Route

🗺️ The Census: Where My Cluster Kept the Old Address

🧠 What This Taught Me

📋 Quick Reference (For Skimmers)

🚀 What's Next

💬 Final Thoughts

🧭 Neovim Thinks in One Directory

🧨 Why "just cd somewhere" does not work

🧭 Decision 1: what is a workspace?

🔍 Decision 2: search is the easy part, so make it nice

🌳 Decision 3: nvim-tree has one root, and that is not negotiable

📍 Decision 4: who owns the cwd?

🔀 Decision 5: plugins that take a cwd versus plugins that are the cwd

🔌 Everything else rides on hooks the plugins already offer

🔒 Hardening, because some of these machines are bastion hosts

🚧 Honest gaps

🧠 What I would tell the next person

📋 Quick Reference (For Skimmers)

💬 Final Thoughts

📊 One File, Twenty-Five Images, Part 2/2 - The Numbers

🔬 The process: build it small, measure it honestly

Measuring, and the several ways I got it wrong first

Four reviewers and a skeptic

📊 Results

Size

Time

The finding that overruled my design

🧪 The clean-slate test

🧭 What I would tell the next person

📦 One File, Twenty-Five Images, Part 1/2 - The Design

🧨 Why the naive approach is expensive

🧭 HLD, decision 1: what is the artifact?

⚖️ HLD, decision 2: the fork that surprised me

🧊 HLD, decision 3: a frozen encoder, and why determinism is the whole game

🧱 LLD: the bundle, in detail

Entry order is a design decision

The tar must be deterministic

bundle-meta.json carries the inventory

Verify while you are already touching the bytes

🔁 LLD: the import side, and a twist I did not plan

🔜 Next

🕳️ HomeLab Chronicles: Episode 9 - Tunnel Vision

🚇 Two Tunnels Diverged in a Yellow Dashboard

🚪 The Part You Don't Get to Skip

🧾 The Fine Print I'm Choosing to Live With

🧠 What This Taught Me

📋 Quick Reference (For Skimmers)

🚀 What's Next

💬 Final Thoughts

🤫 HomeLab Chronicles: Episode 8 - The Silent Treatment

🪦 The Graveyard of Values I Was Setting

🎩 Sidequest: The Job That Vanished Like a Magician

🥷 Sidequest: The Disk That Was "Already In Use"

🧷 Bonus Round: The Password That Broke YAML

🧠 What This Taught Me

📋 Quick Reference (For Skimmers)

🚀 What's Next

💬 Final Thoughts

🔒 HomeLab Chronicles: Episode 7 - Locked Out of My Own House

🚨 Symptom One: Flux Bootstrap Times Out

🚨 Symptom One: `kubectl` Says "Unexpected EOF"

🪪 Sidequest 2: `kubectl logs` and the Other Certificate

`bundle-meta.json` carries the inventory