DEV Community: sanskar arora

🕳️ HomeLab Chronicles: Episode 9 - Tunnel Vision

sanskar arora — Sun, 30 Aug 2026 10:04:13 +0000

Hey all 👋

State of the lab: two nodes, Flux in charge, Airflow alive, backups nightly, RBAC banished to therapy. The UI works from anywhere in the house. But "in the house" is a service radius measured in Wi-Fi bars, and I wanted my Airflow at airflow.sanskararora.in, from anywhere, like a person with a real platform and not two laptops fighting a router for their lives.

The classic move is port forwarding: open a hole in the router, point it at the NodePort, pray. Every part of that sentence should upset you. My router has already been established, across two episodes, as this saga's least reliable character — and now I'd make it my front door too? Also: home IP published to the world, DDoS surface, dynamic-IP DNS jank. No.

Enter Cloudflare Tunnel. The trick that makes it homelab-perfect: a small daemon (cloudflared) runs inside the cluster and dials out to Cloudflare's edge. Traffic to my domain rides that already-open outbound connection back in. Zero inbound ports. Zero public IP. The router is demoted from front door to hallway, which is frankly all it ever deserved.

🚇 Two Tunnels Diverged in a Yellow Dashboard

First fork in the road, and I took both prongs like a professional: Cloudflare tunnels come in two flavors.

Locally-managed: cloudflared tunnel create on your machine, credentials JSON, routing rules in a config file → which for me means in git. GitOps-pure. Chef's kiss.

Remotely-managed: click "create tunnel" in the Zero Trust dashboard, get a token, routing rules live in the UI.

I wrote the manifests for flavor one, then created the tunnel in the dashboard, which is flavor two. The dashboard then helpfully offers you install commands — sudo cloudflared service install eyJ... — which are for running it on a host, i.e., not what a cluster deployment wants at all. The only thing you need from that whole screen is the token. The long eyJhIjoi... string. That's the tunnel's entire identity; treat it like a password, because it is one.

The in-cluster version is refreshingly small — a Deployment, two replicas with pod anti-affinity so the tunnel survives a node dying, cloudflared tunnel run reading TUNNEL_TOKEN from a Secret:

env:
  - name: TUNNEL_TOKEN
    valueFrom:
      secretKeyRef: {name: tunnel-token, key: token}

The token secret is created by hand with stringData: (paste as-is, the API server base64s it — hand-encoding a token that's already opaque gibberish is how you double-encode and then debug "auth errors" for an hour). The manifest holding it never touches git. .gitignore first, file second — that order, specifically, because git add -A has ended better tokens than mine.

Then in the dashboard: Public Hostname → airflow.sanskararora.in → HTTP → the Envoy gateway service's in-cluster FQDN on 8080. DNS record appears automatically. And because my HTTPRoute matches any Host header (Episode 4's hard-won lesson about hostnames), the request flows: Cloudflare edge → tunnel → Envoy → route → Airflow. Login page. On my phone. On mobile data. In a parking lot. I looked deranged and I did not care.

$ kubectl -n cloudflared logs -l app=cloudflared --tail=5
INF Registered tunnel connection ...
INF Registered tunnel connection ...

Four registered connections per replica. The dashboard's sad little "No connectors installed" flips to two healthy connectors. Lovely.

🚪 The Part You Don't Get to Skip

Now the uncomfortable paragraph. What I just built is: my Airflow admin login, on the public internet, guarded by one password, terminating inside my house. Airflow is a genuinely juicy target — it runs arbitrary code on schedule, that's its job — and it has the CVE history of software that important.

And if you're thinking "nobody knows my subdomain exists" — the TLS certificate for it landed in public Certificate Transparency logs the moment it was issued. Scanners watch those logs like RSS feeds. Your "secret" subdomain gets its first uninvited visitors in minutes, not months. Obscurity isn't a moat; it's a countdown. But in order for me to access it freely i am skipping it for now.

🧾 The Fine Print I'm Choosing to Live With

Remotely-managed means the hostname→service routing lives in Cloudflare's dashboard, not in my repo. Which dents the Episode 6 doctrine: a full cluster rebuild now restores everything except the tunnel's routing table. That's one manual step reintroduced into a system I spent three episodes making manual-step-free. It's written in the README, on the rebuild checklist, where future-me will find it and sigh.

The pure path exists — locally-managed tunnel, config in git, SOPS-encrypted token committed alongside it, flux bootstrap --decryption-provider=sops — and it's genuinely the endgame. But the dashboard tunnel was already made, it works, and I have learned (Episodes 5 through 8, inclusive) to stop refactoring things that are currently succeeding.

🧠 What This Taught Me

Outbound-only beats port forwarding in every dimension that matters. No inbound holes, no public IP, no router in the trust path.
Dashboard tunnel = token + rules in UI. CLI tunnel = credentials file + rules in git. Pick one before writing manifests, unlike some people.
Two replicas + anti-affinity makes the tunnel survive exactly the node-death this cluster has proven it can produce.
Certificate Transparency means your subdomain was never secret. Assume discovery is immediate, because it approximately is.
Access in front of everything. Authentication at the edge, before your network — not a nice-to-have, the whole point of fronting with Cloudflare.
stringData for token secrets. Let the API server do the base64. You will mangle it; it won't.

📋 Quick Reference (For Skimmers)

Purpose	Command / Place
Token into cluster	`kubectl -n cloudflared create secret generic tunnel-token --from-literal=token='eyJ...'`
Tunnel healthy?	`kubectl -n cloudflared logs -l app=cloudflared` → `Registered tunnel connection`
Route a hostname	Dashboard → Tunnels → Public Hostname → `HTTP` → `<envoy-svc>.envoy-gateway-system.svc.cluster.local:8080`
Find the Envoy svc name	`kubectl -n envoy-gateway-system get svc`
The moat	Zero Trust → Access → Applications → Self-hosted → Allow: your email
Who already knows your subdomain	crt.sh — search your domain, feel observed

🚀 What's Next

The Pi is still in the drawer. Cilium is still a rumor. But there's now a third-voter-shaped hole in this cluster's HA story, and dqlite quorum math says three is the first number that actually means anything. So: next episode, the drawer opens. Probably. The drawer has heard promises before.

💬 Final Thoughts

Nine episodes ago this was one laptop that couldn't survive a reboot. It's now a two-node, git-defined, nightly-backed-up, tunnel-fronted, edge-authenticated platform that survives power cuts out of spite. The router remains the weakest member of the team and has been architected around accordingly.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated card 🪪, chart-source bookmark 🔖 — and now a domain. The lab is public. Send help, or DAGs.

🤫 HomeLab Chronicles: Episode 8 - The Silent Treatment

sanskar arora — Sun, 30 Aug 2026 09:58:28 +0000

Hey all 👋

Quick recap: power cut (Ep5), GitOps (Ep6), locked myself out of my own control plane (Ep7), rebuilt in twenty minutes because GitOps (Ep6, smugly). Now: getting Airflow back up. Which should be easy, since I've installed it before — Episode 4, eight sidequests, badge earned.

This episode is about a different kind of enemy. Not the loud failure — the silent one. The config that gets accepted, nodded at, and thrown away. Because here's Helm's dirtiest little behavior, in one sentence:

Helm does not validate your values. An unknown key isn't an error. It's a shrug.

Typo a field, use a field from an older chart, invent a field that sounds plausible — Helm renders the templates, your key matches nothing, and the install proceeds looking exactly like success. No warning. No error. Just vibes.

🪦 The Graveyard of Values I Was Setting

Instead of trusting tutorials (or my own Episode 4, ahem), I finally did the thing: fetched the actual chart source — apache/airflow at tag helm-chart/1.22.0, whose appVersion is exactly my Airflow 3.2.2 — and checked every field I was setting against values.yaml and the templates. The census results:

Exhibit A: webserver: — valid key, wrong decade. Chart 1.22.0 ships both webserver: (Airflow 2) and apiServer: (Airflow 3). My replicas and resource limits sat under webserver:, where Airflow 3 never looks. Every limit I thought I'd set on the UI? Decorative. Episode 4 established the service was renamed; it didn't occur to me the values tree split too.

Exhibit B: logs.persistence.accessMode — does not exist. This one's beautiful. The chart's log-PVC template hardcodes accessModes: ["ReadWriteMany"]. Which means my careful Episode 4 setting of ReadWriteOnce — with its confident little comment about single-node clusters — was ignored the entire time. I got RWX whether I asked or not. The chart was protecting me from a config I was proud of.

Exhibit C: webserverSecretKeySecretName — Airflow <3 only. The comment in the chart source says it plainly: that's the Flask secret for the old webserver. Airflow 3 signs JWTs, wants jwtSecretName, and the secret's key must be exactly jwt-secret — that name comes from a template helper, not from you. Get this wrong and the chart generates a fresh key on every render, and your components restart "for no reason." A warning my very first install printed, months of installs ago, that I finally understand.

Exhibit D: knownHosts — a string, not a secret. I'd stuffed a known_hosts file into the SSH secret next to the deploy key, feeling organized. The chart wanted it as a plain values string under dags.gitSync.knownHosts. My copy sat in the secret, unread, while git-sync remained one bad default away from host-key failure. (Also, wait: got superseded by period:. Also-also, the SSH secret's key must be named gitSshKey verbatim — that one Episode 4 got right. Growth.)

Four fields. Four different flavors of silently doing nothing. helm lint caught zero of them, because none of them are syntax errors. They're fluent, grammatical nonsense.

🎩 Sidequest: The Job That Vanished Like a Magician

With values corrected, install. Pods appear! Pods crash-loop. Init container says:

TimeoutError: There are still unapplied migrations after 60 seconds.
MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'1d6611b6ab7c'}

set() — empty. Postgres reachable, schema blank. The migration job never ran. And when I went looking for it:

$ kubectl get jobs -A
No resources found

Nothing. Anywhere. Because by default the chart runs migrations as a Helm hook — it lives inside the install ceremony and gets cleaned up after, success or failure. My install had stalled (see below), Helm aborted early, the hook evaporated, and I was left with pods waiting for a schema that a ghost was supposed to write. Under Flux this deadlocks beautifully: pods can't start without migrations, Helm won't rerun migrations because pods failed.

The chart's own answer for GitOps land:

migrateDatabaseJob:
  enabled: true
  useHelmHooks: false
createUserJob:
  useHelmHooks: false

Now the jobs are ordinary resources: applied like everything else, visible in kubectl get jobs, logs readable when they fail. Debuggable beats ceremonial.

🥷 Sidequest: The Disk That Was "Already In Use"

One more silent saboteur, this time below Kubernetes entirely. Fresh Postgres volume, brand new, zero history — and:

MountVolume.MountDevice failed ... mke2fs 1.47.0
/dev/longhorn/pvc-... is apparently in use by the system;
will not make a filesystem here!

In use? It was born thirty seconds ago. I deleted every PV, PVC, and Longhorn volume in the cluster and got the identical error on the replacement — which is the tell that the problem isn't in the cluster at all.

It's multipathd. Ubuntu ships it enabled, and its hobby is grabbing new block devices the instant they appear, in case they're part of a multipath SAN. I do not have a SAN. I have two laptops. But multipathd claimed each fresh Longhorn device anyway, held it open, and mke2fs politely declined to format a held disk.

sudo systemctl disable --now multipathd multipathd.socket
kubectl -n airflow delete pod airflow-postgresql-0

Postgres: Running. It's a documented Longhorn prerequisite that approximately nobody's tutorial mentions, including — checks notes — my own host-prep list from Episode 4.

🧷 Bonus Round: The Password That Broke YAML

The admin password flows in through a generated values snippet. First version built that YAML with shell string concatenation, which works perfectly until someone's password contains a " — at which point the YAML is invalid and the failure surfaces much later as an inscrutable HelmRelease error, with the password (correctly!) absent from every log. The fix is to stop hand-assembling YAML like it's 1997: serialize with json.dumps — JSON is valid YAML, it's stdlib, and it escapes quotes, backslashes, colons and leading #s without opinion. Tested against the most hostile passwords I could invent. All survived.

🧠 What This Taught Me

Helm ignores unknown values silently. All of them. Forever. The only defense is reading the chart's actual values.yaml — at the tag you're installing — not tutorials, not memory, not last episode's notes.
helm show values <chart> | grep <field> before setting anything load-bearing. Thirty seconds. Cheaper than a weekend.
A valid key from the wrong era is worse than a typo. webserver: parsed fine, rendered fine, and did nothing. Typos at least look wrong.
Helm hooks and GitOps are a bad marriage. useHelmHooks: false turns invisible ceremonies into inspectable resources.
MigrationHead(s) in DB: set() means "empty schema," not "slow schema." The waiter isn't slow; the kitchen never got the order.
multipathd eats block devices on stock Ubuntu. If mke2fs claims a newborn disk is busy, that's your poltergeist.
Never build YAML with string concatenation when a serializer is right there.

📋 Quick Reference (For Skimmers)

Purpose	Command
The ground truth for chart fields	`helm show values apache-airflow/airflow > /tmp/v.yaml` then grep
Even grounder truth	read the chart repo at the release tag — templates don't lie
Airflow 3 UI settings live under	`apiServer:` (not `webserver:`)
JWT secret	`jwtSecretName:` → secret key must be `jwt-secret`
Jobs you can actually see	`migrateDatabaseJob.useHelmHooks: false`
Disk claimed at birth	`systemctl disable --now multipathd multipathd.socket`
Schema actually migrated?	look for `MigrationHead(s) in DB:` ≠ `set()`

🚀 What's Next

Airflow is up, migrations ran, and I can see it from my couch. But couch-radius availability is so Episode 4. Next time: a Cloudflare Tunnel puts it on my actual domain, from anywhere, without opening a single port on the plastic router of doom.

💬 Final Thoughts

Loud failures cost hours. Silent ones cost weekends — and this episode had four values, one hook, one daemon, and one quotation mark, all failing without making a sound.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated coredns card 🪪, and now a bookmark to the chart source. The toolkit is mostly talismans at this point.

🔒 HomeLab Chronicles: Episode 7 - Locked Out of My Own House

sanskar arora — Sun, 30 Aug 2026 09:50:03 +0000

Hey all 👋

Short episode. Painful episode. The kind where the call is coming from inside the house, the house is the control plane, and nobody inside the house is allowed to answer the phone.

Fresh rebuild, Flux repo ready, and I did the responsible thing: enabled RBAC while the cluster was empty. Every best-practices doc agrees — turn on authorization early, before workloads exist, while it's painless.

Reader, it was not painless.

🚨 Symptom One: Flux Bootstrap Times Out

✗ timeout waiting for: [CustomResourceDefinition/kustomizations... status: 'InProgress', ...]

Eleven CRDs, all stuck. But committed and pushed fine — the repo half worked, the cluster half hung. And when I inspected a stuck CRD:

status:
  acceptedNames: {kind: "", plural: ""}
  conditions: null

conditions: null. Not "failing." Not "pending." Never touched. On a healthy cluster the apiserver stamps a CRD as Established within milliseconds. These sat blank, like forms nobody would ever process.

🚨 Symptom Two: A Deployment Frozen in Time

$ kubectl -n kube-system get deploy
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
coredns    0/1     0            0           5h7m

Five hours old. UP-TO-DATE: 0. That column being zero means no ReplicaSet was ever created — which means the deployment controller never even looked at it. And with no DNS pod, everything downstream was queued up to fail alphabetically.

🔍 The Logs Confess Immediately

E... leaderelection.go: "Error retrieving lease lock"
  err="leases... is forbidden: User \"system:kube-controller-manager\"
  cannot get resource \"leases\" in namespace \"kube-system\""

E... "Failed to watch" err="csidrivers... is forbidden:
  User \"system:kube-scheduler\" cannot list resource \"csidrivers\""

Sit with that. The controller-manager — the component whose entire job is running every controller in Kubernetes — was forbidden from acquiring its own leader-election lease. No lease, no leadership; no leadership, no controllers; no controllers, no ReplicaSets, no pods, no anything. The scheduler was equally banned from reading the objects it schedules around.

I enabled authorization, and the first citizens it deported were the government.

🧩 The Part That Made It Weird

The obvious theory: missing ClusterRoleBindings. Checked. They existed. Correct subjects (User: system:kube-controller-manager), correct roles, roles containing exactly the permissions being denied. Checked the client certificates on disk:

subject=CN=system:kube-controller-manager
subject=CN=system:kube-scheduler

Perfect CNs. Perfect bindings. Perfect roles. Still forbidden.

Then the tell. I tried impersonation to test permissions, and even that failed:

selfsubjectaccessreviews is forbidden: User "system:kube-controller-manager"
cannot create resource "selfsubjectaccessreviews"

That permission comes from system:basic-user, bound to system:authenticated — the most universal binding in all of Kubernetes. Literally everyone gets it. When that denies, the story isn't "a binding is missing." The story is the RBAC authorizer isn't loading policy at all. Every rule in the cluster: correct, present, and completely unread.

Meanwhile kubectl get --raw='/readyz?verbose' said readyz check passed, because my admin cert (hello, system:masters) bypasses authorization entirely. The apiserver was healthy. The apiserver was also the only one allowed in the building. Health checks measure the doorman, not whether anyone can get past him.

🧯 The Retreat

sudo microk8s disable rbac
sudo microk8s stop && sudo microk8s start
$ grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver
--authorization-mode=AlwaysAllow

Forbidden errors: gone. Lease: held and renewing. And coredns... still 0/1, UP-TO-DATE 0. Because five hours of objects written into a paralyzed control plane don't spring to life on forgiveness — they'd been created into a void, and some of them (plus a growing chorus of node "msi" not found) were simply inert. At that point the cluster was 24 hours old and held nothing I loved, so I invoked Episode 6's superpower: reset, rejoin, bootstrap. Twenty minutes. This is what the git repo was for.

The rebuilt version skips enable rbac, and adds the check I'll never skip again: coredns must reach 1/1 before anything else happens. It's the cheapest possible proof that the datastore persists writes and the controllers act on them. One deployment, one honest answer.

🧠 What This Taught Me

readyz measures the apiserver, not the cluster. A green apiserver can preside over a completely dead controller plane, especially when your admin cert skips the auth queue.
UP-TO-DATE: 0 on an old deployment is a five-alarm signal. It doesn't mean "unhealthy pods." It means "no controller has ever processed this."
When even system:basic-user is denied, stop auditing bindings. Universal denial means the authorizer itself, not the policy.
A homelab without RBAC is a defensible trade. It governs in-cluster identities, not network exposure; the API server is still cert-authenticated. MicroK8s ships it off by default, and I now understand that as a mercy, not an oversight.
"Enable it while it's painless" assumes it works. Best practice on paper, lockout in my kitchen.

📋 Quick Reference (For Skimmers)

Purpose	Command
Is the controller plane actually alive	`kubectl -n kube-system get lease kube-controller-manager -o jsonpath='{.spec.renewTime}'` — must be seconds old
The canary	`kubectl -n kube-system get deploy coredns` → wants `1/1`
CRD ever processed?	`-o jsonpath='{.status.acceptedNames.kind}'` — empty = never
Current auth mode	`grep authorization-mode /var/snap/microk8s/current/args/kube-apiserver`
Component identities	decode `client-certificate-data` from `/var/snap/microk8s/current/credentials/*.config` → `openssl x509 -noout -subject`

🚀 What's Next

Cluster rebuilt, Flux green, Airflow installing... and then Helm taught me its favorite trick: accepting configuration it has no intention of using, silently, with a straight face. Next episode is about every value I set that did absolutely nothing.

💬 Final Thoughts

I gave my cluster a security system. The security system's first act was arresting the janitor, the electrician, and itself. The house stood perfectly secure and perfectly dark.

Popcorn 🍿, coffee ☕, UPS 🔌, and a laminated card that says check coredns first.

📜 HomeLab Chronicles: Episode 6 - Source of Truth

sanskar arora — Sun, 30 Aug 2026 09:45:04 +0000

Hey all 👋

Last episode a power cut exposed an uncomfortable fact: my cluster's entire memory lived in one SQLite-flavored database, on one laptop, bound to one Wi-Fi address, guarded by one aging battery. Four single points of failure in a trench coat.

The fix isn't making that database unkillable. The fix is making it unimportant. If every manifest lives in git and something reconciles the cluster against git continuously, then "the datastore died" stops being a tragedy and becomes a reboot with extra steps.

So: Flux. Here's the setup, and the four ways I face-planted installing it.

🗂️ The Shape of the Repo

clusters/homelab/
  flux-system/          <- Flux writes this at bootstrap; hands off
  infrastructure.yaml   <- points at infrastructure/
  apps.yaml             <- points at apps/
infrastructure/
  controllers/          <- Longhorn + Envoy Gateway HelmReleases
  configs/              <- GatewayClass, Gateway, StorageClasses
apps/homelab/
  airflow/              <- the actual point of all this

Three Flux Kustomizations, chained: infra-controllers → infra-configs → apps, via dependsOn. That chaining is not decoration. My GatewayClass can't exist until Envoy Gateway's CRDs exist, and the CRDs arrive with the controller's Helm chart. Without dependsOn, Flux sprints ahead, tries to create a GatewayClass into a cluster that's never heard of GatewayClasses, and fails with the enthusiasm of a golden retriever running into a glass door.

dependsOn plus wait: true turns that into: install controllers, wait until healthy, then configs, then apps. Boring. Sequential. Correct. The three great virtues.

🔑 Sidequest 1: The Token Bureaucracy

flux bootstrap github needs a GitHub token, and the docs-diving summary is:

Classic PAT: repo scope. Needed if Flux should create the repo.
Fine-grained PAT (pre-created repo): Contents read/write, Metadata read, and — the one everyone misses — Administration read/write, because Flux installs an SSH deploy key on the repo, and deploy keys are an admin operation.

Here's the nice part: the deploy key is what Flux uses forever after. The PAT is scaffolding. Bootstrap, confirm flux get kustomizations goes green, then revoke the token. Give it a 7-day expiry at creation and let it die of natural causes. Also: I renamed the repo mid-project and exactly zero manifests cared — the repo URL lives only in the bootstrap-generated gotk-sync.yaml. Rename after bootstrapping though, and Flux keeps pulling the old URL. Ask me how I know I almost knew that.

💥 Sidequest 2: The Secret That Must Come First

Some things cannot live in a public-ish git repo: the git deploy key for DAG sync, the Airflow admin password, the JWT signing key. Those get created by hand (well — by script), out-of-band.

And order matters more than I expected. My Airflow HelmRelease pulls its password via valuesFrom with optional: false — no secret, no install. I bootstrapped first and created secrets second, like a person with hope in their heart. Flux tried to install Airflow, found no secret, retried three times, exhausted its retry budget, and parked the release in a failed state.

Then I ran the secrets script and hit this beauty:

✗ Failed to perform remediation: missing target release for rollback:
  cannot remediate failed release

Translation: the first ever install failed, so there's no previous good release to roll back to, so Flux's remediation has nothing to grab. A plain flux reconcile won't retry it. You need --force, or a suspend/resume, or — the actual lesson — create the secrets before you bootstrap. The cluster exists after MicroK8s comes up; Flux doesn't need to be watching yet for you to kubectl create secret.

📌 Sidequest 3: Pin Your Charts Like You Mean It

First draft, every HelmRelease said version: "1.x". Feels flexible. Is flexible. Flexible means "a major chart upgrade can land at 3am while a reconcile loop runs unattended," and for a storage system that's not a version policy, that's a séance.

So I went and found the real versions — including the fun discovery that Airflow's chart 1.22.0 has appVersion: 3.2.2, exactly the Airflow I was running. Now the repo says:

airflow:      version: "1.22.0"
longhorn:     version: "1.12.1"
gateway-helm: version: "v1.9.1"

Upgrades still happen. They just happen when I commit them, on purpose, with coffee.

🫥 Sidequest 4: The Dot of Shame

One evening a reconcile hung with context deadline exceeded and zero pods appearing. Twenty minutes of cluster spelunking later, I looked at my own shell prompt:

╭─sunny@sunny ~/github/lab ‹main●›

That little dot. Uncommitted changes. I'd committed the kustomization entry for a new directory but not the directory. Flux was faithfully applying exactly what I'd pushed, which was a reference to a folder that existed only on my laptop. Flux cannot read your working tree. Flux cannot read your intentions. Flux reads origin/main, and origin/main reads whatever you actually pushed.

The dot was there the whole time. Judging me.

⚖️ What Flux Is Not

Worth saying plainly, because I briefly believed otherwise: Flux is not a backup system. It restores what's declared — Deployments, HelmReleases, routes. It knows nothing about what accumulated at runtime: your Postgres data, DAG run history, Longhorn volume contents, the secrets you created by hand.

So the full recovery story is a two-piece set: Flux rebuilds the skeleton from git, and last episode's nightly dbctl snapshots plus (eventually) Longhorn backup targets restore the organs. Either one alone is half a cluster.

The payoff, though, is real: the rebuild I did mid-way through all this — full microk8s reset, rejoin the Dell, bootstrap, secrets script — took about twenty minutes, most of it image pulls. Episode 5 me would have wept. Episode 6 me made tea.

🧠 What This Taught Me

dependsOn is load-bearing. CRDs before custom resources, controllers before configs, or Flux runs face-first into glass.
Secrets before bootstrap. An optional: false reference plus a missing secret burns the retry budget and needs --force to resurrect.
A first-install failure has no rollback target. "Cannot remediate failed release" means delete-and-retry, not reconcile-harder.
Version ranges are unattended upgrades wearing a disguise. Pin, then upgrade on purpose.
Flux reads the remote, not your working tree. The ● in your prompt is a Flux error message you're rendering locally.
GitOps restores manifests, backups restore data. You need both, and neither is impressed by the other.

📋 Quick Reference (For Skimmers)

Purpose	Command
Bootstrap	`flux bootstrap github --owner=X --repository=Y --branch=main --path=clusters/homelab --personal`
Fine-grained PAT needs	Contents RW, Metadata R, Administration RW
What's reconciling	`flux get kustomizations`
Chart status + real errors	`flux get helmreleases -A`
Push a change now	`flux reconcile kustomization apps --with-source`
Resurrect a dead first install	`flux reconcile helmrelease X -n ns --force`
Why is it stuck	`flux logs --kind=HelmRelease --name=X -n ns`

🚀 What's Next

With the repo in charge, I did the responsible thing and enabled RBAC on the fresh cluster — you know, best practice — and the control plane immediately locked itself out of its own house. Next episode is short, painful, and has a body count of one coredns.

💬 Final Thoughts

Git is now the cluster. The cluster is now a cache. The dot in my prompt is now a trauma response.

Popcorn 🍿, coffee ☕, UPS 🔌, and a pre-push git status — the ritual expands.

🔦 HomeLab Chronicles: Episode 5 - Blackout

sanskar arora — Sun, 30 Aug 2026 09:40:23 +0000

Hey all 👋

Last episode ended on a high. Airflow UI on my phone, logs working, DAGs syncing, and a solemn promise: Dell out of the drawer, Pi out of the other drawer, Cilium out of my dreams and into the cluster.

Here's the scoreboard. The Dell came out of the drawer — actually joined the cluster, same night I published. The Pi remains in its drawer, thriving, unbothered. Cilium remains a concept I respect from a distance. One out of three. In this economy, that's a win.

And then at 05:22 the next morning, the power went out. And when it came back, my cluster had amnesia.

$ kubectl get deployments -A
NAMESPACE     NAME                      READY
kube-system   calico-kube-controllers   1/1

That's it. That's the whole cluster. Airflow: gone. Longhorn: gone. My will to live: buffering.

🕵️ The Crime Scene

First instinct: the datastore got corrupted by the power cut. Reasonable. Dramatic. Wrong — but let's earn that.

MicroK8s doesn't use etcd; it uses dqlite — Raft-replicated SQLite, living at /var/snap/microk8s/current/var/kubernetes/backend/. So that's where I went digging:

$ sudo ls -la /var/snap/microk8s/current/var/kubernetes/backend/
-rw-rw-r-- cluster.crt        Aug 28 21:42
-rw-rw---- cluster.key        Aug 28 21:42
-rw-rw---- metadata1          Aug 28 21:42
-rw-rw---- 0000000000074753-0000000000075264   ...

Read those timestamps again. The database certificates were born at 21:42 the night before. The raft index was in the seventy-thousands — a day of idle heartbeats, not weeks of a running cluster. And:

$ kubectl get ns
NAME              STATUS   AGE
default           Active   16h
kube-node-lease   Active   16h
kube-public       Active   16h
kube-system       Active   16h

Four namespaces. The four a cluster is born with. All sixteen hours old.

The plot twist, delivered by my own filesystem: nothing was lost in the power cut, because nothing was there. When I joined the Dell the previous night, the datastore got reinitialized underneath me. The database I spent the morning mourning was an empty sixteen-hour-old husk. The power cut didn't rob the house. The power cut just knocked on the door of a house that was already empty.

Grief has five stages. ls -la has one.

📡 So What Did the Power Cut Do?

Something sneakier. Look at the dqlite logs from 05:22:

dial: Failed to connect to HTTP endpoint: dial tcp 192.168.68.210:19001:
connect: network is unreachable

Repeated. For a while. Now, the tempting read is "boot race — Wi-Fi wasn't up yet when dqlite started." I almost bought it. But the dqlite process was PID 1919211. Kubelite was PID 2.3 million. You don't get seven-digit PIDs on a fresh boot. That machine had been up for eight hours when this happened.

So it wasn't a boot race. The network vanished mid-run. And here's the geometry of my particular disaster:

My nodes are laptops. My router is not.

When the power cuts, the MSI and the Dell shrug and switch to battery. The router — a small plastic box with the survival instincts of a houseplant — dies instantly. Wi-Fi drops, the interface loses its address, and dqlite, which is bound to 192.168.68.210:19001, suddenly can't reach itself.

And why is it bound to the LAN address at all? Because of the Dell. dqlite binds to 127.0.0.1 by default — a fresh single-node MicroK8s is structurally immune to network outages. The moment a second node joins, dqlite moves to the LAN IP, because the other node has to reach it. Joining the Dell put my datastore on Wi-Fi. The drawer had a curse on it.

🔋 The Villain Nobody Suspects

But wait — a frozen datastore recovers when the network comes back. Frozen isn't corrupted. Where does actual damage come from?

Follow the battery. Power cut → laptops keep running → outage outlasts the battery → laptop hits 0% → hard poweroff. No filesystem sync. No dqlite checkpoint. Just lights out, mid-write. That is the dirty shutdown that corrupts a SQLite-family datastore, and it doesn't happen during the outage. It happens hours later, quietly, while you're asleep and feeling safe.

The fix has absolutely nothing to do with Kubernetes. It's laptop power management:

# /etc/UPower/UPower.conf
UsePercentageForPolicy=true
PercentageLow=25
PercentageCritical=22
PercentageAction=20
CriticalPowerAction=PowerOff

Clean shutdown at 20% instead of a crash at 0%. (UPower insists Low > Critical > Action, so mind the ordering.) Plus telling logind that a closed lid on a server is decor, not a command:

# /etc/systemd/logind.conf.d/99-homelab.conf
[Login]
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
IdleAction=ignore

sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

And a longer shutdown grace so dqlite gets to checkpoint instead of eating SIGKILL at the 90-second default:

# /etc/systemd/system.conf.d/99-shutdown-timeout.conf
[Manager]
DefaultTimeoutStopSec=120s

🧠 The Mental Model I Was Missing

The reason "my deployments are gone" felt so confusing is that three completely different things all get called "the cluster":

Layer	Lives in	Survives a reboot?
Cluster state — Deployments, Services, routes	dqlite, on disk	Yes, if dqlite comes up clean
Data — Postgres, volumes	Longhorn, on disk	Yes
Pods	Nowhere. Vapor.	No — and that's correct

Pods are supposed to vanish and be recreated from Deployments. That's not fragility, that's the design. The only question that ever matters after an outage is: did layer one come back intact? Everything else follows from it.

And ranked by what actually threatens layer one:

Battery draining to a dirty poweroff ← the real one
Network loss while running (freezes it; recovers)
Clean reboot (total non-event)

I'd been emotionally preparing for #3, mildly worried about #2, and completely blind to #1.

💾 And Because Paranoia Is Now a Feature

dqlite has a backup tool nobody talks about:

microk8s dbctl backup /var/backups/microk8s/dqlite-$(date +%Y%m%d-%H%M%S)

Mine now runs nightly from a systemd timer, and building that had its own comedy: /snap/bin isn't on systemd's default PATH, so the naïve version dies at 3am with command not found, silently, forever — a backup job that never runs but that you believe exists is strictly worse than no backup job. Absolute paths, artifact-size verification, a free-space guard, 7-day retention with a keep-minimum floor, and chmod 700 on the directory because a full cluster dump contains every Secret you own.

Oh — and copy them off the node. A backup on the disk that died is a souvenir.

🧠 What This Taught Me

Check timestamps before you grieve. ls -la on the datastore told me in ten seconds what I'd theorized about for an hour.
Joining a second node moves dqlite from loopback to the LAN. Single-node clusters are accidentally immune to network chaos. Multi-node clusters opt into it. Nobody sends you a consent form.
Laptops-as-servers invert the failure mode. The nodes survive the outage; the router is the single point of failure, and the battery turns a power cut into a delayed-action dirty shutdown.
Pods vanishing is fine. The question is always the datastore.
A silent backup job is worse than none. Verify the artifact exists, verify it has bytes, and scream loudly if not.

📋 Quick Reference (For Skimmers)

Purpose	Command
Where dqlite lives	`/var/snap/microk8s/current/var/kubernetes/backend/`
Who dqlite binds to	`sudo cat .../backend/cluster.yaml`
Datastore's actual age	`sudo ls -la .../backend/` — read `cluster.crt`'s date
Snapshot the datastore	`microk8s dbctl backup <path>`
Restore it	`microk8s stop && microk8s dbctl restore <path> && microk8s start`
Stop battery murder	UPower `PercentageAction=20`, `CriticalPowerAction=PowerOff`
Lid ≠ off switch	logind `HandleLidSwitch=ignore` + mask sleep targets

🚀 What's Next

If the cluster can be vaporized by a plastic router and a tired battery, then the cluster can't be the source of truth anymore. Next episode: I put the entire thing in git and make "everything is gone" a twenty-minute inconvenience instead of a personality crisis.

💬 Final Thoughts

The power cut was innocent. The router was negligent. The battery was armed. And the database was already empty before any of them showed up.

Stay tuned. Popcorn 🍿, coffee ☕, and now a UPS 🔌 — the shopping list grows with the wisdom.

🌪️ HomeLab Chronicles: Episode 4 - Turbulence

sanskar arora — Fri, 28 Aug 2026 14:47:14 +0000

Hey all 👋
Sit tight! Because this is a little lengthy one.

Last episode I closed by announcing that I would be adding two more nodes and bringing in Cilium. Confident. Specific. A real roadmap.

The Dell is still in the drawer. The Raspberry Pi is still in a different drawer. Cilium remains a concept I respect from a distance.

What I did instead was install Apache Airflow on the one node I already had, which I estimated at "an evening." Reader, it was not an evening. It was eight distinct sidequests, two of which were caused by me, three by Airflow 3 quietly renaming things, and one by a YAML field that Kubernetes refuses to let you edit after the fact.

Here's all of it.

🎯 Why Kubernetes At All

The honest alternative was Docker Compose on bare Ubuntu, and for a single-box homelab that's a completely defensible choice -- lower overhead, one file, done in twenty minutes.

I went with MicroK8s anyway, for three reasons that I still think hold up:

Task isolation. With KubernetesExecutor, every task run gets its own ephemeral pod with its own dependencies and its own CPU/memory bounds. A runaway DAG starves itself, not the box.
No idle workers. Pods spawn on demand and die when they're done. No permanent Celery workers sitting there eating 1–2GB of RAM to do nothing at 3am.
Production parity. This is roughly how Airflow runs on EKS/GKE/Composer. If I'm going to fight something, I'd rather fight the thing people actually run.

The tuning that follows from that: KubernetesExecutor, Postgres with persistent storage, and statsd / redis / flower all switched off, since with KubernetesExecutor the Kubernetes API is the queue and those three are just RAM tax.

🕹️ Driving From the Couch

First useful realization: Helm is a client-side tool. It talks to the cluster over the Kubernetes API (port 16443 on MicroK8s) using nothing but a kubeconfig. There is no reason to SSH into the server to run it.

So the server exports its config:

sudo microk8s config

And my laptop takes it:

mkdir -p ~/.kube
ssh msi@192.168.68.210 "sudo microk8s config" > ~/.kube/config-microk8s
export KUBECONFIG=~/.kube/config-microk8s
chmod 600 ~/.kube/config-microk8s

kubectl get nodes

One gotcha: if the generated config says 127.0.0.1:16443, swap it for the server's actual LAN IP or you'll spend a confusing minute deploying Airflow to your own laptop.

From here on, everything in this post runs from my laptop. The server just quietly does as it's told.

💥 Sidequest 1: The Migration Job That Wasn't the Problem

First install. Confident. Straight into a wall:

Error: INSTALLATION FAILED: failed post-install: resource Job/airflow/airflow-run-airflow-migrations
not ready. status: InProgress, message: Job in progress
context deadline exceeded

The database migration job timed out. Fine, let's read its logs:

$ kubectl logs airflow-run-airflow-migrations -n airflow
error: error from server (NotFound): pods "airflow-run-airflow-migrations" not found in namespace "airflow"

That's me being sloppy -- that's a Job name, not a Pod name. kubectl logs wants a pod. And by the time I asked, the job controller had blown past its backoff limit and deleted the pod anyway. Use the selector instead:

kubectl logs -n airflow -l component=run-airflow-migrations --tail=100

Which returned nothing, because there was nothing left to return. So I went and looked at what everything else was doing:

$ kubectl get pods -n airflow -o wide
NAME                                     READY   STATUS                  RESTARTS      AGE
airflow-api-server-669b5665d7-5h87d      0/1     Init:CrashLoopBackOff   6 (98s ago)   9m14s
airflow-dag-processor-6d6ffd87cc-lr9n9   0/2     Pending                 0             9m14s
airflow-postgresql-0                     0/1     Pending                 0             9m14s
airflow-scheduler-7d9648844f-g248q       0/2     Init:CrashLoopBackOff   6 (90s ago)   9m14s
airflow-triggerer-0                      0/2     Pending                 0             9m14s

Three pods Pending, two crash-looping on init. The crash-loopers are just waiting for the database, so they're symptoms. Postgres is the interesting one, and describe gave it up immediately:

Warning  FailedScheduling  4m10s  default-scheduler
  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.

There it is. MicroK8s had no default StorageClass. No provisioner meant no PVC could ever bind, which meant Postgres could never schedule, which meant the migration job had nothing to migrate against, which meant Helm reported a migration timeout.

The error I got was four layers downstream of the actual problem. The install didn't fail because of migrations. It failed because I never turned on storage.

sudo microk8s enable hostpath-storage
kubectl get storageclass   # microk8s-hostpath (default)

Lesson filed: when a Helm install fails on a post-install hook, the hook is almost never the thing that's broken. Go find whatever's Pending.

🐄 Sidequest 2: Longhorn, and the Snap Path Nobody Mentions

hostpath-storage unblocked me, and I could have stopped there. I did not stop there, because "hostpath" means "a directory on the node" and I wanted an actual CSI layer with snapshots and a UI before I started putting data I cared about into this thing.

Longhorn needs host-level prep first -- it's iSCSI-based, and that has to exist on the machine, not in the cluster:

sudo apt update
sudo apt install -y open-iscsi nfs-common util-linux
sudo systemctl enable --now iscsid
sudo modprobe iscsi_tcp

Then the values file, which has two things in it that matter enormously and are easy to miss:

persistence:
  defaultClass: true
  defaultClassReplicaCount: 1
  reclaimPolicy: Delete

defaultSettings:
  defaultReplicaCount: 1
  replicaSoftAntiAffinity: true
  storageOverProvisioningPercentage: 200
  storageMinimalAvailablePercentage: 15

csi:
  kubeletRootDir: "/var/snap/microk8s/common/var/lib/kubelet"

Replica count 1, because Longhorn defaults to 3 and on a one-node cluster every single volume will sit there forever politely waiting for two more nodes that are, as established, in drawers.

kubeletRootDir, because MicroK8s is a snap, and snaps don't put kubelet where every Longhorn tutorial on the internet assumes it is. Get this wrong and the CSI plugin comes up looking healthy and then mounts nothing.

And then Helm couldn't even download the chart:

Error: INSTALLATION FAILED: Get "https://release-assets.githubusercontent.com/...longhorn-1.12.1.tgz":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Not a cluster problem at all -- my laptop's connection to GitHub's release CDN was too slow for Helm's built-in client timeout. The fix is to stop asking Helm to be your downloader:

helm pull longhorn/longhorn
helm install longhorn ./longhorn-*.tgz \
  --namespace longhorn-system \
  --create-namespace \
  --values longhorn-values.yaml \
  --timeout 15m

Fetch the tarball with a tool whose entire job is fetching things, then install from disk. Works offline too, which is a nice side effect.

🚪 Sidequest 3: Two Owners, One CRD

Airflow was up. Now I wanted to reach it from my phone without kubectl port-forward, which meant an ingress, and I decided on Envoy Gateway -- the CNCF one, driven by the Gateway API instead of a pile of NGINX annotations.

Step one: install the Gateway API CRDs. Step two: install Envoy Gateway. Both from the docs, both correct in isolation:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.4 --namespace envoy-gateway-system --create-namespace

Error: INSTALLATION FAILED: failed to install CRD crds/gatewayapi-crds.yaml: conflict occurred while
applying object /gatewayclasses.gateway.networking.k8s.io: Apply failed with 3 conflicts:
conflicts with "kubectl-client-side-apply" using apiextensions.k8s.io/v1:
- .metadata.annotations.gateway.networking.k8s.io/bundle-version
- .metadata.annotations.gateway.networking.k8s.io/channel
- .spec.versions

This one's genuinely interesting. The Envoy Gateway chart ships the Gateway API CRDs itself. So I installed them twice: once via kubectl apply (which registers a field manager called kubectl-client-side-apply), then again via Helm's server-side apply. Kubernetes tracks who owns which field, saw two different managers claiming the same annotations, and refused to guess.

Two ways out. Let Helm own them:

helm uninstall eg -n envoy-gateway-system 2>/dev/null
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml
# reinstall -- the chart brings its own

Or keep mine and tell the chart to stay out of it:

helm install eg oci://docker.io/envoyproxy/gateway-helm --skip-crds ...

I went with letting Helm own them, so version compatibility stays somebody else's problem on future upgrades. Worth noting the delete nukes any existing Gateway/HTTPRoute objects -- harmless on a fresh cluster, less harmless later.

🧭 The Manifests, And Two Things The Obvious Version Gets Wrong

The GatewayClass registers Envoy's controller. The Gateway defines a listener, and creating it makes Envoy Gateway provision an actual Envoy data-plane Deployment and Service. Standard:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-homelab-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
      allowedRoutes:
        namespaces:
          from: All

allowedRoutes.namespaces.from: All is what lets the HTTPRoute live over in the airflow namespace and still attach here.

Now the two traps.

Trap one: you cannot put an IP address in hostnames. The Gateway API Hostname type explicitly excludes numeric IPs. The CRD's regex is loose enough that 192.168.68.210 slips past admission -- each octet looks like a valid DNS label -- so nothing errors and you assume it worked. It is not supported and the matching behaviour isn't guaranteed. (This one bit me later. See sidequest 5.)

Trap two: don't kubectl patch the Envoy Service. Every guide tells you to find the generated service and patch it to NodePort:

kubectl patch svc $ENVOY_SVC -n envoy-gateway-system -p '{"spec": {"type": "NodePort", ...}}'

That works, right up until it doesn't. That Service is controller-owned -- Envoy Gateway recomputes it from the Gateway spec on controller restarts, leader-election events, and Gateway updates. Nothing in the cluster knows you wanted NodePort, so the next reconcile quietly resets it to ClusterIP. It'll break at the worst possible moment and look like a networking problem.

The durable way is an EnvoyProxy resource wired into the GatewayClass via parametersRef, which also lets you pin the nodePort declaratively:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
                - name: http-8080
                  port: 8080
                  protocol: TCP
                  targetPort: 8080
                  nodePort: 30080

Then add the parametersRef to the GatewayClass pointing at it. Now NodePort 30080 is part of the desired state, and reconciliation enforces it instead of undoing it.

🕵️ Sidequest 4: A 500 From A Service That Doesn't Exist

Gateway green. Route applied. Browser: nothing.

Start at the top and work down. Gateway first:

$ kubectl get gateway envoy-homelab-gateway -n envoy-gateway-system -o yaml
status:
  addresses:
  - value: 192.168.68.210
  conditions:
  - reason: Accepted    status: "True"
  - reason: Programmed  status: "True"   message: Address assigned, 1/1 envoy replicas available

Perfect. Service is NodePort on 30080, Envoy pod 2/2 Running. So the whole ingress layer is fine. Then the route:

$ kubectl get httproute airflow-route -n airflow -o yaml
status:
  parents:
  - conditions:
    - reason: Accepted           status: "True"    message: Route is accepted
    - reason: BackendNotFound    status: "False"   message: Service airflow/airflow-webserver not found

Service airflow/airflow-webserver not found. And sure enough:

$ kubectl get svc -n airflow
NAME                    TYPE        CLUSTER-IP       PORT(S)
airflow-api-server      ClusterIP   10.152.183.153   8080/TCP
airflow-postgresql      ClusterIP   10.152.183.194   5432/TCP
airflow-postgresql-hl   ClusterIP   None             5432/TCP
airflow-triggerer       ClusterIP   None             8794/TCP

There is no airflow-webserver. Airflow 3.x deleted the webserver. The UI got folded into the API server, so the chart now creates <release>-api-server. Every tutorial written for Airflow 2 points at a service that no longer exists.

This is also why the local curl returned a 500 rather than a connection error -- Envoy was alive and listening, it just had no valid backend cluster to route to, so it answered every request with an empty 500. The proxy was working perfectly. It had simply been told to forward traffic to a ghost.

One word changed in backendRefs, and:

$ curl -v -H "Host: airflow.local" http://localhost:30080/
< HTTP/1.1 200 OK
< server: uvicorn
<!doctype html>
<title>Airflow</title>

🌐 Sidequest 5: curl Works, The Browser 404s

Victory lap over to the browser on my phone. http://192.168.68.210:30080/:

Looks like there's a problem with this site Error code: 404 Not Found

Two problems stacked, and I'd caused both.

First, I'd been typing 192.168.68.21 into the browser. The node is 192.168.68.210. I stared at that for longer than I want to admit.

Second, and more instructive: my curl was lying to me. Look at what I'd been testing with:

curl -H "Host: airflow.local" http://localhost:30080/

I was manually forcing the Host header to the one value my HTTPRoute matched. The browser sends Host: 192.168.68.210:30080, my route only had hostnames: ["airflow.local"], so Envoy had no matching route and correctly 404'd. Note the failure mode here -- a 404 from the browser is Chrome rendering a real HTTP response from Envoy, not a connection failure. The packet arrived. The routing table just didn't want it.

Since the Gateway API won't let me legitimately match an IP anyway (trap one, from earlier), the fix is to stop matching hostnames entirely. An HTTPRoute with no hostnames field matches any Host header:

spec:
  parentRefs:
    - name: envoy-homelab-gateway
      namespace: envoy-gateway-system
      sectionName: http
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: airflow-api-server
          port: 8080

Airflow UI, on my phone, on the couch. Roughly six hours after "this should take an evening."

📂 Sidequest 6: Where Do DAGs Even Go?

Next: actually put a DAG in it. I went looking for the DAGs folder on the scheduler:

$ kubectl describe pod -n airflow -l component=scheduler
Volumes:
  config:  ConfigMap
  logs:    EmptyDir
  kube-api-access-7zh2t: Projected

No DAGs volume. At all. Not empty -- absent.

Because Airflow 3 also split DAG parsing out of the scheduler into its own dag-processor component. The scheduler genuinely doesn't need DAG files anymore. The volume was on the pod I hadn't looked at:

$ kubectl describe pod airflow-dag-processor-... -n airflow
Mounts:
  /opt/airflow/dags from dags (rw)
Volumes:
  dags:
    Type:       PersistentVolumeClaim
    ClaimName:  airflow-dags

That's two components Airflow 3 renamed or relocated out from under every guide I was reading. Worth internalizing if you're following Airflow 2 material: webserver → api-server, and DAG parsing → dag-processor.

I could have kubectl cp'd files into that PVC. I wanted git-sync instead, from a private repo, which needs a read-only deploy key:

ssh-keygen -t ed25519 -C "airflow-gitsync" -f ./airflow-gitsync-key -N ""

kubectl create secret generic airflow-git-ssh-key-secret \
  -n airflow --from-file=gitSshKey=./airflow-gitsync-key

The secret key must be named gitSshKey -- that's the exact name the chart looks for. Public half goes on the repo as a deploy key with write access unchecked; git-sync only ever pulls.

Then in values:

dags:
  persistence:
    enabled: true      # keep this ON -- git-sync syncs *into* the PVC
    size: 5Gi
    accessMode: ReadWriteOnce
  gitSync:
    enabled: true
    repo: "git@github.com:<user>/<private-repo>.git"
    branch: "main"
    subPath: "dags"
    sshKeySecret: "airflow-git-ssh-key-secret"
    wait: 10

Git-sync runs as a sidecar on the dag-processor, worker, and triggerer pods -- not the scheduler, for the reason above.

📜 Sidequest 7: Logs That Point At Ghosts

Triggered a DAG. It ran. Clicked the logs tab:

Could not read served logs: HTTPConnectionPool(host='test-branch-and-bash-option-a-2njq669w', port=8793):
Max retries exceeded ... Failed to resolve 'test-branch-and-bash-option-a-2njq669w'
([Errno -2] Name or service not known)

Two episodes ago I spent an evening on a ghost that answered pings. This time it's a hostname that used to be a pod.

Here's the mechanism: by default the API server fetches task logs by making an HTTP request directly to the pod that ran the task, on port 8793. Under KubernetesExecutor, task pods are deleted the instant they finish. So the API server is trying to resolve the DNS name of something that stopped existing several seconds ago -- and a bare pod name isn't independently resolvable anyway without a governing Service.

The fix is to stop making it a network problem. Give everything a shared volume and let the API server read the file off disk:

logs:
  persistence:
    enabled: true
    size: 5Gi

The usual caveat is that this wants ReadWriteMany and most simple provisioners don't do RWX. On a single-node cluster it doesn't matter -- RWO means "mountable by one node", not one pod, and every pod here lands on the same node regardless. One-node homelabs get to skip a whole class of storage problem.

🧊 Sidequest 8: The StatefulSet That Wouldn't Budge

Applied that. Helm said no:

Error: UPGRADE FAILED: server-side apply failed for object airflow/airflow-triggerer
apps/v1, Kind=StatefulSet: StatefulSet.apps "airflow-triggerer" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template',
'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

Not a chart bug, not a values mistake -- a hard Kubernetes rule. volumeClaimTemplates on a StatefulSet is immutable after creation. The triggerer had been created with its own per-pod log volume. Turning on shared log persistence removes that block entirely. Kubernetes doesn't allow that field to change on a live object, full stop.

The only move is to delete the object and let Helm rebuild it:

kubectl delete statefulset airflow-triggerer -n airflow
helm upgrade airflow apache-airflow/airflow -n airflow --values airflow-values.yaml

The pod blips for a few seconds, deferred-task processing pauses, nothing is lost. StatefulSets don't garbage-collect their PVCs, so the old per-pod log volume survives the delete as an orphan -- worth cleaning up once you've confirmed the new setup is healthy:

kubectl get pvc -n airflow          # look for logs-airflow-triggerer-0
kubectl delete pvc logs-airflow-triggerer-0 -n airflow

🚧 And Then, A Cliff to jump off of

or at-least it looked like one

Logs working. DAGs syncing. UI reachable from every device in the house. I typed in the admin password, hit enter, and:

Bad Request The CSRF session token is missing.

Created tens of theories hundreds of stories but at a end one patient moment solved it. When i tried longing in after a few seconds it worked, my working hypothesis as of is that the pod to pod auth/authz was not initialised and once it got properly synced i was error free for a few mins.

🧠 What This Taught Me

A failing Helm hook is usually a symptom, not a cause. The migration job "failed" because Postgres couldn't schedule because there was no StorageClass. Go find whatever's Pending before you read a single hook log.
kubectl logs <job-name> doesn't work. Jobs aren't pods. Use -l component=<label>, and do it before the controller cleans up the evidence.
Installing the same CRDs twice is a real failure mode. Field managers collide, and "the docs told me to" doesn't help. Check whether your chart already ships them.
If a controller generates a resource, don't kubectl patch it. Your change lives until the next reconcile and then vanishes without a trace. Put it in the spec the controller reads.
Curl with a forced -H "Host:" is not the same test as a browser. I proved my stack worked for a request no real client would ever send.
Airflow 3 renamed the furniture. webserver → api-server, DAG parsing → dag-processor, [scheduler] dag_dir_list_interval → [dag_processor]. Most tutorials out there are Airflow 2 and will point you at things that no longer exist.
volumeClaimTemplates is immutable. Some Helm upgrades simply require deleting the StatefulSet. That's not a workaround, that's the procedure.
Single-node clusters make RWO vs RWX a non-problem. One of the very few advantages of having exactly one node.
Read the status.conditions block. BackendNotFound, ResolvedRefs: False, FailedScheduling -- the Gateway API and Kubernetes both told me exactly what was wrong, in plain English, every single time. I just had to look.

📋 Quick Reference (For Skimmers)

Purpose	Command
Pull kubeconfig off the server	`ssh user@host "sudo microk8s config" > ~/.kube/config-microk8s`
Enable storage before anything else	`sudo microk8s enable hostpath-storage`
Find what's actually broken	`kubectl get pods -n airflow -o wide` then `describe` the `Pending` one
Logs from a Job's pod	`kubectl logs -n airflow -l component=<label> --tail=100`
Check a Gateway is live	`kubectl get gateway <name> -n <ns> -o yaml` → `Accepted` + `Programmed`
Check a route resolved its backend	`kubectl get httproute <name> -n <ns> -o yaml` → `ResolvedRefs`
Find the generated Envoy service	`kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=<gw>`
Confirm the real service names	`kubectl get svc -n airflow` (it's `api-server`, not `webserver`)
Create the git-sync key secret	`kubectl create secret generic <name> -n airflow --from-file=gitSshKey=<keyfile>`
Watch git-sync do its thing	`kubectl logs -n airflow -l component=dag-processor -c git-sync`
Escape an immutable StatefulSet	`kubectl delete statefulset <name> -n airflow`, then `helm upgrade`

🚀 What's Next

Genuinely, this time -- the Dell and the Pi come out of the drawers. Three nodes. Cilium. The thing I promised you last episode and will keep promising until it becomes true or I run out of drawers.

💬 Final Thoughts

Eight sidequests. Two self-inflicted, three courtesy of Airflow 3's renaming spree, one Kubernetes immutability rule, one CRD ownership fight, and one CDN that was too slow for Helm's patience.

The actual Airflow install -- the part I'd budgeted the evening for -- took about four minutes and worked on the first try.

Stay tuned. Popcorn 🍿 this time optional, but tea is now mandatory 🫖.

🏠 HomeLab Chronicles: Episode 3 - Ghost in the Mesh

sanskar arora — Sun, 23 Aug 2026 22:21:33 +0000

Hey all 👋

Last episode I said the network was going to need luck instead of the laptop. I'd like to formally apologize to the network. It didn't need luck. It needed an exorcism.

The task itself was supposed to be the single most boring item on the entire mission list: Step 2, remote SSH access. Generate a key, copy it over, done, never touch a keyboard directly on this machine again. Ten minutes, tops. I've done it a hundred times on a hundred boxes.

This time it took an evening, a diagram, and a newfound distrust of my own WiFi.

🔑 The Part That Was Supposed to Be Boring

Standard stuff, nothing to see here:

ssh-keygen -t ed25519 -C "msi@homelab"
ssh-copy-id msi@192.168.68.110
ssh msi@192.168.68.110

Key on the client, key copied to the server, connect, no password, go live my life. That's the whole plan. It fell apart before I even got through it.

🚫 Connection Refused, Apparently

$ ssh msi@192.168.68.110
ssh: connect to host 192.168.68.110 port 22: Connection refused

Refused. Not timed out — refused, like the server had looked at me personally and said no. Which felt extra unfair, because when I actually checked the server, it had done nothing wrong:

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Active: active (running)

$ sudo ss -tlnp | grep :22
LISTEN  0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",...))

sshd up, listening on every interface, no firewall even configured yet to blame. By every metric the server had access to about itself, it was innocent. And yet.

📡 Clue One: A Ping With Main Character Energy

Before spiraling into SSH configs, I did the normal thing and pinged it.

$ ping 192.168.68.110
64 bytes from 192.168.68.110: icmp_seq=1 ttl=64 time=312 ms
64 bytes from 192.168.68.110: icmp_seq=2 ttl=64 time=298 ms

It answered! Great, host's up, must be an SSH-level problem — except 300ms on a LAN isn't a number, it's a cry for help. Same-subnet pings on wired gear should be sub-millisecond. 300ms is what you get pinging a different continent, not a laptop three feet away on the same switch. I almost let that slide. I am glad I didn't.

🕵️ Clue Two: The ARP Table Doesn't Lie (Unlike Ping, Apparently)

Here's the part nobody warns you about: a ping reply only proves something answered at that IP. It doesn't prove it was the thing you meant. On a LAN, IP-to-hardware resolution happens over ARP, and your machine caches whatever answer it got first. Wrong cache, wrong destination — for every protocol, ping included.

$ ip neigh show 192.168.68.110
192.168.68.110 dev eth0 lladdr aa:bb:cc:11:22:33 REACHABLE

Then, straight from the server's own mouth:

$ ip link show enp0s3
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
    link/ether aa:bb:cc:44:55:66 brd ff:ff:ff:ff:ff:ff

aa:bb:cc:11:22:33 versus aa:bb:cc:44:55:66. Two different pieces of silicon, one IP address, and my laptop had confidently bet on the wrong one.

👻 So Who's the Impostor?

Somewhere on my network, something else was also answering to .110. And once I actually thought about my network topology instead of just interrogating my server, I had a solid suspect.

Here's the setup: fiber comes into a modem, the modem feeds a TP-Link mesh router, and that mesh router has an extender hanging off it to cover the dead zone at the far end of the apartment.

Internet → Modem → TP-Link Mesh (main node)
                         │
                         ├── extender ── tp-link device, same IP as my server
                         │
                         └── my server (when the mesh feels like it)

Mesh systems are supposed to make all of this invisible — one big happy network, roam anywhere, nobody notices the seams. In practice, an extender bolted onto a mesh is still, underneath, its own little hop with its own opinions about who's connected where. My working theory: something on the extender's side ended up holding .110 at the same time my server did, and depending on which hop answered the ARP broadcast first, my laptop's cache latched onto the extender's guest instead of my actual box.

Ping worked because whatever was squatting on .110 answered ICMP just fine — a real, alive device, just not mine.
SSH got refused, not timed out, because a refusal means the SYN reached a live host that slammed the door on port 22 — nothing was listening there. A timeout would've meant something dropping packets silently. A refusal meant I'd reached a machine. Just not the machine.
The 300ms latency fits the theory too — an extra wireless hop through an extender is exactly the kind of detour that turns sub-millisecond LAN pings into "did this packet go on vacation" pings.

Nothing here was actually broken. It was all working exactly as designed. The design just never accounted for me.

🛠️ The Fix: Stop Negotiating, Just Move

I could've gone spelunking through the mesh app's client list hunting for the impostor. I did not have that kind of patience left in me. I gave the server a new, boring, unclaimed IP instead.

sudo nano /etc/netplan/99-static.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: false
      dhcp6: false
      addresses:
        - 192.168.68.230/24
      routes:
        - to: default
          via: 192.168.68.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 8.8.8.8

sudo netplan try

netplan try deserves a fan club. It applies the change immediately and auto-reverts after 120 seconds unless you confirm — extremely relevant when you're one YAML indent away from locking yourself out over the very connection you're using to make the change. Confirmed the box was still reachable, then made it permanent:

sudo netplan apply

✅ Receipts

$ ping 192.168.68.210
64 bytes from 192.168.68.210: icmp_seq=1 ttl=64 time=0.91 ms

$ ip neigh show 192.168.68.210
192.168.68.210 dev eth0 lladdr aa:bb:cc:44:55:66 REACHABLE

Sub-millisecond, and the MAC finally matches the server's actual NIC. Proof, at last, that I was talking to my own hardware.

🔑 Back to the Boring Part, Finally

ssh-copy-id msi@192.168.68.210
ssh msi@192.168.68.210

Straight in. No password. The ten-minute task, delivered roughly one evening and one network diagram late:

sudo nano /etc/ssh/sshd_config
# PasswordAuthentication no

sudo systemctl restart ssh

🧠 What This Taught Me

Ping proves a device answered. Not which device. Don't let it clear a network problem it never actually investigated.
"Refused" and "timed out" are different diagnoses wearing the same "it didn't work" costume — refused means a live host said no, timeout means something's silently eating your packets. Learn to tell them apart fast.
Weird LAN latency is a symptom, not background noise. Sub-millisecond is the bar; three hundred milliseconds is a red flag wearing a ping reply as a disguise.
ip neigh plus ip link settles an identity crisis in about ten seconds.
Mesh WiFi plus a bolted-on extender is a surprisingly good recipe for exactly this kind of ghost — worth remembering before blaming the server for anything again.
Reserve the address at the router level (MAC-based DHCP reservation), not just on the box. A static IP the router doesn't know about is one collision away from becoming somebody else's problem — namely mine, again.

📋 Quick Reference (For Skimmers)

Purpose	Command
Confirm sshd is listening	`sudo ss -tlnp \
Check the MAC your client has cached	{% raw %}`ip neigh show <ip>`
Check a device's real MAC	`ip link show <interface>`
Clear a stale ARP entry	`sudo ip neigh flush dev <interface>` or `sudo arp -d <ip>`
Apply a network change with a safety net	`sudo netplan try`
Copy your public key to a server	`ssh-copy-id user@host`

🚀 What's Next

Given how well "one server, one IP" just went, the obvious next move is clearly to add more servers and more IPs. So: the Dell Inspiron isn't staying benched after all — it's getting reflashed with the same minimal Ubuntu Server + LXD/LXC combo that worked the first time, except this time with a job description it can actually survive, alongside an old Raspberry Pi I dug out of a drawer. Three nodes, one fledgling cluster.

And since apparently I haven't suffered enough at the hands of networking this episode, the plan is to bring in Cilium — eBPF-based pod networking, or as I now think of it, round two against the concept of "IP address" as a stable idea — to actually handle traffic across all of it instead of hoping defaults figure it out on their own.

Given this episode's track record, I would like the record to reflect I have concerns.

💬 Final Thoughts

Ghost: identified, evicted. SSH: finally, actually working. Trust in my own home WiFi: still recovering.

Stay tuned. Popcorn 🍿 mandatory this time, not optional.

🏠 HomeLab Chronicles: Episode 2 - The Frankenstack

sanskar arora — Sun, 23 Aug 2026 22:09:05 +0000

Hey all 👋

Quick recap for anyone walking in mid-movie: last time, I ragequit my garbage collector project (StopLang, RIP, still not sorry), decided Arch Linux would be the homelab OS "because I hate myself but in a fun way," and set out to Frankenstein an old Dell Inspiron into a server.

Reader. Arch won. Again.

🏴‍☠️ Arch, Round Two (And Some Other News)

I would like to say the Arch install went fine and I switched to Ubuntu for Strategic Reasons. That would be a lie. What actually happened was a rerun of the exact 2024 highlight reel from last episode, a bootloader that needed convincing, drivers that needed convincing, and me, at 1am, needing convincing this was a good use of a Tuesday. Turns out "Arch-nemesis" wasn't just a clever episode title, it was a prophecy, and apparently my prophecies about my own life choices self-fulfill on a pretty tight schedule.

So: Ubuntu Server, similar as my daily driver (debian), same as the "Easy Life Gang" I already confessed to joining. No shame left to lose here.

While I was in there rethinking the OS, I also did some honest math on what this box actually needed to run; a K8s cluster, several flavors of containers, a metrics stack, all at once — and concluded that "8GB of dreams" was not going to survive contact with that plan either.

The Dell Inspiron got benched — not buried, though I didn't know that at the time. In its place, for now:

CPU: 10th-gen i5, quad-core — not a data center chip, but it can hold a conversation
RAM: 16GB — double the dreams, still cheaper than therapy
GPU: RTX 2050 — currently doing nothing but idling. This is foreshadowing. I don't know for what yet, but it's foreshadowing.
Storage: 1TB SSD — room for several Kubernetes clusters and all my feelings about them

Still a laptop cosplaying as a server. Just one with a better résumé.

🐧 Ubuntu Server, Minimal, No Notes

No desktop environment, no bloat, nothing I'm going to get emotionally attached to this time. Just a prompt and my own poor decisions:

lscpu | grep -E "Model name|CPU\(s\)"
free -h
lsblk

Four cores, sixteen gigs, a terabyte of blank canvas. A reasonable enough starting point for whatever's about to happen to it.

🏗️ Building the Stack (or: How I Learned to Stop Worrying and Run Five Container Tools At Once)

Episode 1's Step 3 was, and I quote myself, embarrassingly: "No kubeadm, no fancy tools. Just pure, uncut YAML & CLI pain."

That promise survived for about as long as it took me to remember I have a job and a sleep schedule to protect. Here's what actually landed on the box. (the pure pain plan is postponed for later not scrapped)

📦 LXC/LXD — containers that pretend to be VMs

First up, LXD (built on LXC) for anything that wants to feel like a real machine instead of a single disposable process. Docker-style containers are great for "run this one thing," but sometimes you want something that boots, keeps its own init system, and behaves like a small persistent VM without the overhead of actually virtualizing hardware. That's LXD's whole pitch, and it's a genuinely different tool from everything else on this list.

sudo snap install lxd
sudo lxd init

lxd init walks you through storage backend and networking choices. I took most of the defaults and made a mental note to regret that later, which is extremely on-brand for this project.

🐳 MicroK8s - K8s, but I broke my own rule immediately

This is the "K8s or bust" step from last time. It bust, slightly — I said no fancy tools, then installed the fanciest, most single-command Kubernetes distro Canonical makes:

sudo snap install microk8s --classic
microk8s status --wait-ready
microk8s enable dns storage

MicroK8s packs an entire lightweight control plane into one snap, which felt like cheating right up until I remembered this laptop has four cores total and would like to keep a couple for itself. Efficiency won. Pride lost. I've made peace with it.

🔧 CRI-O — because the default runtime felt too easy

MicroK8s ships with containerd as its runtime out of the box, which would've been the sensible thing to just use. Instead I swapped the runtime over to CRI-O — built specifically around Kubernetes' CRI spec rather than being a general-purpose runtime that also happens to speak CRI. Mostly this meant repointing the kubelet at CRI-O's socket instead of containerd's, then losing an evening to why nothing was pulling images anymore, which feels like a homelab rite of passage at this point.

The honest reason: I wanted to actually understand the CRI boundary instead of pretending containerd is the only runtime that exists. The slightly less honest reason: it was there, and I have a problem.

🦭 Podman — for when I don't need the whole orchestra

Not everything needs to go through Kubernetes. Sometimes I just want to run one container, right now, without asking a control plane's permission first. That's Podman — daemonless, rootless by default, and CLI-compatible enough with Docker that alias docker=podman is basically a rite of passage in the container world.

sudo apt install -y podman
podman run --rm hello-world

It lives alongside the MicroK8s/CRI-O side of the house as the quick-and-dirty option, entirely separate from anything the cluster is doing.

📊 Prometheus — watching the chaos happen, in graphs

With four different ways to run a container now active on one laptop, I needed to actually see what was happening instead of vibing based on fan noise. Prometheus scrapes metrics on a schedule and stores them as time series, which beats my previous monitoring strategy of "is it hot, and did it crash."

Right now it's mostly watching node-level stuff — CPU, memory, disk. Container- and cluster-level scraping is still on the list, and a dashboard on top of all this is next, because collecting metrics nobody's looking at is functionally the same as not collecting them. But that's a problem for a laptop that isn't currently on fire.

🤔 Why Not Just Pick One Tool?

Fair question. LXD, Podman, and MicroK8s-with-CRI-O all solve "run an isolated workload" in overlapping ways, and in a real production environment picking one and defending that choice in a design doc would be the entire job.

This is not a production environment. This is a laptop that used to belong to an 8th-gen i5's better-funded cousin, sitting on a shelf in my apartment. The whole point of a homelab is that nobody's stopping me from running four competing container philosophies at once, just to see how they actually differ from the inside instead of from a comparison blog post. The only incident review here is me, at 1am, asking myself why I did this.

🚀 What's Next

Per the original mission list, Step 2 was supposed to be remote SSH access — "because reaching across the room is so 2020." Should be the easy part. Generate a key, copy it over, done in ten minutes.

Reader, it was not ten minutes. That's Episode 3.

(Step 4, the firewall, is still sitting untouched on the list, quietly judging me from a distance. One nemesis at a time.)

💬 Final Thoughts

Hardware: upgraded. Arch: lost, again, exactly as the title predicted. OS: Ubuntu, minimal, unbothered. Stack: five tools deep and actively questioning my life choices. GPU: still idling, still foreshadowing.

This episode was somehow both "I installed five pieces of infrastructure" and "I still haven't done the one thing that was supposed to be the easy step." That checks out. That's the HomeLab life, apparently.

Stay tuned, grab popcorn 🍿, and this time wish the network luck instead of the laptop. It's going to need it.

☠️ HomeLab Chronicles: Bash, Bootloaders and BooBoos

sanskar arora — Sun, 08 Jun 2025 15:20:53 +0000

Ahoy, me nerdy hearties! ⚓

Today, we sail deeper into the digital abyss, where the seas are encrypted and the only map we follow is man pages. Welcome to Part 1 of this chaos: Getting Arch Installed & Ready for SSH Shenanigans. 🧑‍✈️💻

Let the plundering begin.

🪓 Step 1: Flashin’ Be Plunderin’

Get yerself a fresh Arch ISO (ye ol’ treasure map 🗺️) and flash it onto a thumb drive. Use whatever tool ye fancy — dd, balenaEtcher, or magic. Doesn’t matter. Just don’t accidentally nuke your primary drive or the Kraken (a.k.a. your data) shall rise.

⚰️ Step 2: Boot Ye Olde Rig

Plug in yer USB, boot that dusty Dell (or whatever sea-worn vessel ye got), and sail straight into the BIOS. Change boot order like a rebellious pirate ignoring port law, and let the Arch ISO take the wheel. 🚢

📡 Step 3: Cast the Wi-Fi Net

Time to connect to the seven signals. Enter the iwctl tavern and speak these sacred incantations:

iwctl
station wlan0 scan
station wlan0 get-networks
station wlan0 connect wifi_name
station wlan0 show

Once you're connected, exit like a ghost (exit), and if you’re the suspicious type (which you should be), test your signal with ping archlinux.org -c 3, if the ping pings, the wind is in our sails. 🌬️🏴‍☠️

🔐 Step 4: SSH or Walk the Plank

Install OpenSSH (if it’s not already on board — usually is) pacman -S openssh.
Set a password for the root user (yes, ye be root now — absolute pirate king 🧑‍✈️). Let’s say: passwd choose something like "pass" if you're feeling lucky 🍀.

Now SSH into the box from yer main ship:

ssh root@<ip_address>

If your OS barks at ye, find an equivalent command or walk the plank.

🪚 Step 5: Slicing Up the Drive Like a Mango

Time to swashbuckle that drive:

/boot — 1GB ⚓
EFI — 1GB 🧭
The rest? LVM party time. (But don’t forget to save a few gigs for future chaos.) Estimated ~125GB for LVM. ⚔️

We use the old-school blade: fdisk. Because real pirates don’t use fancy new TUIs.
We're encrypting the big LVM partition, not the little ones — because we hate pain, but not that much.

🧙 Step 6: Manual Install? Aye, Because Pain is the Best Teacher

Could we run archinstall and be done? Sure.
Do we want to? HA. No. Where’s the fun in that, matey?
We do this the hard way. Because learning builds character (and for once, not craving trauma).

🔗 Important Runes (You Will Forget Otherwise)

Trust me, sailor, these are hard-earned battle scars:

🔩 Load Device Mapper before formatting yer volumes:

  modprobe dm_mod

🪛 Update mkinitcpio hooks (or prepare to scream): In /etc/mkinitcpio.conf, modify:

  HOOKS=(base udev autodetect ... encrypt lvm2 filesystems)

⚙️ Tweak GRUB so it speaks encryption: In /etc/default/grub, update:

  GRUB_CMDLINE_LINUX_DEFAULT="... cryptdevice=/dev/<drive>:<volgroup> ..."

💣 MOST IMPORTANT: Generate that GRUB config or the boot gods shall smite you:

  grub-mkconfig -o /boot/grub/grub.cfg

If you skip this, your bootloader will throw a tantrum and “not find vmlinuz or initrd,” and you’ll be mounting USBs like a panicked sailor chasing his lost rum barrel.

(Yes, I did that. Yes, it sucked. Yes, I survived. Barely. 🥲)

🐳 What’s Next?

Now, Arch is alive and answering pings like a loyal parrot, we’ll move on to the Big Fish™: setting up Kubernetes.

But for now… pour yourself a drink, pat your encrypted SSD, and take pride. You’ve sailed through the cursed waters of manual Arch install. 🍻💾

Until next time, mateys.
Keep your drives encrypted and your SSH keys close.

And remember, the name is Captain Jack Sparrow 🏴‍☠️

EDIT (a day later)

Today, while on the bus, I had a thought, To kubernetes I need at least two runners - 1 master and 1 worker node, and guess what, I only have 1 system to experiment with.
I have three options:

buy another machine (Too expensive)
setup a virtualisation solution to get around it (Considering Proxmox and HyperV for now, could move to another hypervisor as well)
hybrid setup - with master node on my machine and workers on VMS in cloud (Azure/GCP)

I am going ahead with option 2 (Hypervisor) because option 1 is expensive, and option 3 will have too many components, and if something goes wrong, there would be multiple points to check on (n/w concerns me the most). So, for now, I will start with hypervisors and then, after successful deployments and testing on my assembled cluster, probably include hybrid nodes (Azure, GCP, etc).

🏠 HomeLab Chronicles: Episode 1 - Arch-nemesis

sanskar arora — Sat, 07 Jun 2025 21:26:41 +0000

Hey all 👋

So, I was supposed to be working on my garbage collector project and the very stoppable StopLang, but... you know when your brain says C, and your heart screams "Sea ya later!"? Yeah. That.

So instead, I’m kicking off a new project and a new series I’m calling:

🧪 HomeLab: Because what's better than chaos in your own home?

💻 The Plan: Frankensteining an Old Laptop

The victim: a brave old Dell Inspiron 🧟‍♂️
Specs:

CPU: 8thgen i5 processor (not the freshest fruit in the basket, but it'll do)
Memory: 8GB of dreams
Disk Space: 256GB of potential

I’m going to torch this machine 🔥 with a Linux distro (no GUI, because who needs fancy pixels?) and then just vibe with it.

🏴‍☠️ Why Arch? Because I hate myself (but in a fun way)

Arch Linux is my pick for this culinary disaster of a home server. Here's why:

No bloatware. Just vibes.
Comes bare minimum out of the box (just like me on Mondays).
Most importantly, PewDiePie uses Arch.
- And honestly, it bruised my tech ego that while I’m here running Ubuntu, he’s flexing with Arch. Not okay.

🧠 Backstory:
I was actually an Arch user until November 2024 (RIP). Then I got a shiny new laptop and realized I no longer had the time (or sanity) to fix bootloader issues for the 100th time or recompile audio drivers like it’s Groundhog Day. So I reluctantly joined the Ubuntu™ Easy Life Gang.

🏗️ The Mission: Milestone 1

Here’s what I’ll be doing in Phase 1 of this HomeLab Madness™:

✅ Step 1: Install Arch (with just enough to not regret it instantly)

Barebones setup only. Think minimal. Think IKEA flat-pack with no manual.

✅ Step 2: Remote Control, Baby 🚀

Get remote SSH access, because reaching across the room is so 2020.

✅ Step 3: K8s or Bust 🐳

Set up a Kubernetes cluster from scratch.
- No kubeadm, no fancy tools.
- Just pure, uncut YAML & CLI pain.
- Because apparently I like suffering.

✅ Step 4: 🔥 Wall Building (Digital Only)

Set up firewalls to protect this digital potato.
But I want more than just ufw incantations — I’ll aim to build something slightly more usable, because who wants to memorize commands when you can just push a button?

🚀 After That?

Big plans, my friends.

I’ll be building a platform where I can deploy workloads on this cluster using:

A Docker image URL (or a local name if I’m feeling nostalgic),
A few config options to tweak runtime behaviour & exposure,
And some classification magic to keep things sorted.

Basically: "Give me a container, and I shall give it a home."

💬 Final Thoughts

This series is going to be weird, wild, and probably involve more swearing at YAML than anyone should legally be allowed to do. But hey — that’s (I guess) the HomeLab life.

Stay tuned, grab popcorn 🍿, and wish my Dell laptop luck. It’s going to need it.

Birth of StopLang – A Useless Language With a Very Specific Purpose

sanskar arora — Thu, 05 Jun 2025 19:49:32 +0000

So, I wanted to implement a garbage collector.

Cool idea, right? Memory management, pointer wizardry, existential dread—the whole buffet.

But then came the realization™️:

“Uhhh, how the hell do I even test this thing?”

You can’t really test a garbage collector in a vacuum (though that would collect garbage nicely). I needed some code, some variables, some actual garbage. Something to allocate memory, throw things around, and see if the collector actually does its job—or rage quits.

Option A: Plug it into an existing language

Like a real adult. Take some interpreter written in C, rip out its memory management, and bolt mine in.

But nah.

That way lies integration hell.

Do I look like someone who enjoys linker errors and segfaults from code I didn’t write? (No. No, I do not.)

Option B: Just build my own language

Like an unhinged adult.

So that’s what I did.

Okay so here’s the tea:

🛠️ It’ll be compiled, because I want my errors during dev time, not as jump scares during runtime.
⚙️ It’s in C, because:
- C++ is too friendly (a little too comfy, not enough pain).
- Assembly is too sadistic (I still like my sanity).
- C is that spicy sweet spot of power and pain.

We're going full DIY on this one. Think of it as a language that’s just smart enough to test our GC, but dumb enough that we don’t have to spend years writing a parser.

Now, we needed a name. Something iconic. Something that tells you what this language is really about.

And so, I give you:

🛑 `StopLang`

Because:

It’s the opposite of Go. It doesn’t want to “go” anywhere.
It’s useless, and proudly so.

It’s not fast.

It’s not smart.

It’s barely a language.

But it’s ours, and it will be just good enough to build and test garbage collectors.

Bonus: It will be easier to read than Brainfuck, and more coherent than Brainrot. (Which honestly sounds like medical conditions at this point.)

📌 What’s Coming Next?

This blog would turn into a research paper if I tried to cover everything here. So we’re breaking it down:

This post: Why StopLang exists and how I got here (hello chaos)
Next post(s): Building the core of StopLang—basic syntax, data structures, compiler setup, maybe a sad little REPL
Then: Plugging in our garbage collectors and watching them either work… or work once and then segfault spectacularly

If you’ve read this far, bless your soul.

Thanks for bearing with the puns, the rambling, and the “what if I just wrote a language for no reason?” energy.

Stay tuned, fellow code gremlins.

More nonsense incoming soon. 😈🧹

Building a Garbage Collector in C: Because Who Needs Sleep Anyway?

sanskar arora — Mon, 02 Jun 2025 22:12:34 +0000

Let’s be honest: most people’s idea of a good time doesn’t involve poking around in the underbelly of C, trying to make memory management less… well, terrifying. But here I am, rolling up my sleeves and diving into the world of garbage collectors. Why? Because understanding how the engine works under the bonnet is way more fun than actually driving the car. (And also, I’m only a little cracked—not batshit crazy, so no assembly for now.)

Why C? Why Not?

You might ask, “Why C? Why not something friendlier, like Python, or at least C++ with its fancy destructors?” Well, I want to keep things low-level. Real low. Like, “I can see the bits and bytes from here” low. Plus, C gives you the delightful opportunity to shoot yourself in the foot with malloc and free, which is basically a rite of passage for any programmer.

Step 1: Accept Your Fate

Before you even write a line of code, accept that you will spend hours debugging segfaults, chasing dangling pointers, and questioning your life choices. This is normal. This is growth. This is C.

Step 2: Plan Your Collector

There are many flavours of garbage collectors—reference counting, mark-and-sweep, generational… I’ll probably start with something simple, like reference counting and end with mark-and-sweep, because I like the idea of pretending to be a janitor for my code. (“Sweep the heap, Daniel-san!”) And yes, I am only going to do 2 types.

Step 3: Write Some Code, Break Some Code

Here’s where the real fun begins. You’ll write a function to allocate memory, another to mark what’s still in use, and a third to sweep away the rest. Inevitably, you’ll forget to mark something, and your program will crash spectacularly. Don’t worry—it’s all part of the process.

Step 4: Document the Madness

As I go, I’ll document every step, every bug, and every existential crisis. Because if you can’t laugh at your own memory leaks, what can you laugh at?

The Journey Continues…

So, join me as I attempt to tame the wild beast that is manual memory management in C. Will I succeed? Will I lose my mind? Will I finally understand how the engine works beneath the bonnet? Only time—and a lot of printf debugging—will tell.

Stay tuned, and remember: in C, the real garbage collector… is you.