🤫 HomeLab Chronicles: Episode 8 - The Silent Treatment

Hey all 👋

Quick recap: power cut (Ep5), GitOps (Ep6), locked myself out of my own control plane (Ep7), rebuilt in twenty minutes because GitOps (Ep6, smugly). Now: getting Airflow back up. Which should be easy, since I've installed it before — Episode 4, eight sidequests, badge earned.

This episode is about a different kind of enemy. Not the loud failure — the silent one. The config that gets accepted, nodded at, and thrown away. Because here's Helm's dirtiest little behavior, in one sentence:

Helm does not validate your values. An unknown key isn't an error. It's a shrug.

Typo a field, use a field from an older chart, invent a field that sounds plausible — Helm renders the templates, your key matches nothing, and the install proceeds looking exactly like success. No warning. No error. Just vibes.

---

🪦 The Graveyard of Values I Was Setting

Instead of trusting tutorials (or my own Episode 4, ahem), I finally did the thing: fetched the actual chart source — apache/airflow at tag helm-chart/1.22.0, whose appVersion is exactly my Airflow 3.2.2 — and checked every field I was setting against values.yaml and the templates. The census results:

Exhibit A: webserver: — valid key, wrong decade. Chart 1.22.0 ships both webserver: (Airflow 2) and apiServer: (Airflow 3). My replicas and resource limits sat under webserver:, where Airflow 3 never looks. Every limit I thought I'd set on the UI? Decorative. Episode 4 established the service was renamed; it didn't occur to me the values tree split too.

Exhibit B: logs.persistence.accessMode — does not exist. This one's beautiful. The chart's log-PVC template hardcodes accessModes: ["ReadWriteMany"]. Which means my careful Episode 4 setting of ReadWriteOnce — with its confident little comment about single-node clusters — was ignored the entire time. I got RWX whether I asked or not. The chart was protecting me from a config I was proud of.

Exhibit C: webserverSecretKeySecretName — Airflow <3 only. The comment in the chart source says it plainly: that's the Flask secret for the old webserver. Airflow 3 signs JWTs, wants jwtSecretName, and the secret's key must be exactly jwt-secret — that name comes from a template helper, not from you. Get this wrong and the chart generates a fresh key on every render, and your components restart "for no reason." A warning my very first install printed, months of installs ago, that I finally understand.

Exhibit D: knownHosts — a string, not a secret. I'd stuffed a known_hosts file into the SSH secret next to the deploy key, feeling organized. The chart wanted it as a plain values string under dags.gitSync.knownHosts. My copy sat in the secret, unread, while git-sync remained one bad default away from host-key failure. (Also, wait: got superseded by period:. Also-also, the SSH secret's key must be named gitSshKey verbatim — that one Episode 4 got right. Growth.)

Four fields. Four different flavors of silently doing nothing. helm lint caught zero of them, because none of them are syntax errors. They're fluent, grammatical nonsense.

---

🎩 Sidequest: The Job That Vanished Like a Magician

With values corrected, install. Pods appear! Pods crash-loop. Init container says:

TimeoutError: There are still unapplied migrations after 60 seconds.
MigrationHead(s) in DB: set() | Migration Head(s) in Source Code: {'1d6611b6ab7c'}

set() — empty. Postgres reachable, schema blank. The migration job never ran. And when I went looking for it:

$ kubectl get jobs -A
No resources found

Nothing. Anywhere. Because by default the chart runs migrations as a Helm hook — it lives inside the install ceremony and gets cleaned up after, success or failure. My install had stalled (see below), Helm aborted early, the hook evaporated, and I was left with pods waiting for a schema that a ghost was supposed to write. Under Flux this deadlocks beautifully: pods can't start without migrations, Helm won't rerun migrations because pods failed.

The chart's own answer for GitOps land:

migrateDatabaseJob:
  enabled: true
  useHelmHooks: false
createUserJob:
  useHelmHooks: false

Now the jobs are ordinary resources: applied like everything else, visible in kubectl get jobs, logs readable when they fail. Debuggable beats ceremonial.

---

🥷 Sidequest: The Disk That Was "Already In Use"

One more silent saboteur, this time below Kubernetes entirely. Fresh Postgres volume, brand new, zero history — and:

MountVolume.MountDevice failed ... mke2fs 1.47.0
/dev/longhorn/pvc-... is apparently in use by the system;
will not make a filesystem here!

In use? It was born thirty seconds ago. I deleted every PV, PVC, and Longhorn volume in the cluster and got the identical error on the replacement — which is the tell that the problem isn't in the cluster at all.

It's multipathd. Ubuntu ships it enabled, and its hobby is grabbing new block devices the instant they appear, in case they're part of a multipath SAN. I do not have a SAN. I have two laptops. But multipathd claimed each fresh Longhorn device anyway, held it open, and mke2fs politely declined to format a held disk.

sudo systemctl disable --now multipathd multipathd.socket
kubectl -n airflow delete pod airflow-postgresql-0

Postgres: Running. It's a documented Longhorn prerequisite that approximately nobody's tutorial mentions, including — checks notes — my own host-prep list from Episode 4.

---

🧷 Bonus Round: The Password That Broke YAML

The admin password flows in through a generated values snippet. First version built that YAML with shell string concatenation, which works perfectly until someone's password contains a " — at which point the YAML is invalid and the failure surfaces much later as an inscrutable HelmRelease error, with the password (correctly!) absent from every log. The fix is to stop hand-assembling YAML like it's 1997: serialize with json.dumps — JSON is valid YAML, it's stdlib, and it escapes quotes, backslashes, colons and leading #s without opinion. Tested against the most hostile passwords I could invent. All survived.

---

🧠 What This Taught Me

• Helm ignores unknown values silently. All of them. Forever. The only defense is reading the chart's actual values.yaml — at the tag you're installing — not tutorials, not memory, not last episode's notes.
• helm show values <chart> | grep <field> before setting anything load-bearing. Thirty seconds. Cheaper than a weekend.
• A valid key from the wrong era is worse than a typo. webserver: parsed fine, rendered fine, and did nothing. Typos at least look wrong.
• Helm hooks and GitOps are a bad marriage. useHelmHooks: false turns invisible ceremonies into inspectable resources.
• MigrationHead(s) in DB: set() means "empty schema," not "slow schema." The waiter isn't slow; the kitchen never got the order.
• multipathd eats block devices on stock Ubuntu. If mke2fs claims a newborn disk is busy, that's your poltergeist.
• Never build YAML with string concatenation when a serializer is right there.

📋 Quick Reference (For Skimmers)

Purpose	Command
The ground truth for chart fields	helm show values apache-airflow/airflow > /tmp/v.yaml then grep
Even grounder truth	read the chart repo at the release tag — templates don't lie
Airflow 3 UI settings live under	apiServer: (not webserver:)
JWT secret	jwtSecretName: → secret key must be jwt-secret
Jobs you can actually see	migrateDatabaseJob.useHelmHooks: false
Disk claimed at birth	systemctl disable --now multipathd multipathd.socket
Schema actually migrated?	look for MigrationHead(s) in DB: ≠ set()

🚀 What's Next

Airflow is up, migrations ran, and I can see it from my couch. But couch-radius availability is so Episode 4. Next time: a Cloudflare Tunnel puts it on my actual domain, from anywhere, without opening a single port on the plastic router of doom.

💬 Final Thoughts

Loud failures cost hours. Silent ones cost weekends — and this episode had four values, one hook, one daemon, and one quotation mark, all failing without making a sound.

Popcorn 🍿, coffee ☕, UPS 🔌, laminated coredns card 🪪, and now a bookmark to the chart source. The toolkit is mostly talismans at this point.