DEV Community

sanskar arora
sanskar arora

Posted on

đŸŒĒī¸ HomeLab Chronicles: Episode 4 - Turbulence

#airflow #kubernetes #helm #devops

Hey all 👋
Sit tight! Because this is a little lengthy one.

Last episode I closed by announcing that I would be adding two more nodes and bringing in Cilium. Confident. Specific. A real roadmap.

The Dell is still in the drawer. The Raspberry Pi is still in a different drawer. Cilium remains a concept I respect from a distance.

What I did instead was install Apache Airflow on the one node I already had, which I estimated at "an evening." Reader, it was not an evening. It was eight distinct sidequests, two of which were caused by me, three by Airflow 3 quietly renaming things, and one by a YAML field that Kubernetes refuses to let you edit after the fact.

Here's all of it.

đŸŽ¯ Why Kubernetes At All

The honest alternative was Docker Compose on bare Ubuntu, and for a single-box homelab that's a completely defensible choice -- lower overhead, one file, done in twenty minutes.

I went with MicroK8s anyway, for three reasons that I still think hold up:

  • Task isolation. With KubernetesExecutor, every task run gets its own ephemeral pod with its own dependencies and its own CPU/memory bounds. A runaway DAG starves itself, not the box.
  • No idle workers. Pods spawn on demand and die when they're done. No permanent Celery workers sitting there eating 1–2GB of RAM to do nothing at 3am.
  • Production parity. This is roughly how Airflow runs on EKS/GKE/Composer. If I'm going to fight something, I'd rather fight the thing people actually run.

The tuning that follows from that: KubernetesExecutor, Postgres with persistent storage, and statsd / redis / flower all switched off, since with KubernetesExecutor the Kubernetes API is the queue and those three are just RAM tax.

🕹ī¸ Driving From the Couch

First useful realization: Helm is a client-side tool. It talks to the cluster over the Kubernetes API (port 16443 on MicroK8s) using nothing but a kubeconfig. There is no reason to SSH into the server to run it.

So the server exports its config:

sudo microk8s config
Enter fullscreen mode Exit fullscreen mode

And my laptop takes it:

mkdir -p ~/.kube
ssh msi@192.168.68.210 "sudo microk8s config" > ~/.kube/config-microk8s
export KUBECONFIG=~/.kube/config-microk8s
chmod 600 ~/.kube/config-microk8s

kubectl get nodes
Enter fullscreen mode Exit fullscreen mode

One gotcha: if the generated config says 127.0.0.1:16443, swap it for the server's actual LAN IP or you'll spend a confusing minute deploying Airflow to your own laptop.

From here on, everything in this post runs from my laptop. The server just quietly does as it's told.

đŸ’Ĩ Sidequest 1: The Migration Job That Wasn't the Problem

First install. Confident. Straight into a wall:

Error: INSTALLATION FAILED: failed post-install: resource Job/airflow/airflow-run-airflow-migrations
not ready. status: InProgress, message: Job in progress
context deadline exceeded
Enter fullscreen mode Exit fullscreen mode

The database migration job timed out. Fine, let's read its logs:

$ kubectl logs airflow-run-airflow-migrations -n airflow
error: error from server (NotFound): pods "airflow-run-airflow-migrations" not found in namespace "airflow"
Enter fullscreen mode Exit fullscreen mode

That's me being sloppy -- that's a Job name, not a Pod name. kubectl logs wants a pod. And by the time I asked, the job controller had blown past its backoff limit and deleted the pod anyway. Use the selector instead:

kubectl logs -n airflow -l component=run-airflow-migrations --tail=100
Enter fullscreen mode Exit fullscreen mode

Which returned nothing, because there was nothing left to return. So I went and looked at what everything else was doing:

$ kubectl get pods -n airflow -o wide
NAME                                     READY   STATUS                  RESTARTS      AGE
airflow-api-server-669b5665d7-5h87d      0/1     Init:CrashLoopBackOff   6 (98s ago)   9m14s
airflow-dag-processor-6d6ffd87cc-lr9n9   0/2     Pending                 0             9m14s
airflow-postgresql-0                     0/1     Pending                 0             9m14s
airflow-scheduler-7d9648844f-g248q       0/2     Init:CrashLoopBackOff   6 (90s ago)   9m14s
airflow-triggerer-0                      0/2     Pending                 0             9m14s
Enter fullscreen mode Exit fullscreen mode

Three pods Pending, two crash-looping on init. The crash-loopers are just waiting for the database, so they're symptoms. Postgres is the interesting one, and describe gave it up immediately:

Warning  FailedScheduling  4m10s  default-scheduler
  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.
Enter fullscreen mode Exit fullscreen mode

There it is. MicroK8s had no default StorageClass. No provisioner meant no PVC could ever bind, which meant Postgres could never schedule, which meant the migration job had nothing to migrate against, which meant Helm reported a migration timeout.

The error I got was four layers downstream of the actual problem. The install didn't fail because of migrations. It failed because I never turned on storage.

sudo microk8s enable hostpath-storage
kubectl get storageclass   # microk8s-hostpath (default)
Enter fullscreen mode Exit fullscreen mode

Lesson filed: when a Helm install fails on a post-install hook, the hook is almost never the thing that's broken. Go find whatever's Pending.

🐄 Sidequest 2: Longhorn, and the Snap Path Nobody Mentions

hostpath-storage unblocked me, and I could have stopped there. I did not stop there, because "hostpath" means "a directory on the node" and I wanted an actual CSI layer with snapshots and a UI before I started putting data I cared about into this thing.

Longhorn needs host-level prep first -- it's iSCSI-based, and that has to exist on the machine, not in the cluster:

sudo apt update
sudo apt install -y open-iscsi nfs-common util-linux
sudo systemctl enable --now iscsid
sudo modprobe iscsi_tcp
Enter fullscreen mode Exit fullscreen mode

Then the values file, which has two things in it that matter enormously and are easy to miss:

persistence:
  defaultClass: true
  defaultClassReplicaCount: 1
  reclaimPolicy: Delete

defaultSettings:
  defaultReplicaCount: 1
  replicaSoftAntiAffinity: true
  storageOverProvisioningPercentage: 200
  storageMinimalAvailablePercentage: 15

csi:
  kubeletRootDir: "/var/snap/microk8s/common/var/lib/kubelet"
Enter fullscreen mode Exit fullscreen mode

Replica count 1, because Longhorn defaults to 3 and on a one-node cluster every single volume will sit there forever politely waiting for two more nodes that are, as established, in drawers.

kubeletRootDir, because MicroK8s is a snap, and snaps don't put kubelet where every Longhorn tutorial on the internet assumes it is. Get this wrong and the CSI plugin comes up looking healthy and then mounts nothing.

And then Helm couldn't even download the chart:

Error: INSTALLATION FAILED: Get "https://release-assets.githubusercontent.com/...longhorn-1.12.1.tgz":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Enter fullscreen mode Exit fullscreen mode

Not a cluster problem at all -- my laptop's connection to GitHub's release CDN was too slow for Helm's built-in client timeout. The fix is to stop asking Helm to be your downloader:

helm pull longhorn/longhorn
helm install longhorn ./longhorn-*.tgz \
  --namespace longhorn-system \
  --create-namespace \
  --values longhorn-values.yaml \
  --timeout 15m
Enter fullscreen mode Exit fullscreen mode

Fetch the tarball with a tool whose entire job is fetching things, then install from disk. Works offline too, which is a nice side effect.

đŸšĒ Sidequest 3: Two Owners, One CRD

Airflow was up. Now I wanted to reach it from my phone without kubectl port-forward, which meant an ingress, and I decided on Envoy Gateway -- the CNCF one, driven by the Gateway API instead of a pile of NGINX annotations.

Step one: install the Gateway API CRDs. Step two: install Envoy Gateway. Both from the docs, both correct in isolation:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.4 --namespace envoy-gateway-system --create-namespace
Enter fullscreen mode Exit fullscreen mode 
Error: INSTALLATION FAILED: failed to install CRD crds/gatewayapi-crds.yaml: conflict occurred while
applying object /gatewayclasses.gateway.networking.k8s.io: Apply failed with 3 conflicts:
conflicts with "kubectl-client-side-apply" using apiextensions.k8s.io/v1:
- .metadata.annotations.gateway.networking.k8s.io/bundle-version
- .metadata.annotations.gateway.networking.k8s.io/channel
- .spec.versions
Enter fullscreen mode Exit fullscreen mode

This one's genuinely interesting. The Envoy Gateway chart ships the Gateway API CRDs itself. So I installed them twice: once via kubectl apply (which registers a field manager called kubectl-client-side-apply), then again via Helm's server-side apply. Kubernetes tracks who owns which field, saw two different managers claiming the same annotations, and refused to guess.

Two ways out. Let Helm own them:

helm uninstall eg -n envoy-gateway-system 2>/dev/null
kubectl delete -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml
# reinstall -- the chart brings its own
Enter fullscreen mode Exit fullscreen mode

Or keep mine and tell the chart to stay out of it:

helm install eg oci://docker.io/envoyproxy/gateway-helm --skip-crds ...
Enter fullscreen mode Exit fullscreen mode

I went with letting Helm own them, so version compatibility stays somebody else's problem on future upgrades. Worth noting the delete nukes any existing Gateway/HTTPRoute objects -- harmless on a fresh cluster, less harmless later.

🧭 The Manifests, And Two Things The Obvious Version Gets Wrong

The GatewayClass registers Envoy's controller. The Gateway defines a listener, and creating it makes Envoy Gateway provision an actual Envoy data-plane Deployment and Service. Standard:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: envoy-homelab-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
      allowedRoutes:
        namespaces:
          from: All
Enter fullscreen mode Exit fullscreen mode

allowedRoutes.namespaces.from: All is what lets the HTTPRoute live over in the airflow namespace and still attach here.

Now the two traps.

Trap one: you cannot put an IP address in hostnames. The Gateway API Hostname type explicitly excludes numeric IPs. The CRD's regex is loose enough that 192.168.68.210 slips past admission -- each octet looks like a valid DNS label -- so nothing errors and you assume it worked. It is not supported and the matching behaviour isn't guaranteed. (This one bit me later. See sidequest 5.)

Trap two: don't kubectl patch the Envoy Service. Every guide tells you to find the generated service and patch it to NodePort:

kubectl patch svc $ENVOY_SVC -n envoy-gateway-system -p '{"spec": {"type": "NodePort", ...}}'
Enter fullscreen mode Exit fullscreen mode

That works, right up until it doesn't. That Service is controller-owned -- Envoy Gateway recomputes it from the Gateway spec on controller restarts, leader-election events, and Gateway updates. Nothing in the cluster knows you wanted NodePort, so the next reconcile quietly resets it to ClusterIP. It'll break at the worst possible moment and look like a networking problem.

The durable way is an EnvoyProxy resource wired into the GatewayClass via parametersRef, which also lets you pin the nodePort declaratively:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: NodePort
        patch:
          type: StrategicMerge
          value:
            spec:
              ports:
                - name: http-8080
                  port: 8080
                  protocol: TCP
                  targetPort: 8080
                  nodePort: 30080
Enter fullscreen mode Exit fullscreen mode

Then add the parametersRef to the GatewayClass pointing at it. Now NodePort 30080 is part of the desired state, and reconciliation enforces it instead of undoing it.

đŸ•ĩī¸ Sidequest 4: A 500 From A Service That Doesn't Exist

Gateway green. Route applied. Browser: nothing.

Start at the top and work down. Gateway first:

$ kubectl get gateway envoy-homelab-gateway -n envoy-gateway-system -o yaml
status:
  addresses:
  - value: 192.168.68.210
  conditions:
  - reason: Accepted    status: "True"
  - reason: Programmed  status: "True"   message: Address assigned, 1/1 envoy replicas available
Enter fullscreen mode Exit fullscreen mode

Perfect. Service is NodePort on 30080, Envoy pod 2/2 Running. So the whole ingress layer is fine. Then the route:

$ kubectl get httproute airflow-route -n airflow -o yaml
status:
  parents:
  - conditions:
    - reason: Accepted           status: "True"    message: Route is accepted
    - reason: BackendNotFound    status: "False"   message: Service airflow/airflow-webserver not found
Enter fullscreen mode Exit fullscreen mode

Service airflow/airflow-webserver not found. And sure enough:

$ kubectl get svc -n airflow
NAME                    TYPE        CLUSTER-IP       PORT(S)
airflow-api-server      ClusterIP   10.152.183.153   8080/TCP
airflow-postgresql      ClusterIP   10.152.183.194   5432/TCP
airflow-postgresql-hl   ClusterIP   None             5432/TCP
airflow-triggerer       ClusterIP   None             8794/TCP
Enter fullscreen mode Exit fullscreen mode

There is no airflow-webserver. Airflow 3.x deleted the webserver. The UI got folded into the API server, so the chart now creates <release>-api-server. Every tutorial written for Airflow 2 points at a service that no longer exists.

This is also why the local curl returned a 500 rather than a connection error -- Envoy was alive and listening, it just had no valid backend cluster to route to, so it answered every request with an empty 500. The proxy was working perfectly. It had simply been told to forward traffic to a ghost.

One word changed in backendRefs, and:

$ curl -v -H "Host: airflow.local" http://localhost:30080/
< HTTP/1.1 200 OK
< server: uvicorn
<!doctype html>
<title>Airflow</title>
Enter fullscreen mode Exit fullscreen mode

🌐 Sidequest 5: curl Works, The Browser 404s

Victory lap over to the browser on my phone. http://192.168.68.210:30080/:

Looks like there's a problem with this site Error code: 404 Not Found

Two problems stacked, and I'd caused both.

First, I'd been typing 192.168.68.21 into the browser. The node is 192.168.68.210. I stared at that for longer than I want to admit.

Second, and more instructive: my curl was lying to me. Look at what I'd been testing with:

curl -H "Host: airflow.local" http://localhost:30080/
Enter fullscreen mode Exit fullscreen mode

I was manually forcing the Host header to the one value my HTTPRoute matched. The browser sends Host: 192.168.68.210:30080, my route only had hostnames: ["airflow.local"], so Envoy had no matching route and correctly 404'd. Note the failure mode here -- a 404 from the browser is Chrome rendering a real HTTP response from Envoy, not a connection failure. The packet arrived. The routing table just didn't want it.

Since the Gateway API won't let me legitimately match an IP anyway (trap one, from earlier), the fix is to stop matching hostnames entirely. An HTTPRoute with no hostnames field matches any Host header:

spec:
  parentRefs:
    - name: envoy-homelab-gateway
      namespace: envoy-gateway-system
      sectionName: http
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: airflow-api-server
          port: 8080
Enter fullscreen mode Exit fullscreen mode

Airflow UI, on my phone, on the couch. Roughly six hours after "this should take an evening."

📂 Sidequest 6: Where Do DAGs Even Go?

Next: actually put a DAG in it. I went looking for the DAGs folder on the scheduler:

$ kubectl describe pod -n airflow -l component=scheduler
Volumes:
  config:  ConfigMap
  logs:    EmptyDir
  kube-api-access-7zh2t: Projected
Enter fullscreen mode Exit fullscreen mode

No DAGs volume. At all. Not empty -- absent.

Because Airflow 3 also split DAG parsing out of the scheduler into its own dag-processor component. The scheduler genuinely doesn't need DAG files anymore. The volume was on the pod I hadn't looked at:

$ kubectl describe pod airflow-dag-processor-... -n airflow
Mounts:
  /opt/airflow/dags from dags (rw)
Volumes:
  dags:
    Type:       PersistentVolumeClaim
    ClaimName:  airflow-dags
Enter fullscreen mode Exit fullscreen mode

That's two components Airflow 3 renamed or relocated out from under every guide I was reading. Worth internalizing if you're following Airflow 2 material: webserver → api-server, and DAG parsing → dag-processor.

I could have kubectl cp'd files into that PVC. I wanted git-sync instead, from a private repo, which needs a read-only deploy key:

ssh-keygen -t ed25519 -C "airflow-gitsync" -f ./airflow-gitsync-key -N ""

kubectl create secret generic airflow-git-ssh-key-secret \
  -n airflow --from-file=gitSshKey=./airflow-gitsync-key
Enter fullscreen mode Exit fullscreen mode

The secret key must be named gitSshKey -- that's the exact name the chart looks for. Public half goes on the repo as a deploy key with write access unchecked; git-sync only ever pulls.

Then in values:

dags:
  persistence:
    enabled: true      # keep this ON -- git-sync syncs *into* the PVC
    size: 5Gi
    accessMode: ReadWriteOnce
  gitSync:
    enabled: true
    repo: "git@github.com:<user>/<private-repo>.git"
    branch: "main"
    subPath: "dags"
    sshKeySecret: "airflow-git-ssh-key-secret"
    wait: 10
Enter fullscreen mode Exit fullscreen mode

Git-sync runs as a sidecar on the dag-processor, worker, and triggerer pods -- not the scheduler, for the reason above.

📜 Sidequest 7: Logs That Point At Ghosts

Triggered a DAG. It ran. Clicked the logs tab:

Could not read served logs: HTTPConnectionPool(host='test-branch-and-bash-option-a-2njq669w', port=8793):
Max retries exceeded ... Failed to resolve 'test-branch-and-bash-option-a-2njq669w'
([Errno -2] Name or service not known)
Enter fullscreen mode Exit fullscreen mode

Two episodes ago I spent an evening on a ghost that answered pings. This time it's a hostname that used to be a pod.

Here's the mechanism: by default the API server fetches task logs by making an HTTP request directly to the pod that ran the task, on port 8793. Under KubernetesExecutor, task pods are deleted the instant they finish. So the API server is trying to resolve the DNS name of something that stopped existing several seconds ago -- and a bare pod name isn't independently resolvable anyway without a governing Service.

The fix is to stop making it a network problem. Give everything a shared volume and let the API server read the file off disk:

logs:
  persistence:
    enabled: true
    size: 5Gi
Enter fullscreen mode Exit fullscreen mode

The usual caveat is that this wants ReadWriteMany and most simple provisioners don't do RWX. On a single-node cluster it doesn't matter -- RWO means "mountable by one node", not one pod, and every pod here lands on the same node regardless. One-node homelabs get to skip a whole class of storage problem.

🧊 Sidequest 8: The StatefulSet That Wouldn't Budge

Applied that. Helm said no:

Error: UPGRADE FAILED: server-side apply failed for object airflow/airflow-triggerer
apps/v1, Kind=StatefulSet: StatefulSet.apps "airflow-triggerer" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template',
'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden
Enter fullscreen mode Exit fullscreen mode

Not a chart bug, not a values mistake -- a hard Kubernetes rule. volumeClaimTemplates on a StatefulSet is immutable after creation. The triggerer had been created with its own per-pod log volume. Turning on shared log persistence removes that block entirely. Kubernetes doesn't allow that field to change on a live object, full stop.

The only move is to delete the object and let Helm rebuild it:

kubectl delete statefulset airflow-triggerer -n airflow
helm upgrade airflow apache-airflow/airflow -n airflow --values airflow-values.yaml
Enter fullscreen mode Exit fullscreen mode

The pod blips for a few seconds, deferred-task processing pauses, nothing is lost. StatefulSets don't garbage-collect their PVCs, so the old per-pod log volume survives the delete as an orphan -- worth cleaning up once you've confirmed the new setup is healthy:

kubectl get pvc -n airflow          # look for logs-airflow-triggerer-0
kubectl delete pvc logs-airflow-triggerer-0 -n airflow
Enter fullscreen mode Exit fullscreen mode

🚧 And Then, A Cliff to jump off of

or at-least it looked like one

Logs working. DAGs syncing. UI reachable from every device in the house. I typed in the admin password, hit enter, and:

Bad Request The CSRF session token is missing.

Created tens of theories hundreds of stories but at a end one patient moment solved it. When i tried longing in after a few seconds it worked, my working hypothesis as of is that the pod to pod auth/authz was not initialised and once it got properly synced i was error free for a few mins.

🧠 What This Taught Me

  • A failing Helm hook is usually a symptom, not a cause. The migration job "failed" because Postgres couldn't schedule because there was no StorageClass. Go find whatever's Pending before you read a single hook log.
  • kubectl logs <job-name> doesn't work. Jobs aren't pods. Use -l component=<label>, and do it before the controller cleans up the evidence.
  • Installing the same CRDs twice is a real failure mode. Field managers collide, and "the docs told me to" doesn't help. Check whether your chart already ships them.
  • If a controller generates a resource, don't kubectl patch it. Your change lives until the next reconcile and then vanishes without a trace. Put it in the spec the controller reads.
  • Curl with a forced -H "Host:" is not the same test as a browser. I proved my stack worked for a request no real client would ever send.
  • Airflow 3 renamed the furniture. webserver → api-server, DAG parsing → dag-processor, [scheduler] dag_dir_list_interval → [dag_processor]. Most tutorials out there are Airflow 2 and will point you at things that no longer exist.
  • volumeClaimTemplates is immutable. Some Helm upgrades simply require deleting the StatefulSet. That's not a workaround, that's the procedure.
  • Single-node clusters make RWO vs RWX a non-problem. One of the very few advantages of having exactly one node.
  • Read the status.conditions block. BackendNotFound, ResolvedRefs: False, FailedScheduling -- the Gateway API and Kubernetes both told me exactly what was wrong, in plain English, every single time. I just had to look.

📋 Quick Reference (For Skimmers)

Purpose Command
Pull kubeconfig off the server ssh user@host "sudo microk8s config" > ~/.kube/config-microk8s
Enable storage before anything else sudo microk8s enable hostpath-storage
Find what's actually broken kubectl get pods -n airflow -o wide then describe the Pending one
Logs from a Job's pod kubectl logs -n airflow -l component=<label> --tail=100
Check a Gateway is live kubectl get gateway <name> -n <ns> -o yaml → Accepted + Programmed
Check a route resolved its backend kubectl get httproute <name> -n <ns> -o yaml → ResolvedRefs
Find the generated Envoy service kubectl get svc -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=<gw>
Confirm the real service names kubectl get svc -n airflow (it's api-server, not webserver)
Create the git-sync key secret kubectl create secret generic <name> -n airflow --from-file=gitSshKey=<keyfile>
Watch git-sync do its thing kubectl logs -n airflow -l component=dag-processor -c git-sync
Escape an immutable StatefulSet kubectl delete statefulset <name> -n airflow, then helm upgrade

🚀 What's Next

Genuinely, this time -- the Dell and the Pi come out of the drawers. Three nodes. Cilium. The thing I promised you last episode and will keep promising until it becomes true or I run out of drawers.

đŸ’Ŧ Final Thoughts

Eight sidequests. Two self-inflicted, three courtesy of Airflow 3's renaming spree, one Kubernetes immutability rule, one CRD ownership fight, and one CDN that was too slow for Helm's patience.

The actual Airflow install -- the part I'd budgeted the evening for -- took about four minutes and worked on the first try.

Stay tuned. Popcorn đŸŋ this time optional, but tea is now mandatory đŸĢ–.

Top comments (0)