How I Made My Homelab Declarative

Motivation

Prior to this, all of the applications that were installed in devata (my homelab cluster) were installed through helm install. I wanted to explore GitOps and wanted to adopt a declarative approach rather than an imperative approach.

The imperative setup worked, but it had a shape I did not like. The only record of what was running and how it was configured lived inside the cluster itself, in Helm release secrets, and in my shell history. If I wanted to upgrade a chart, I had to remember which values I had passed months ago. If the cluster died, the recovery plan was "reinstall everything from memory". And every change was a one-off command that left no trace anywhere I could review, diff, or roll back.

The instinct came from somewhere. During my LFX term I worked on meshery/schemas, the source of truth for the whole Meshery project: every construct is defined once there, and every component downstream takes reference from it. I wanted my cluster to work the same way.

That arrangement, applied to a cluster, is GitOps. The desired state of the cluster lives in a git repository, and a reconciler running inside the cluster continuously pulls that state and applies it. Git becomes the source of truth: a change is a commit, a review is a pull request, a rollback is a revert. If something drifts from what git says, the reconciler puts it back. If a change is not in a file, it does not really exist.

Everything below lives in my public lab repository, so you can read the real manifests as we go.

Table of Contents

• Argo CD
• The repository layout
• The app-of-apps
• Migrating the imperative stack
• The adoption drill
• What each component taught me
• The proof

Argo CD

I chose Argo CD for this. It is the reconciler: it watches a git repository, compares what is declared there against what is actually running in the cluster, and syncs the difference.

For the install itself I first went with the raw upstream install.yaml, committed straight into the repo, instead of the Helm chart or Autopilot. The reasoning was transparency: you can read every resource the installer creates, and committing it means even the installer is versioned. That choice did not survive contact with automation. Once Renovate started watching the repo, a 33,000 line vendored manifest was invisible to it; a pinned reference is not. So the install is now a small kustomization pointing at the exact upstream base that install.yaml is generated from, pinned by tag:

# kubernetes/bootstrap/argocd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - github.com/argoproj/argo-cd/manifests/cluster-install?ref=v3.4.4

The property I actually cared about survives: the installer is still versioned, because the pin is the version. What changed is the shape of an upgrade. It used to be a regenerated 33,000 line file that no one truly reviews; now it is a one line diff that Renovate opens as a PR and CI renders before merge. Bootstrapping a fresh cluster is still two commands:

kubectl create namespace argocd
kubectl apply -k kubernetes/bootstrap/argocd

The neat part is what happens next: Argo CD manages itself. One of the Applications it reconciles points right back at the directory that defines its own installation. Upgrading Argo CD is now editing a file and pushing.

# kubernetes/clusters/devata/argocd.yaml
spec:
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/bootstrap/argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: false  # never let argocd prune its own install
    syncOptions:
      - ServerSideApply=true

Two details in that snippet carry weight. prune: false means Argo CD will never delete pieces of its own installation, even if they disappear from git; a bad commit should not be able to take down the engine that would fix it. And ServerSideApply=true is there because Argo CD's CRDs are so large they blow past the size limit of the client-side last-applied-configuration annotation.

The repository layout

The repo is split into three planes, by who applies them:

lab/
├── talos/              # machine configs, applied by talosctl
├── kubernetes/         # the ONLY path GitOps watches
│   ├── bootstrap/      # argocd install + the root app
│   ├── clusters/
│   │   └── devata/     # one child Application per component
│   ├── infra/          # values and manifests per component
│   │   ├── networking/     # cilium, metallb
│   │   ├── observability/  # kube-prometheus-stack, loki, promtail
│   │   └── controllers/    # nvidia-device-plugin, sealed-secrets
│   └── apps/           # workloads
└── lab-experiments/    # sandbox, never reconciled

kubernetes/ is the single GitOps-watched path. talos/ is applied by talosctl from my workstation, because the machine layer has to exist before there is a cluster to reconcile anything. And lab-experiments/ is the escape hatch: a place to try things without the reconciler ever touching them.

The app-of-apps

Argo CD's unit of work is an Application: one CRD instance that says "this git path renders to these resources, keep them in this namespace". Instead of registering every component by hand in the UI, I use the app-of-apps pattern: a single root Application that watches a directory of other Application manifests.

# kubernetes/bootstrap/root.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: devata-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/PragalvaXFREZ/lab.git
    targetRevision: main
    path: kubernetes/clusters/devata
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

devata-root watches kubernetes/clusters/devata/. Every YAML file in that directory is itself an Application pointing at a component. Adding a component to the cluster is now: commit a child Application file, push, done. The root notices the new file, creates the Application, and that Application syncs the component. Deleting the file prunes it. The whole cluster inventory is one ls:

argocd.yaml  cilium.yaml  hello.yaml  kps.yaml  loki.yaml
metallb.yaml  nvidia-device-plugin.yaml  promtail.yaml

This root app is the only thing I ever applied manually with kubectl apply -f root.yaml. Everything after that arrived through git.

Migrating the imperative stack

Bootstrapping the engine on a fresh cluster is the easy half. My problem was the brownfield half: devata already had six components running as hand-installed Helm releases, some of them load-bearing.

$ helm list -A
NAME                    NAMESPACE
cilium                  kube-system
kps                     monitoring
loki                    logging
metallb                 metallb-system
nvidia-device-plugin    nvidia-device-plugin
promtail                logging

The naive move would be a big-bang takeover: point Argo CD at everything at once and let it sync. On a cluster where one of those releases is the CNI (the thing pods need to have network at all) and another is the entire monitoring stack, that is how you turn a learning exercise into an outage. So the migration went one component at a time, least blast radius first, and each one followed the same drill.

The key realization that makes adoption safe: Argo CD does not care how resources got into the cluster. If the manifests it renders from git match what is already running, the diff is empty and syncing changes nothing. So the whole game is to reproduce, in git, exactly what Helm had installed, and to prove the diff is empty before letting the reconciler touch anything.

The adoption drill

Per component, the workflow looked like this:

1. Set up the GitHub repository, the app-of-apps root Application, and configure your repository to serve as the source of truth (once, see above).
2. helm list -A to find the next release to migrate.
3. helm get values <release> -n <namespace> to recover the exact values the release was installed with. This is the step that saves you: those values are the configuration you would otherwise have to remember.
4. Commit two things: the recovered values at kubernetes/infra/<domain>/<component>/values.yaml, and a child Application in kubernetes/clusters/devata/. The Application uses Argo CD's multi-source pattern: one source is the upstream chart repository with a pinned version, the other is my git repo providing the values file.
5. Automation stays off at first. The Application appears in Argo CD, but nothing syncs.
6. argocd app diff <name> until the diff is empty. Every line in that diff is a discrepancy between what git declares and what is running, and each one gets fixed in git, not in the cluster.
7. Flip automation on (selfHeal: true, prune: true), then drift-test it: change something trivial with kubectl and watch the reconciler put it back.
8. Retire the Helm release record. Helm stores its bookkeeping as a Secret named sh.helm.release.v1.<release>.v<N> in the release namespace. Deleting it removes the component from helm list for good. Helm no longer owns this component, git does.

Here is the first multi-source Application I wrote:

# kubernetes/clusters/devata/nvidia-device-plugin.yaml
spec:
  sources:
    - repoURL: https://nvidia.github.io/k8s-device-plugin
      chart: nvidia-device-plugin
      targetRevision: 0.19.3  # pinned; upgrading is editing this line
      helm:
        valueFiles:
          - $values/kubernetes/infra/controllers/nvidia-device-plugin/values.yaml
    - repoURL: https://github.com/PragalvaXFREZ/lab.git
      targetRevision: main
      ref: values

The second source contributes no manifests; ref: values just makes my repo addressable as $values, so the chart from NVIDIA's repo renders with the values file from mine.

Lesson

Name the Application after the release, not the chart. Charts embed the release name into resource names: my kube-prometheus-stack release is kps, so the resources are kps-grafana, kps-operator, and so on. If I had named the Application kube-prometheus-stack, Argo CD would have rendered a second, parallel stack with new names, and the diff could never reach empty.

What each component taught me

The drill was the same six times, but every component had its own personality.

nvidia-device-plugin went first as the safe rehearsal: a single DaemonSet, four lines of values, and (since the GPU in that node turned out to have a hardware fault) a component that advertises nothing anyway. Perfect crash test dummy. Its diff was empty on the very first render.

kube-prometheus-stack was the first component that was not plain, in two ways. Its CRDs are enormous, so like Argo CD itself it needs ServerSideApply=true. And the recovered values had a surprise in them: helm get values kps returned the Grafana admin password in plaintext. It had been sitting in the Helm release secret the whole time. That password never entered git. It moved into a manually created grafana-admin Secret in the cluster, values.yaml points the chart at it through grafana.admin.existingSecret, and retiring the Helm release records at the end of the drill had a satisfying side effect: it destroyed the plaintext copies. Sealing that Secret into git properly, encrypted, is the sealed-secrets workstream that comes next.

loki and promtail were the routine reps: recover values, commit, diff to empty, flip automation. By this point the drill felt mechanical, which is exactly what you want from your logging stack migration.

metallb is tracked as the upstream native manifests plus my address-pool config rather than a chart, and it introduced the next class of problem: fields that are supposed to differ from git. The controller injects caBundle certificates into its webhook configurations at runtime. Git says the field is empty, the cluster says it is a certificate, and that diff will never close because it is not drift, it is the component working as designed. The answer is ignoreDifferences:

ignoreDifferences:
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    jqPathExpressions:
      - .spec.conversion.webhook.clientConfig.caBundle
syncPolicy:
  syncOptions:
    - RespectIgnoreDifferences=true

RespectIgnoreDifferences=true matters: without it, ignored fields still get overwritten on sync. With it, Argo CD neither reports them as drift nor touches them.

cilium went last, deliberately. It is the CNI; if adopting it goes wrong, every pod on the cluster loses networking, including Argo CD itself. And it had the hardest version of the metallb problem: the chart's default hubble.tls.auto.method=helm generates fresh Hubble TLS certificates on every render. So cilium-ca and both Hubble cert Secrets differed on every single diff, and with self-heal on, the reconciler would have rotated Hubble's TLS forever, once per sync. Two clean options existed: switch cert generation to the chart's cronJob method (Cilium's documented GitOps pattern), or ignoreDifferences on the three Secrets. I went with ignoreDifferences plus RespectIgnoreDifferences, the same shape as metallb: the certs are runtime state that Helm happened to generate, not configuration I want git to own. Diff to empty, automation on, sh.helm.release.v1.cilium.v3 deleted.

The proof

With few pull requests and tinkering done here and there, I was able to get it done at the end:

Figure 1. helm list -A on devata returns nothing. All of the applications have been migrated to be declarative with the help of Argo CD.

An empty helm list -A does not mean Helm is gone; the charts are still rendered by Helm, inside Argo CD. What is gone is Helm as the owner of state on my cluster. The owner is git now:

$ argocd app list
NAME                  SYNC    HEALTH
argocd                Synced  Healthy
cilium                Synced  Healthy
devata-root           Synced  Healthy
hello                 Synced  Healthy
kps                   Synced  Healthy
loki                  Synced  Healthy
metallb               Synced  Healthy
nvidia-device-plugin  Synced  Healthy
promtail              Synced  Healthy

Every one of those is a file in the repo. Upgrading a chart is editing one pinned version line. Changing a dashboard is a pull request. And if devata dies tomorrow, the recovery plan is no longer "reinstall from memory", it is: build the machines with talosctl, apply two manifests, and watch Argo CD rebuild everything else from git. Pretty cool, huh ?

---

Originally published at pragalva.me.