From "kubectl apply" to a self-driving platform: a GitOps homelab, evolved

#devops #kubernetes #gitops #argocd

Long read. This is the story of how a laptop cluster went from "I threw some
manifests at it" to a platform that ships and heals itself - and, more
importantly, how the thinking changed at each step. If you're weighing
Argo CD, Istio, Vault or just "how do I structure a GitOps repo that won't
rot", this is the map I wish I'd had.

I run a four-node kind cluster on my desktop. Early on I made one rule: I
don't touch the cluster. I touch git. No kubectl apply by hand, no "let me
just patch this one thing on the side." If it isn't in a repo, it doesn't
exist. Everything below is what that rule forced me to learn.

Phase 1 - the foundation: the app must not know about the cluster

The first real design decision is the one everything else hangs off, so I'll
state it plainly:

The application knows nothing about the cluster. The cluster knows how to
run the application.

Concretely: my two demo services (front, back) live in their own repos.
Their CI builds an image (Kaniko, no Docker daemon), packages a Helm chart, and
pushes both to the GitLab OCI registry. Then it does the only thing that
touches the platform - it writes a single number, the new chart version, into a
JSON file in the GitOps repo:

{ "name": "back", "chart": "backend", "chartVersion": "0.1.10" }

An Argo CD ApplicationSet watches those files and turns each one into an
Application that pulls the chart straight from the registry. The app repo
never holds a kubeconfig, never runs kubectl, never knows the gateway exists.
It ships a chart version; the platform decides how that runs.

The platform itself is the app-of-apps pattern: one root Application,
applied once, that renders only control-plane objects - projects, the
ApplicationSet, and one small app.yaml per component. Each of those is its own
Application owning its own manifests. So the entire cluster is kubectl apply -f root.yaml, once, and after that Argo manages everything, including Argo.

Why it matters: this split is what lets a robot bump a version and a human
never log in. It's also what keeps the app charts portable - they describe a
workload and a Service, nothing environment-specific.

Phase 2 - maturing the GitOps logic: structure, ordering, and deleting things

A working app-of-apps is step one. Making it not rot is the real work, and it
came in three moves.

Delete the dead weight. I used to run self-hosted GitLab inside the
cluster - its own Postgres, Redis, MinIO. For a homelab that's just mass. I
ripped it out, moved CI to gitlab.com, and dropped the GitLab database and roles
from Postgres. I also killed a custom image-updater the moment CI started
writing the chart version itself. Half of GitOps maturity is realizing what you
can remove - every component you delete is one you never have to reconcile,
secure, or debug at 3am.

Group by domain. The flat platform/* pile grew to ~14 components and got
unreadable. I regrouped into platform/{mesh,security,data,identity,ci,core}/,
with cluster/ for Argo's own governance and apps/ for workloads. The quiet
lesson here: Argo tracks Applications by name, not path, so git mv +
fixing each source.path + the root glob is a zero-downtime refactor - nothing
gets recreated. (I froze the root app - selfHeal: false, prune: false - during
the move, so a bad glob couldn't prune half the cluster before I noticed. That
two-line insurance is not optional.)

Add guardrails and ordering. AppProjects became allow-lists: which repos a
component may pull from, which namespaces it may write to, which cluster-scoped
kinds it may touch. Boring until the day a typo tries to land in kube-system
and the project simply says no. And Argo sync-waves encode the ordering that
actually matters - the CloudNativePG operator comes up in one wave, the Postgres
Cluster in the next, because the CRDs and webhooks have to exist before you
can create a resource against them.

Phase 3 - the hardcore infra: mesh, state, and secrets that never touch git

With the logic clean, the platform got real capabilities.

Ingress ? Gateway API on Istio. I replaced ingress-nginx with the Gateway
API served by Istio: one Gateway on port 80, one HTTPRoute per host. The
interesting part was kind - no LoadBalancer, so the gateway pod has to bind
hostPort: 80 on the control-plane node, exactly how nginx did. Getting there
meant teaching the istio-generated Deployment a nodeSelector, a control-plane
toleration, the host port, and a Recreate strategy (hostPort + RollingUpdate
deadlocks - the new pod can't bind a port the old one still holds). Slightly
outside pure GitOps because Istio owns that Deployment, but it works and Istio
leaves the extra fields alone.

State: CloudNativePG. Postgres runs as a CNPG Cluster, hosting the app
database. It's the backbone that makes stateful demos possible and survives pod
restarts - the boring, reliable core the rest leans on.

Secrets, evolved. This is the part I'm most happy with, because it shows a
progression rather than a single tool:

v1: SealedSecrets - encrypted secrets committed to git, decrypted only in-cluster. Great for bootstrap material.
v2: Vault + External Secrets Operator. The database password lives in Vault (source of truth). ESO authenticates to Vault with the Kubernetes auth method - it presents a ServiceAccount token, Vault validates it via TokenReview and maps it to a policy that may read exactly one path, and ESO materializes a Kubernetes Secret the app consumes. The password never touches git and is never typed into a manifest. The wiring is declarative (a SecretStore + an ExternalSecret); the value is fetched at runtime.

To make all of this tangible I rebuilt the demo app into a small guestbook:
nginx frontend ? /api ? a Flask backend that runs its own SQL migrations on
boot and reads/writes CloudNativePG, with its DB credentials delivered through
that Vault ? ESO chain. Every layer of the platform is visible in one page -
DATABASE: connected, a visit counter, and messages persisted in Postgres:

Screenshot - the guestbook: DATABASE: connected, a live visit counter, and a message wall persisted in CloudNativePG. (add 01-app-guestbook.png)

That "DATABASE: connected" pill is the whole platform working end to end - CI
built the image with Kaniko, Argo deployed it, ESO pulled the password out of
Vault, and CloudNativePG is holding the rows.

Screenshot - Argo CD: all 19 applications Synced + Healthy. (add 02-argo-apps.png)

War stories - because troubleshooting is half the job

Pretty YAML is easy. The signal that you can run this stuff is what happens when
it breaks.

The day Argo tried to delete itself. Argo manages Argo here, so when I
bumped the argo-cd chart, it tried to reconcile its own workloads. The new
chart changed the pod selectorLabels - and Deployment.spec.selector is
immutable. Kubernetes rejected the update on every Argo component, the
Services quietly adopted the new selectors while the Pods kept the old labels,
and a Service whose selector matches nothing has zero endpoints.
argocd-server: no endpoints, 503 - and the tool that's supposed to fix Argo
couldn't reach its own repo-server to render itself. I hand-recreated the
control plane (delete + recreate so the immutable selectors could change) while
the UI threw 503s at me. Lesson, permanently: bumping the Argo chart across a
selector change means recreating its workloads by hand.

The quieter ones. Istiod injects a webhook caBundle at runtime, so Argo saw
perpetual drift until I added ignoreDifferences. ESO's CRDs are big enough to
blow past the 256 KiB last-applied-configuration annotation limit, so its
controller crash-looped until I switched that Application to
ServerSideApply=true. Each one obvious in hindsight, each one only learnable
by getting bitten.

Reality check - the trade-offs I chose on purpose

Senior isn't "it's all perfect," it's "I know exactly where the sharp edges
are":

Vault auto-unseals itself via a CronJob and a single hand-created unseal key (shares=1). That's a lab decision - it trades real security for the cluster coming back on its own after a restart. In production the unseal key never sits next to the thing it unseals.
CloudNativePG runs a single instance. State survives pod restarts, but there's no HA - one instance, one backup story I haven't built yet. Fine for a lab, not for anything with an SLA.
The gateway hostPort patch lives slightly outside GitOps because Istio owns that Deployment. It's declarative enough and stable, but it's the one seam where I traded purity for "works in kind."
DB creds are static-in-Vault, not dynamic yet. The database engine is wired; dynamic per-connection users are the next step, and they need a shared owner role so rotated users still see the schema. Deliberately deferred, not forgotten.

Where it landed

Nineteen applications, all green. CI ships images, Argo ships the platform, the
platform ships the apps, Vault hands out the secrets, and I run none of it - I
run git. The whole thing rebuilds from kubectl apply -f root.yaml and a
handful of git pushes.

It broke, I fixed it, and now it's boring to operate. Boring is the trophy.

Want the guts - the app-of-apps root, the Vault?ESO wiring, the gateway
hostPort hack? Say the word and I'll drop a follow-up on any one of them.