DEV Community

Cover image for Rebuilding an On-Prem Kubernetes Platform With GitOps After a Storage Wipe
Syed Muhammad Jamal Uddin Gilani
Syed Muhammad Jamal Uddin Gilani

Posted on

Rebuilding an On-Prem Kubernetes Platform With GitOps After a Storage Wipe

#kubernetes #git #docker #architecture

description: How we rebuilt an RKE2 on-prem cluster from Argo CD only — OpenBao, External Secrets, TrueNAS CSI, CloudNativePG, Harbor, MinIO, and GitHub Actions runners.

Rebuilding an On-Prem Kubernetes Platform With GitOps After a Storage Wipe

We wiped the old HDD-era cluster and rebuilt on SSD-backed storage. The rule was simple: Argo CD is the only thing applied by hand. Everything else — operators, storage, secrets plumbing, Postgres, the registry, ingress, CI runners — had to come back from git.

This is the story of that rebuild: what we put in GitOps, what we deliberately left out of git, and the failures that taught us more than the happy path.

I work on an on-prem platform team. The cluster is RKE2 (Canal, a typical RKE2 pod CIDR) on a private LAN. A TrueNAS box provides iSCSI, NFS, and S3. Platform hostnames (argocd, vault, harbor, s3) terminate on a MetalLB VIP. Source of truth is a private GitOps repo, watched by Argo CD on main.

The constraint that kept us honest

Empty-cluster bootstrap is one command:

kubectl apply -k infra/<cluster>/argocd/bootstrap
Enter fullscreen mode Exit fullscreen mode

That applies official Argo CD v3.5.2 install.yaml, an AppProject, and a root Application. The root app points at argocd/applications/infra, which owns:

  • the infra app-of-apps (infra/<cluster>)
  • the Gateway
  • CloudNativePG Postgres
  • a dump client

Enabling a component is adding a directory to infra/<cluster>/kustomization.yaml and pushing. We did not kubectl apply Harbor, CSI, or runners “just this once.” When something was wrong, we fixed git or OpenBao, not the live object — except one-time secret material that cannot live in git.

kubectl apply (once)
  └── Argo CD + AppProject + root Application
        └── applications/infra
              ├── on-prem-infra          → operators, storage, Harbor, ARC
              ├── infra-gateway          → Gateway API + TLS
              ├── general-postgres       → CNPG cluster
              └── general-postgres-dump  → dump job/client
Enter fullscreen mode Exit fullscreen mode

How the repo is shaped

One folder per component. Typical layout:

infra/<cluster>/<component>/
  application.yaml      # Argo Application (Helm multi-source or Kustomize)
  values.yaml           # chart values, referenced as $values/...
  external-secret.yaml  # ESO → OpenBao → Kubernetes Secret
  kustomization.yaml
Enter fullscreen mode Exit fullscreen mode

Platform workloads (Postgres, dump) are sibling Applications, not nested inside the infra tree. That keeps “cluster plumbing” separate from “data plane.”

The AppProject lists every destination namespace and every Helm repoURL. Miss one and Argo returns ComparisonError or permission denied. We hit that more than once when adding Harbor (public Harbor Helm repo) and extra runner namespaces.

Network first: VIP, DNS, and why pods cannot “just reach” the NAS

MetalLB L2 advertises a small reserved range with autoAssign: false. The Gateway VIP is a single pinned address in that range. DNS A records for platform names must point there — not a jump host, not a tunnel endpoint. We learned that the hard way when Harbor’s registry returned HTTP 500 and S3 XML parse errors: the S3 hostname was resolving to the wrong box and serving HTML 404 instead of MinIO.

TLS is Gateway API (NGINX Gateway Fabric) plus cert-manager Cloudflare DNS-01 for a wildcard on our public zone.

Pods live on the cluster pod CIDR and do not SNAT onto the LAN. The NAS must have a route back to that CIDR via the node IPs. If a CSI controller times out talking to the NAS API, it is almost always routing, not “the chart is wrong.” Node plugins for iSCSI/NFS already use hostNetwork where the chart sets it. We did not flip the controller onto hostNetwork as a shortcut.

MinIO itself is not in-cluster. We published a Service + EndpointSlice to the NAS S3 port and an HTTPRoute for the S3 hostname. A subtle Gateway API rule: the HTTPRoute parent namespace must match the Gateway namespace. A wrong parent left the route Progressing forever.

Secrets: OpenBao is the store, ESO is the only door

OpenBao (Raft, Shamir 5 shares / threshold 3) is the secret engine. External Secrets Operator is the only supported way to materialize Kubernetes Secrets.
ClusterSecretStore a single cluster-scoped store
KV secret/ (v2)
Auth Kubernetes mount, dedicated ESO role
Audience the in-cluster API server audience
ESO SA the operator ServiceAccount in its namespace

Path convention:

secret/<env>/<name>
Enter fullscreen mode Exit fullscreen mode

Examples: CSI driver configs, Harbor admin/DB/S3, GitHub App credentials for ARC, DNS-01 token, Harbor pull secret for the dump image.

Never committed: NAS API keys, SSH private keys, Harbor passwords, GitHub App PEMs, unseal shares, tokens.

Init, unseal, and a new Raft

After a wipe, the Raft PVC is new. Old unseal keys are dead. We ran bao operator init, stored the JSON offline, unsealed with three keys, enabled KV and Kubernetes auth, and bound an ESO role to the operator ServiceAccount.

ESO then failed with 403 on Kubernetes login until audience + role matched. Connection refused meant OpenBao was sealed or the pod was down — not an ESO bug.

Audit that the API will not let you enable

We wanted file audit. bao audit enable returned 400 (declarative config owns audit devices). The working pattern is an HCL stanza in Helm values:

audit "file" "file-audit" {
  options = {
    file_path = "/openbao/audit/audit.log"
  }
}
Enter fullscreen mode Exit fullscreen mode

OpenBao’s Helm chart uses OnDelete. Changing the ConfigMap does not roll the pod. Delete the OpenBao pod, wait for recreate, unseal again with the same keys. Empty bao audit list usually meant options {} never made it into the running config.

We created a human CLI user with an admin policy for day-2 work. Tokens stay out of git.

Storage: Longhorn, then TrueNAS CSI

StorageClass Backend Use
SSD Longhorn RWO Longhorn general RWO, OpenBao audit volume
SSD iSCSI democratic-csi / TrueNAS iSCSI OpenBao Raft, CNPG
HDD iSCSI democratic-csi / TrueNAS iSCSI Harbor PVCs, dump PVC
HDD NFS democratic-csi / TrueNAS NFS shared / bulk HDD

CSI driver names must stay unique (one name per backend/protocol). Each chart uses existingConfigSecret. ESO fills driver-config-file.yaml from OpenBao. YAML spaces only — tabs in an SSH privateKey CrashLoop the driver.

StorageClass parameters are immutable

We changed fsType from ext4 to xfs on the database class. Kubernetes rejected the patch. Delete the StorageClass, let Argo recreate it. Existing PVs keep the old parameters.

NAS 401

HDD iSCSI CrashLooped with 401 Invalid API key. The key had been rotated or mistyped in OpenBao; it also showed up in driver logs (rotate immediately if that happens). After the Secret hash changed, we deleted the CSI pods so they remounted config.

The NFS bug that looks like a chart bug

HDD NFS came up as:

initializing csi driver: undefined
Error: invalid csi driver: undefined
Enter fullscreen mode Exit fullscreen mode

The ExternalSecret was Ready. The Secret existed. Grepping the decoded Secret showed:

apiVersion: v1
kind: Secret
stringData:
Enter fullscreen mode Exit fullscreen mode

We had stored a full Kubernetes Secret manifest in OpenBao. ESO copies a property. Democratic-csi reads a driver config file. It expects a top-level driver: freenas-nfs (or freenas-iscsi). It does not understand apiVersion: v1.

The fix is not in git. Unwrap the file:

# This is what the OpenBao property must contain.
# Not a Secret wrapper. No real hosts or keys.
driver: freenas-nfs
instance_id: ""
httpConnection:
  # host, apiKey — values live only in the secret store
sshConnection:
  # host, username, privateKey — same
nfs:
  shareHost: <nas-address>
  # datasetParentName, …
Enter fullscreen mode Exit fullscreen mode

Then:

bao kv put secret/<env>/<csi-nfs-config> \
  driver-config-file.yaml=@/tmp/nfs-driver.yaml

kubectl -n <csi-namespace> annotate externalsecret <nfs-config> \
  force-sync="$(date +%s)" --overwrite
Enter fullscreen mode Exit fullscreen mode

Confirm the live Secret’s first key is driver:, then delete the NFS CSI pods. If logs still show a timestamp from before the put, you are looking at a dead container.

Also watch NFS shareAllowedNetworks. An empty list can lock out both LAN and pod CIDR. You typically want the node/LAN range and, if mounts come from pods, the pod CIDR. The NAS still needs zfs allow for the CSI dataset parent.

Postgres with CloudNativePG

Order mattered: CSI SSD healthy → CNPG operator → cluster (Postgres 16.1, data/WAL on SSD iSCSI) → PgBouncer pooler → Harbor role/database.

Apps connect through the in-cluster pooler Service.

Harbor needs:

  • a managed role (password from an ESO Secret)
  • a registry database (CNPG Database CR)

When the Database CR never appeared, the cause was a kustomization referencing a file that did not exist. Argo failed the whole app with a manifest generation error. Commenting or adding the missing resource unblocked sync.

Harbor core then failed with Postgres 08P01 / password failures until ALTER USER matched the Secret, and then CREATE DATABASE registry because the CR had not created it yet.

CNPG volumeClaimTemplates are immutable. Size or StorageClass changes need an operator replace strategy and usually PVC recreate. Plan storage before the first PVC.

A dump client uses a private registry image and a PVC on HDD iSCSI. ImagePullBackOff was either “tag not in the registry yet” or a missing dockerconfigjson pull Secret from OpenBao. The ExternalSecret maps username/password into a Docker config; the path name in OpenBao must match what ESO asks for.

Harbor and S3

Harbor chart (1.19.x) with in-cluster Redis/Trivy, external Postgres, S3 on the NAS via the Gateway hostname (dedicated bucket).

HTTPRoute parent: the Gateway’s namespace.

If docker push returns 500 and registry logs talk about S3 XML, curl the S3 hostname from a node. You want S3 XML, not an HTML 404 from a reverse proxy in another network.

GitHub Actions runners (ARC)

Controller gha-runner-scale-set-controller 0.14.1 in its own namespace.

Two gotchas:

  1. CRDs. The controller chart was installed with skipCrds: true. Without a separate CRD Application, scale sets fail on missing EphemeralRunnerSet.
  2. One controller ServiceAccount. Every scale set must set controllerServiceAccount.name to the actual controller SA. A generated SA name looks fine until the listener cannot read the GitHub App Secret (can-i is no; pods Pending “secret not found” even when the Secret exists).

We run several org scale sets (minRunners: 0): work only appears when a job is queued. A listener should still exist once credentials and RBAC are correct. runnerScaleSetName overrides keep runs-on labels stable.

How we actually sequenced the empty cluster

  1. Apply Argo bootstrap from a jump host whose kubeconfig is this cluster.
  2. Register the git repo (URL includes .git).
  3. Let the root app sync. Wait for CRDs/webhooks (MetalLB-style configs need skip-dry-run patterns).
  4. Confirm the Gateway VIP and DNS.
  5. Init / unseal OpenBao. Enable KV + Kubernetes auth + the ESO role.
  6. Put CSI and app secrets in OpenBao before expecting ExternalSecrets to go Ready.
  7. NAS: API reachability from the pod CIDR, zfs allow, iSCSI initiator IQNs.
  8. Enable components one line at a time in kustomization.yaml: MetalLB → Longhorn → cert-manager → Gateway → OpenBao → ESO → SSD CSI → CNPG → Postgres → HDD CSI → dump → Harbor → MinIO route → ARC.

One change, sync, verify pods/PVCs, then the next. That sounds slow. It is faster than debugging five CrashLoops at once.

Day-2 rules we wrote down after we paid for them

  • Do not “fix values to make it work” without an explicit change. Prefer names and enable/disable lines. Drift from the pasted chart values is how you lose the next rebuild.
  • Helm OnDelete (OpenBao): delete the pod and unseal. ConfigMaps lie until then.
  • AppProject is part of the API surface. New namespace or chart repo = patch the project first.
  • Missing file in resources: fails the entire Application.
  • CSI CrashLoop order: (1) Secret YAML parse / wrapper vs inner driver:, (2) NAS 401, (3) timeout from the pod CIDR to the NAS.
  • Rotate anything that hit a log line or a chat paste: NAS API keys, SSH keys.

Adding the next component (the template we reuse)

  1. Create infra/<cluster>/<name>/.
  2. Add namespace + Helm repo to the AppProject.
  3. Add - <name> to the infra kustomization.yaml.
  4. If it needs a secret: ExternalSecret + OpenBao path — never the value in git.
  5. Push main. Synced/Healthy before starting the next one.

New ARC org: copy a runner folder, change names and githubConfigUrl only, same controller ServiceAccount.

What this rebuild proved

GitOps is not “Argo exists.” GitOps is one manual apply, secrets that are not in the repo, and the discipline to treat OpenBao payloads as part of the interface (inner YAML, not a Secret wrapper).

The cluster is boring again: Gateway VIP, Longhorn, three CSI drivers, OpenBao + ESO, CNPG, Harbor on external PG + S3, MinIO HTTPRoute, ARC listeners. The interesting parts were routing, immutable StorageClasses, Raft unseal after wipe, and a CSI factory that printed undefined because we stored the wrong document type.

If you rebuild after a storage wipe, write the bootstrap order down before you are tired. The next wipe will not wait for you to remember why S3 returned HTML.

Top comments (1)

Collapse
 
abhinavs1920 profile image
Abhinav Sharma •

The bootstrap order is the useful part here. Keeping the manual secrets steps beside the Git-managed steps makes the recovery checklist much more honest than just saying "everything is in Git".