DEV Community: George Ezejiofor

Canary Releases for an ML Model on KServe: Replayed Traffic, Sidecar Metrics and Automatic Rollback

George Ezejiofor — Thu, 29 Oct 2026 22:19:45 +0000

Part of the homelab build series. A football match-outcome model ships every day through Kubeflow Pipelines and KServe on RKE2. Each new model is staged next to the live one, tested, given 10% and then 50% of replayed traffic, measured revision by revision, and promoted or rolled back by the pipeline itself. The service stays highly available throughout: the serving revision keeps its traffic until the new one has earned it.

Canary Releases for an ML Model on KServe: the new revision's share of replayed requests at each stage, from staged at 0% through 58 of 600 at the 10% step and 305 of 600 at the 50% step to promoted at 100%, with any failed check rolling back to 0%

Introduction

A canary with no traffic is a coin you never flip.

The textbook canary goes like this. Send 10% of live requests to the new model, watch error rates and latency against the old one, step up, promote. Every guide says it, and every guide assumes one thing: that there are live requests.

My model has almost none. It predicts Premier League results once a day. The pipeline writes the predictions to the bucket, and the website reads them from there. Nobody calls the model server in normal use. A 10% split of nothing is nothing, so a canary built from the guides would pass every time without measuring anything.

So this canary brings its own traffic, measures each revision from its own sidecar, and decides on its own: promote, or put everything back the way it was. This article is how that works, what KServe actually does under the words "canary rollout", and the traps that look like model failures but aren't.

Where it sits

The model is an XGBoost classifier, served by KServe's built-in XGBoost server from a model directory on Ceph. A Kubeflow pipeline runs every morning:

Step	What it does
ingest and publish	new results into an Iceberg table through a Nessie branch
tune (most days skipped)	Katib's best parameters from the last sweep
finalize	retrain, compare the new recipe with the serving one on a walk-forward backtest, register the model
deploy	stage the new model next to the serving one, at 0% of traffic
smoke test	one known request with a known answer, then a burst of 200
canary	10%, then 50% of replayed traffic, judged step by step
promote	100% to the new model, the old one scales down

The last four are this article. They run only if finalize decided the new model is worth serving at all; a model that can't beat the old one on the backtest never gets staged.

The rollout: the new model is staged at 0%, smoke tested, given 10% then 50% of replayed traffic, and promoted to 100%; any failure rolls it back and the old model keeps all the traffic

What KServe means by "canary"

Most of the confusion I ran into, mine included, comes from two different KServe APIs with the same word in them.

The old v1alpha2 API had a separate canary block next to the default predictor. The current v1beta1 API has no such block. A canary in v1beta1 is one field on the predictor, canaryTrafficPercent, plus the fact that KServe keeps the last fully rolled-out revision around:

spec:
  predictor:
    canaryTrafficPercent: 10
    model:
      modelFormat:
        name: xgboost
      storageUri: s3://mlops-data-lake/football/models/<new-version>

Change the model and set the field, and KServe creates a new revision for the new model, gives it 10%, and leaves the previous rolled-out revision serving the other 90%. Then the field means three different things depending on its value:

`canaryTrafficPercent`	Meaning
`0`	the new revision exists and gets nothing: a holding position, and the rollback
`10`, `50`, …	a traffic split between the new and the rolled-out revision
removed	promote: the newest revision becomes the rolled-out one and takes 100%

The last row is the one to remember. Rolling back means setting the field to 0. Deleting it, which sounds like "cancel the canary", does the opposite and ships the new model to everyone. An AI assistant I tried got exactly this backwards, along with the v1alpha2 block, which would have shipped the bad model during an incident.

The payload: replayed matches

Without callers, the canary needs a payload, and the right payload is real inputs with known answers. Finalize already has both. After training it takes the 500 most recent matches from the training set and writes three things per match to the bucket: the features the model sees, the real result, and the new model's own answer.

Real inputs, artificial caller. The canary sends those rows to the model, 600 requests per step, 8 at a time.

Two things this payload can and can't tell you:

It can catch a broken model: one that loads but answers garbage, answers slowly, or errors on real feature rows.
It can't tell you the new model forecasts better. Both models were trained on those matches, so comparing them there measures fit, not forecasting. That question is answered earlier, by finalize's walk-forward backtest, before anything is staged.

Staging: the new model at 0%

The deploy step patches the InferenceService with the new model and canaryTrafficPercent: 0, then waits until the new model is actually running. "Actually" is the hard part.

Straight after the patch, the InferenceService still reports the previous revision as Ready, so a plain readiness check returns immediately and the pipeline goes green while the new model may never load. KServe's observedGeneration is no help either; on this version it reported a number higher than the object's own generation. The check that works is to find the revision by the model it carries: KServe annotates every revision with its storage location, so staging is done when the latest ready revision carries the new model's location.

# the annotation the deploy step matches on, per Knative revision
metadata:
  annotations:
    internal.serving.kserve.io/storage-initializer-sourceuri: s3://mlops-data-lake/football/models/<new-version>

The smoke test

Before any traffic moves, a pod in the same namespace calls the new revision directly, on its private service, with one known request. The answer must match finalize's answer for that request within 1e-4. Then it sends a burst of 200 requests from 8 threads: all must succeed, all with the same answer, p95 under a second.

If any of that fails, the pipeline stops there. The field stays at 0 and the old model keeps 100%. I tested this end to end with a one-off pipeline version whose tolerance could never be met: the smoke test failed, promotion never ran, and the serving revision kept all the traffic.

The canary: two steps, measured per revision

The canary step runs as one job and does four things before it moves any traffic:

asks both revisions directly for every replay row, and checks the new one gives exactly the trained model's answers;
compares the two models' fit on the replay set (log loss and accuracy against the real results);
finds each revision's pods;
checks the shared route answers at all.

Then, for 10% and then 50%:

step:
  set: canaryTrafficPercent = <step>
  wait: until Knative reports the split, plus a few seconds for the sidecars
  replay: 600 requests through the shared route, 8 at a time
  measure: each revision's requests, errors and latency, from its own sidecar
  judge: fail the step if any rule below is broken

How the canary measures each revision: the canary Job replays past matches through Knative's weighted route to both revisions, reads each revision's Istio sidecar counters before and after every step, then rolls back to 0% or lets the promote step remove the field

Measuring per revision without help from the response

The response doesn't say which revision answered it. On a day with no new results, both revisions are the same model and give identical answers, so the answers can't tell them apart either.

The sidecars can. Every model pod runs an Istio sidecar that counts the requests it receives, by caller. The canary reads each pod's counters just before and just after the replay and keeps only its own job's requests. The difference is exact: how many requests each revision got, how many failed, and their latency.

The rules

Before any traffic moves, the run stops if the new revision's answers differ from the trained model's, or if its fit on the replay set is more than 0.02 log loss worse than the serving model's. After that, a step fails if any of these hold:

Rule	Why
any answer matches neither revision's own answer	a third, unknown model is answering
requests failed and more than 1% never reached either revision	the route is broken, not a model
the new revision got under a quarter of its expected share	the split isn't working
its error rate is more than 1 point above the serving model's	the new model errors
its mean latency is over 3× the serving model's and over 25 ms slower	the new model is slow, by a margin that matters

The two-part latency rule matters at this scale. Both models answer in a few milliseconds, so "3× slower" alone would fail a release over a 4 ms difference.

Rollback and promotion

Once traffic has moved, every way out that isn't a pass rolls back: a broken rule, an exception, even the job being killed. Rollback sets the field to 0 and waits for Knative to confirm the serving revision has everything again.

I tested the rollback the same way as the smoke test, with a one-off pipeline version whose error rule could never be met. The 10% step ran first: the new revision took 70 of the 600 requests, all answered correctly. Then the step failed, the job set the field back to 0, Knative confirmed the serving revision held 100% again, and the run failed before the promote step could start. The failure was forced, not caused by a genuinely bad model, but every move after the decision was the real one.

On a pass, the promote step removes the field. It has to send an explicit null for that:

# JSON merge patch body, in YAML form
spec:
  predictor:
    canaryTrafficPercent: null

The KServe Python SDK can't send that. It leaves out every field that is unset when it builds the request, so "set to nothing" never reaches the cluster, and an SDK-based promote silently does nothing. The pipeline uses the plain Kubernetes client with a merge patch instead.

Promote doesn't stop at "Ready" either. It waits until the revision carrying its own model location holds 100% of the traffic, the same check staging uses. That one line turned out to matter more than I expected; see Two releases at once.

When the inputs change

The canary assumes both revisions read the same rows. Most releases keep that true. One didn't: adding bookmaker odds as features took the model from 23 inputs to 26, and the serving model rejected every 26-input replay row with an HTTP error. That isn't a failure of the new model. There is simply nothing to compare it with, and a 10% split would send a tenth of the traffic to a model that can only fail.

So the canary checks the serving model before any traffic moves. Every revision KServe creates is annotated with the model location it serves, so the canary loads that model file and asks it how many inputs it takes. If the number differs from the replay rows, the input schema changed and the canary skips the split:

canary: "SKIPPED: the serving revision takes 23 inputs and the replay rows have 26, so the input schema changed"
result: canary=skipped
promote: on the strength of the checks that did run

The first version of this check didn't read the model at all. It skipped whenever the serving revision answered the replay rows with an HTTP error, and that turned out to be far too wide; see Two releases at once.

The new revision has, by then, answered every replay row exactly as the trained model does, and passed the smoke test. That is what it is promoted on. It happens once, on the release that changes the inputs. From the next run, both revisions read the same rows again and the split comes back.

Before any traffic moves: the canary's seven checks in order, which of them skip the split and which fail the run, and the serving model keeping 100% until the first split

Two releases at once

The pipeline runs once a day, so I never wrote a lock. Then the lab came up late after a cold boot, I started a manual run to stand in for the missed one, and two minutes later the scheduler started the missed run by itself. Two releases, one InferenceService.

Run	What happened
the manual run	staged its model and passed the smoke test; the serving revision, not yet up after the cold boot, answered the canary's first requests with 503; the canary skipped the split and removed the field
the scheduled run	staged its own model a minute later, on top of the first
the scheduled run's canary	found its own revision already rolled out, nothing earlier to compare with, and skipped the split
the scheduled run's promote	its model at 100%: passed
the manual run's promote	waited for its model to reach 100%, saw the other run's revision there instead, and failed after ten minutes

Two releases at once: the manual run and the scheduler's catch-up run on one InferenceService, which run's canary skipped and why, the promote that failed, and the fixes

That first skip was a bug of mine, not the race. The input-change check, as it was then, treated any HTTP error from the serving revision as "different inputs". The real input change answered 500; this was a 503, a pod that wasn't ready yet. An escape hatch that wide skips the split whenever the serving model is briefly unavailable, which is exactly when a careful rollout should wait, not wave the new model through.

The fix takes HTTP errors out of that decision altogether. Whether the inputs changed now comes from the serving model's own file, as described above. When the inputs match, the serving revision has to answer: a 502, 503 or 504, a refused connection or a timeout is retried with backoff for up to three minutes, and if it still can't answer, the canary fails before any traffic moves. Any other error fails at once. The serving model keeps 100% either way, and the run fails loudly instead of shipping without a split. The first release with the new check split normally: 76 of 600 requests to the new revision at 10%, 305 of 600 at 50%, no errors.

Nothing broke for callers: a model that had passed its own smoke test served the whole time. The part that worked is the promote check. A promote that only waited for "Ready" and 100% would have reported success for a model that was never serving. The part that didn't is the canary: the second run skipped its split, because the first run had already moved the field. Both models were the same recipe on the same data that day, so nothing was lost. On a day with a real change, the second model would have shipped without a split.

The fix is cheap and the order matters: check whether the scheduler is about to catch up before starting a stand-in run. A lock on the InferenceService would close it properly, and it's on the list.

The 403 that wasn't the model

The first canary runs failed with 403 on every request through the shared route, while the same requests to each revision's private service worked. Nothing was wrong with either model.

The namespace's mesh policy only admits callers from inside the namespace. The private service goes straight to the pod, carrying the canary job's own identity: allowed. The shared route, though, went through Knative's activator, the component that buffers requests while pods scale. Through the activator, a request arrives carrying the activator's identity, from another namespace: refused.

The 403 that wasn't the model: the canary Job reaches a revision directly as its own namespace and is allowed, but through Knative's activator the request arrives as knative-serving and is refused, until target-burst-capacity "0" takes the activator out of the path

The fix is one annotation on the predictor, which takes the activator out of the path whenever a pod is up:

spec:
  predictor:
    annotations:
      autoscaling.knative.dev/target-burst-capacity: "0"

The staging step sets it on every new revision. A serving revision staged before the annotation existed still had the activator in front of it, so the first run after the change skipped the split and said so, and every run since has split normally.

Who owns the InferenceService: YAML or the pipeline

The InferenceService is not in ArgoCD, and that's deliberate. A canary is a loop: set 10%, measure, decide, set 50%, measure, decide. Git describes an end state, not a loop. If ArgoCD owned this object with self-healing on, it would put the Git values back in the middle of a run.

So the pipeline owns it after the first deploy, and the YAML file in the repo is a first-deploy file only, with a loud header saying so. It matters: before I rewrote it, that file pointed at a three-week-old model and lacked the activator annotation. One kubectl apply would have rolled the model back three weeks and broken every canary run with 403s. The file now matches the live object except for the model location, which the pipeline moves on every release. So kubectl diff against it should show exactly one changed line, storageUri, and anything more is drift.

Watching it in Grafana

The decision is the canary job's, from exact before-and-after sidecar counts. Prometheus is for looking back afterwards, so the Model Serving dashboard has a "Canary runs" row: the canary's error share and p95 latency by revision, one point per run.

Two traps made those panels lie before they were right:

Empty is not zero. An error-share query divides errors by requests. On a clean run there are no error samples, so the division returns nothing, and a clean run looks exactly like a broken query. The query now falls back to zero wherever the revision had canary requests, and the same formula pointed at known-bad traffic shows 100%, so it can show errors.
Not every error is the model's. The activator's own requests, all failing with 503 URX,UF (nothing listening yet) or 0 DC (it hung up), come in bursts whenever a pod starts. They are cold boots, not model errors. Counting them would show 100% errors on every revision, so the canary panels count only the canary job's requests.

The dashboard's top row is now a set of traffic lights, green, orange and red, for 5xx share and p95 latency across callers. The first trap came straight back there: with zero errors, the 5xx tile said "no traffic" until its numerator fell back to zero too. A second one was new:

Last is not now. A stat tile over a seven-day range, reduced to "last value", kept showing a p95 of 488 ms hours after traffic had stopped. Every tile is now an instant query, and "no requests in the last ten minutes" reads as "no traffic" rather than as an old number.

The Model Serving dashboard over the half hour of a scheduled run: the top row reads no real traffic, and the traffic split shows the canary's replay reaching both revisions, 00074 and 00075, with xgboost-canary as the caller

The Canary runs row for the same run: error share 0% on both revisions, and p95 latency between 23.6 and 24.3 ms per revision

A real run

A scheduled daily run, end to end, with nobody watching. It promoted revision 00075 over 00074.

The scheduled run in Kubeflow Pipelines, every step green: backfill matches, publish to Iceberg, Katib sweep, final train and register, stage on KServe at 0%, smoke test, canary at 10% then 50%, promote to 100%

Stage	Result
smoke test	known answer matched; 200 requests from 8 threads in 1.6 s, all identical, p95 103 ms
fit on the 500 replay matches	new 0.8779 log loss, serving 0.8779; accuracy 62.8% both (no new results that day, same recipe)
the two revisions' answers	the same on 500 of 500 rows
step 10%	new revision 58 requests, serving 542; 0 errors; mean 6.8 ms vs 7.7 ms
step 50%	new revision 305, serving 295; 0 errors; mean 9.7 ms vs 10.2 ms
all 1,200 replayed answers	matched one of the two revisions; none foreign
promote	new revision at 100%, previous one scaled down
Prometheus, the canary's requests that day	not one non-200 response; p95 under 25 ms on both revisions

The measured split tracked the requested one: 58 of 600 requests at 10%, 305 of 600 at 50%. Knative splits by weight per request, so a few requests either way is normal; the rule only fails a step below a quarter of the expected share.

The canary step's own log for that run: the replay set, equal fit for both models, the same answer on 500 of 500 rows, 58 then 305 of 600 requests to the new revision with no errors, and canary=passed

How reliable is it?

The model service stays available through every release: the serving revision keeps its traffic until the new one has passed every check, and a failure at any stage leaves it exactly as it was.

If this happens	What the platform does	Evidence
the new model can't beat the serving one on the backtest	it is never staged; the serving model is untouched	finalize's gate decides before deploy runs
the new model fails to load	staging times out with the reason; the serving revision keeps 100%	the deploy step waits for the revision that carries the new model, not for "Ready"
the new model answers wrongly or slowly	the smoke test fails; nothing is promoted	live negative test: smoke failed, promote never ran, old revision kept 100%
the new model is worse under split traffic	the canary rolls back to 0% and the run fails	live negative test: failed at 10% after 70 requests had reached the new revision; field back to 0, serving revision 100%, promote never ran
the canary job is killed mid-step	it rolls back on the way out	rollback runs on every exit that isn't a pass, including termination
Knative's activator sits in front of the serving revision	the canary skips the split instead of failing on 403s	seen once, on the first run after the annotation change
the new model takes different inputs from the serving one	the canary skips the split once and promotes on the smoke test and the replay check	seen live when the inputs went from 23 to 26 columns; decided from the serving model's own input count
the serving revision is briefly unavailable	the canary retries for up to three minutes, then fails with no traffic moved; the serving revision keeps 100%	covered by tests; before the fix, a 503 after a cold boot skipped the split instead
two releases run at once	the later one wins; the earlier one's promote refuses to report success for a model that isn't serving	seen live after a cold boot: one run passed, the other failed in promote, callers were served throughout
the namespace quota refuses the smoke or canary Job	the step fails within five minutes and prints the quota's used and hard figures; the serving revision keeps 100%	the Job runner reads the quota's FailedCreate events instead of waiting for a pod that never comes
someone re-applies the YAML from Git	prevented by the file's ownership note; drift shows in `kubectl diff`	the diff shows only `storageUri`, which the pipeline moves on every release

Honest limits

The wait-then-fail path for an unavailable serving revision hasn't happened live yet. Tests cover it; the live case so far is the one from before the fix.
Both failure paths were tested with forced failures. The smoke test and the canary each failed a live run on an impossible threshold. A genuinely bad model would take the same path, but none has come along yet.
It catches broken models, not worse forecasts. Both models have seen the replayed matches. Forecasting quality is decided earlier, by the walk-forward gate.
On a day with no new results, both revisions are the same model. The canary then tests the rollout machinery more than the model.
A release needs room for two predictors. The new revision runs next to the serving one, so the namespace quota must fit both for the length of a run.
One release at a time is assumed, not enforced. There's no lock on the InferenceService. Two concurrent runs end with one model serving and the other run failed, which is safe, but the second run's canary is skipped.
Some releases skip the split by design. A first deploy and a change of inputs both promote on the smoke test and the replay check alone. The input change is read from the serving model's file, so a serving revision that is briefly unavailable no longer counts as one.
One pod per revision. Each revision runs a single pod, two pods in total while a canary runs, and nothing has been load-tested beyond the canary's own 600 requests. That suits a model called by its own pipeline, not a public endpoint.

To follow the rest of this build, 🚀 Follow me on LinkedIn

Author: George Ezejiofor

GRC Open-Source Tools: Falco, Wazuh & Probo (CrowdStrike/Vanta Alternative, Zero SaaS Bills)

George Ezejiofor — Thu, 29 Oct 2026 17:49:59 +0000

Part of the homelab build series. Governance, risk and compliance on open source only: Kyverno blocks, Falco detects live threats, Wazuh correlates and maps them to MITRE ATT&CK and SOC 2, and Probo holds the evidence. A brownfield build on an RKE2 cluster that already runs Istio, Keycloak, Ceph, Argo CD, Kyverno, KubeArmor and Trivy. Highly available: three Wazuh indexers, one per Proxmox host, and three Wazuh managers.

Open-source GRC: Kyverno, Falco, Wazuh and Probo

Introduction

My cluster had 41 Kyverno policies, KubeArmor enforcing AppArmor profiles and Trivy scanning every image. If someone had opened a shell inside one of my containers, none of it would have told me.

That gap is what companies pay CrowdStrike to close. They pay Vanta to turn it into audit evidence, and Azure Policy to govern the whole thing, as long as it lives in Azure. All three are good products, and all three keep your data on their platform and bill you every year to keep it there.

I wanted the same capability on hardware I already own, with every rule and mapping in Git. So I added three open-source tools to the platform. Falco watches every system call on the nodes. Wazuh collects those alerts next to the host logs and tags each one with a MITRE technique and a SOC 2 criterion. Probo tracks the SOC 2 controls and which part of the platform answers each of them. Kyverno, already running, stays the gate at admission.

It works. A shell opened in a container now reaches the SIEM in under a second, posts to Slack and maps to SOC 2 control CC7.2. Getting there took more tuning than building. The first hour produced about 18,500 alerts, and I had to bring that down to 176 without hiding a single real attack.

To be clear about what this is: it replaces the tools, not the people behind a managed service. I'll come back to what that costs you near the end.

What you are locked into

With CrowdStrike Falcon you pay per device, billed annually and up front, so every node you add raises the bill. Your telemetry lives in their cloud. You can write your own detections, but you can't read theirs, and the threat intelligence stops the day you cancel.

Vanta doesn't publish prices. Buyers say the quote grows with headcount and with every framework you add. Over time your control mappings, your evidence and your audit history all build up inside it, collected through its integrations, so leaving means rebuilding evidence collection somewhere else.

Azure Policy governs Azure. A cluster outside Azure can join through Azure Arc, which installs Azure Policy as an extension running Gatekeeper, but Azure's control plane stays in the loop and the policies are written in Azure's JSON format with Azure resource aliases. They don't come with you.

In this build every rule is Falco YAML, every mapping is Wazuh XML and every control is a line in a Git file. If I moved clusters tomorrow, all of it would come along.

Architecture

Architecture: Falco on each node, Wazuh agents on the hosts, the Istio gateway, the Wazuh cluster, Slack and Probo

I think of it as four layers, each answering one question:

Layer	Tool	Question	Stands in for
Prevent	Kyverno	May this run at all?	Azure Policy
Detect	Falco	What is running right now, and is it an attack?	CrowdStrike
Correlate	Wazuh	What does it mean, and which technique and criterion does it hit?	the SIEM
Attest	Probo	Which control does it satisfy, and where's the evidence?	Vanta

Kyverno came earlier in this series, so I don't rebuild it here. Its policy reports don't reach Wazuh yet, which is why that arrow is dashed.

Where the agents run

Wazuh documents two ways to run its agent in Kubernetes, as a DaemonSet or as a sidecar. I didn't use either. Ansible installs the agent on the host itself, on all 20 RKE2 VMs and the 3 Proxmox hosts.

The reasoning is simple: whatever watches the cluster shouldn't fall over when the cluster does. A DaemonSet agent only sees its own container unless you give it the host filesystem, privileged mode and the host PID namespace, at which point your security agent is the most privileged pod on every node. If the cluster goes down, a DaemonSet agent goes with it, and a dead node just looks like silence. The documented DaemonSet also mounts the Docker socket, and RKE2 runs containerd. And the Proxmox hosts aren't Kubernetes nodes at all, so only a host install reaches them.

So Falco covers the containers, and the host agent covers the host and carries Falco's alerts out.

Deploying Wazuh into a brownfield cluster

I vendored Wazuh's Kubernetes manifests at v4.14.9 without touching them, and change them only through patches in a Kustomize overlay that Argo CD applies. Upstream assumes a cloud and an empty cluster. Mine is neither, and several things broke.

The first surprise was four type: LoadBalancer Services. With MetalLB in the cluster, the dashboard, the manager API, the workers and the indexer API on 9200 would each have taken an address on my LAN. All four became ClusterIP, and only the Istio gateway is exposed. The manifests also ship default credentials, admin:SecretPassword and an enrolment password of password, plus four demo users with known hashes. Terraform generates every certificate, password and hash into Vault instead, and the Vault Secrets Operator syncs them into Secrets with the names upstream expects. Upstream's secret files never make it into the overlay.

Then the service mesh broke Wazuh's own clustering. With Istio sidecars, every connection reached the master from 127.0.0.6, so the two workers looked like one, only two of the three indexers joined, and Filebeat failed TLS. Wazuh already encrypts every hop, so I took the namespace out of the mesh. Upstream also sets vm.max_map_count from a privileged init container. Ansible sets it on the database nodes now, the init container is gone, and the namespace enforces Pod Security baseline. Restricted isn't possible, because the manager runs as root.

The one that cost me most was disk. Each manager downloads a 9.2 GB vulnerability feed, and upstream gives it 500Mi. I'd raised that to 10Gi, and within 15 minutes all three volumes were full, wazuh-db couldn't open its database, and the master stopped taking events. They have 40Gi each on Ceph now, on ceph-block instead of upstream's cloud StorageClass.

Agents reach the managers through the same Istio ingress gateway as everything else, with ports 1514 (events) and 1515 (enrolment) added as TCP routes. On the first enrolment, agents registered with the master and then connected to a worker that didn't have their key yet. They got "Invalid ID", tried again, were refused as duplicates, and settled after a few minutes.

At the end of it I had 23 agents active, 3 indexers green, a master and 2 workers.

Wazuh Endpoints: 23 agents active, 20 Ubuntu RKE2 VMs and 3 Debian Proxmox hosts, all on Wazuh v4.14.9

Single sign-on with Keycloak

The dashboard signs in through Keycloak, like every other app on the platform. Terraform creates a wazuh realm with Google and GitHub sign-in and an email allow-list, so any other account is turned away at first login. The indexer maps the allowed emails to all_access, and the internal admin from Vault stays as a break-glass login.

The Wazuh login page: the internal login stays as a break-glass, and "Sign in with Keycloak" is how people get in

Keycloak realm wazuh: sign in with Google or GitHub; only allow-listed emails get through

Bringing Falco back

Falco had been switched off on this cluster for a while. It was using 875 to 956 millicores per node against a 100m request, and I needed the CPU.

When I turned it back on, every pod crash-looped. Falco itself started fine, but its container plugin (0.6.3, bundled with chart 8.0.2) panicked on a nil pointer: containerd 2.3 leaves a field empty that the plugin reads without checking. Pinning the plugin to 0.7.4 fixed it without a chart upgrade.

collectors:
  containerEngine:
    pluginRef: ghcr.io/falcosecurity/plugins/plugin/container:0.7.4

A changed ConfigMap doesn't restart a DaemonSet, so I had to restart the pods before they picked it up.

Routing Falco alerts to Wazuh

My first idea was Falcosidekick posting straight to Wazuh. That doesn't work, because the manager listens for syslog, not HTTP, and every alert would arrive with no idea which host it came from.

What I did instead needs no new component. Falco writes each alert as one JSON line to a file on the node, and the Wazuh agent that's already on that host reads it. Every alert reaches Wazuh tagged with the right host, through the same gateway and TLS as everything else.

falco:
  file_output:
    enabled: true
    keep_alive: false
    filename: /var/log/falco/events.json

mounts:
  volumes:
    - name: falco-events
      hostPath:
        path: /var/log/falco
        type: DirectoryOrCreate
  volumeMounts:
    - name: falco-events
      mountPath: /var/log/falco

I didn't edit 23 hosts to point the agents at that file. Wazuh's shared configuration does it from the master:

<agent_config>
  <localfile>
    <log_format>json</log_format>
    <location>/var/log/falco/events.json</location>
  </localfile>
</agent_config>

A small set of rules, IDs 100300 to 100399, turns Falco's priorities into Wazuh levels and tags each alert with a MITRE technique and a SOC 2 criterion:

<rule id="100311" level="8">
  <if_sid>100302</if_sid>
  <field name="rule">^Read sensitive file untrusted$|^Read Sensitive File in Container$</field>
  <description>Falco: sensitive file read in $(output_fields.k8s.ns.name)/$(output_fields.k8s.pod.name)</description>
  <mitre>
    <id>T1003.008</id>
  </mitre>
  <group>tsc_CC7.2,tsc_CC7.3,</group>
</rule>

That keep_alive: false in the Falco config looks like a detail. It isn't, and the tuning section explains why. Two other things caught me out here. The master syncs new rules to the workers, but the workers keep using the old ones until analysisd reloads (PUT /cluster/analysisd/reload), so for a while half the cluster looked silent. And the file needs rotating, which logrotate does with copytruncate, rolled out by Ansible. The agent notices the truncation and reads from the top again.

Alerts in Slack

Anything at level 8 or above goes to Slack, into the channel Alertmanager already posts to. Wazuh has a built-in Slack integration, so the message carries the mapped alert: host, pod, MITRE technique and SOC 2 criterion.

The webhook is a secret, and ossec.conf can't read secrets from a file. So the Vault Secrets Operator renders the whole integration block, and an init container appends it to the manager's config when the pod starts:

transformation:
  excludeRaw: true
  templates:
    slack.conf:
      text: |-
        <ossec_config>
          <integration>
            <name>slack</name>
            <hook_url>{{ get .Secrets "hook_url" | trim }}</hook_url>
            <level>8</level>
            <alert_format>json</alert_format>
          </integration>
        </ossec_config>

Terraform fills that Vault value from the same variable as Alertmanager's webhook, so rotating one rotates both. While setting this up I also worked out why I'd once missed an Alertmanager alert. A Slack incoming webhook posts to the channel it was created for and ignores the channel in your config. My alerts had been arriving all along, just not in the channel the config named, and the one I needed was buried among routine warnings.

Something to know before you copy this: Wazuh writes the full webhook URL to /var/ossec/logs/integrations.log every time it sends. Don't screenshot that file, and assume anyone who can exec into the managers has the URL.

monitoring-stack-alert: the shell test as Wazuh rule 100310, level 8, with agent and location. Below it, one of the manager start-up false alarms that the wazuh-syscheckd exemption later removed

Validation

Nothing here runs on a schedule. Falco looks at every system call as it happens. To check the whole chain, I ran three real attacks against a pod that was already in the cluster:

# Read the password hashes
kubectl -n probo exec deploy/probo-chrome -- cat /etc/shadow

# An interactive shell inside a container
script -q /dev/null kubectl -n probo exec -it deploy/probo-chrome -- sh

# Tamper with the container's configuration
kubectl -n probo exec deploy/probo-chrome -- sh -c 'echo t > /etc/falco-write-test; rm -f /etc/falco-write-test'

All three arrived in Wazuh with the right host, pod and namespace, and so did a real privileged container launch:

Action	Falco rule	Wazuh rule	Level	MITRE
`cat /etc/shadow`	Read sensitive file untrusted	100311	8	T1003.008
`kubectl exec -it … sh`	Terminal shell in container	100310	8	T1059
Write below `/etc`	Write to /etc in Container	100302	5	none
Privileged pod replaced	Launch Privileged Container in Homelab	100322	3	T1611

From Falco seeing the system call to Wazuh storing the alert took 0.72 seconds.

Wazuh Discover, filtered to Falco alerts from the probo namespace: the shadow read (100311, T1003.008), the interactive shell (100310, T1059) and the /etc write (100302), each tagged with the host agent that shipped it

Tuning the noise

The first hour was about 18,500 Falco alerts, nearly all of them the platform going about its normal business. Nobody can read that, so most of this project ended up being tuning.

Falco alerts per hour before and after tuning, and the 332 s to 0.72 s delay fix

I used the same loop every time. Find the top image and namespace for each rule in Wazuh, decide whether it's a threat or a known workload, fix it in Git, then compare the hour before with the hour after.

The biggest single fix was embarrassing. Our own "Read Sensitive File in Container" rule listed /etc/passwd, which fired about 3,600 times an hour as MySQL, Postgres, Ceph and Trino looked up user names. That file is world-readable on purpose. I took it out and put /etc/gshadow in. Close behind it was Wazuh's own rootcheck reading /etc/shadow on every host, which is the security agent tripping the security rule. I exempted it by its executable path, /var/ossec/bin/wazuh-modulesd, rather than a process name anything could copy. Later the managers' file-integrity daemon needed the same treatment, after every manager restart posted three false alarms to Slack.

Two rules were broken in ways I'd never have guessed. "Launch Privileged Container" had no event type in its condition, so it matched every system call of a privileged container that was already running, about 3,600 times an hour from one CSI driver. Falco even warns about this at load time (LOAD_NO_EVTTYPE). It now matches only a container's first process, and deleting one of those CSI pods produced exactly one alert. The kubectl rule excluded (ci_cd_images, kubectl_images), but neither list was defined anywhere, and Falco quietly reads an undefined name as a plain string. That exclusion had never matched anything, which is why the kube-apiserver's own health probe was firing thousands of times an hour.

For everything else I exempted by namespace and image together. Controllers that talk to the Kubernetes API by design, such as Tekton, Kiali, Kyverno and Argo CD, are allowed to only in their own namespace with their own image. KubeArmor, the Prometheus config reloader and Istio's pilot-agent may write their own config below /etc, but only in their own directory. A reverse-shell rule that matched -i anywhere in a bash command had to be narrowed too, because Rook-Ceph's health probes run curl -i and produced over a thousand false Critical alerts in three hours. The last one turned up while I was writing this. Our GitHub Actions self-hosted runners download kubectl, crane and yq during a job and run them, which is exactly what Falco's "drop and execute a new binary" rule looks for. One CI run raised 594 Critical alerts, every one of which would have gone to Slack. The runners are now exempt for binaries under their own home directory, and only there.

After every change I ran the attack tests again, and they all still fired. That mattered more to me than the drop in numbers.

The same story is on a dashboard, kept in Git like everything else. A small generator writes the saved objects, and a Job imports them after every Argo CD sync.

Falco runtime security: every Falco alert carries SOC 2 CC7.2; the 2,607 at level 8 and above also carry CC7.3 and go to Slack

The last 24 hours in half-hour bars: about 10,000 alerts every 30 minutes before tuning, then a trickle. The MITRE techniques sit beside them

Where the alerts came from, by namespace and image, and the level 8+ events with their MITRE technique

Getting rid of the noise exposed a problem I hadn't seen before. I wrote a test file below /etc and watched it appear on Falco's stdout straight away, then waited. It reached Wazuh 332 seconds later, and only because I restarted that Falco pod. With keep_alive: true, Falco keeps the output file open and buffers its writes, even with buffered_outputs: false. At 18,500 alerts an hour the buffer filled every few seconds, so I never noticed. At a few dozen an hour, a real alert could sit there for minutes. Switching to keep_alive: false makes Falco write each alert as it happens, and the same test took 0.72 seconds. If you tune a pipeline like this, measure the timing again afterwards.

There was a nice side effect, too. Before tuning, Falco averaged 294 millicores per node. After, the same 14 pods average 28.

Compliance mapping in Wazuh

This is where the security work turns into compliance evidence. Wazuh's own ruleset tags its checks with PCI DSS, NIST 800-53, HIPAA, GDPR and SOC 2, so a failed check arrives already attached to named controls.

Here's a real one. Wazuh runs the CIS Ubuntu 24.04 benchmark on every host agent, and this check fails on all 20 of my RKE2 VMs:

Field	Value
Check	CIS Ubuntu Linux 24.04 LTS 3.3.5: Ensure ICMP redirects are not accepted
Wazuh rule	19007, level 7
SOC 2	CC6.3, CC6.6, CC7.1, CC7.2
PCI DSS v4	1.2.5, 2.2.4, 6.4.1
ISO 27001:2013	A.13.1.3
NIST 800-53	CM.1
MITRE ATT&CK	T1557, mitigations M1030 and M1042

That's as far as Wazuh goes. It produces the mapped technical evidence, but it doesn't track who owns a control or what an auditor has already seen. For that I use Probo.

Evidence in Probo

Probo is an open-source compliance tool. It runs in the same cluster and signs in through Keycloak with SAML. Its frameworks come with their controls already written, and I added all 13 to my organisation, 1,008 controls in total:

Framework	Controls	Measures linked
SOC 2	61	11
PCI DSS	331	not yet
ISO 27001 (2022)	123	not yet
ISO 27701 (2025)	108	not yet
ISO 42001 (2023)	70	not yet
DORA	65	not yet
GDPR	62	not yet
HIPAA	60	not yet
HDS	31	not yet
FERPA	29	not yet
21 CFR Part 11	28	not yet
CCPA	23	not yet
NIS 2	17	not yet

Keycloak realm Probo: sign in with Google or GitHub; only allow-listed emails get through

Created George Homelab as organisation

I started with SOC 2, because Wazuh already tags every alert with its criteria.

The 13 frameworks added to my Probo organisation; only SOC 2 has measures linked so far

In Probo, each control the platform really runs becomes a measure, linked to the criteria it answers. I didn't want to click those together in a UI, so they live in a YAML file in Git:

- code: IC-MON-01
  name: Runtime threat detection (Falco → Wazuh)
  category: Monitoring
  control_type: DETECTIVE
  operating_frequency:
    mode: CONTINUOUS
  implementation_status: OPERATING
  soc2:
    - CC7.2
    - CC7.3

A short script, probo-sync.py, reads that file and talks to Probo's MCP API. It shows a plan first, only applies with --apply, and never deletes anything. The first run created 11 measures and 19 links, covering runtime detection, host assessment, sign-in, secrets, network segmentation, admission policy, GitOps change control, the image supply chain and tested backups. Where I know a measure has a gap, it says "in progress". When I add ISO 27001, I'll link the same measures to its controls. The evidence doesn't change, only the framework it's mapped to.

SOC 2 control CC7.2 in Probo, answered by three measures: runtime threat detection and the Wazuh host assessment, both implemented, and Prometheus alerting, still in progress

All 11 measures in Probo, created from soc2-mapping.yaml: seven implemented, four still in progress where I know there's a gap

One event, end to end

One shell in a container, from the system call to Slack and the SOC 2 control

Following one shell through the whole chain: Falco matches "Terminal shell in container" on the node where the pod runs. The Wazuh agent on that node reads the JSON line and sends it through the Istio gateway on port 1514. The manager applies rule 100310, which makes it level 8 and tags it T1059 and SOC 2 CC7.2 and CC7.3. The alert is stored 0.72 seconds after the system call and posted to Slack. In Probo, CC7.2 is answered by the runtime detection measure, which names this pipeline as its evidence.

The last step is the weak one. The measure shows the control exists and runs, but one alert doesn't flow into Probo by itself. That's what I'll build next.

How reliable is it?

The lab powers off every night and cold-boots every morning, so every part of this restarts from scratch daily, which turns out to be a decent test in itself.

What goes wrong	What happens	Evidence
All three hosts cold-boot	Wazuh, Falco and Probo come back with no manual step	23 agents active and 3 indexers green after the morning boot
A Wazuh manager restarts	Its agents reconnect to the managers still running	All three managers restarted one after another: all 23 agents reconnected, and every test alert sent during the restarts arrived
Falco restarts on every node (14 pods)	Each pod is replaced in turn and picks up where it stopped	Several rolling restarts during tuning; every attack test run afterwards fired
An indexer pod dies	The cluster drops to yellow, never red, and alerts keep landing. The pod is back in about 10 seconds	Green again 60 seconds after the delete; an attack test sent during the outage was stored 0.21 s after Falco saw it. The first run went red: Wazuh creates its inventory and vulnerability indices with 0 replicas. An hourly job now keeps one replica on them
One Proxmox host goes down	[PENDING: shut down one host, check agents fail over to the remaining workers and the indexer keeps 2 of 3]	[PENDING]

What this doesn't do

CrowdStrike sells more than a sensor. You also get managed threat hunting and threat intelligence drawn from their whole customer base. Falco and Wazuh give me the detection, but none of that service.

Detection is automatic here, and response isn't. When an alert lands in Slack, a person decides what happens next. Falco Talon could kill a pod on its own, but I haven't deployed it.

Vanta collects evidence through hundreds of integrations. Here, only the measures are synced from Git, and linking evidence is still manual. Only SOC 2 is mapped so far. The other 12 frameworks are in Probo with all their controls, waiting. Kyverno's policy reports don't reach Wazuh yet either, so I can't show a compliance score per framework the way Azure Policy can, and the admission measure stays "in progress". Falco also isn't running on the database and GPU nodes yet; it covers 14 of the 20 VMs until I add tolerations for those pools.

And you run all of it yourself. Upgrades, storage, certificates and noise tuning are your job. The tuning section above is a fair picture of what that takes.

Cost and sovereignty

There's no licence fee. What it costs is the capacity it uses, measured on the live cluster:

Component	Footprint
Wazuh: 3 indexers, 3 managers, dashboard	about 7 GiB memory, under 0.1 CPU core idle
Wazuh storage on Ceph	3 × 30Gi indexer, 3 × 40Gi manager
Falco on 14 nodes	about 28 millicores per node after tuning
Probo with its Postgres	about 0.5 GiB memory

Here's roughly what the same coverage would cost from the vendors, at their published or reported prices:

Product	Price basis	For this lab
CrowdStrike Falcon Enterprise (the tier with full EDR)	$184.99 per device per year, list price (CrowdStrike pricing)	23 hosts: about $4,250 a year. Container and Kubernetes protection is priced separately and isn't published
Vanta	No published price. Median contract $20,000 a year; $12,000 to $28,000 for 1 to 50 employees on one framework (Vendr, from anonymised purchases)	at least $12,000 a year, plus the auditor: $8,000 to $25,000+ for SOC 2, not included
Azure Arc + Azure Policy	Kubernetes configuration: first 6 vCPUs free per subscription; Azure Policy guest configuration $6 per server per month outside Azure; other Azure Policy at no extra cost (Azure Arc pricing)	the smallest line, but every policy and audit log then lives in Azure

That comes to $16,000 or more a year for a homelab, before anyone has paid an auditor. Self-hosting doesn't get rid of the auditor either.

It doesn't get rid of the penetration test, either. PCI DSS still wants one every year, plenty of customers ask for the report, and third-party testers aren't cheap. In my experience, even moving a booked pentest to another date can cost 25% of the fee. What this stack changes is what you're paying the tester for. The findings a scanner would hand you, like the ICMP-redirect check failing on every node, are fixed before anyone arrives, so the hours go into the problems only a person can find. And while they work, every technique they try should show up in Wazuh and in Slack, which turns the pentest into a test of your detection as well.

For me, though, the stronger reason is sovereignty. My security telemetry and compliance evidence never leave my hardware. No third party processes them, so the question of cross-border transfers never comes up.

That carries more weight in the EU. DORA makes financial firms manage the risk of every ICT provider they rely on and keep a register of them. NIS 2 asks essential and important entities to secure their supply chain and to report a significant incident with an early warning within 24 hours and a fuller notification within 72. A self-hosted stack means fewer providers on that register, and the logs you'd report from stay on your own hardware. Probo ships both frameworks, too.

What's Next?

The next piece closes the gap in that last step: a small job that pushes Wazuh alert counts for each SOC 2 criterion into Probo as evidence, so the audit view updates itself. After that I'll send Kyverno's policy reports into Wazuh, give Falco tolerations for the database and GPU nodes, and link ISO 27001 to the measures that already exist.

This is the part of the platform that proves the rest of it to someone looking in from outside: the Terraform, RKE2 with Ansible, the hardening series, the signed supply chain, storage, Vault and observability. It does live detection, mapped evidence and audit tracking, all in Git, all on hardware I control.

To follow the rest of this build, 🚀 Follow me on LinkedIn

Author: George Ezejiofor

Brownfield Kubeflow on RKE2: Swapping Dex & SeaweedFS for Keycloak & Ceph RGW

George Ezejiofor — Sat, 17 Oct 2026 19:16:43 +0000

Upstream Kubeflow assumes a greenfield cluster and ships its own Dex, SeaweedFS and MySQL, with its own oauth2-proxy in front. My homelab already had better versions of most of those. This is the **brownfield* deploy on an HA RKE2 cluster: every reference component I ripped out (and the one I haven't yet), what I wired in its place, and the auth chain that ate most of the debugging time.*

The Kubeflow Central Dashboard, logged in through Keycloak SSO (not Dex), with the whole platform iframed into one tab.

Introduction

When the platform assumes it's the only thing in the cluster

Kubeflow was designed as if it would have the cluster to itself, and it still shows. Point the reference installer at an empty cluster and it works well: nine controllers, four web UIs, two workflow engines, an inference layer and a per-user namespace system, all iframed into one dashboard and running in twenty minutes.

Point it at a cluster that already has an identity provider, an object store, a database fleet, a service mesh and an SSO layer guarding ten other apps, and you're in for a fight. Kubeflow never asks what you already run. The v1.11 multi-user install brings Dex for identity (behind its own oauth2-proxy, wired into Istio as an external authorizer), SeaweedFS for object storage, MySQL for state, and a mesh-wide trust model that doesn't match any real cluster I know of.

That's the gap between a greenfield deploy and a brownfield one. Greenfield, you accept whatever the platform brings. Brownfield, the platform moves into a neighbourhood that was there first. Mine already had Keycloak issuing every realm, Rook-Ceph serving S3 to half the stack, CloudNativePG holding everyone's relational state, and oauth2-proxy in front of ten apps.

I could run two of every backend, or rewrite Kubeflow's integration layer to talk to what was already in the building. I picked the rewrite. It cost roughly six weekends and a kustomize overlay per component, and the platform now fits the cluster it runs on.

Here's the full list: every reference component upstream Kubeflow ships, what I put in its place, and why the stock one didn't fit.

Stock Kubeflow manifests ship…	I substituted…	Why the stock one didn't fit
Dex (bundled OIDC identity provider)	Keycloak, central realm `kubeflow`	One IdP already issues every app's realm. A Kubeflow-only Dex is a second identity island to run, patch and secure.
oauth2-proxy + Dex as Istio ext_authz (every request checked by an external authorizer at the gateway)	My own `oauth2-proxy-kubeflow` in reverse-proxy mode against Keycloak, with ext_authz kept only for `/notebook/*`	It's the same OIDC pattern that already gates ~10 apps on per-app cookie scopes, and proxy mode lets one instance route all the static-URL UIs.
SeaweedFS (KFP artifact store, exposed behind the `minio-service` name; MinIO in older releases)	Rook-Ceph RGW (S3) via an `ObjectBucketClaim`	One storage floor already serves S3 to MLflow and the rest of the data stack. Every KFP S3 client points straight at `https://s3.georgehomelab.com:443` (RGW is TLS-only); VSO maps `AWS_*` → `accesskey`/`secretkey`.
MySQL (KFP metadata DB)	Not swapped yet: still the bundled MySQL. CloudNativePG `kubeflow_metadata_db` is provisioned and waiting	CNPG already hosts everyone's relational state with backups to Ceph RGW. The database and its Vault credentials exist; the KFP Deployments don't point at them yet (I checked live).
Principal-scoped AuthorizationPolicies (allow only the ingress-gateway service account)	Path-based AP rules (`paths: ["/*"]`)	A PERMISSIVE mesh doesn't populate the SPIFFE principal reliably, so principal-scoped rules silently deny everything.
Bundled cert-manager + self-signed Issuer	Existing cert-manager + Let's Encrypt (Cloudflare DNS-01)	Real, browser-trusted certs are already automated cluster-wide; no self-signed CA to hand out.
Kubeflow's Spark-on-K8s overlay	Existing mlops-spark-operator	The MLOps stack already runs a Spark operator; two of them fight over the same CRDs.
Istio bundled by the installer	Existing RKE2 Istio 1.30 mesh (PERMISSIVE)	The mesh predates Kubeflow and runs everything else. Kubeflow gets injected into it; an ML installer doesn't get to replace the service mesh.

The rest of this piece walks through those swaps, the "huge CRD" trap that breaks client-side apply, the per-Profile governance Kyverno generates, and the Auth Chain Saga, where RBAC: access denied turned up at five different layers over several days before login finally worked.

Why Kubeflow, and which release

I pinned to v1.11.0 of the kubeflow/manifests repo, the most recent immutable release tag when I started. Don't be fooled by releases/release-1.11/ on master: it's a release-process tracker (a README about dates and the team). The real installer entry point is example/kustomization.yaml, and that differs between master HEAD (a moving target) and v1.11.0 (immutable). I wanted a build I could reproduce, so I took the tag.

Most of that time went on the auth chain (the Auth Chain Saga and the notebook routing pivot below). The heavy KFP overlay and the Model Registry integration were the runners-up.

The honest head-to-head: Kubeflow vs SageMaker/Vertex, Flyte, Metaflow, Ray, and MLflow

People ask me "should I use Kubeflow or MLflow?" a lot, and the question doesn't quite work. It's like asking whether to buy a kitchen or a knife. Kubeflow is a nine-component platform, and almost everything people compare it to replaces exactly one of those components. To compare fairly you have to sort the field by the layer each tool occupies:

Whole self-hosted platform (notebooks + pipelines + training + HPO + serving + registry + multi-tenant portal): Kubeflow, plus the open all-in-ones MLRun and Determined AI.
Managed cloud platform (all of the above, on someone else's hardware): AWS SageMaker, GCP Vertex AI, Azure ML.
Pipeline / orchestration only: Flyte, Metaflow, Airflow, Prefect, Dagster. These compete with KFP, which is one of the nine boxes.
Experiment tracking + registry only: MLflow, Weights & Biases, ClearML. A component, not a platform (I run MLflow inside this Kubeflow).
Serving only: KServe (which is part of Kubeflow), Seldon, BentoML, Triton.
Distributed compute only: Ray (KubeRay), which runs alongside Kubeflow rather than replacing it.

In other words, MLflow, Seldon and Ray are things you run inside Kubeflow, not alternatives to it. The only real peers to the whole platform are the managed clouds and a couple of open all-in-ones:

	Notebooks	Pipelines	HPO	Serving	Registry	Multi-tenant	Self-hosted	Best at
Kubeflow	✅	✅ KFP	✅ Katib	✅ KServe	✅	✅ Profiles	✅	Breadth, on your own hardware, no lock-in
SageMaker / Vertex	✅	✅	✅	✅	✅	✅	❌ cloud	Zero-ops DX, if you'll pay + go off-prem
Flyte	❌	✅✅ typed	⚠️	❌	⚠️	✅	✅	Best pipeline authoring: typed, reproducible
Metaflow	❌	✅	⚠️	❌	✅ tracking	⚠️	✅	Data-scientist Python DX; hide the infra
Ray (KubeRay)	⚠️	✅ tasks	✅ Tune	✅ Serve	❌	⚠️	✅	Distributed compute (train/tune at scale)
MLflow	❌	❌	❌	⚠️	✅✅	❌	✅	Experiment tracking + model registry

Where each one wins:

SageMaker / Vertex / Azure ML: the best managed DX, zero platform ops, elastic GPU infra, wired into cloud data and IAM. If you already live in a cloud and care more about time-to-value than control, cost and data sovereignty, managed wins clearly. Kubeflow's answer is the flip side: no cloud lock-in, no per-GPU-hour rental markup, data never leaves my rack, and it runs on hardware I own. Think of it as a self-hosted SageMaker for people who want to own the platform.
Flyte: honestly, a better pipeline engine than KFP. Strongly-typed Python, first-class versioning and caching, cleaner local dev. If pipelines are the whole job and you don't need notebooks, serving, HPO or a multi-tenant portal, Flyte is the nicer tool. Kubeflow wins on breadth: the nine boxes plus the portal that puts them in one tab.
Metaflow: a pleasant Python DX from Netflix, aimed at data scientists who don't want to think about infrastructure. It covers orchestration and tracking, not serving, HPO or multi-tenancy, and its Kubernetes support is thinner than its AWS support.
Ray (KubeRay): the best tool for distributed compute: training big models, hyperparameter tuning, RL, distributed data. It doesn't replace Kubeflow, it sits under it (Katib overlaps Ray Tune; KServe can serve Ray). If your bottleneck is Python-native distributed compute, run Ray alongside Kubeflow.
MLflow / W&B: the best experiment tracking and registry around. That's why MLflow is a tile in my Central Dashboard rather than something I picked instead of Kubeflow.
Seldon / BentoML / Triton: serving specialists. KServe covers most cases, but for advanced inference graphs or GPU-optimized Triton runtimes they can beat it on serving alone.

Why Kubeflow here, then? It's the one open-source platform that gives me all nine boxes (Jupyter notebooks, KFP pipelines, Trainer V2, Katib HPO, KServe serving, Model Registry and per-user Profiles) on my own RKE2 + GPU hardware, integrated with the stacks I already run (Keycloak, Ceph, CNPG, Istio), with no cloud lock-in. I couldn't find anything else in open source that complete and also self-hosted.

The catch is complexity. Kubeflow has a lot of moving parts, an auth chain that took several days to untangle, and an installer that assumes a greenfield cluster. If all you need is pipelines, Flyte has a fraction of the operational surface. If you'll pay for managed, SageMaker takes the ops away entirely.

Kubeflow only makes sense when you want the full platform, self-hosted, on your own hardware, and you're prepared to pay the complexity cost once. If you don't need all that, a specialist tool or a managed service on a credit card is the better call.

The cluster Kubeflow is landing on

The facts that matter:

RKE2 with five masters and a mix of worker pools (app, data, GPU). HA control plane via kube-vip. The whole lab powers off every night and cold-boots in the morning, which matters more than I expected (see the cold-boot race below).
Cilium in eBPF mode, with Hubble enabled.
Istio 1.30 as the service mesh. The default mode is PERMISSIVE, and per-namespace PeerAuthentication CRs flip to STRICT where it matters. istio-system's default stays PERMISSIVE because the API server has to reach admission webhooks in injected namespaces without holding a client cert.
Rook-Ceph for storage, with three SCs: ceph-block (RBD, RWO), ceph-filesystem (CephFS, RWX), ceph-bucket (RGW, S3 via OBC).
CloudNativePG for relational state. The MLOps cluster postgres-mlops-cnpg-cluster hosts kubeflow_metadata_db, ready for Kubeflow Pipelines but not in use yet.
Vault + Vault Secrets Operator, auto-unsealed by a separate transit Vault in its own vault-transit namespace (Shamir, 1 share).
Keycloak as the OIDC provider for every app. Realms and clients are managed by Terraform; there are no Keycloak operator CRDs on this cluster.
ArgoCD, driven by AppSets. Each major area (mlops-stack, kubeflow-stack, cicd-stack, security-stack, monitoring-stack) has its own AppSet generating a child App per component.
Kyverno for admission policy and generate rules. At the time, the cluster-wide make-configmaps-immutable policy was the one that bit hardest on this build.

The kubeflow-stack.yaml AppSet in kubernetes-addons/0-georgehomelab-argocd-projects/2-argocd-applicationsets/ phases the rollout. Sync waves run from 1 (foundation) to 8 (the SSO front door), across 24 child Applications. Each component is its own Application, so I can debug it in isolation.

Kubeflow on RKE2, what I replaced from upstream: the cluster's Istio gateway, oauth2-proxy as a reverse proxy and Keycloak in front of the Central Dashboard, its six apps, and a foundation of GPUs, bundled MySQL, Ceph object storage and CloudNativePG, each marked replaced, upstream or added

Kubeflow as deployed: oauth2-proxy → Central dashboard → apps, MLflow beside it, and every swap from upstream tagged.

The ArgoCD UI filtered to kubeflow-*: the 24 child Applications from the kubeflow-stack AppSet, all Synced/Healthy, in sync-wave order.

Prerequisites first

Phase 0 is everything that has to exist before Phase 1's AppSet even sees the cluster. Five things:

The CNPG kubeflow_metadata_db Database CR. It lives in kubernetes-addons/Data-platform-stacks/mlops-cnpg/postgres-mlops-cnpg-cluster.yaml next to the other MLOps databases. The owner is the kubeflow Postgres role, with its password in Vault. Applying the Database CR creates the database on the already-running CNPG cluster. It's there for the MySQL → CNPG move, which hasn't happened yet (see the KFP overlay section).
Vault paths, populated by Terraform. Three matter:
- secret/homelab/kubeflow/kubeflow-metadata-db: {username: kubeflow, password: ...}
- secret/homelab/kubeflow/oauth2-proxy: {client-id, client-secret, cookie-secret} for the Keycloak kubeflow realm client.
- secret/homelab/kubeflow/centraldashboard-session (optional): the Central Dashboard session secret. Terraform writes these from .secrets.auto.tfvars. No human writes Vault paths by hand.
The Keycloak kubeflow realm. Terraform creates the realm, its Google and GitHub identity providers and the oauth2-proxy client, and it reads that client's secret from the same Vault path oauth2-proxy uses, so the two can't drift. Before the SSO front door goes in, I check two things directly: the realm's .well-known/openid-configuration returns 200 with a valid issuer, and the Vault path holds all three keys.
Google OAuth client. Add kubeflow.georgehomelab.com/oauth2/callback to the authorized redirect URIs on the existing OAuth client.
DNS and a dedicated certificate. kubeflow.georgehomelab.com → 10.0.1.21 (the kube-vip ingress VIP), managed in Cloudflare. Kubeflow gets its own cert-manager Certificate (kubeflow-georgehomelab, issued via Let's Encrypt DNS-01), which writes the kubeflow-georgehomelab-tls-cert Secret the Kubeflow Gateway serves.

With those five in place, Phase 1 can fire.

Laying the foundation

Phase 1 is five Apps at sync waves 1 and 2.

kubeflow-namespace creates the kubeflow and kubeflow-system namespaces with pod-security.kubernetes.io/enforce: privileged. Upstream uses restricted, and my cluster default is also restricted. But every istio-injected namespace needs privileged, because istio-init runs with CAP_NET_ADMIN + CAP_NET_RAW to program iptables. Without privileged, istio-init fails admission and every pod in the namespace hits FailedCreate with no obvious cause. CNPG's initdb Job hits the same trap.

kubeflow-cert-issuer installs a selfSigned ClusterIssuer for Kubeflow's internal admission-webhook certs. Public TLS doesn't use it; that comes from the dedicated Let's Encrypt certificate.

kubeflow-roles holds the kubeflow-admin / kubeflow-edit / kubeflow-view ClusterRole aggregates. The Profiles controller binds them in per-user namespaces when it reconciles a Profile CR.

kubeflow-istio-resources is the dedicated kubeflow-gateway Gateway, which I put in istio-system rather than in the kubeflow namespace where upstream puts it. It binds kubeflow.georgehomelab.com specifically and serves the kubeflow-georgehomelab-tls-cert Secret, so I can rotate the Kubeflow cert without touching anything else.

The plan was failure-domain isolation from the shared homelab-gateway, and that only half worked. homelab-gateway's *.georgehomelab.com server overlaps this host on the same ingress gateway port 443, and Istio picks one per ingress replica. It showed up as intermittent 404s: curl worked, the browser didn't. The front-door VirtualService now binds both gateways, so the route exists whichever one Istio picks.

The same App later picked up the notebook ext_authz policy and its redirect EnvoyFilter (see the notebook routing section), but in Phase 1 it's just the Gateway and the RBAC aggregates.

kubeflow-kfp-infrastructure holds KFP's prerequisites. It creates the kfp-artifacts ObjectBucketClaim (Rook-Ceph provisions a bucket on RGW with its own access keys), plus the VaultAuth + VaultStaticSecret that sync the CNPG kubeflow user's credentials into a kubeflow-metadata-db Secret in the kubeflow namespace.

Those credentials are ready and unused. The ml-pipeline Deployment that lands in Phase 3 pulls the OBC-produced kfp-artifacts-bucket Secret via envFrom, but still reads its database credentials from the bundled MySQL's mysql-secret (I checked live). The same App also carries a daily backup CronJob that dumps both bundled MySQL servers (KFP and Katib) and ships them to a kubeflow-db-backups bucket on RGW. I only added it after noticing that neither database had a backup at all.

After Phase 1 there are no application pods yet, just CRDs, OBCs, Secrets and the gateway. That's on purpose. Phase 1 only lays the substrate.

Knative Serving and Eventing

Phase 2 is Knative. Serving is a KServe dependency (KServe's InferenceService Pod is built on Knative's Service CR). Eventing is in for completeness, in case a later workload wants Trigger / Broker to fan events out.

The deploy itself is uneventful; upstream's overlays land cleanly. The trouble is the CRD spec.conversion blocks. Some Knative CRDs (pingsources.sources.knative.dev, eventtypes.eventing.knative.dev) ship with webhookClientConfig set but spec.conversion.strategy missing. ArgoCD's ServerSideDiff dry-run rejects that with "Required value / Forbidden", and the App sits in ComparisonError, stuck on Unknown.

The fix goes in the AppSet template: argocd.argoproj.io/compare-options: ServerSideDiff=false. Falling back to the legacy 3-way merge skips the dry-run. You lose mutation-webhook injection in the diff, which I can live with because those webhooks are already in the broader ignoreDifferences block.

The second Knative trap: the controller rewrites the entire .webhooks[].rules[] array at runtime. It consolidates per-resource entries into wildcards, adds status subresources and uses apiVersion *, while the source manifest's rules block is a stub. Result: permanent OutOfSync. The fix is ignoreDifferences on the whole .webhooks array, scoped to the named Knative webhooks (webhook.serving.knative.dev, validation.webhook.serving.knative.dev).

A third one was on the risk register but never bit: the cluster-local-gateway. Knative needs an internal Istio gateway for in-cluster routing, and the homelab already had one in istio-system from an earlier deploy (30+ day uptime, helm-managed). Installing it again from the Knative manifests would fight the existing helm field-manager, so the Knative phase leaves the cluster-local-gateway out and relies on the one that's there.

The operators

Phase 3 is seven Apps at sync waves 4 and 5: admission-webhook, trainer, katib, pipelines (KFP), kserve, kserve-models-web-app and model-registry.

admission-webhook is the PodDefaults admission controller. It mutates pods at admission to inject volumes, env vars and secrets. Tiny footprint, no surprises, and it turns out to be the piece users notice most (more on that in the governance section).

trainer is Kubeflow Trainer V2 (formerly Training Operator V2). In 1.11 it's pure backend with no UI: the TrainJob / TrainingRuntime / ClusterTrainingRuntime CRDs and a controller manager. Users create TrainJob CRs from notebooks with the Python SDK (kubeflow.trainer).

Trainer has a kustomize trap. The upstream overlay sets app.kubernetes.io/part-of: kubeflow-training on both the Deployment's selector and the pod template. My overlay's labels: block (which stamps app.kubernetes.io/part-of: kubeflow-stack) with includeSelectors:false includeTemplates:true only touches the template, so selector and template disagree and the apiserver rejects the Deployment. The fix was to drop the labels: block from the trainer overlay.

katib is the hyperparameter optimization service: Experiment / Suggestion / Trial CRDs, a controller, a UI and a bundled MySQL backend. I left that MySQL alone. Katib only stores short-lived experiment metadata, and I saw no reason to put it on CNPG.

kserve and kserve-models-web-app are the inference layer. KServe's controller ships ~30 CRDs (InferenceService, InferenceGraph, ClusterServingRuntime and so on), and the Models Web App is a UI for listing and configuring InferenceService CRs. model-registry gets its own section further down.

The KFP heavy overlay

This is the most invasive overlay in the whole deploy. Upstream's applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user overlay assumes:

SeaweedFS for KFP artifacts: a bundled S3-compatible store that upstream exposes under the old minio-service Service name, so KFP's MinIO-era config keeps working. (Older releases shipped MinIO itself.)
MySQL for the ml-pipeline metadata database: bundled Deployment, Service and PVC.
A mlpipeline-minio-artifact Secret with hardcoded minio / minio123 creds.

I deleted the SeaweedFS workloads and replaced the Secret. MySQL is still running (more on that below).

SeaweedFS → RGW. Every KFP component that talks S3 now goes straight to RGW's public TLS endpoint, s3.georgehomelab.com:443. There are five of them:

ml-pipeline (the API server) and cache-server, through env vars;
pipeline-install-config (minioServiceHost);
the Argo artifactRepository;
the pipelines profile-controller's sync.py, through AWS_ENDPOINT_URL.

The first version of this overlay worked differently. It turned upstream's minio-service into an ExternalName pointing at the in-cluster RGW Service on plain port 80. That broke the day RGW went TLS-only and port 80 disappeared.

The ExternalName is still there, now pointing at s3.georgehomelab.com:443, but it's vestigial. Its only job is to stop upstream's minio-service, whose selector targets the SeaweedFS pod I deleted, from dangling. Nothing should depend on it. The minio-service.kubeflow* names were dropped from RGW's hosting.dnsNames, so a path-style request through that alias gets a 403.

Host and port alone aren't enough. ml-pipeline and cache-server read OBJECTSTORECONFIG_* straight from their env and ignore the ConfigMap's minioServicePort. Point them at port 443 without turning TLS on and they speak plain HTTP to a TLS port, and ensureBucketExists dies with upstream connect error or disconnect/reset before headers. The patch sets all three:

- name: OBJECTSTORECONFIG_HOST
  value: "s3.georgehomelab.com"
- name: OBJECTSTORECONFIG_PORT
  value: "443"
- name: OBJECTSTORECONFIG_SECURE
  value: "true"

The artifact Secret. Upstream ships mlpipeline-minio-artifact with minio/minio123 in it. If you leave that Secret in the kustomize render, VSO's overwrite: true only merges. The placeholder keys survive, KFP reads accesskey=minio, and RGW answers "access key ID does not exist".

So the overlay deletes the upstream Secret with $patch: delete, and VSO creates it fresh from Vault with only the two keys KFP reads, accesskey and secretkey. The KFP Deployments reference the Secret by name, so they don't care who created it.

MySQL → CNPG (planned, not live). The cluster still runs the bundled mysql Deployment, and pipeline-install-config still says dbType=mysql. Upstream ships no multi-user Postgres overlay and the data is small, so the daily dump to RGW closed the real gap (no backups at all) first.

The CNPG database and its credentials are provisioned, and the patch below is the plan. The ml-pipeline, cache-server, metadata-grpc-deployment, ml-pipeline-persistenceagent, ml-pipeline-scheduledworkflow, ml-pipeline-ui and ml-pipeline-viewer-crd Deployments all take the database from env vars, and each gets patched to use the CNPG-provisioned creds:

env:
  - name: DBCONFIG_USER
    value: kubeflow
  - name: DBCONFIG_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kubeflow-metadata-db
        key: password
  - name: DBCONFIG_HOST
    value: postgres-mlops-cnpg-cluster-rw.mlops.svc.cluster.local
  - name: DBCONFIG_PORT
    value: "5432"
  - name: DBCONFIG_NAME
    value: kubeflow_metadata_db

The kubeflow-metadata-db Secret is VSO-synced from secret/homelab/kubeflow/kubeflow-metadata-db. Once this lands, the bundled MySQL Deployment goes via $patch: delete.

The istio init race. ml-pipeline-api-server calls glog.Fatalf on its first S3 connection failure, with no retry. Without proxy.istio.io/config: holdApplicationUntilProxyStarts: true, the apiserver gets "connection refused" because istio-proxy hasn't opened its outbound listeners yet, the app exits, and you have a CrashLoop. The annotation makes the main container wait for the sidecar's readiness probe. The same trap comes back with Model Registry's golang-migrate flow.

The cold-boot race. The lab powers off every night, so every morning Kubeflow starts from nothing. KFP's scheduler came up before ML Metadata, saw that the daily run's slot had been missed, and fired it straight away. The run's first step needs ML Metadata, so it failed before doing any work. I added an init container to the scheduler that waits for ML Metadata, MySQL, the cache server, Kyverno and a healthy API server, then gives everything two more minutes.

The first version of that wait didn't wait at all. It opened a TCP connection to each Service, and inside the mesh that always succeeds: the pod's own Envoy sidecar accepts the connection, whether or not anything is behind the Service. I only caught it by pointing the check at a Service with no endpoints and watching it pass. The fix is traffic.sidecar.istio.io/excludeOutboundPorts on the scheduler pod, so those checks go straight to the real Service. On the next cold boot it held the scheduler for 220 seconds while ML Metadata came up, and the run that followed succeeded.

template:
  metadata:
    annotations:
      traffic.sidecar.istio.io/excludeOutboundPorts: "8080,3306,443"

A KFP run's artifact panel showing an s3://kfp-artifacts/... URI (served by Ceph RGW, not SeaweedFS), or the Ceph dashboard → Object Gateway → Buckets view with kfp-artifacts listed.

The "huge CRD" trap

Kubeflow CRDs are big. clustertrainingruntimes.trainer.kubeflow.org weighs ~280KB, trainjobs.trainer.kubeflow.org ~220KB, and the InferenceService and Knative ones aren't far behind.

The Kubernetes apiserver caps metadata.annotations at 262144 bytes (256KB). ArgoCD's default CSA-to-SSA migration writes a kubectl.kubernetes.io/last-applied-configuration annotation during the handover, and on these CRDs that annotation alone is >256KB. You get metadata.annotations: Too long: may not be more than 262144 bytes.

I stacked two fixes:

Per-CRD SSA via a kustomize patch. Put the argocd.argoproj.io/sync-options: ServerSideApply=true annotation on each problem CRD. ArgoCD then skips CSA and goes straight to SSA, which doesn't write the annotation.
App-level ClientSideApplyMigration=false. This turns the migration off entirely; it's set in the AppSet template's syncOptions. The CRD becomes argocd-controller-owned on first sync and stays SSA-managed.

Workspace UIs

Phase 4 is six Apps at sync wave 6: notebook-controller, jupyter-web-app, volumes-web-app, tensorboard-controller, tensorboards-web-app and pvcviewer-controller.

The web apps are SPAs and the controllers are headless. Between them, users can:

spawn JupyterLab notebooks from the Jupyter UI;
browse and create PVCs in the Volumes UI;
launch TensorBoard instances against TensorBoard log dirs;
view files in a PVC through the PVCViewer's per-PVC file browser.

Two integration points matter:

Every UI reads a USERID_HEADER env to find out who's making the request. Upstream's default is kubeflow-userid; I override it to X-Forwarded-Email (the Auth Chain Saga explains why).
Each UI's AuthorizationPolicy originally allowed requests only from cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account. Under a PERMISSIVE mesh that principal-based ALLOW matches nothing, so I replaced it with a path-based ALLOW.

The notebook-controller's ISTIO_GATEWAY env defaults to kubeflow/kubeflow-gateway, a stub Gateway in the kubeflow ns that isn't bound to any ingress IP in my setup. I override it to istio-system/kubeflow-gateway with a kustomize configMapGenerator using behavior:merge. The per-notebook VirtualService the controller emits then binds to the real ingress, and traffic to /notebook/<ns>/<name>/ routes correctly.

Ten surfaces read the user header: centraldashboard, kfam, manager, jupyter-web-app, katib-ui, kserve-models-web-app, volumes-web-app, tensorboards-web-app, ml-pipeline-ui and ml-pipeline-api-server. Most use USERID_HEADER; ml-pipeline-ui and ml-pipeline-api-server use KUBEFLOW_USERID_HEADER. Get it wrong on any single component and that UI answers "no user identity in header" on every request, even though oauth2-proxy has "authenticated" the user.

One more quirk. jupyter-web-app's /api/namespaces/<ns>/notebooks listing endpoint returns a 500 for any Notebook CR with a container that has no volumeMounts field. The web UI always adds a workspace volume to the notebooks it spawns, so you only see this with directly-applied CRs with bare containers. The bug is at utils.py:145, which does cntr["volumeMounts"] without a key check. Workaround: always add at least one volumeMount, even an emptyDir.

The Jupyter web app's notebook list in a Profile namespace, with a running GPU notebook (nvidia.com/gpu: 1) and its Connect button.

Two notebooks, one shared volume

A notebook's storage is only as useful as your ability to get at it from somewhere else. A teammate's notebook, a second kernel, a training job that needs the same dataset. So I tested the thing the Jupyter form quietly offers: a volume that more than one notebook can mount and write to at the same time.

The storage underneath is CephFS, through the ceph-filesystem storage class with ReadWriteMany. The default block class is ReadWriteOnce, which gives one pod its own disk and nobody else.

Create the first notebook with two shared volumes. In the "New notebook" form, untick "Use default class" on the workspace volume, pick ceph-filesystem and ReadWriteMany, then add one data volume the same way:

Volume	Storage class	Access mode	Size	Mounted at
`pod-nfs-shared-nb-workspace`	`ceph-filesystem`	ReadWriteMany	5Gi	`/home/jovyan`
`pod-nfs-shared-nb-shared`	`ceph-filesystem`	ReadWriteMany	10Gi	`/home/jovyan/pod-nfs-shared-nb-shared`

Ignore the "nfs" in the name. I named the notebook before I'd settled on the backend, and there is no NFS server in this cluster.

The "New notebook" form with the workspace volume set to ceph-filesystem and ReadWriteMany.

Check that both volumes are mounted, and that they really are two volumes. Open a terminal in JupyterLab:

df -hT /home/jovyan /home/jovyan/pod-nfs-shared-nb-shared

Filesystem                                                       Type  Size  Used Avail Use% Mounted on
csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-ebf4d59a-...  ceph  5.0G     0  5.0G   0% /home/jovyan
csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-198d5e31-...  ceph   10G     0   10G   0% /home/jovyan/pod-nfs-shared-nb-shared

Two different csi-vol IDs and two different sizes. It isn't one volume with a subfolder in it.

Write a marker file from the first notebook:

echo "written by $(hostname) at $(date)" > /home/jovyan/pod-nfs-shared-nb-shared/proof.txt
cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>

Create a second notebook and attach the same volume. I called it share-test. Under Data Volumes, choose "Attach existing volume" and pick pod-nfs-shared-nb-shared. The dropdown lists every claim in the namespace next to its access mode, which is a quiet but useful piece of UI: you can see which volumes are shareable before you try.

Read the file from the second notebook, then reply. Both notebooks are running:

cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt
echo "reply from $(hostname)" >> /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>

Back in the first notebook, read it again:

cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>
reply from share-test-0

Two lines, two hostnames, one file, and neither notebook was restarted in between.

The two JupyterLab terminals side by side, each showing both lines of proof.txt. Crop out the browser tabs and bookmarks bar.

Ask Kubernetes who is holding the claim:

kubectl -n terranetes describe pvc pod-nfs-shared-nb-shared | grep -A2 "Used By"

Used By:       pod-nfs-shared-nb-0
               share-test-0

Three things worth knowing before you hand this to a team:

You don't need to own the namespace. The kubeflow-edit role, which is what a contributor gets, can create notebooks and create and delete volume claims. kubeflow-view can only look.
The quota is what stops you. My Kyverno-generated quota gives each user namespace 10 claims and 100Gi. Every notebook with its own workspace eats into that.
Every notebook here asks for a GPU. With three cards in the cluster, a forgotten share-test notebook is a third of the GPU capacity doing nothing. Stop it when you're done.

Central Dashboard, Profiles and Kyverno governance

Phase 5 brings the portal and multi-user setup. Central Dashboard is the iframe-style UI portal. Its sidebar tile list lives in a ConfigMap that the centraldashboard pod reads at runtime through the K8s API. Tiles come in four kinds: menuLinks (iframed sub-UIs), externalLinks (opened in new tabs), documentationItems and quickLinks.

Profiles handle multi-user, run by the Profiles controller and KFAM (Kubeflow Access Manager). A Profile CR defines a user. The controller creates a namespace with the same name as the Profile's metadata.name, binds the appropriate ClusterRoles, generates a ns-owner-access-istio AuthorizationPolicy and registers the namespace with KFAM.

My Kyverno governance policies fire on app.kubernetes.io/part-of: kubeflow-profile namespaces and generate:

ResourceQuota (kubeflow-user-quota): cpu 4 requested / 16 limit, memory 8/16 Gi, storage 100 Gi, 20 pods, 10 PVCs. The CPU limit started at 8 and that was too tight: a running notebook holds about 2 CPUs of limit, and the pipeline's training Jobs were refused for the rest. A refused Job gets no pod and no error, only a FailedCreate event, so the step simply waited until it timed out. The pipeline step now reads those events and fails within five minutes with the quota's numbers in the message.
LimitRange (kubeflow-user-limits): default cpu 500m / mem 512Mi, max cpu 4 / mem 8 Gi.
Two NetworkPolicies: default-deny-all as the baseline, then allow-in-ns-and-kubeflow, which lets traffic in from the namespace itself and the Kubeflow control plane (kubeflow, kubeflow-system, knative-serving, istio-system), and out to that same control plane plus mlops (MLflow), rook-ceph (RGW) and DNS in kube-system.
Two CiliumNetworkPolicies: one allowing egress to the kube-apiserver (a plain NetworkPolicy can't express that reliably under Cilium) and one allowing egress to PyPI by FQDN, so pip install works in a notebook and nothing else on the internet does.

A separate Kyverno mutate policy, mutate-pss-privileged-kubeflow-user-ns, stamps every Profile namespace with PSS privileged at admission. User namespaces are istio-injected too, so istio-init needs the same capabilities there as in the control plane.

So user namespaces are bounded by default. A user can't reach anyone else's namespace, oversubscribe the cluster, or deploy anything bigger than the LimitRange allows. They can talk to the Kubeflow platform services, MLflow, Ceph RGW and PyPI.

The same policy file also generates three PodDefaults in every Profile namespace, and they're the best user-facing feature of the whole build. A PodDefault is a tick-box in the Jupyter spawn form: one for KFP access (a projected service-account token so the KFP SDK can call the API from a notebook), one for S3 (the Ceph RGW endpoint and bucket credentials as AWS_* env vars), and one for MLflow (MLFLOW_TRACKING_URI and the RGW endpoint). A user ticks "MLflow" and their notebook can log runs without anyone pasting a URL or a key into a cell.

Central Dashboard runs with REGISTRATION_FLOW=true, so a first-time user with no Profile gets a "pick a namespace" form instead of RBAC: access denied. I still pre-create Profiles in Git (22-user-namespace/profile-<alias>.yaml), because a Git-managed Profile carries the kubeflow-profile label the governance policies key on, and it's reviewed like any other change. KFAM provisions the namespace and RBAC, and Kyverno fires the generate policies.

The one in the repo looks like this (owner changed for the article):

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: kubeflow-user-george
  labels:
    app.kubernetes.io/part-of: kubeflow-profile
spec:
  owner:
    kind: User
    name: user@example.com
  resourceQuotaSpec: {}   # Kyverno generates this; leave empty here

That's the whole onboarding flow: add the user in Keycloak, add a Profile YAML, commit and push. ArgoCD reconciles, and by the next refresh the user has a namespace with quotas, limits, network isolation and PodDefaults.

Live, there are three Profiles: upstream's default kubeflow-user-example-com, the Git-managed kubeflow-user-george, and terranetes, which I created by hand early on and which isn't in Git yet. That last one is where the real ML work runs.

The Kubeflow login redirecting to the Keycloak realm login page (proof Dex is gone), then landing on the Central Dashboard.

The SSO front door

Phase 6, the last piece. The oauth2-proxy that gates the whole platform lives in the kubeflow-sso namespace and is an ordinary AppSet element, central-dashboard-sso, at sync wave 8. It was briefly parked while I re-verified the Keycloak realm and the Vault secret, and re-enabled a day later.

The overlay sits at kubernetes-addons/0-boostrap/oauth2-proxy/overlays/kubeflow-centraldashboard/. It builds on the base oauth2-proxy chart and adds:

a VirtualService for host kubeflow.georgehomelab.com, bound to both istio-system/kubeflow-gateway and istio-system/homelab-gateway (the overlapping-SNI story from Phase 1);
a multi-upstream config (below);
the cookie domain set to kubeflow.georgehomelab.com, with cookie name _oauth2_proxy_kubeflow;
a Redis session store, so the browser cookie is a small session ID rather than the whole token set (Failure 9 below explains why that matters);
a VaultStaticSecret syncing the client secret and cookie secret from secret/homelab/kubeflow/oauth2-proxy;
the oauth2-proxy-kubeflow-userid-header EnvoyFilter (Lua), which aliases X-Forwarded-Email → kubeflow-userid on outbound (see the Auth Chain Saga);
the oauth2-proxy-kubeflow-authz-path-normalize EnvoyFilter, which only matters for notebooks (see the notebook routing section).

The multi-upstream config is the heart of it. Central Dashboard iframes every sub-UI at a fixed path: /jupyter/, /katib/, /pipeline/, /kserve-endpoints/, /tensorboards/, /volumes/, /model-registry/. Each one is a separate backend Service, and oauth2-proxy has to know about all of them:

args:
  - --upstream=http://centraldashboard.kubeflow.svc.cluster.local:80/
  - --upstream=http://jupyter-web-app-service.kubeflow.svc.cluster.local:80/jupyter/
  - --upstream=http://katib-ui.kubeflow.svc.cluster.local:80/katib/
  - --upstream=http://ml-pipeline-ui.kubeflow.svc.cluster.local:80/pipeline/
  - --upstream=http://kserve-models-web-app.kubeflow.svc.cluster.local:80/kserve-endpoints/
  - --upstream=http://tensorboards-web-app-service.kubeflow.svc.cluster.local:80/tensorboards/
  - --upstream=http://volumes-web-app-service.kubeflow.svc.cluster.local:80/volumes/
  - --upstream=http://profiles-kfam.kubeflow.svc.cluster.local:8081/kfam/
  - --upstream=http://metadata-envoy-service.kubeflow.svc.cluster.local:9090/ml_metadata.MetadataStoreService/
  - --upstream=http://model-registry-ui-service.kubeflow.svc.cluster.local:8080/model-registry/
  - --pass-host-header=false

Upstream's per-component overlays each ship a VirtualService bound to kubeflow-gateway with hosts: ['*'] and prefix: /jupyter/ (and the equivalent for every other component). With my kubeflow-gateway bound to kubeflow.georgehomelab.com, those wildcard-host VSes collide with my specific-host oauth2-proxy VS. Istio's route selection picks the wildcard route, requests skip oauth2-proxy and hit the components directly with no auth headers, and you get an "RBAC: access denied" body with HTTP 403.

The fix is to delete every upstream hosts: ['*'] VS with a kustomize $patch: delete and let oauth2-proxy do the path-based routing.

Login (no Dex) view: People → kubeflow-gateway → oauth2-proxy → Keycloak → Google, then oauth2-proxy → Central dashboard → each app with X-Forwarded-Email.

The Auth Chain Saga

I'll be re-reading this section every time I deploy another OIDC-fronted, multi-component platform. It's the longest in the article because the debugging took the longest.

Here's the auth flow end to end. Every step is a place it can break:

One request from the browser to jupyter-web-app, with the failure points pinned to the stage each one hits.

The browser asks for kubeflow.georgehomelab.com/_/jupyter/?ns=..., and TLS terminates at the istio-ingressgateway.
Istio routes it to the oauth2-proxy-kubeflow VirtualService (catch-all prefix /).
oauth2-proxy checks the _oauth2_proxy_kubeflow cookie. No cookie means a 302 to /oauth2/start, the Keycloak login, the callback, and a fresh cookie. With a cookie, the request goes upstream.
With --pass-host-header=false, oauth2-proxy rewrites Host to the upstream service FQDN.
Istio sends the outbound request through an internal rewrite VS that strips /jupyter/ to /.
An EnvoyFilter Lua script reads X-Forwarded-Email and adds the kubeflow-userid header.
The request reaches the jupyter-web-app pod, which reads the USERID_HEADER env to find the identity header.
An Istio AuthorizationPolicy checks the path, and the request is handled.

That's eight steps, and between them they produced nine distinct failures.

Failure 1: the wrong upstream Host header. With the default --pass-host-header=true, oauth2-proxy forwards Host: kubeflow.georgehomelab.com to every backend. Istio's outbound listener looks up the vhost by Host header, and the public FQDN matches no in-cluster cluster, so traffic falls through to PassthroughCluster (raw TCP, no HTTP filters).

The rewrite VS never fires, the backend receives /jupyter/static/main.js unrewritten, and the SPA serves index.html (text/html) as a 200 OK. The browser gets HTML where it expected JS and shows a blank page. Fix: --pass-host-header=false, so Host gets rewritten to the upstream service FQDN.

Failure 2: the wrong header name (X-Forwarded-Email vs X-Auth-Request-Email). I first patched the Kubeflow components to read USERID_HEADER=X-Auth-Request-Email. That's the response header oauth2-proxy sets when it's answering an auth check. In proxy mode, which is what the dashboard path runs, the request going upstream carries X-Forwarded-Email (set by --pass-user-headers=true). A component reading the wrong header logs no user identity in request and every API call 401s. Login looks fine; the UI is blank.

If you spot X-Auth-Request-Email later in the notebook section, that isn't a contradiction. Proxy mode passes X-Forwarded-Email upstream; the ext_authz path only gets X-Auth-Request-Email, because that's the header the Istio provider copies onto the allowed request through headersToUpstreamOnAllow.

Failure 3: principal-based APs under a PERMISSIVE mesh. The upstream AuthorizationPolicies require principals: cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account. Under my PERMISSIVE mesh the principal isn't reliably populated, the AP rule doesn't match, and you get 403 RBAC: access denied. Fix: replace it with a path-based ALLOW (paths: ["/*"]).

Failure 4: KUBEFLOW_USERID_HEADER vs USERID_HEADER. Most components read USERID_HEADER. ml-pipeline-ui reads KUBEFLOW_USERID_HEADER, a different env name. And ml-pipeline-api (the backend) has its own KUBEFLOW_USERID_HEADER to override, which is the easy one to miss. The symptom: the KFP UI loads, but /apis/v1beta1/{pipelines,experiments,runs,recurringruns} returns "Request header error: there is no user identity header."

Failure 5: kustomize vs VSO on the secret. The oauth2-proxy base used to ship the Secret with placeholder values (REPLACE_VIA_SETUP_SECRET) for the client secret and cookie secret. Every sync put the placeholders back, VSO's next reconcile put the real values back, and in between oauth2-proxy was reading a placeholder. Keycloak answered Invalid client credentials on the callback.

The fix was to stop shipping the placeholders. The base generator now creates the Secret with client-id only, and VSO (create: false, overwrite: true) is the sole writer of client-secret and cookie-secret. With nothing in Git to revert to, the tug-of-war between ArgoCD and VSO stopped.

Failure 6: Vault sealed, so every secret is stale. The transit Vault was sealed, which left main Vault crashlooping and VSO unable to sync. Every Secret stayed at its last-known value, which at the time meant the placeholders from Failure 5. Unseal transit, main Vault recovers, VSO catches up. There's a second trap inside this one: kubectl exec -it ... vault operator unseal quietly sends empty stdin, so use the positional-argument form.

Failure 7: Katib only serves at /katib/. Most Kubeflow web apps serve at both / and /<prefix>/. Katib serves only at /katib/, and its index.html emits absolute asset paths (src="/katib/static/main.js"). The Istio rewrite VS that strips /katib/ → / broke it (404 at /). Fix: no rewrite VS for katib. oauth2-proxy forwards /katib/ to the backend unchanged.

Failure 8: the EnvoyFilter Lua reading the wrong header. The Lua filter that aliases the user email to kubeflow-userid (to satisfy Model Registry's hardcoded header) originally read x-auth-request-email. Same bug as Failure 2, in EnvoyFilter form. model-registry-ui returned 400 "missing required kubeflow-userid header" on every API call and the SPA showed "Error communicating with server". Fix: read x-forwarded-email.

Failure 9: HTTP 431, the cookie that was too big. This one hid for a long time because the dashboard shell always loaded. Notebooks, Endpoints, Tensorboards and Volumes didn't. Those four are Tornado-based web apps that cap request headers at about 8 KB, and a Keycloak session stored in the cookie (access, ID and refresh tokens) came to 12–16 KB. They answered 431 Request Header Fields Too Large before ever looking at auth.

I measured it by sending each upstream Service the same request with different cookie sizes: those four flipped to 431 above 8 KB, while the dashboard, Katib, KFP, KFAM and Model Registry stayed at 200. The fix was moving oauth2-proxy's session store to Redis, so the browser only carries a short session ID. The overlay's deployment-patch.yaml keeps the measurement table as evidence.

Each of those took me somewhere between a few hours and half a day to find. I wrote every one down as a trap note the day it was fixed, which is the only reason this section exists in this much detail.

The Model Registry UI listing a RegisteredModel + its versions, reached from inside a Profile namespace via the Kyverno-generated alias Service.

Adding Model Registry

Model Registry is five pieces, all in the kubeflow namespace: model-registry-deployment (REST + gRPC API server), model-registry-db (bundled postgres 16), model-registry-ui (SPA + BFF), model-catalog-server (catalog API) and model-catalog-postgres (catalog DB). It's a Wave-5 App, registered in the AppSet.

The deploy is three upstream overlays (overlays/postgres for core + postgres, options/ui/overlays/kubeflow for the UI in kubeflow mode, options/catalog/base for the catalog) plus kustomize patches for:

appProtocol: http on three Services (model-registry-ui-service, model-registry-service, model-catalog). Without it Istio routes them through PassthroughCluster, the same trap as Failure 1 in the Auth Chain Saga.
Deleting the upstream hosts: ['*'] VS (the collision pattern).
A path-based AP in place of the principal-based ALLOW.
holdApplicationUntilProxyStarts: true on model-registry-deployment and model-catalog-server. Both use golang-migrate, which sets schema_migrations.dirty=true the moment a migration starts. If the pod gets killed mid-migration (say, istio-proxy isn't ready yet), the next start aborts with Dirty database version N. Fix and force version. and CrashLoops forever.

The UI is the hard part. In --deployment-mode=kubeflow it scopes service discovery to the user's Profile namespace via SubjectAccessReview, then filters Services by:

label component=model-registry;
a port named http-api (or https-api); port 8080 alone isn't enough;
a valid ClusterIP (ExternalName Services are rejected);
and it hardcodes the auth header to kubeflow-userid, so it isn't USERID_HEADER-configurable.

The hardcoded header is what made the EnvoyFilter Lua essential, and it's how Failure 8 in the Auth Chain Saga finally got noticed once MR was deployed.

Then the discovery filter. My Model Registry is one shared instance in the kubeflow namespace. The UI looks for MR Services in the user's own Profile namespace, finds none, and shows the "No registries found" empty state.

The fix is a Kyverno generate policy that creates, in every Profile namespace, a Service named model-registry-service (ClusterIP, no selector, port name http-api, the right labels) plus an Endpoints object pointing at the cluster-wide MR's ClusterIP. Kube-proxy chains it: user-ns ClusterIP, DNAT to the kubeflow-ns ClusterIP, DNAT to the real pod IP. The shared MR's ClusterIP survives pod restarts (pod IPs don't), so the chain holds.

The policy lives in kubernetes-addons/security-stack/kyverno/policies/kubeflow-user-namespace-governance.yaml, next to the quota, limit, network and PodDefault generators. Its apiCall context resolves the kubeflow-ns ClusterIP at evaluation time, so the Endpoints carries the actual IP rather than a placeholder.

Wiring this turned up one more Kyverno trap. Kyverno's validate-policy admission webhook rejects generate rules if the kyverno-background-controller lacks RBAC for the target resource. For Services and Endpoints, that meant aggregating a new ClusterRole into the background-controller's compiled role with the label rbac.kyverno.io/aggregate-to-background-controller: "true". The AuthorizationPolicy generator in the notebook routing section needed the same treatment.

End-to-end test from a notebook pod over REST:

MR=http://model-registry-service:8080
# 1. Register the model
curl -X POST "$MR/api/model_registry/v1alpha3/registered_models" \
  -H "Content-Type: application/json" \
  -d '{"name":"iris-classifier", "owner":"user@example.com"}'
# RegisteredModel id=1

# 2. Add a version (registeredModelId required in body too)
curl -X POST "$MR/api/model_registry/v1alpha3/registered_models/1/versions" \
  -H "Content-Type: application/json" \
  -d '{"registeredModelId":"1", "name":"v1.0.0", "author":"user@example.com"}'
# ModelVersion id=2

# 3. Add the artifact URI
curl -X POST "$MR/api/model_registry/v1alpha3/model_versions/2/artifacts" \
  -H "Content-Type: application/json" \
  -d '{"artifactType":"model-artifact", "name":"iris-classifier-onnx",
       "uri":"s3://kfp-artifacts/sample-models/iris/v1/model.onnx",
       "modelFormatName":"onnx"}'

All three POSTs return 200 from a notebook pod in the user namespace, and the model shows up in the UI straight away. The SDK path doesn't need an SSO round-trip.

Notebook routing: from bypass to ext_authz

This is where the architecture changed direction halfway through.

The bypass. Per-notebook VirtualServices (emitted by notebook-controller) bind to istio-system/kubeflow-gateway and route /notebook/<ns>/<name>/ straight to the user's notebook pod. They have to, because oauth2-proxy's --upstream is a static prefix-to-host map, while the notebook URL has a dynamic <ns> and <name>. There's no fixed backend, and no wildcard upstream entry can work out where to send the request.

The easy route was to send notebook traffic directly through Istio and skip oauth2-proxy entirely. The per-notebook VS already existed and only needed binding to the right gateway. So I did that, and added a Kyverno-generated path-based AP in user namespaces to let the traffic in.

The problem. With that bypass, /notebook/<ns>/<name>/ URLs are reachable with no auth check at all. Anyone on the LAN who guesses the URL pattern can open anyone's notebook. Tolerable for a single-user homelab, not for multi-tenant.

The fix is ext_authz, using Istio's extensionProviders MeshConfig feature:

Register a new extensionProvider named oauth2-proxy-kubeflow in MeshConfig, pointing at oauth2-proxy-kubeflow.kubeflow-sso.svc:80 with pathPrefix: /oauth2/auth.
Create a CUSTOM AuthorizationPolicy on istio-ingressgateway (in istio-system), scoped to /notebook/* paths and calling the oauth2-proxy-kubeflow provider.
Add an EnvoyFilter Lua that turns ext_authz 401s into a 302 with Location: /oauth2/start?rd=<original-url>, so unauthenticated users bounce through Keycloak login and back to their notebook.
Tighten the Kyverno-generated user-ns AP from path-only to requiring the X-Auth-Request-Email header. That header is in headersToUpstreamOnAllow on the provider, so its presence proves the request went through ext_authz.

Where the pieces live:

MeshConfig: kubernetes-addons/0-boostrap/istio-service-mesh/istiod/...-values.yaml
CUSTOM AP + redirect EnvoyFilter: kubernetes-addons/Data-platform-stacks/kubeflow/04-istio-resources/notebook-ext-authz.yaml
Method-normalising EnvoyFilter: envoyfilter-authz-path-normalize.yaml in the oauth2-proxy overlay
Kyverno tightening: the same kubeflow-user-namespace-governance.yaml as before.

Two follow-ups came after the first version shipped. The redirect Lua originally rewrote every 401 on the ingress gateway into a login redirect, which was fine until I found it had been breaking Harbor image pushes (the Docker registry protocol relies on a real 401 challenge). It's now scoped to /notebook/ only.

The second was terminals and kernels. Envoy builds the auth check by appending the original path to /oauth2/auth and keeping the original method, so opening a terminal sent POST /oauth2/auth/notebook/... to oauth2-proxy. That missed the auth endpoint, fell through to the dashboard upstream, and came back 404, which ext_authz treated as a deny. The notebook UI loaded, but every terminal and kernel start failed. The path-normalize EnvoyFilter rewrites the method of those checks to GET, which the authenticated dashboard answers with a 200, so POST and PUT now pass.

After the change, the flow looks like this:

the notebook path after the pivot, with the ext_authz side call to oauth2-proxy.

The browser asks for /notebook/<ns>/<name>/lab, and the notebook VS matches at the istio-ingressgateway.
The CUSTOM AuthorizationPolicy fires: the gateway makes an ext_authz call to oauth2-proxy's /oauth2/auth, forwarding the cookie.
oauth2-proxy answers 200 OK, and X-Auth-Request-Email is attached to the upstream request.
The request goes over mTLS to the notebook pod.
The user-namespace AP allows it only when X-Auth-Request-Email is present.
The notebook serves /lab.

So the build is hybrid: proxy mode for apps with static URLs, ext_authz for apps with dynamic ones. Going all-in on ext_authz would mean an extra hop on every request, losing cookie isolation and rebuilding the UX. A single uniform pattern isn't worth that.

Central Dashboard tiles: MLflow

The dashboard tiles live in a ConfigMap that centraldashboard reads at runtime through the K8s API (not a volume mount). At the time, two facts made updating it annoying:

Kyverno's make-configmaps-immutable policy marked every new CM immutable, so you couldn't kubectl edit or kubectl apply your way to new data.
The kubelet treats immutable as "cache forever", so even if you get past Kyverno somehow, pods that mount the CM (centraldashboard doesn't, other components do) won't see the update.

What I did instead:

Create a separate CM (centraldashboard-tiles) with the merged tile list (Kubeflow defaults plus the MLflow external link). A different name from centraldashboard-config means I can manage it independently.
Patch the centraldashboard Deployment's DASHBOARD_CONFIGMAP env to point at the new CM.
Generate the CM with disableNameSuffixHash: true (a stable name) and the argocd.argoproj.io/sync-options: Replace=true,Force=true annotation.

Replace=true,Force=true is the pair that makes it work. Replace=true alone runs kubectl replace -f, which is still an UPDATE and gets rejected on immutable resources. Replace=true,Force=true runs kubectl replace --force -f, which deletes and recreates, and that works. It took me an hour to find. The error (field is immutable when immutable is set) pointed clearly enough at the cause that I expected the obvious fix to work. It didn't.

The kubeflow namespace is now excluded from that Kyverno policy (Profile namespaces too, through a namespaceSelector), so new ConfigMaps there stay editable. ConfigMaps that were already made immutable stay frozen, which is why the replace-and-force pattern is still worth knowing.

MLflow is the only external link in the sidebar. There used to be an Airflow tile too, but Airflow was retired, so it went.

The Central Dashboard sidebar showing the MLflow external link.

How reliable is it?

The cluster is HA. Kubeflow on it mostly isn't, and I'd rather say that than imply otherwise. Here's what the live cluster shows:

Layer	What runs	What that means
Kubernetes control plane	Five masters, etcd on each, kube-vip for the API address	Survives losing a master
The front door	`oauth2-proxy-kubeflow` with 2 replicas, sessions in Redis	Survives losing one proxy pod; nobody gets logged out
Kubeflow itself	33 Deployments in `kubeflow`, one replica each, no PodDisruptionBudgets	A lost node means a reschedule and a short gap for the apps on it
GitOps	24 ArgoCD Applications, all Synced and Healthy	Drift is put back by ArgoCD's self-heal within seconds
KFP and Katib databases	The bundled MySQL, one pod each, dumped to Ceph RGW every day	The latest daily dump succeeded; a restore loses at most a day
The daily training run	A KFP RecurringRun every morning	The last three runs succeeded. The failures before them were the namespace quota and the cold-boot race, both fixed above
Cold boots	The whole lab, every morning	The scheduler now waits for its dependencies instead of firing into a half-started platform

The single replicas are a choice for a one-person homelab, not an accident: the Kubeflow controllers reconcile from Kubernetes state, so a restart costs a gap, not data. The databases are the part I'd fix first, which is why the CNPG move is at the top of the list below.

Kubeflow traps that will bite you

If you're putting Kubeflow on a cluster that isn't the reference one, these are the ones I'd check first:

Set --pass-host-header=false on oauth2-proxy. The default forwards the public Host, Istio's vhost lookup fails, traffic lands in PassthroughCluster, and you get blank pages with 200s.
Delete every upstream hosts: ['*'] VirtualService. They outrank your specific-host route and send requests around the proxy with no identity.
Replace principal-based AuthorizationPolicies with path-based ones under a PERMISSIVE mesh, or everything is RBAC: access denied.
Set the user header on all ten surfaces, and remember KFP's UI and API read KUBEFLOW_USERID_HEADER, not USERID_HEADER.
Keep the session out of the cookie. Keycloak tokens push it past the ~8 KB header limit of the Tornado-based UIs and you get HTTP 431.
Huge CRDs need per-CRD ServerSideApply and ClientSideApplyMigration=false, or the 256KB annotation limit stops the sync.
Every istio-injected namespace needs PSS privileged, Profile namespaces included.
Watch for overlapping wildcard Gateways on the same port. If another Gateway's *.domain server covers your host, bind your VirtualService to both or live with intermittent 404s.
Don't trust a TCP check inside the mesh. The pod's own sidecar accepts the connection, so "the port is open" proves nothing. Exclude those ports from the sidecar, or check at the HTTP level, and test the check against a Service with nothing behind it.
Size the Profile quota for notebooks and pipelines together. A Job the quota refuses never gets a pod and never errors; it just waits.

Conclusion

Here's what's running today:

Five master nodes, HA control plane, and three RTX A1000 GPUs available to notebooks and workloads.
Knative Serving + Eventing.
Kubeflow admission webhook, Trainer V2 (jobset-controller + trainer-controller), Katib, KFP (api-server + ui + persistenceagent + scheduledworkflow + viewer-crd + metadata-grpc + metadata-envoy + cache-server + workflow-controller), with a daily backup of KFP's and Katib's MySQL to RGW.
KServe controller + LocalModel controller + Models Web App, serving a real model: football-match-outcome in the terranetes namespace, retrained and redeployed by a daily XGBoost KFP RecurringRun.
Notebook controller + Jupyter Web App + Volumes Web App + TensorBoard controller + TensorBoards Web App + PVCViewer controller.
Central Dashboard + Profiles (kfam + manager containers) + per-user namespace governance and PodDefaults via Kyverno generate policies.
Model Registry + Model Registry UI + Model Catalog server + their two postgres Deployments.
oauth2-proxy-kubeflow in the kubeflow-sso ns, plus the per-component config (multi-upstream + pass-host-header=false + Redis sessions + EnvoyFilter Lua + ext_authz for /notebook/*).
Three Profiles: the upstream default, kubeflow-user-george from Git, and the hand-made terranetes.
The MLflow external link in the Central Dashboard sidebar.

How that served model gets trained, gated and promoted is its own story, in my separate article on the EPL match-prediction pipeline.

What isn't done yet:

KFP metadata on CNPG (still the bundled MySQL, backed up daily).
More than one replica for the Kubeflow components (see the reliability table).
Spark on Kubeflow (I use the mlops-spark-operator from the MLOps stack instead of Kubeflow's bundled overlay; they share infrastructure).
A real multi-tenant test. The Profiles exist, but I'm the only person using them, so isolation between two real users hasn't been exercised.
terranetes in Git. It works, but it's the one Profile that would vanish in a rebuild.
Network Policies on the Kubeflow control plane (user namespaces have them via Kyverno; the platform namespaces are open).

If I did this build again, I'd change five things:

Start with the auth chain, not finish with it. In the plan, Phase 6 was the front door. In practice it's where most of the debugging time went. Knowing the shape of the auth chain up front would have shaped every component overlay: the USERID_HEADER choices, the AP structure and the EnvoyFilter Lua would all have been settled in advance instead of fought one component at a time.
Build the EnvoyFilter Lua early. The Lua that aliases X-Forwarded-Email to kubeflow-userid is what holds cross-component auth together. I built it once, hit the wrong-header bug, fixed it, and now it works. Building it on day 1 with the right header would have saved half a dozen "why doesn't X have a user" sessions.
Write the trap notes as I go, not afterwards. Most of my notes on this build were written after the bug was fixed. Writing them during the debug, even as rough notes, would have made the next bug quicker to find.
Test with more than one user earlier. A single user (me) hides every cross-namespace issue. The Kyverno generate policies, the MR alias services and the user-ns governance have only ever been exercised by one person.
Lean on what I already knew about the substrate. At the start I had notes covering Vault, Keycloak, oauth2-proxy patterns, Istio STRICT-vs-PERMISSIVE and Kyverno, and none covering Kubeflow. Most of the surprises in this build came from the substrate, not from Kubeflow itself, so the old notes were more useful than I expected.

Practical takeaways if you're putting Kubeflow Platform 1.11 on a non-reference substrate:

Pin to v1.11.0, not master. A reproducible build beats the newest commit.
Design the auth chain first. Settle the OIDC provider, the proxy mode, the header names, the session store and the AP shape before you patch a single component overlay.
PSS=privileged on every Kubeflow namespace, user namespaces included. istio-init needs CAP_NET_ADMIN/NET_RAW wherever the sidecar is injected, and Profile namespaces are injected too. I enforce it with a Kyverno mutate policy so a new Profile can't come up restricted and fail every pod.
Disable the ClientSideApply migration at the AppSet level. Kubeflow CRDs are too big for the 256KB annotation limit.
Put Kyverno governance on Profile namespaces from the start. The generated quota, limits, network policies, PodDefaults, Model Registry alias and notebook AP all live in one governance file. Bolting them on afterwards is much harder than having them in place at Phase 5.
Write the traps down. The next platform with substrate swaps will hit a good share of the same ones.

What's Next?

The main thing I took from this build: a platform's reference architecture is a starting point, and you don't have to keep it. Kubeflow ships Dex, SeaweedFS and MySQL because it has to ship something. Each of them can be swapped out and wired to the stacks you already trust. Brownfield costs more up front, and in return the platform fits the cluster you already run.

The model-serving loop is already working, and it's covered in the EPL match-prediction pipeline article. What's left on this platform, roughly in order:

Move KFP metadata to CNPG. The database and credentials are waiting; the work is writing the Postgres patches upstream doesn't ship for multi-user mode.
Onboard a second tenant and test isolation properly. A second real user, with their own Profile in Git, trying to reach my namespace, my notebooks and my models, and failing.
Wire up OpenLineage so every pipeline run shows its inputs and outputs in a lineage graph, not just in MLflow tags.

If you're facing Kubeflow on a cluster that already has opinions of its own, the swap table at the top is a good place to start.

Follow me on LinkedIn

Author: George Ezejiofor

Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

George Ezejiofor — Fri, 16 Oct 2026 23:23:18 +0000

Self-Hosted Databricks Alternative: Homelab Data Lakehouse

Part of the homelab build series. Trino, Apache Iceberg, Project Nessie, dbt, Spark, MLflow
and OpenMetadata on RKE2, with highly available state on Ceph and CloudNativePG, secrets from
Vault and sign-in through Keycloak. Raw data comes in at one door, gets built and inspected
on the line, and leaves as tables and models I can trust. Nobody sends an invoice.

Introduction

A factory you rent by the hour, or one you own

Raw data is worth about as much as raw steel. Nobody wants it until someone has cut it,
joined it, checked it and stamped it.

Snowflake, Databricks and BigQuery will rent you an excellent factory for that, and the
meter runs for as long as the lights are on. I wanted the factory without the meter, so I
built one on three mini-PCs in my homelab, on the same RKE2 cluster as everything else in
this series. Every station is a real open-source tool doing a real job.

It's built to make one thing: machine-learning models trained on data I can name down to
the exact version. One is already serving, a football predictor that a scheduled pipeline
refits and re-tests every day.

Here is the floor, in the order the material moves:

The loading dock: Ceph object storage. Everything that arrives lands here.
The stock ledger: Project Nessie. Every table is an Apache Iceberg table, and Nessie records every version it has ever had. It's Git for data.
The assembly line and quality control: dbt. Tables are built and tested on a branch, so a batch that fails inspection never reaches the shelf.
The front counter: Trino. Anyone with access asks for what they need in plain SQL.
The showroom: Superset. Charts and dashboards over the finished tables, for people who'd rather not write SQL. It asks for data in their name, never its own.
The records office: OpenMetadata. It traces any table back to its raw material.
The machine shop: Kubeflow. Notebooks, pipelines and hyperparameter searches, on GPUs when a job needs them.
Finished goods: MLflow and KServe. Every run is tracked, and the model that passes its checks is served over HTTP.

The utilities were already in the building: Ceph, CloudNativePG, Vault, Keycloak, Istio and
ArgoCD.

It's highly available where it holds state. Postgres runs three instances, one in each
datacenter, the ledger runs two replicas and search runs three nodes. Trino, Superset,
OpenMetadata and MLflow run a single replica each, and I come back to that below. One bay is still
empty: a Hermes AI copilot that will read the lineage over OpenMetadata's MCP
interface, waiting on a DGX Spark that hasn't arrived.

If you know Databricks, you already know the layout:

Databricks	This homelab lakehouse
Databricks Runtime (Spark)	Kubeflow spark-operator (installed, not on the daily line yet)
Delta Lake	Apache Iceberg
Unity Catalog	Nessie (catalog + Git-for-data) + OpenMetadata (governance/lineage)
Databricks SQL / Photon	Trino
dbt	dbt (dbt-trino)
Managed MLflow	MLflow
Notebooks	Kubeflow notebooks / Jupyter
Workflows	dbt/OpenMetadata CronJobs + Kubeflow Pipelines RecurringRuns
S3 / ADLS	Ceph RGW
SSO / SCIM	Keycloak
Databricks Assistant / agents	Hermes `mlops-copilot` over OpenMetadata MCP (designed, not running yet)

It's an open-source equivalent, not Databricks, which is a product and a trademark, and
there's no affiliation. This is the second design: the first used Hive Metastore and
Airflow, and I say below why I replaced both.

The brief

The brief I set myself: an MLOps and big-data platform on the existing homelab RKE2 cluster
that a real data team could plausibly use. The homelab's existing backends (Rook-Ceph,
CloudNativePG, Vault, Keycloak) had to thread through every component, and each piece had
to land in GitOps and come back after a node reboot with nobody touching it.

What landed is two tiers. The lakehouse: Ceph RGW object storage, Apache Iceberg
tables, a Project Nessie catalog (state in CNPG nessie_db), Trino from Stackable's
coordinated 25.3.0 release, dbt-trino for tested marts, and OpenMetadata for
governance, with the Kubeflow spark-operator installed for the day the data outgrows
SQL. MLOps on top: MLflow Tracking and the Model
Registry, backed by CNPG mlflow_backend_db and the mlflow-artifacts bucket, behind
oauth2-proxy for the browser and reachable directly over in-cluster DNS from Kubeflow
notebooks.

The constraints that shaped every decision:

No HDFS, no ZooKeeper. Ceph RGW is my S3, and nothing in the lakehouse needs ZK. The RAM saving alone justified leaving both out; the complexity avoided mattered more.
No bundled databases. Every chart wants to ship its own MySQL or Postgres or Mongo. I ripped each one out of the lakehouse and MLflow charts and pointed them at CNPG. There is one Postgres cluster for this stack (postgres-mlops-cnpg-cluster, three instances across three DCs, each on its own zone-local volume) and it hosts the lakehouse and MLflow databases. (Kubeflow is the exception; it keeps its bundled MySQL, katib-mysql and Model Registry's Postgres for now.)
No v1/Secret literals in Git. Every credential (application user passwords, OAuth2 client secrets, OIDC cookie keys) flows from Terraform into a Vault path, then into a VaultStaticSecret CR, then into the destination namespace. Kustomize may declare a placeholder Secret to keep references valid, but VSO always wins on the data values.
Mirror production patterns, even when it's harder. Real OIDC, real mTLS, real RBAC. No "developer convenience" backdoors.

The rest of this article covers how each piece lands, what I cut, and which
traps cost the most time.

The architecture

If the word is new, the lakehouse is the third step in a short history. A warehouse
gives fast SQL over the tables you planned for, in a format you rent. A lake keeps any
file cheaply but has no table rules, so teams copy the data into a warehouse anyway and
end up with two versions of the truth. A lakehouse puts the table rules (Iceberg) and
the versions (Nessie) on the open files themselves, so every engine reads one copy.

Warehouse, lake, lakehouse: a warehouse sells storage and engine together, a lake plus a warehouse keeps two copies, and the lakehouse built here keeps one copy of open files in Ceph with Iceberg tables and Nessie versions read by Trino, dbt, Kubeflow and MLflow

Here is how this one is put together:

The data factory:** dbt, Trino, Nessie, Kubeflow, MLflow and KServe on the line, Superset and OpenMetadata above it behind Keycloak, and Ceph, CloudNativePG, Vault, Istio and Argo CD underneath

With five guided views: the query path, the daily pipeline, governance, train-and-serve,
and where secrets come from. Blog platforms can't embed the HTML, so host it and link it.

Kubeflow appears here as one station, the machine shop; how I changed it from upstream
is in the SSO section. Read the diagram bottom-up, because that's the order the
dependencies run:

Foundation. Ceph RGW holds every byte of table data (s3://mlops-data-lake, TLS-only at s3.georgehomelab.com). One CloudNativePG cluster holds the lakehouse and MLflow stores (nessie_db, mlflow_backend_db, openmetadata_db). Vault + VSO hand out every credential.
Catalog. Project Nessie answers "where does table X live, and at which commit?" over Iceberg REST, versioning the whole catalog like Git.
Query and transform. Trino reads and writes the Iceberg tables; dbt builds and tests the marts through Trino every morning, on a Nessie branch dbt-audit that only merges into main if the tests pass.
Govern. OpenMetadata 2.0.2 (search on its own OpenSearch 3.4.0) crawls Trino at 10:00 and ingests dbt's artifacts at 10:15. That's where the lineage comes from. People reach it through Keycloak (Google or GitHub).
Consume. Superset 6.1 gives people charts, dashboards and a SQL editor over Trino. It signs people in through Keycloak and asks Trino to run each query as the person, so Trino's rules and its audit table see a name, never a shared login.
Train and serve. MLflow tracks runs in Postgres and models in RGW; a KServe InferenceService serves the promoted model.

Everything in the foundation layer predates this work. Most of my effort went into
making sure the upper layers didn't reinvent any of it. The one deliberate exception is
OpenMetadata's own OpenSearch; the going-live section explains why that stopped being
optional.

Stackable operators, and what I dropped

Stackable ships a coordinated release of operators for Hadoop-ecosystem services. It's
tempting to take all of them. I only took the parts that fit:

Stackable component	Use it?	Why / Why not
`commons-operator`	yes	Hard prerequisite: ships `S3Connection` / `AuthenticationClass` CRDs that every workload operator references.
`listener-operator`	yes	Hard prerequisite: ships `ListenerClass` CRDs the workload operators reference.
`secret-operator`	yes (minimal)	Hard prerequisite for CRD presence, because the workload admission webhooks reference `SecretClass`. The chart's default `tls` SecretClass issues Trino's internal TLS. No credentials go through it.
`trino-operator`	yes	The SQL query engine over Iceberg. It was "deferred" in v1; the lakehouse pivot promoted it to the primary query surface.
`hive-operator`	no (retired)	Was the metastore/catalog; Project Nessie replaced it (Iceberg REST + Git-for-data).
`airflow-operator`	no (retired)	Orchestration dropped. dbt CronJobs and Kubeflow Pipelines handle scheduling. One fewer stateful service to babysit.
`spark-k8s-operator`	no	I run the Kubeflow spark-operator instead: same `SparkApplication` shape, already deployed for the ML platform.
`zookeeper-operator`	no	Nothing in the lakehouse needs ZK.
`hdfs-operator`	no	Ceph RGW is my S3. HDFS would just duplicate storage and chew RAM.
`kafka-operator`	deferred	No broker runs today; both Strimzi and Redpanda are parked. Skip until there's a real streaming workload.

The Stackable operators pin to 25.3.0 and release as a set, so I bump them as a set
(never Trino alone). Project Nessie and OpenMetadata are not Stackable products; they
deploy from their own Helm charts, the price of a best-of-breed catalog and governance layer.

The secret-operator needs explaining. The workload operators' admission webhooks
reference the SecretClass CRD, and without it workload CR creation fails. So I install
the operator but create no SecretClass of my own. The chart's default tls SecretClass
issues Trino's internal (autoTls) certificates, and the dbt and ingestion pods mount its
CA to verify them. Credentials never go through it; they're plain v1/Secrets synced by
VSO from Vault.

Stackable workloads also brought the first Istio trap. Under STRICT mTLS (the cluster
default), the API server's webhook calls can't reach the operator pods, because the API
server has no mesh client cert. The fix is a PeerAuthentication: PERMISSIVE in the
mlops namespace. Don't turn off injection; relax the mode for that namespace and
nothing else.

Storage: Rook-Ceph all the way down

The homelab already runs Rook-Ceph with four storage classes: ceph-block (RBD, RWO,
three copies, the default for notebook PVCs and most stateful pods), ceph-block-pg-local
(RBD, one copy, pinned to a datacenter, for Postgres only), ceph-filesystem (CephFS, RWX,
only when a workload really needs shared scratch) and ceph-bucket (RGW, S3), the data lake.

RGW is TLS-only. Clients use https://s3.georgehomelab.com,
which terminates on the Istio gateway with a public Let's Encrypt certificate (no private
CA to distribute), and path-style addressing still works. The old plain-HTTP
…rook-ceph.svc:80 endpoint is gone.

Buckets come from Rook's ObjectBucketClaim CR. The OBC controller provisions the bucket
on RGW and creates a per-bucket Secret with AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY keys, named after the claim: Spark reads mlops-data-lake, MLflow
reads mlflow-artifacts, KFP reads kfp-artifacts. Nothing fans out from a single root key.

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: mlops-data-lake
  namespace: mlops
spec:
  bucketName: mlops-data-lake
  storageClassName: ceph-bucket

Apply that and the bucket, its owner user and the Secret all exist, ready for envFrom.

The Ceph dashboard → Object Gateway → Buckets: mlops-data-lake, mlflow-artifacts and kfp-artifacts, each with its own OBC-issued owner.

Databases: one CNPG cluster, many logical databases

The easy path is to leave every chart's postgresql.enabled: true on and accept the
extra StatefulSets. I didn't: backups, upgrades, monitoring and HA belong on one tier, not
nine. The homelab's CNPG pattern already runs Keycloak, SonarQube, Tekton and others, so
one more cluster meant copying an existing manifest (trimmed here; the real one adds
managed roles, affinity and resources):

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-mlops-cnpg-cluster
  namespace: mlops
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:17
  bootstrap:
    initdb:
      database: mlops_placeholder       # real DBs come from Database CRs
      owner: postgres
      secret:
        name: mlops-cnpg-bootstrap      # VSO-synced from Vault
  storage:
    storageClass: ceph-block-pg-local   # one Ceph copy, in the instance's own DC
    size: 40Gi
  postgresql:
    parameters:
      max_connections: "200"
  backup:
    retentionPolicy: "7d"
    barmanObjectStore:
      destinationPath: "s3://cnpg-backups/postgres-mlops-cnpg-cluster"   # Ceph RGW

Three instances, one per DC, automatic failover, and continuous WAL archival to RGW.

The storage class is worth a sentence. Postgres already keeps three copies of the data,
one per instance. On the default ceph-block, Ceph kept three copies of each of those,
nine in all, and every commit waited on a write to another datacenter. So Postgres now
sits on ceph-block-pg-local: one Ceph copy, in the same datacenter as the instance.
Postgres does the replicating and Ceph just holds the disk. A 4K synced write on a
database node went from about 34 ms to between 15 and 24 ms. The price is that losing a
drive loses that instance, and the operator has to clone a new one from the primary.
CNPG's Database CR (its per-logical-DB declaration) provisions one logical database per
app, each with its own owner role whose password lives at its own Vault path:

Vault path	Owner of
`secret/homelab/mlops/cnpg-bootstrap`	`postgres` superuser (used only by `initdb`)
`secret/homelab/mlops/nessie-db`	`nessie_db` (Project Nessie catalog version store)
`secret/homelab/mlops/mlflow-db`	`mlflow_backend_db`
`secret/homelab/mlops/openmetadata-db`	`openmetadata_db` (OpenMetadata knowledge graph)
`secret/homelab/kubeflow/kubeflow-metadata-db`	`kubeflow_metadata_db` (lives in Kubeflow scope; provisioned, not yet used by KFP)

The lakehouse pivot added nessie_db and openmetadata_db with a short Database CR
each, and no new stateful tier. The old hive_metastore_db and airflow_metadata_db
(and their hive / airflow roles) are retained but no longer used: their Database
CRs set databaseReclaimPolicy: retain, so nothing drops them by accident.

Each app sees only its own DB, with its own creds. The superuser never lands on
an application Pod.

kubectl get database.postgresql.cnpg.io -n mlops: one Database per app on the single postgres-mlops-cnpg-cluster, all APPLIED true, including the retained (unused) hive_metastore_db and airflow_metadata_db.

Two Istio nuances are specific to CNPG. First, its istio-injected pods cycle their
sidecars whenever Istio config changes, so the READY column bounces between 1/2 and
2/2 and the operator reports READY < INSTANCES. Postgres is fine; it's sidecar churn.
Don't triage CNPG; bump the sidecar resources or ignore it.

Second, any istio-injected namespace must run at PSS=privileged, because istio-init
needs NET_ADMIN/NET_RAW. CNPG's initdb Job lives in such a namespace. If PSS is
baseline, the Job goes FailedCreate with no obvious cause. That's why the mlops
namespace is PSS=privileged.

Secrets: Vault all the way down

Vault is the single source of truth for credentials, and Terraform is the only thing that
writes to it. The chain for any credential: Terraform reads the value from a gitignored
.secrets.auto.tfvars and writes a Vault KV-v2 path; a VaultStaticSecret CR names
that path, the Vault role and the Secret to produce; VSO reconciles it hourly into a
plain v1/Secret (optionally reshaping keys with a template); the workload mounts it.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: mlops-mlflow-db
  namespace: mlops
spec:
  vaultAuthRef: mlops-mlflow
  mount: secret
  type: kv-v2
  path: homelab/mlops/mlflow-db
  refreshAfter: 1h
  destination:
    name: mlops-mlflow-db
    create: true
    overwrite: true

Don't drop the overwrite: true. If the same Secret is referenced from both a kustomize
secretGenerator (with placeholder data) and a VSO sync, every kubectl apply -k briefly
stomps VSO's data with the kustomize placeholders. With overwrite: true on the VSS, VSO
puts the real values back on its next reconcile. Without it, oauth2-proxy gets
unauthorized_client from Keycloak and you spend two hours blaming the realm import.

Each app has its own ServiceAccount, its own Vault role (Kubernetes auth, policy scoped to
its own paths) and its own VSS, so no credential crosses an app boundary.

The Vault UI → secret/ → homelab/mlops/: the list of per-app paths (mlflow-db, nessie-db, openmetadata-db, …). ⚠️ Show the path list only; .

SSO: per-app oauth2-proxy with Keycloak upstream

Every UI in this stack sits behind Keycloak, using the standard
Authorization-Code-with-PKCE flow. For MLflow: the Istio gateway routes
mlflow.georgehomelab.com to its oauth2-proxy, which looks for the _oauth2_proxy_mlflow
cookie. Without it, the browser goes to
https://keycloak.georgehomelab.com/realms/mlflow/protocol/openid-connect/auth?..., signs in
with Google, and comes back to /oauth2/callback. oauth2-proxy validates the token, sets the
cookie, and from then on reverse-proxies to mlflow.mlops.svc.cluster.local:5000 with
X-Forwarded-Email added.

Each app gets its own realm too: MLflow's proxy uses realm mlflow, and OpenMetadata
(which speaks OIDC natively) uses realm openmetadata.

The decision that isn't obvious is running one oauth2-proxy per app rather than one
shared proxy. One shared proxy means one shared cookie. A leaked MLflow session becomes a
leaked session for OpenMetadata, Kiali, Hubble UI and every other gated app at once.
Per-app instances cost a little RAM (oauth2-proxy is ~30 MiB) and buy real session
isolation. Each overlay sets OAUTH2_PROXY_COOKIE_NAME (_oauth2_proxy_<app>) and
OAUTH2_PROXY_COOKIE_DOMAINS (its own host). The base values use a default that collides,
so without the override the first app to set a cookie "captures" it and later apps return

The per-app client secret follows the usual chain: Terraform writes Vault, a VaultStaticSecret syncs it into the namespace, and oauth2-proxy reads it via envFrom.

Two more lessons came out of this. In oauth2-proxy's proxy mode (which I use for
MLflow), the user email reaches the upstream as X-Forwarded-Email, not
X-Auth-Request-Email. The latter is a response header for auth_request-style
integrations. Get it wrong and the upstream app sees no user identity even though
authentication "worked."

And Keycloak doesn't substitute environment variables at realm import. The realm JSON
includes the oauth2-proxy client, but its secret is the literal placeholder string
${TOOL_OAUTH2_PROXY_CLIENT_SECRET}. Every new realm needs a post-import
kcadm update clients/<id> -s secret=<actual> patch with the same value that's in Vault.
post-deploy-fixes.sh automates it, but "Invalid client credentials" on a fresh realm
still catches you out the first time.

The Keycloak sign-in page for the mlflow realm, with the "Sign in with Google" and "GitHub" buttons. Every UI in this stack sends you to a page like this first.

Brownfield Kubeflow: the same pattern, applied to a stack that ships its own auth

Upstream Kubeflow ships its own login stack: Dex as the identity provider, with
oauth2-proxy acting as an Istio ext_authz check in front of it. I don't run Dex at all.

What changed, checked against the live cluster:

Dex → Keycloak. Login goes Google → Keycloak realm kubeflow → oauth2-proxy (kubeflow-sso/oauth2-proxy-kubeflow, 2 replicas) in reverse-proxy mode, with one --upstream per app. The upstream per-app hosts: ['*'] VirtualServices are deleted.
The user header. Every app reads X-Forwarded-Email instead of upstream's kubeflow-userid. That's nine separate places to patch, and all of them must agree.
Authorization. The AuthorizationPolicies are path-based (paths: ["/*"]) instead of principal-based, because the mesh is PERMISSIVE. /notebook/* is gated by a CUSTOM ext_authz policy on the gateway.
Istio. The cluster's shared Istio, with its own istio-system/kubeflow-gateway and a Let's Encrypt certificate.
Storage. KFP artifacts go to an ObjectBucketClaim on Ceph RGW instead of the bundled SeaweedFS/MinIO, with the secret from Vault via VSO; every PVC is on ceph-block.
GPUs. The NVIDIA gpu-operator runs three RTX A1000 cards, and notebooks claim one each.
Still upstream. KFP and ML Metadata run on the bundled MySQL (a CNPG database exists for them but nothing uses it yet), Katib keeps katib-mysql, and Model Registry keeps its own Postgres. The bundled MySQL is at least backed up now: a daily CronJob dumps it to its own bucket on RGW.

The catalog: Project Nessie (Iceberg REST + Git-for-data)

The catalog is the small service everything else depends on. It remembers that "table events_curated
is an Iceberg table whose current metadata lives at s3://mlops-data-lake/warehouse/...,
with these columns, these snapshots, these partitions." Replacing the Hive Metastore with
Project Nessie was the main reason for the pivot.

Why Nessie over HMS:

It's an Iceberg REST catalog, the modern standard. Trino, Spark and dbt all speak Iceberg REST; HMS's Thrift is the legacy path.
Git-for-data. Nessie versions the entire catalog: branches, tags, commits, merges. You can cut a branch, run an ETL job, validate it, and merge atomically, which gives you multi-table transactions HMS can't do. On an ML platform that means reproducible data: tag the exact catalog state a model trained on. Both halves are in real use: dbt builds on a branch, and the football model trains on a tag (more on both below).
Credential vending. Nessie holds the S3 creds and vends them to the engines when they load a table. (Without STS on Ceph, "vending" hands out the same bucket-owner key, and as the going-live section shows, vending alone isn't enough for Trino to create a table.)

Nessie deploys from its own Helm chart (charts.projectnessie.org), not Stackable. Its
version store is Postgres (CNPG nessie_db), and its warehouse is the same
mlops-data-lake bucket Spark writes to:

versionStoreType: JDBC2
jdbc:
  jdbcUrl: "jdbc:postgresql://postgres-mlops-cnpg-cluster-rw.mlops.svc:5432/nessie_db"
  secret:
    name: mlops-nessie-db
    username: username
    password: password
catalog:
  enabled: true
  iceberg:
    defaultWarehouse: warehouse
    warehouses:
      - name: warehouse
        location: s3://mlops-data-lake/warehouse
  storage:
    s3:
      defaultOptions:
        endpoint: https://s3.georgehomelab.com
        pathStyleAccess: true
        accessKeySecret:
          name: mlops-data-lake
          awsAccessKeyId: AWS_ACCESS_KEY_ID
          awsSecretAccessKey: AWS_SECRET_ACCESS_KEY

Trino reaches it at http://nessie.mlops.svc:19120/iceberg. Spark uses
org.apache.iceberg.nessie.NessieCatalog against .../api/v2 and can create and merge
branches from a job.

The Nessie UI on main: the commit log and the namespaces (raw, raw_football, marts_staging, marts_marts, marts_football) that dbt and Trino have created.

Why Nessie and not lakeFS? Nessie already gives the Iceberg tables Git-style
branches, tags and atomic multi-table commits; a second version-control layer over the
same tables would be two sources of truth for one question. lakeFS earns its place for
unstructured files (I have none yet), and its Iceberg REST catalog, SSO and RBAC are
paid-edition features. So I'll revisit it as a replacement, never an addition, if
Nessie stalls or unstructured data shows up.

Garbage collection. Versioning has a cost that's easy to miss: every dbt build and
pipeline publish replaces table snapshots, and the old files stay forever unless something
removes them. Nessie's separate GC tool runs as a weekly CronJob (Sundays, 13:00 UTC).

It marks the files still reachable (30 days of history per branch, and just the tagged
commit for each train-football-… tag) and treats the rest of each table's files as
candidates. It only looks under Iceberg table locations, so the landing Parquet is out of
its reach. It runs with --defer-deletes for now, recording candidates in nessie_db
until I've reviewed a few runs. The first walked 3 references and 131 commits and found 8
candidates, all old table metadata.

A daily tag-retention job sits in front of it, so the pins don't pile up either. It
keeps any tag a Model Registry version names, plus the last 30 days, and is also in
dry-run mode for now.

MLflow: tracking and model registry

MLflow is the simplest part of the stack to deploy and the one I use most day to day.
The chart is the official one from the MLflow repo, vendored at mlflow/charts/mlflow
(chart 0.1.0, appVersion 3.12.0). It takes a Postgres backend and an S3 artifact store.

values.yaml excerpt (trimmed):

mlflow:
  backendStoreUriFrom:
    secretKeyRef:
      name: mlops-mlflow-db          # VSO builds the `uri` key from Vault
      key: uri
  defaultArtifactRoot: mlflow-artifacts:/       # clients use the proxy scheme
  artifactsDestination: s3://mlflow-artifacts/  # the server does the S3 I/O

env:
  - name: MLFLOW_SERVER_ENABLE_JOB_EXECUTION
    value: "false"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: mlflow-artifacts
        key: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: mlflow-artifacts
        key: AWS_SECRET_ACCESS_KEY
  - name: MLFLOW_S3_ENDPOINT_URL
    value: https://s3.georgehomelab.com
  - name: MLFLOW_S3_IGNORE_TLS
    value: "false"

Artifacts are proxied: clients log to mlflow-artifacts:/ and the server writes to the
bucket with its own OBC credentials, so notebooks only need the tracking URI, not S3 keys.
The MLFLOW_S3_ENDPOINT_URL override points the server at RGW instead of AWS. The
endpoint has a public certificate, so MLFLOW_S3_IGNORE_TLS stays false: TLS is
genuinely verified. Path-style addressing works because the host is in the
CephObjectStore's hosting.dnsNames.

MLFLOW_SERVER_ENABLE_JOB_EXECUTION=false is there because MLflow 3.x's job subsystem is
on by default and starts eight huey consumers, which ate about 1.5Gi while completely idle
and kept OOM-killing the pod. I don't use server-side jobs, so turning it off beat raising
the limit.

The MLflow UI sits behind an oauth2-proxy in the mlflow-sso namespace, with the
per-app cookie scoping from the SSO section. Inside the cluster (a Kubeflow notebook
in a kubeflow-user-* ns), the tracking client talks to the service directly:

import mlflow
mlflow.set_tracking_uri("http://mlflow.mlops.svc.cluster.local:5000")
mlflow.set_experiment("homelab-iris-v1")
with mlflow.start_run():
    mlflow.log_param("lr", 1e-3)
    mlflow.log_metric("val_acc", 0.97)
    mlflow.sklearn.log_model(model, "model")

There's no SSO round-trip on the SDK path; any pod the Network Policy allow-lists can reach
the in-cluster service. Only the browser UI goes through Keycloak.

The MLflow UI → experiment football-match-outcome: the daily runs with wf_logloss and deploy_approved as columns.

Trino, dbt and OpenMetadata: query, transform, govern

v1 had an Airflow tier here, which the pivot deleted. Airflow isn't bad, but on a homelab
with one operator a persistent scheduler is a lot of surface area for what's really three
jobs a day. In its place is a query/transform/govern tier that does more and needs less
babysitting.

Trino: SQL over Iceberg. The Stackable TrinoCluster runs coordinator-only: workers
are scaled to 0 and node-scheduler.include-coordinator is true, so the coordinator
runs the splits itself. No app node has room for a 4Gi/1-CPU worker right now; workers
come back with more hardware. Trino queries Iceberg through the Nessie REST catalog. In
25.3.0 the first-class iceberg connector models the Hive metastore, so the REST wiring
goes through the generic connector.

Nessie vends credentials for reads, but Trino also needs its own S3 keys to create
tables (more on that in the going-live section). They come from the Rook bucket Secret via
valueFromSecret, never as literals:

spec:
  connector:
    generic:
      connectorName: iceberg
      properties:
        iceberg.catalog.type:
          value: "rest"
        iceberg.rest-catalog.uri:
          value: "http://nessie.mlops.svc:19120/iceberg/"
        iceberg.rest-catalog.vended-credentials-enabled:
          value: "true"
        s3.endpoint:
          value: "https://s3.georgehomelab.com"
        s3.path-style-access:
          value: "true"
        s3.aws-access-key:
          valueFromSecret:
            name: mlops-data-lake
            key: AWS_ACCESS_KEY_ID
        s3.aws-secret-key:
          valueFromSecret:
            name: mlops-data-lake
            key: AWS_SECRET_ACCESS_KEY

dbt: tested transformations. dbt-trino turns raw Iceberg tables into modeled,
tested, documented marts. It runs as a CronJob at 09:30 UTC. The project ships
as a kustomize-generated dbt-project ConfigMap (no git clone at runtime), and a
non-root container runs dbt build. dbt's not_null, unique, accepted_values and
accepted_range tests are the data-quality gate; there's no separate Soda/GE.

The dbt profile needs no secret at all. dbt proves who it is with a short-lived
Kubernetes token for its own service account, mounted into the pod and sent as a JWT,
and Trino's rules then decide what dbt-runner may read and write (how that works is
the subject of the RBAC article in this series). It also verifies Trino's certificate
against the Stackable tls CA, mounted per pod. After each run it publishes manifest.json, catalog.json and
run_results.json to a small CephFS volume for OpenMetadata.

dbt writes on a branch, then publishes (write-audit-publish). At first, each
dbt build landed in main as roughly 15 separate Nessie commits (__dbt_tmp renames,
__dbt_backup drops), so a query at the wrong moment could catch a half-built mart. Now
the 09:30 CronJob does three things:

It resets a Nessie branch, dbt-audit, to main's head through Nessie's REST API v2.
It runs dbt build --target audit through a second Trino catalog, audit, whose REST URI ends in .../iceberg/dbt-audit/. Nessie reads the ref from the URI path, so the same tables appear on a different branch. (It's audit and not dbt_audit because Kubernetes names can't contain _, and Stackable uses metadata.name as the Trino catalog name.)
Only if the build passes does it merge dbt-audit into main.

main now gets one atomic "Merged dbt-audit" commit per build. A failure leaves main
untouched and the branch sitting there for me to inspect. A normal run ends PASS=23
(3 models and 20 tests) and merges cleanly.

spec:
  connector:
    generic:
      connectorName: iceberg
      properties:
        iceberg.catalog.type:
          value: "rest"
        iceberg.rest-catalog.uri:
          value: "http://nessie.mlops.svc:19120/iceberg/dbt-audit/"

A few details kept this from leaking into the rest of the stack. OpenMetadata's manifest
still comes from dbt docs generate --target prod, so lineage keeps its iceberg.*
names, and its Trino crawl only includes ^iceberg$, so the audit catalog never
shows up as a duplicate. The dbt runner got an Istio AuthorizationPolicy
rule that allows only /api/v2/trees*, not the Iceberg REST path that vends S3
credentials. And because adding a Trino catalog restarts the coordinator under Stackable,
I rolled it out between the scheduled runs, not during one.

Bronze, silver, gold. The tables fall into the three layers Databricks calls a
medallion architecture: bronze is data as it lands, silver is typed and cleaned, gold is
tested and ready to use. Here every layer is a set of Iceberg tables, and the gate
between them is the dbt-audit branch. OpenMetadata labels each table with its layer
(its Certification field), so a reader can filter the whole catalog down to Gold.

This is ELT rather than the warehouse's ETL: data is loaded first, almost as it arrives,
and transformed afterwards inside the lakehouse. Keeping the raw layer is what makes that
safe: when a dbt model is wrong, I fix the model and rebuild from bronze, without fetching
anything again.

Bronze, silver, gold: raw_football.matches and raw.events are bronze, stg_events is silver, the two marts are gold; dbt builds silver and gold on the dbt-audit branch, the football model trains on bronze at a pinned tag, and people read gold through Trino and Superset, with lineage in OpenMetadata.

Two details I'd rather show than smooth over. Football has no silver step: dbt builds
the gold mart straight from the raw table. And the football model trains on bronze,
at a pinned tag, because it needs the exact rows it saw, not a cleaned view that could
change underneath it. So raw_football.matches is Bronze and Tier 1 at the same time:
the layer says how refined a table is, and the tier says how much depends on it.

OpenMetadata → Explore → Data Assets: Table, Certification: Gold. The two gold marts with their badge, tier and owner.

OpenMetadata: governance, lineage and MCP. The top layer is a metadata knowledge
graph that crawls Trino and ingests dbt's artifacts to produce end-to-end lineage,
descriptions and data-quality results. It runs 2.0.2, with CNPG for its store, its
own 3-node OpenSearch 3.4.0 for search (operator-managed, on the database node pool),
and Keycloak realm openmetadata for OIDC; login redirects straight to Keycloak's
Google/GitHub page.

Ingestion runs as two CronJobs (om-ingest-trino at 10:00, om-ingest-dbt at
10:15), not the bundled Airflow (I'm not bringing back what I just removed). They
authenticate with the ingestion-bot token Terraform writes to Vault. OpenMetadata is in
this homelab at all because it ships an MCP server, which lets the Hermes agents query
certified, lineage-aware data in natural language. The rest of the stack is building
towards that.

OpenMetadata → marts_marts.daily_active_users → Data Quality tab: the dbt tests with their latest results.

Spark: installed, and not on the line yet

Every lakehouse diagram has Spark in the middle, so I should be straight about mine. The
Kubeflow spark-operator is installed and has run test jobs, but nothing on the daily line
uses it. My tables are small and Trino builds them in seconds, so Spark on that line would
be a forklift carrying one box. It joins in its own article, on a job that needs it:
real-time fraud detection with Kafka, once the cluster has more memory. It will read and
write the same Iceberg tables through Nessie, so nothing about the catalog or the storage
has to change.

SPARK JOB DEMO WITH SPARK CUSTOM IMAGE ON TEKTON AND HARBOR BUILT THROUGH SECURED CI/CD SUPPLY CHAIN

How reliable is it?

A factory that stops when one machine does isn't much of a factory. Here's what this one
survives, and how I know:

What goes wrong	What happens	How I know
A Postgres instance, or the node under it, is lost	Another instance is promoted and the apps reconnect through the same service name	After a cold boot the cluster promoted a different instance and kept serving; all six databases stayed applied
A database is damaged or dropped	It can be restored to a point in time from object storage	A nightly base backup plus continuous WAL archiving, both reporting healthy, with seven days kept
A catalog pod dies	The other replica answers; the ledger itself is in Postgres	Nessie runs 2 replicas and keeps no state of its own
A search node dies	The other two carry on, and the index can be rebuilt from Postgres anyway	OpenSearch runs 3 nodes, one in each datacenter
A dbt build fails its tests	`main` is untouched and the failed batch stays on its branch for inspection	One "Merged dbt-audit" commit per passing build, and nothing on a failing one
The lab is powered off overnight	Jobs are scheduled in the day, and a missed slot is skipped, not replayed at boot	Each job has a ten-minute start deadline; the build, crawl and lineage jobs all ran on schedule after the last cold boot
A new model is worse than the one serving	It is logged but not promoted; the current model keeps serving	Every run records `deploy_approved` in MLflow, and the served revision only changes on a pass
A new model passes the gate but misbehaves once served (Canary deployment strategy with Kserve)	It is tested at 0% of traffic, then at 10% and 50%, and traffic returns to the old model if it does worse	A rollout forced to fail put all traffic back on the old model and failed the run

And the honest limits:

Four stations are single replicas. Trino, Superset, OpenMetadata and MLflow each run one pod. If its node dies the pod is rescheduled and that station is down for a minute or two. Nothing is lost, because their state lives in Postgres and object storage.
A demoted Postgres primary doesn't always come back by itself. After that same cold boot, the old primary couldn't rejoin: its history had diverged from the new primary's. The database ran on two instances until I deleted the stale one's volume and pod. The operator then cloned a fresh replica in the same datacenter and had it streaming within a few minutes. Failover is automatic; that repair was not.
Each Postgres volume is one Ceph copy. That was a deliberate trade for write speed, covered in the databases section. Losing a drive loses an instance, not data.
Snapshot cleanup is still deferred. The tag clean-up deletes for real now, but the catalog's garbage collector only records which files it would remove and deletes nothing yet.
The daily training run can lose a race with the morning boot. If it starts before the cluster has settled, it fails and waits for the next day. About half of the recent scheduled runs went that way, which is next on my list to fix. A rerun by hand takes about six minutes.

kubectl get cluster.postgresql.cnpg.io -n mlops: postgres-mlops-cnpg-cluster with 3 instances, 3 ready, and its current primary.

The traps I hit

Everything above was designed before it ran. Once it ran, it met a cluster that powers
off at night and defaults that fail without an error. Three traps shaped the platform
enough to belong here. The rest, from an operator chart pointing at an image that was
never published to a catalog that would hand storage credentials to any pod that asked,
have their own article in this series.

Trino could query, but never create a table. Every CREATE TABLE failed:

Failed checking new table's location: s3://mlops-data-lake/warehouse/raw/...
  -> Failed to list location
  -> SdkClientException: Unable to load credentials from any of the providers in the chain

Trino checks a new table's location with its own credentials before Nessie gets
involved, and it had none. That gap alone is why dbt had never built anything. The fix
gives Trino the same bucket key Nessie already vends (so no new access).

The cron that never fired. dbt-build had been scheduled for 03:00 UTC for eleven
days and never ran once, with no error, event or Job. Thanos had the answer: the lab
powers off overnight, and there isn't a single metric sample at 03:00 in the retention
window. Every CronJob moved into daytime (09:30 dbt → 10:00 Trino crawl → 10:15 dbt
lineage), with startingDeadlineSeconds so a missed slot is skipped, not replayed at boot.

A quota on limits can block a model rollout. Every Kubeflow profile namespace gets a
ResourceQuota, generated by a Kyverno policy, and mine capped CPU limits at 8. A KServe
rollout runs the old and the new model server side by side until the new one is ready.
Each asks for 2.5 CPU of limit, and one of those is the Istio sidecar's. Add one running
notebook and the pipeline step itself and the total is 8.2. So the new revision was
refused, the deploy step waited fifteen minutes and timed out, and the run failed with a
perfectly good model sitting in the bucket. The symptom is a revision stuck on
FailedCreate with exceeded quota in its events and no pod to look at. I raised the
limit quota to 16 and left the request quota at 4, because requests are what reserve
real capacity. Then the change hit a second trap: ArgoCD reported the policy synced while
the live object was still on its first generation with the old value. Check
metadata.generation on a policy before believing a sync.

One day on the factory floor: every schedule in the lakehouse in the order it fires, from the morning power-on to the afternoon backups

End-to-end workflow

From a data scientist's seat, the platform looks like this:

Ingest (on a branch). A job writes new raw data as an Iceberg table on its own Nessie branch, checks it, and merges the branch into main in one commit, so nobody ever reads a half-loaded table. The football pipeline below ingests exactly this way. The other raw table, raw.events, is a seeded demo table with no scheduled ingest.
Transform and test. The 09:30 dbt-build CronJob runs raw.events → marts_staging.stg_events (view) → marts_marts.daily_active_users (Iceberg) on the dbt-audit branch, so a failed test never reaches main. The doubled-up schema names are dbt's default (custom schema appended to the target schema); overriding generate_schema_name would move the live tables and orphan their OpenMetadata history, so they stay.
Query. Anyone with Trino access runs ad-hoc SQL over the marts (SELECT * FROM iceberg.marts_marts.daily_active_users), something v1 couldn't do without Spark in the loop.
Train. A Kubeflow notebook reads the curated Iceberg table (via Trino or the Spark Nessie catalog), trains a model and logs to MLflow:

   mlflow.set_tracking_uri("http://mlflow.mlops.svc.cluster.local:5000")
   mlflow.sklearn.log_model(model, "model", registered_model_name="event-clf")

Because the data lives on a Nessie tag, the run is reproducible: months later you
can point back at the exact catalog state. That works through Nessie (pyiceberg reads
the table at the tag). Trino's catalogs are each pinned to one branch, so a SQL query
sees the current version, not last month's.

Register and govern. The best run is promoted in MLflow's Model Registry, and OpenMetadata's 10:00 and 10:15 CronJobs stitch the lineage together. Humans browse it today; the Hermes agents will query it over MCP once the copilot lands.
Serve. A KServe InferenceService serves the promoted model (the football match-outcome predictor is the first one live).

The worked example: a football model trained on a pinned snapshot

From match results to a served model: football-data.org to landing Parquet, published to an Iceberg table through a Nessie branch, then Katib, the promotion gate, KServe, MLflow and the predictions site

A KFP RecurringRun fires at 10:00 London time every day. A Job in mlops, the only
namespace with internet egress, pulls Premier League results into s3://mlops-data-lake
as Parquet. The lakehouse part is the publish step:

It writes iceberg.raw_football.matches on a fresh Nessie branch.
It audits that branch before anything else sees it: match ids are unique, finished matches have scores, every season is present, and the finished-match count never shrinks.
It merges into main in one commit and tags the commit train-football-<UTC>.

Katib and the training step read exactly that snapshot, and MLflow and the Kubeflow Model
Registry record the tag, the commit and the Iceberg snapshot id beside the model. dbt's
football_matches model reads the same table.

The model is modest, and I'd distrust anything better: a walk-forward log-loss of 1.050
against 1.082 for always guessing the usual split, over 587 matches, and the right result
47% of the time. A new model has to pass a promotion gate, a smoke test and a canary at
10% then 50% of traffic before it serves. The full story, from the features to the rollout,
is its own article in this series.

epl-predict.georgehomelab.com: the matchday predictions with the scorecard above them, and the goals tables with their "does not beat that base rate" line.

This workflow is how I know the integration points actually work: S3 endpoints resolve,
Nessie answers Iceberg REST, dbt tests gate the marts, MLflow logs to Postgres and S3, and
a notebook in a different namespace reaches everything over in-cluster DNS.

Superset's SQL Lab is a Trino SQL client

Apache Superset is best known for charts and dashboards, but it also ships SQL Lab: a SQL editor in the browser. Point it at Trino and it becomes a SQL client for the whole lakehouse, with no desktop tool to install and no password to hand out.

Here it signs you in through Keycloak, then sends each query to Trino as you, not as a shared Superset account. Trino applies the same per-person rules it applies everywhere else, and the query lands in the audit table under your email.

Apache Superset client connected to Trino, running SELECT * FROM iceberg.marts_marts.daily_active_users and returning rows: the query v1 couldn't run without Spark.

What I deferred

A list, with reasons. Most of these deserve an article of their own.

The Hermes data copilot. The AI-native part, an agent that reads the catalog, lineage and data over OpenMetadata's MCP and flags tables it can't trust, is waiting on the DGX Spark for local model inference.
Generic MLflow → KServe promotion. One model is served today through its own pipeline. The general controller ("watch the registry for a promoted version, materialize an InferenceService") is still to write.
Trino workers. Trino runs coordinator-only, which is plenty for tables this size. Workers come back with more hardware.
Soda/Great Expectations. dbt tests cover data quality today; these stay deferred until they aren't enough.
Feast feature store. It waits until a model needs online inference with sub-100ms feature retrieval.
Streaming, and Spark with it. No broker runs today: Strimzi and Redpanda are both parked, and Spark has no job. They come back together in their own article on real-time fraud detection with Kafka and Spark, once the hardware is there.
Multi-tenant MLflow and a Spark History Server. Not worth it for 1.x users; kubectl logs on the driver pod covers most debugging.

Conclusion

This build cost about three weekends of focused work, and most of the hours went on the
traps, not the happy paths.

What I got back is close enough to production that the lessons carry over: real OIDC with
per-app cookies, RBAC enforced in the data plane, continuous WAL archiving and
three-instance Postgres, Vault behind every secret, GitOps behind every change, and metrics
from day one.

What I'd tell anyone trying this on their own metal:

Say no to bundled defaults. Every chart wants to ship its own database, storage and auth, and every "yes" is technical debt.
One source of truth per concern. Vault for secrets, CNPG for relational state, Rook-Ceph for storage, Keycloak for identity. Two of any of these is an integration problem, not a feature.
The hard part is the seams, not the install. Nessie → Postgres → Vault, MLflow → RGW, oauth2-proxy → Keycloak → Google. That's where the documentation is thinnest.
Defer aggressively, then ship when the workload arrives. Trino was deferred until I needed ad-hoc SQL, and then it pulled Nessie, dbt and OpenMetadata in with it.
Don't be afraid to rip out something everything depends on. GitOps turned the Hive + Airflow swap into a series of reviewable commits instead of a rebuild.

The part I lean on most is Git-for-data. dbt publishes each build to main as one tested
commit, and the football model trains on a tagged snapshot that MLflow and the Model
Registry both remember. It stopped being a slide.

The ArgoCD UI filtered to mlops-* & kubeflow-*: the whole lakehouse (Trino, Nessie, dbt, OpenMetadata, MLflow, CNPG) Synced and Healthy.

What's Next?

The next milestones, and probably the next few articles:

The Hermes copilot, once the DGX Spark lands: local inference, read-only verification first, proposed writes later.
Streaming with Kafka and Spark: real-time fraud detection, with card transactions in and a model trained in Kubeflow out. This is the job that puts Spark on the line, and it waits for more hardware.
Generic MLflow → KServe promotion, so any registered model can be served without its own bespoke pipeline.
Turning the garbage collector on for real: Nessie GC without --defer-deletes, after a few weeks of clean logs. Tag retention already deletes.
Unstructured data, probably arriving with the DGX Spark. That's the point where I revisit lakeFS, as a replacement for Nessie and not as a second layer on top of it.

If you want to see how those go, including the parts that break, follow along.

Follow me on LinkedIn for the next posts in this series.

Author: George Ezejiofor

Cilium Hubble on RKE2: Microservice Traffic Observability (Seeing the Invisible)

George Ezejiofor — Mon, 12 Oct 2026 20:40:33 +0000

Part of the homelab build series. Every packet in my cluster already passes through Cilium's eBPF datapath on all 29 nodes. Hubble turns that into a live map of who talks to whom, with no sidecars and no code changes. This is how I switched it on for RKE2, the two traps that crash-looped it, and the uncomfortable thing I found when I went looking for the HTTP metrics.

The Hubble UI service map for one namespace, drawn live from real traffic.

Introduction

The map you already had and couldn't see

Ask anyone to draw their cluster's network and you'll get a confident whiteboard sketch that's quietly wrong.

The frontend talks to the API, the API talks to Postgres, everything talks to DNS. Sure. But which pod is really hammering that service? Was that "connection refused" a crashed pod, a missing endpoint, or a NetworkPolicy nobody remembers writing? The real traffic graph, the one packets actually follow, is invisible. So we debug it the old way: kubectl exec into a pod, run tcpdump, squint, and guess.

Here's the thing I'd missed for months. My CNI is Cilium. It's eBPF, it is the datapath, and it was already seeing every single packet on every node. I just wasn't looking.

Hubble is how you look. It turns Cilium's datapath into a stream of flows that say "kubeflow/notebook → kube-system/coredns:53, FORWARDED", not "10.42.3.7 → 10.42.1.9". No sidecar, no code change, and no app knows it's being watched.

So I switched it on. Within an afternoon I had a live service map behind Keycloak, hubble observe streaming every flow on the cluster, and network-policy decisions I could read instead of guess. It also cost me a week of debugging, and it taught me exactly where its vision stops.

What Cilium and Hubble actually are

Two layers, one on top of the other:

Cilium is the CNI. It wires pod networking and enforces NetworkPolicy with eBPF programs in the kernel rather than iptables rules. Every workload gets a stable identity derived from its labels, not its IP, which is why its telemetry reads like Kubernetes instead of a wall of addresses. Here it's version 1.19.1, bundled with RKE2, using a VXLAN overlay alongside kube-proxy.
Hubble is the observability built on top. Each cilium-agent exposes its flows on a socket. Hubble Relay gathers every node's stream into one cluster-wide feed. Hubble UI draws it as a live map.

You get three ways in: the CLI (hubble observe, grep for the network), the UI (the live service map), and Prometheus metrics (flow rates, drops, DNS).

The mental model I use: Hubble is tcpdump with Kubernetes identity, across the whole cluster, at almost no cost. It sees layers 3 and 4 for everything. Layer 7 is a different story, and I'll get to it.

How Hubble sees the cluster: cilium-agent on every node, Relay in the middle, the UI behind oauth2-proxy and Keycloak.

How eBPF sees a packet

"It's eBPF" gets said a lot and explained rarely, so here is what happens to one packet.

First, what it is. eBPF is a feature of the Linux kernel, not a product. Cilium is one of many tools built on it.

eBPF lets you load a small program into the running Linux kernel and attach it to a hook. The kernel checks the program before it runs, so it can't crash the node or loop forever. Cilium attaches its programs to the network hook on each pod's virtual interface, the veth pair that joins the pod to the node. Nothing can enter or leave a pod without passing one.

When a packet arrives, the program does three things, all inside the kernel:

It works out who is talking. It looks up the source and destination addresses in an eBPF map, a hash table shared between the kernel and the agent, and gets back two identities. An identity is a number that stands for a set of labels, so app=orders in one namespace is one identity however many pods carry it.
It decides. A second map holds the policy for that pod: which identities may reach which ports. One lookup gives the verdict, forward or drop. That is the difference from iptables, which walks a list of rules until one matches and so gets slower as policies grow.
It reports. The program writes a short event into a buffer the agent reads: the two identities, the ports, and the verdict. The packet itself is never copied out of the kernel.

Hubble is what reads those events. The agent turns the identities back into pod names and labels, and keeps the result as a flow. So the traffic flow you see in hubble observe is the same decision the kernel made, written down as it was made, not a second system guessing from the outside.

That is also why the service map draws itself. Cilium sees every connection, and it already knows which pod owns which address because it handed the addresses out. Nothing has to be registered or annotated: an edge appears on the map because traffic passed. The other side of that is that a service which hasn't talked recently has no edge.

Two details explain the cost. Cilium doesn't report every packet. By default it reports when a connection opens, changes state or closes, which is why a node pushing a lot of traffic still produces only a few hundred flows a second. And nothing is added to the pod: no sidecar, no extra network hop, no restart.

The same word, different tools. If you've used OpenTelemetry's eBPF instrumentation, this is not that. Both load programs into the kernel, but they attach in different places. Cilium hooks the pod's network interface and sees packets. OpenTelemetry's instrumentation hooks the application process and sees requests: method, path, status and duration, read before the application encrypts them. So one can block traffic and is blind inside TLS, and the other only observes but can see inside a service mesh. "It uses eBPF" tells you how a tool collects its data, not what data it collects.

That design also sets the limit. The kernel program reads addresses, ports and TCP flags. It does not read what is inside the connection. An HTTP path or a status code is layer 7, and for that Cilium has to hand the connection to a proxy.

Switching it on in RKE2

RKE2 ships Cilium as a bundled chart, so you don't install Cilium yourself. You layer values onto it with a HelmChartConfig called rke2-cilium in kube-system, and RKE2's helm controller merges and re-applies it within about 30 seconds. My whole Hubble setup is one file. Here's the heart of it:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium          # must match the bundled release
  namespace: kube-system
spec:
  valuesContent: |-
    hubble:
      enabled: true
      relay: { enabled: true, replicas: 1 }
      ui:    { enabled: true, replicas: 1, ingress: { enabled: false } }
      metrics:
        enabled:
          - dns
          - drop
          - tcp
          - flow
          - port-distribution
          - icmp
          - "httpV2:exemplars=true;labelsContext=source_namespace,source_workload,destination_namespace,destination_workload,traffic_direction"
      tls:
        auto: { enabled: true, method: cronJob }

A few choices worth borrowing:

The UI has no ingress, on purpose. Hubble UI shows all your cluster's traffic, so you never expose it raw. An oauth2-proxy in front of Keycloak serves it at hubble.georgehomelab.com, the same single sign-on pattern every other UI here uses.
TLS between the pieces uses Cilium's own CronJob, so kube-system doesn't depend on cert-manager.
It's applied by hand, not by ArgoCD. kube-system is too foundational for a reconcile loop: one bad sync here can cut a node off the network. So it's a reviewed kubectl apply, and you watch the agents roll.

The three things it's actually good for

The live service map. Open the UI, pick a namespace, and the graph draws itself from real traffic. It's not a diagram someone drew; it's the actual edges, appearing as packets flow.

The Hubble UI filtered to one namespace (kubeflow), with per-edge protocols and flow counts.

Reading the network's decisions instead of guessing them. When a connection fails, the question is always "is this supposed to work, and if not, what blocked it?" Hubble answers directly:

hubble observe --namespace kubeflow --verdict DROPPED --last 50

The kernel already made the decision, and Hubble reports it as it happened: FORWARDED or DROPPED, and why. No more simulating NetworkPolicy rules in your head. That matters here: there are 48 Kubernetes NetworkPolicies and 6 Cilium policies on this cluster, and I can't hold all of them in my head.

For what it's worth, the cluster is quiet. When I swept all 20 nodes for drops, I found 24, all UNSUPPORTED_L3_PROTOCOL: non-IP frames Cilium simply doesn't route. Not one was a policy denial. That's a nice thing to be able to prove rather than assume.

DNS, where the policy asks for it. Cilium can see DNS queries and answers, but only for traffic passing through its DNS proxy. Here that means the Kubeflow profile namespaces, where a Cilium policy lets notebooks look up the package index. In those namespaces Hubble shows every lookup and its answer, which is exactly what you want when pip install hangs.

The thing I found when I went looking for HTTP

The draft of this article promised "which HTTP call 500'd". Then I went and checked.

Across all 20 nodes, Hubble had seen zero HTTP flows.

There are two reasons, and both are worth knowing before you enable the same httpV2 metric I did:

Cilium only parses HTTP when traffic goes through its layer-7 proxy, and it only does that for flows a policy tells it to inspect. None of my policies have HTTP rules, so Cilium never looks inside.
Most of my traffic is inside an Istio mesh. The sidecars encrypt pod-to-pod traffic with mTLS before it reaches Cilium's datapath, so even with an L7 rule, all Cilium would see is TLS.

So for the meshed workloads, which is most of this cluster, Hubble is a layer 3, layer 4 and DNS tool, and it's a very good one. HTTP behaviour comes from the mesh instead: Kiali and Istio's telemetry see every request, status code and retry, because the sidecars are the HTTP endpoints. That's not a failure of Hubble; it's the two tools working at different layers. But I'd have liked to know before I put HTTP in the article's subtitle.

Making it show HTTP

Zero flows is a fact about my policies, not about Hubble. To prove it, I gave Cilium one flow it was allowed to read.

The demo is two small workloads in a namespace that is deliberately outside the mesh: storefront calls orders every two seconds. Both are plain busybox: orders runs httpd on port 8080 with two small files to serve, and storefront loops over wget. The part that matters is the policy on orders:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: orders-http
  namespace: hubble-l7-demo
spec:
  endpointSelector:
    matchLabels:
      app: orders
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: storefront
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /
              - method: GET
                path: /api/.*

Take away the rules.http block and this is an ordinary layer 4 rule: storefront may reach port 8080. With it, Cilium redirects that port through the Envoy proxy built into each agent, and the proxy reads every request. The policy now allows GET / and GET on anything under /api/, and nothing else.

The client sends four requests each round: two that should work, one for a file that doesn't exist, and a POST the policy doesn't allow.

hubble observe --namespace hubble-l7-demo --protocol http

storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  FORWARDED (HTTP/1.1 GET http://orders:8080/api/orders.json)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 200 2ms (GET http://orders:8080/api/orders.json))
storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  FORWARDED (HTTP/1.1 GET http://orders:8080/api/missing)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 404 4ms (GET http://orders:8080/api/missing))
storefront-556fcd89c7-gs8k9 -> orders-5b87b7c4bc-5bcq4:8080 http-request  DROPPED   (HTTP/1.1 POST http://orders:8080/api/orders.json)
storefront-556fcd89c7-gs8k9 <- orders-5b87b7c4bc-5bcq4:8080 http-response FORWARDED (HTTP/1.1 403 0ms (POST http://orders:8080/api/orders.json))

That's the real output, with the timestamps, namespace prefix and identity numbers trimmed so it fits. Three different stories in six lines:

200 in 2 ms. Allowed, served, and timed. The latency is measured by the proxy, per request.
404. The policy allowed it and the application said no. The network was fine; the file isn't there.
403, and DROPPED on the request. The policy refused it. Cilium's proxy answered with the 403 itself, in under a millisecond, and the request never reached orders.

That last pair is the one worth having. A 403 from your application and a 403 from the network look the same to the caller. Hubble shows which it was.

The Hubble UI on the hubble-l7-demo namespace: the storefront → orders edge, and the flows table with the HTTP method, path and the 200, 404 and 403 status codes.

The httpV2 metric I'd switched on months earlier finally had something to count, too. Straight from one agent's metrics port:

hubble_http_requests_total{destination_workload="orders",method="GET",source_workload="storefront",status="200"} 15
hubble_http_requests_total{destination_workload="orders",method="GET",source_workload="storefront",status="404"} 9
hubble_http_requests_total{destination_workload="orders",method="POST",source_workload="storefront",status="403"} 8

Request rate, error rate and a latency histogram per workload pair, with no sidecar and no change to the application. I read those from the agent directly, because nothing scrapes them yet (see the traps).

Before you copy this onto a real service, three things it changes:

The policy is also a firewall. The moment a pod is selected by an ingress policy, everything that policy doesn't allow is refused. orders now accepts storefront on port 8080 and nothing else. Write an HTTP rule to get visibility and forget another caller, and you've caused an outage.
The traffic takes a detour. Requests to that port now pass through a proxy on the node, so they pick up a little latency and depend on that proxy. It's one proxy per node, not one per pod, which is still far lighter than a sidecar.
Flows now carry request details. The URL and headers are in the flow record, visible to anyone who can open the Hubble UI. Cilium can redact them, and I'd turn that on before pointing this at anything with tokens in a query string.

And it doesn't change the answer for the mesh. An HTTP rule on a meshed pod would still show Cilium a TLS stream. On this cluster, layer 7 through Hubble is for the workloads Istio doesn't cover.

How reliable is it?

Here's what the networking layer survives, and the evidence for each:

What goes wrong	What happens	How I know
A node reboots or the lab loses power	Each node's cilium-agent comes back and rejoins	`cilium status`: 20 of 20 nodes reachable, and 34 of 34 controllers healthy on the agent I asked
One node's agent dies	Only that node's pods are affected; every other node keeps its own datapath	One agent per node, as a DaemonSet: 20 desired, 20 ready
KubeArmor tries to confine the agent	It can't: the agent runs unconfined	The `cilium-agent: unconfined` annotation is on the live DaemonSet (see the traps)
The Relay lands on a bad node	It can't: it's kept off the two nodes where it failed	A node anti-affinity in the values file
A node runs short of memory	The scheduler knows what the agent really uses	512Mi memory request on every agent (see the traps)

And the honest limits. The Relay and the UI are single replicas, so if the Relay is down you lose the cluster-wide view, though every node's agent keeps working. Each node keeps 4,095 flows in memory. A quiet node sees about 10 flows a second and a busy one close to 300, so that's anywhere from six minutes of history down to fifteen seconds, and about 45 seconds on a typical node. And nothing scrapes the metrics yet. All three are on the list at the end.

cilium status from one agent: Cluster health: 20/20 reachable, Hubble: Ok.

The traps that taught me

KubeArmor crash-looped my CNI

After I enabled Hubble, cilium-agent on two nodes went into CrashLoopBackOff with permission denied trying to create /sys/fs/bpf/tc. It couldn't pin its own eBPF maps. Nothing in the Cilium config was wrong.

The culprit was KubeArmor, the runtime security layer from the security episode. It generates a default-deny AppArmor profile for every pod, and that profile blocked the exact BPF calls Cilium needs. My CNI was being confined by my security tool: two layers of the stack fighting in the kernel.

The fix is one annotation, and the hard part is where it goes:

podAnnotations:
  container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined

It has to be top-level podAnnotations. daemon.podAnnotations looks right and isn't a real Cilium key, so it's silently ignored, and the agent gets confined again on the next roll. It also has to be this per-container annotation; a pod-level appArmorProfile: Unconfined isn't enough, because KubeArmor keys off the container one. Without it, the two nodes wouldn't recover even after a reboot.

That fix got undone more than once. An older copy of this manifest sitting on the control-plane disks kept putting the broken version back.

The Relay that crash-looped on two nodes

The Relay kept crash-looping, 105 restarts, with:

dial tcp <hubble-peer ClusterIP>:443: connect: connection refused

It only happened when the Relay landed on two particular nodes. On those two, Cilium's datapath answers the hubble-peer service with ICMP port-unreachable. The service is internalTrafficPolicy: Local, hard-coded in the chart, so the Relay can only reach the peer service through its own node, and on those two nodes that fails. On every other node it works.

I still don't know the root cause. I ruled out kube-proxy, socket load-balancing, Cilium's state database and the kernel. So the fix is a placement workaround, and I'm saying so plainly:

relay:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/hostname
                operator: NotIn
                values: [rke2-worker-dc3-app-a, rke2-worker-dc3-db-a]

Setting affinity replaces the chart's default, which includes a pod affinity that keeps the Relay next to a Cilium agent. Restate that too, or you'll break it differently. Since the workaround, the Relay has stayed on healthy nodes.

8 GiB of memory the scheduler couldn't see

The cilium-agent ran with no memory request at all, while actually using 217 to 460 MiB per node. Across 20 nodes that's about 8.3 GiB of real memory the scheduler didn't know existed. That's how nodes hit real memory pressure while still reporting headroom.

The fix is a request, and deliberately no limit:

resources:
  requests:
    cpu: 100m
    memory: 512Mi

512Mi sits just above the highest use I measured. There's no limit on purpose: a memory limit on the CNI agent is an off switch for the node's network. It made every node look about 512Mi fuller to the scheduler, and that's the point. The memory was already in use; now the scheduler knows.

The metrics nobody reads

All those metrics are switched on, and nothing scrapes them. There's no ServiceMonitor for Cilium or Hubble yet, so the drop and DNS rates exist and vanish. It costs almost nothing, but it's the first thing on the list.

Hubble or something else?

"Isn't this just what Kiali does?" is the fair question. The answer is no, and I run both. They work at different layers:

eBPF network observability, no sidecar: Cilium Hubble, Microsoft Retina, Inspektor Gadget.
Service-mesh observability, sidecar-based: Kiali with Istio, Linkerd Viz.
Packet capture: tcpdump, Wireshark, ksniff.
Commercial network monitoring: Datadog Network Monitoring, Kentik.

	Layer	Sidecar?	HTTP depth	Knows Kubernetes identity	Best at
Hubble	eBPF datapath	❌	⚠️ only where a policy has HTTP rules, and not through mTLS	✅	flow map and policy decisions at almost no cost
Kiali / Istio	the mesh	✅	✅ every request	✅	HTTP behaviour of meshed services
tcpdump / Wireshark	raw packets	n/a	✅ if you parse it	❌	one flow in full detail
Retina	eBPF, any CNI	❌	⚠️	⚠️	flow visibility on a non-Cilium cluster
Datadog NPM	agent	❌	⚠️	✅	managed and correlated, if you'll pay

Kiali and Istio win for per-request behaviour: status codes, retries, latency, mTLS. On this cluster that's where HTTP visibility comes from, because the sidecars are the HTTP endpoints.
tcpdump is still the tool for "is this TLS handshake actually broken". It just doesn't scale past one flow.
Retina is the pick if you're not on Cilium.
Datadog wins on zero operations, if you'll send your flows to their cloud and pay per host.

Why Hubble here: it's eBPF, it knows Kubernetes identity, it's free, and Cilium was already my CNI, so it cost a few Helm values. For "what's talking to what, and did the network allow it", nothing else comes close for the price.

What's Next?

The invisible is now a live map behind a Keycloak login, and the network's decisions are one command away. What I'm doing next:

Scrape the metrics: a ServiceMonitor for Cilium and Hubble, and a dashboard for drops and DNS errors.
Keep more than a minute: export flows to OpenObserve, so "what dropped at 3 a.m." is a query, not luck.
Run two Relays, so the cluster-wide view survives losing one.
Find the real cause behind the two nodes that refuse the Relay, and drop the workaround.
Let the SRE agents use it: an agent that reads DROPPED flows and proposes the NetworkPolicy fix.

If you're on Cilium and you've never switched Hubble on, you're sitting on an X-ray machine you haven't plugged in. Just check what it can see through before you promise anyone HTTP.

Follow me on LinkedIn

Author: George Ezejiofor

NVIDIA GPU Operator on RKE2: Passthrough to Kubeflow (The Sports Car in the Garage)

George Ezejiofor — Mon, 12 Oct 2026 16:31:01 +0000

Episode 7 of the homelab build series. The foundation is poured, the walls are up, the alarms are wired, the storage floor is down and the safe is locked. Now for the fun room: three NVIDIA GPUs, one in each machine, so GPU capacity stays highly available when any one of them is down, managed by the NVIDIA GPU Operator and wired into Kubeflow.

nvidia-smi side by side on the three GPU nodes, one per datacenter.

Introduction

Three GPUs. Three datacenters. Zero schedulable.

I had three NVIDIA cards in three machines, and Kubernetes swore I had none. I asked for one GPU in a test pod and got this:

0/21 nodes are available: 21 Insufficient nvidia.com/gpu.

The Proxmox hosts could see them. lspci could see them. nvidia-smi on the host lit up. But to the cluster, three cards' worth of Ampere silicon were parked, idle and invisible. That's the most expensive kind of idle there is.

And the whole MLOps platform from the earlier episodes (Kubeflow, Katib, KServe, the training pipelines) was built to train and serve real models. You can't schedule what the scheduler doesn't know exists.

Fitting the cards took a Saturday. Getting the kernel module, the container runtime, the device plugin and the scheduler to agree took a lot longer, and taught me thirteen different ways a GPU can hide from Kubernetes.

Here's where it ended up. Three RTX A1000s, one in each "datacenter", which here means one in each of three mini PCs on a shelf. Lose a whole machine and two cards are still there. The NVIDIA GPU Operator builds the driver, wires up the runtime and advertises the cards on its own. And the part I'm proud of: when the whole lab loses power, all three cards come back and pass their CUDA self-test without me touching anything.

This is how I got there, every trap along the way, and the parts that still aren't done.

And at the end there's a real job on a real card: a 7-billion-parameter language model answering at about 30 tokens a second, against 4 on the CPU beside it.

Where this sits in the build

Foundation: Terraform on Proxmox.
Framing: Ansible building the five-master HA RKE2 cluster.
Security: Kyverno, KubeArmor and Trivy, with Falco parked for now to save CPU.
Workshop: Tekton, Harbor, SonarQube and Chains.
Storage floor: Rook-Ceph and CloudNativePG.
The safe: HA HashiCorp Vault.
The sports car: this one.

This is where the cluster stops being general-purpose Kubernetes and gains an accelerator pool. Before the operator went in, the three GPU workers were scheduled like any other node, by CPU and memory, with no idea a GPU was attached.

The repo side is small: one namespace, one chart, one values file, and one ClusterPolicy the operator builds everything from.

kubernetes-addons/0-boostrap/gpu-operator/
  namespace.yaml             # gpu-operator namespace, privileged Pod Security
  gpu-operator-values.yaml   # annotated Helm values; every trap fix lives here
  dcgm-custom-metrics.yaml   # extra DCGM fields for the dashboard
  nvidia-dcgm-dashboard.yaml # Grafana dashboard
  examples/                  # multi-GPU best-practice manifests

The hardware in the garage

The cards: three NVIDIA RTX A1000s. Ampere GA107GL, 8 GB of GDDR6, 2,304 CUDA cores, about 50 W. It draws all its power from the PCIe slot, with no extra connector, which is exactly why it fits where a bigger card wouldn't. The node label says it plainly:

$ kubectl get node rke2-worker-dc1-gpu-a \
    -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product} {.metadata.labels.nvidia\.com/gpu\.memory}MiB'
NVIDIA-RTX-A1000 8188MiB

The hosts: three Minisforum MS-01s. Intel i9-13900H, 96 GB of DDR5, and, rarely for a mini PC, one real half-height PCIe slot wired to the CPU: x16 in length, x8 electrically, which is plenty for this card. That slot is the whole reason this build exists.

Datacenter	Proxmox host	GPU worker	GPU	VRAM
dc1	proxmox-dc-1	rke2-worker-dc1-gpu-a	RTX A1000	8 GB
dc2	proxmox-dc-2	rke2-worker-dc2-gpu-a	RTX A1000	8 GB
dc3	proxmox-dc-3	rke2-worker-dc3-gpu-a	RTX A1000	8 GB

Three cards, three machines: PCIe passthrough from Proxmox host to RKE2 worker.

That's 24 GB of GPU memory across the fleet. One 8 GB card runs a 4-bit 7B model with room to spare; I measure that further down. It's great for inference, fine for small fine-tunes, and marginal for training from scratch, which is a rented-A100 job anyway.

Why one card per datacenter instead of three in one box? The same answer as storage: failure domains. If a machine goes down, GPU work still has two healthy places to land. In a homelab, rack density is worth nothing and HA is worth a lot.

Why bother with Kubernetes when the GPU is right there on the host? Three reasons. Each project gets its CUDA version from an image tag instead of eight CUDA installs fighting each other. The runtime becomes a pinned image, so results are reproducible. And the scheduler is the only sane way to say "this job gets one card, that one gets another".

PCIe passthrough: from Proxmox to the VM

Each card is bound to vfio-pci on its Proxmox host and passed into exactly one VM. An Ansible playbook handles the nouveau blacklist, the vfio binding, IOMMU and the reboot. All three cards sit at the same address, 0000:01:00, which keeps the Terraform pleasingly uniform.

The passthrough itself is Terraform with the bpg provider, through a gpu_pci_hosts map per datacenter. One gotcha deserves its own sentence: raw hostpci needs the root@pam Proxmox user. A normal API token can't assign raw PCI devices, and the error doesn't tell you that.

cd terraform/environment/prod-rke2/bpg
terraform plan  -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu
# 0 to add, 2 to change, 0 to destroy   (in place: +machine=q35, +hostpci0)
terraform apply -target=module.rke2_worker_dc2_gpu -target=module.rke2_worker_dc3_gpu

machine=q35 and hostpci0 only take effect after a full power cycle of the VM, not a soft reboot.

The best moment of the whole build came here. Once dc1 worked, I passed the cards through on dc2 and dc3 and rebooted them. The card appeared, the node got labelled, and the operator rolled the entire stack onto both nodes by itself. The first node took me twelve hours. The other two took about ninety seconds each, and I didn't touch them.

Three hardware notes that earned their place:

Driver version is pinned to 570.133.20. The version stays put. What has to follow the kernel is the module, which the driver container rebuilds against whatever kernel the VM is running.
No vGPU, no MIG. The A1000 supports neither.
Secure Boot never gets a say. The GPU VMs boot with SeaBIOS, not UEFI, so the unsigned NVIDIA module loads without any key enrolment. The Proxmox hosts themselves do have Secure Boot on; it doesn't matter, because the module loads inside the VM. If your VMs use UEFI with Secure Boot, you'll have to sign the module or switch it off there.

What the GPU Operator actually does

What the GPU Operator runs: a tower where every floor waits for the one below.

One Helm release of nvidia/gpu-operator (chart v26.3.2) runs a set of DaemonSets from a single ClusterPolicy. Sticking with the garage:

Component	In plain terms	What it really does
Operator	the garage manager	watches the `ClusterPolicy` and keeps everything below in shape
NFD (Node Feature Discovery)	the inspector	scans hardware and labels nodes that have an NVIDIA PCI device
GFD (GPU Feature Discovery)	the spec sheet	adds product, memory, count, driver and CUDA labels
Driver	the mechanic	compiles and loads `nvidia.ko` against the running kernel, inside a container
Container Toolkit	the licence plate	registers the `nvidia` runtime with containerd
Device Plugin	the parking attendant	advertises `nvidia.com/gpu` to the kubelet
DCGM Exporter	the dashboard	per-GPU metrics for Prometheus
Validator	the MOT	runs a CUDA test on every card and reports the result

The dependency chain is the thing to remember. NFD labels the nodes that have a GPU. Everything else only runs where that label is. So the driver, toolkit and device plugin land on the three GPU workers and nowhere else. No NFD label, no driver. No driver, no GPU.

And every other piece waits for the driver. When the bottom of that tower is broken, everything above it sits in Init. That gave me the rule that saved the most time: debug from the driver up. There's no point staring at the device plugin while the driver is still crash-looping.

What I deliberately don't run: the MIG manager (the A1000 has no MIG), the vGPU manager (that's for licensed shared vGPU), the sandbox and Kata pieces (the card is passed through once, at Proxmox), GPUDirect Storage (no hardware for it), and a standalone DCGM (the exporter brings its own, and two would fight over one socket).

The stack lives in the gpu-operator namespace with privileged Pod Security. The driver, toolkit and device plugin load kernel modules and touch host paths, and a restricted namespace blocks every one of them.

kubectl get pods -n gpu-operator mid-cascade: a wall of Init:0/1 and one very sad nvidia-driver-daemonset. The "before" shot.

From PCIe slot to scheduled pod

From PCIe slot to scheduled pod: once per node, then once per pod.

Once per node:

The worker boots with the GPU present as a PCI device.
NFD labels the node.
The driver pod compiles nvidia.ko for the running kernel and loads it.
The toolkit registers the nvidia runtime with containerd.
The device plugin tells the kubelet the node has nvidia.com/gpu: 1.
GFD adds nvidia.com/gpu.product=NVIDIA-RTX-A1000 and friends.
The DCGM exporter starts and metrics flow.

Then once per pod:

A workload asks for nvidia.com/gpu: 1.
Kyverno adds the taint toleration and the GPU node rule (more on that in the next section).
The scheduler finds a GPU node with a free card and binds the pod.
The NVIDIA runtime injects the device and driver libraries.
nvidia-smi inside the container sees its card.

All of it is automatic once the operator is in. That's the whole argument against a hand-rolled Ansible role that builds kernel modules per node and breaks on every kernel upgrade.

Keeping GPU pods on GPU nodes

Two bits of metadata mark these nodes, and you want both:

A label so workloads can find them: nodepool=gpu, plus everything NFD and GFD add.
A taint so ordinary workloads can't drift onto them and starve a GPU job: nvidia.com/gpu=true:NoSchedule.

$ kubectl get node rke2-worker-dc1-gpu-a -o jsonpath='{.spec.taints}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","value":"true"}]

Normally that means every GPU pod needs a toleration, a node selector and a resource limit, and forgetting any one gives you a confusing failure. On this cluster a Kyverno policy, gpu-pods-to-gpu-nodes, does the first two for you. Any pod that asks for nvidia.com/gpu is mutated to tolerate the taint and require a nodepool=gpu node. So a GPU pod here needs exactly one line:

resources:
  limits:
    nvidia.com/gpu: 1

The operator's own DaemonSets carry the toleration in the values file, and NFD's master also needs a control-plane toleration:

daemonsets:
  priorityClassName: system-node-critical
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

nfd:
  master:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Equal
        value: ""
        effect: NoSchedule

Prove it works

The quickest proof is NVIDIA's own CUDA sample. It adds two vectors on the card and prints one line if the whole chain held: driver, runtime, device plugin, scheduler.

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
  namespace: default
spec:
  restartPolicy: OnFailure
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
      resources:
        limits:
          nvidia.com/gpu: 1

Look at what isn't there: no toleration, no node selector, no runtimeClassName. The only GPU line is the limit. The securityContext blocks have nothing to do with the GPU; my default namespace enforces the restricted Pod Security Standard, so every pod there needs them.

$ kubectl apply -f cuda-vectoradd.yaml
$ kubectl get pod cuda-vectoradd -o wide
NAME             READY   STATUS      RESTARTS   AGE   NODE
cuda-vectoradd   0/1     Completed   0          16s   rke2-worker-dc1-gpu-a

$ kubectl logs cuda-vectoradd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Sixteen seconds from apply to Completed, image pull included. The pod landed on a GPU worker without being told to, and you can see what Kyverno wrote into it on the way in:

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.tolerations}'
[{"effect":"NoSchedule","key":"nvidia.com/gpu","operator":"Equal","value":"true"}]

$ kubectl get pod cuda-vectoradd -o jsonpath='{.spec.affinity.nodeAffinity}'
{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"nodepool","operator":"In","values":["gpu"]}]}]}}

Delete the pod when you're done. It has finished, so it isn't holding the card, but it's clutter.

kubectl logs cuda-vectoradd ending in Test PASSED. The hero shot.

The runtime: getting containerd to know about GPUs

containerd doesn't know what a GPU is. Something has to register an nvidia runtime that injects the device and driver libraries into containers. The classic way is to patch containerd's config.toml, and RKE2 regenerates that file from a template, throwing your patch away on every restart.

Two things make it work here. CDI is on, so the toolkit writes a device spec at /etc/cdi/nvidia.yaml that containerd reads at container start:

cdi:
  enabled: true
  nriPluginEnabled: false

And the toolkit is pointed at RKE2's own containerd rather than the default one:

toolkit:
  env:
    - name: CONTAINERD_CONFIG
      value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
    - name: CONTAINERD_RUNTIME_CLASS
      value: nvidia
    - name: CONTAINERD_SET_AS_DEFAULT
      value: "true"

The subtle bit is that CONTAINERD_CONFIG points at the generated config.toml, not the .tmpl. The toolkit re-applies its setup every time it starts, so when a reboot regenerates the config without nvidia, the toolkit simply puts it back. That's what makes it heal itself across reboots. Point it at the template and the live config sits there without a GPU runtime until someone restarts rke2-agent by hand.

Because nvidia is the default runtime, workloads here don't need runtimeClassName: nvidia. On a cluster without that setting they do, and that one line is what confuses people copying manifests between clusters.

Sharing the cards

Each A1000 advertises exactly one nvidia.com/gpu, so by default one pod owns the whole card.

Mode	What it does	Isolation	Here
Exclusive	one pod owns the card	full	✅ what runs today
Time-slicing	several pods take turns on the card	compute only, memory is shared	available, not switched on
MIG	hardware partitions with their own memory	full	❌ the A1000 has no MIG
MPS	processes share one GPU context	partial	available, not used

The node labels confirm it: nvidia.com/gpu.replicas=1 and sharing-strategy=none on all three.

Time-slicing is a ConfigMap plus a patch to the ClusterPolicy:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  a1000: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4        # one A1000 looks like four schedulable GPUs

kubectl patch clusterpolicies.nvidia.com cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"a1000"}}}}'

Create the ConfigMap first, then patch. I ran the patch on its own once, and the device plugin's replacement pod sat in Init with configmap "time-slicing-config" not found, which took that node's card from one schedulable GPU to zero until I cleared the reference again. The DaemonSet replaces one pod at a time, so the rollout stalled there and the other two cards kept working.

Before you turn it on, remember it's oversubscription, not partitioning:

No memory isolation. Four pods share 8 GB. Four 2 GB models fit; a fourth 3 GB model dies with CUDA out-of-memory.
No fault isolation. One pod that wedges the card affects its neighbours.
Sharing, not guarantees. Right for notebooks and bursty inference, wrong for anything with a latency target.

My plan is to time-slice dc1 and dc2 and leave dc3 whole for jobs that need all 8 GB.

How reliable is it?

Here's what the GPU tier survives, and the evidence for each:

What goes wrong	What happens	How I know
A whole machine goes down	Its card is gone, and so is any job that was running on it; new GPU pods land on the other two	One card per machine (dc1, dc2, dc3), each advertising `nvidia.com/gpu: 1`
The whole lab loses power	Every node rebuilds its GPU stack and re-tests its card, hands-off	After the last power cut, the toolkit, device plugin, DCGM exporter and validators came back on all three nodes by themselves, and the CUDA validator Succeeded on every card
RKE2 regenerates containerd's config on reboot	The toolkit re-applies the `nvidia` runtime	`CONTAINERD_CONFIG` points at the generated file, not the template
Someone forgets the GPU toleration	Kyverno adds it, plus the GPU node affinity	`gpu-pods-to-gpu-nodes` mutation policy
A notebook is abandoned with a card	It should be stopped after 24 hours idle and the card freed	Config only: `ENABLE_CULLING=true`, `CULL_IDLE_TIME=1440`. I have not seen it fire. The lab is powered off overnight, and two notebooks sat on cards until I stopped them by hand
A driver or runtime problem	The operator stops and reports it	`ClusterPolicy` state `ready`; validators gate everything above the driver

And the honest limits. "Highly available" here means GPU capacity: a job running on a card dies with its machine and has to start again elsewhere. The operator itself is a single replica; if it's down, running GPU pods carry on and nothing new gets configured. A kernel upgrade on a GPU node still needs a driver rebuild (see the traps), there are no GPU alerts wired yet, and nothing caps how many cards one namespace can take. Those last three are on the list at the end.

kubectl get pods -n gpu-operator -l app=nvidia-cuda-validator: three validators, one per datacenter, all Completed.

The traps, from the driver up

Thirteen ways a GPU hides from Kubernetes, by layer, from the driver up.

Thirteen of them, grouped by layer, in the order you should debug them. Most of them bit me. A few I only know from other people's write-ups, and those say so. Every piece waits on the driver, so a broken foundation makes everything above it look broken too.

The driver

The driver gets OOM-killed, exit 137. The log dies right after an nvidia-drm warning, then nvidia-smi: not found, and it crash-loops. It looks like a driver bug. It isn't: the driver container compiles the kernel module, running gcc against the kernel headers, and a 512Mi limit kills it mid-compile.

driver:
  resources:
    limits:
      cpu: "2"
      memory: 4Gi     # 512Mi OOMs the kernel-module compile

failed to install nvidia kernel module: distro 'ubuntu24.04' not supported. The driver image ships pre-built modules for a fixed list of distro and kernel pairs, and mine wasn't on it. Build from source instead:

driver:
  kernelModuleType: auto      # open modules first, proprietary as fallback
  usePrecompiled: false       # build from source, about 3 minutes on first boot

nvidia: Unknown symbol __register_chrdev after a host kernel upgrade. The cached module was built for the old kernel. Force a rebuild:

kubectl -n gpu-operator rollout restart daemonset/nvidia-driver-daemonset

The real lesson is a runbook entry, not a config change: a GPU node's kernel upgrade is a special operation, with a driver restart on the checklist. A plain power cut doesn't need it, because the kernel hasn't changed.

Missing kernel headers. The driver builds against the running kernel, so a kernel upgraded without linux-headers-$(uname -r) fails the build. The Ansible role for the GPU pool installs them, and the error is plain in the driver pod log.

The validator

nvidia-validator: command not found, exit 127. Version skew, and the error tells you nothing. Older charts shipped the validator as a separate image; the 26.x chart puts the binary inside the main operator image. Values written for one layout against a chart of the other produce a missing binary. Pin the chart (v26.3.2 here), deploy it from the NVIDIA repo rather than a stale vendored copy, and check what's actually running with helm list -n gpu-operator, not what your values file assumes.

The toolkit and runtime

no runtime for "nvidia" is configured. The RKE2 trap the generic docs miss. The toolkit looks for /run/containerd/containerd.sock; RKE2 runs its own containerd at /run/k3s/containerd/containerd.sock. The fix is the toolkit.env block above.

Failed to initialize NVML: Driver/library version mismatch inside a pod, while the host shows the same version. The toolkit had patched config.toml directly and RKE2 regenerated it. CDI or the generated-config setting fixes it.

KubeArmor's AppArmor profile blocks the toolkit. The same family of trap that once bit Cilium here: the toolkit can't write what it needs, containerd is silently never configured, and pods report the nvidia RuntimeClass as missing. If you hit it, run the toolkit container unconfined. Today it runs fine under its own KubeArmor-generated profile.

The toolkit writes the CDI spec and containerd never reads it. Pods schedule, but nvidia-smi hangs. CDI needs containerd 1.7 or newer, so check what your RKE2 release ships before you switch it on. I didn't hit this one: my containerd is 2.x.

Missing runtimeClassName: nvidia on a cluster where nvidia isn't the default runtime. The pod schedules, runs with no GPU injected, and looks exactly like broken hardware. Not an issue here because of CONTAINERD_SET_AS_DEFAULT, which is why it's worth knowing: the requirement depends on the cluster.

The device plugin and scheduling

Insufficient nvidia.com/gpu while three healthy GPUs sit idle. The driver and toolkit were running, but the device plugin was Pending:

0/21 nodes are available: 21 node(s) had untolerated taint
{nvidia.com/gpu: true}, that the pod didn't tolerate.

No device-plugin pod means nothing tells the kubelet about the card. The upstream chart ships the right toleration, but a trimmed values file copied from a blog about an untainted cluster quietly drops it. Same trap one layer down: NFD's master without its control-plane toleration never labels anything, so nothing else schedules either.

NFD takes a minute on first install. The operator comes up before NFD's first scan, so for about a minute it looks like the operator is running and nothing is happening. It isn't broken. Don't start changing things.

Telemetry

The DCGM exporter gets OOM-killed right after logging DCGM successfully initialized!. It loads its whole field set into memory and 256Mi isn't enough:

dcgmExporter:
  resources:
    limits:
      memory: 1Gi

All of these fixes live in gpu-operator-values.yaml, .

Who holds a card, and how to get it back

This is the part I'd most have wanted to read, because it's invisible until a GPU job won't schedule.

nvidia.com/gpu can't be oversubscribed. Without time-slicing, a pod with nvidia.com/gpu: 1 owns that physical card for its whole life. Three cards means three GPU pods at once, full stop.

That makes long-lived notebooks the quiet problem. At one point two Kubeflow notebooks were each holding a whole A1000, one of them for three and a half months, while asking for 0.6 and 1.2 CPUs. A notebook that asks for less than one CPU and holds an entire GPU is the shape of the problem. It also quietly capped a Katib sweep at one GPU trial at a time. And they cost more than cards: together they used over half of their namespace's CPU quota, enough to make a daily training pipeline time out because its last job could never be scheduled.

The claim is on the pod, not the Notebook, the StatefulSet or the volume. So stopping a notebook frees the card. Kubeflow's "stop" sets an annotation, the controller scales to zero, and the pod goes away:

kubectl -n <profile> annotate notebook <name> \
  kubeflow-resource-stopped="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

I sampled the free-GPU count every ten seconds while doing it:

[01] sts replicas=0   pod=Terminating   FREE GPUs=1
[02] sts replicas=0   pod=gone          FREE GPUs=2

Look at the first line. The notebook was "stopped" by every API you'd think to check, and the card was still claimed, because the pod hadn't finished terminating. replicas=0 isn't proof a card came back. On a cluster where KubeArmor can wedge pods in Terminating, that matters. The reliable check counts from the pods and cross-checks the scheduler:

# what pods claim
kubectl get pods -A -o json | jq '[.items[] | select(.status.phase=="Running" or .status.phase=="Pending")
  | .spec.containers[].resources.limits["nvidia.com/gpu"] // "0" | tonumber] | add'

# what the scheduler has allocated on a node
kubectl describe node <gpu-node> | awk '/Allocated resources/,/Events/' | grep nvidia.com/gpu

As I write this, both GPU notebooks are stopped and all three cards are free.

Culling is on as a safety net. The notebook controller stops notebooks that have been idle for 24 hours. It stops, it never deletes: the volume and every file survive, and restarting is one click. What's lost is in-memory kernel state. A running cell counts as busy, so a long training run won't be culled. And the setting is global and not GPU-aware, so it catches abandoned notebooks; it doesn't manage contention. The better lever is not giving notebooks a GPU by default.

⚠️ One more surprise: nvidia.com/gpu isn't in the profile namespaces' ResourceQuota, and a quota only limits what it names. So GPU use per namespace is unbounded even though CPU and memory are capped.

Putting the GPUs to work

Making nvidia.com/gpu schedulable was the prerequisite, not the goal.

Who on the platform can ask for a card

Nothing on this platform needs a GPU to run. Every Kubeflow component is a CPU service; what changes is the work users launch through them. Here's each door to a card, and whether anything has walked through it yet:

Platform piece	How it asks for a card	Has it used one here?
Kubeflow Notebooks	a GPU dropdown in the spawner form	Yes, and too well: two notebooks held cards while idle
Katib	`nvidia.com/gpu` in the trial template	No. Both sweeps ran on CPU
Kubeflow Trainer	`resourcesPerNode` on a `TrainJob`	No. The runtimes are installed (`torch-distributed`, three torchtune LLM fine-tunes); no job has run
KServe	a GPU limit on the predictor	No. The one live model is XGBoost on CPU
Kubeflow Pipelines	a GPU limit on a single step	No. The daily pipeline is all CPU
Spark	executor GPU settings plus the RAPIDS plugin	No, and not planned: these jobs are small
A plain pod	one line, `nvidia.com/gpu: 1`	Yes: the LLM benchmark below

Everything else in the data platform (Kafka, Airflow, MLflow, the databases) never touches a card and shouldn't.

So the honest summary is two real users so far: notebooks, which took cards and did nothing with them, and one benchmark pod, which took a card and drove it at 98%.

Hyperparameter search with Katib

Katib runs an Experiment that launches trials as pods, each reporting a metric back through an injected sidecar. Getting it to run at all taught me more than the GPUs did.

It couldn't run in any Kubeflow profile namespace. An Experiment sat at Created with zero trials, and the UI only said "Couldn't find any successful Trial." The real error was two levels down, on a ReplicaSet:

pods "<exp>-tpe-..." is forbidden:
  minimum memory usage per Container is 16Mi, but request is 10Mi

Katib asks for 10Mi of memory for its suggestion pods, and the Kyverno-generated LimitRange in every profile namespace has a 16Mi floor. Eleven of twelve algorithms had no explicit resources, so eleven of twelve couldn't schedule. Fixing those exposed the same problem on the trial pods, because the metrics sidecar is injected by a webhook and its default never shows up in the template. The fix is to raise Katib to meet the floor, not lower the floor:

runtime:
  suggestions:
    - algorithmName: tpe
      image: ghcr.io/kubeflow/katib/suggestion-hyperopt:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
  metricsCollectors:
    - kind: StdOut
      image: ghcr.io/kubeflow/katib/file-metrics-collector:v0.19.0
      resources:
        requests:
          cpu: 50m
          memory: 32Mi
        limits:
          cpu: 200m
          memory: 128Mi

Three more gotchas from getting a sweep to finish:

katib-config is mounted with subPath, which never receives updates. Restart the Katib controller after every config change; waiting does nothing.
Never put a multi-line python3 -c script in a trial template. The stdout collector rewrites the command into a shell string, and the inline script doesn't survive. Pass a file path and flags instead.
A sub-second trial can finish before the collector attaches, which shows up as random failures with an empty observation. Pace the objective and set backoffLimit: 1.

With all of that fixed, a 12-trial TPE sweep finished 12 out of 12 and found the known optimum, on CPU, because for a demonstration sweep a GPU changes nothing (more on that below). And with no time-slicing, the real ceiling for GPU trials is three at a time, one per card. A parallelTrialCount higher than that just leaves trials Pending.

The Katib trial leaderboard with the best trial highlighted.

Training and serving

A GPU training job in Kubeflow Trainer V2 looks like this. I haven't run this one yet, so treat it as the shape, not a tested manifest. With one card per machine, two workers have to land on two different machines:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: resnet-finetune
  namespace: terranetes
spec:
  runtimeRef:
    name: torch-distributed
  trainer:
    numNodes: 2
    resourcesPerNode:
      limits:
        nvidia.com/gpu: 1

For serving, a KServe InferenceService with minReplicas: 0 gives the card back when the model is idle and cold-starts it on demand. That works because the GPU claim is per pod: scale to zero, the pod goes, the card is free. With three cards, scale-to-zero is the difference between one always-on model and a fleet of models sharing three GPUs.

My live models today don't need a GPU at all. The football match predictor is an XGBoost model on tabular data, and it trains and serves on CPU in seconds. The first real GPU TrainJob and a properly served LLM are both on the list at the end.

The first real job: a 7B model on one card

For three months these cards passed their self-test and did nothing else. Prometheus has the receipts: 0% utilisation and 0 MiB of GPU memory on all three, the whole time. So before writing another word about what an A1000 can do, I made one do it.

The job is deliberately plain. One pod, one card, Ollama running Mistral 7B in 4-bit (Q4_K_M, a 4.1 GB download), the same prompt every time with a fixed seed. The whole request for hardware is one line, nvidia.com/gpu: 1. Kyverno added the toleration and the node rule, the scheduler put it on dc2, and nvidia-smi inside the container saw the card.

NAME          SIZE      PROCESSOR    CONTEXT
mistral:7b    5.8 GB    100% GPU     4096

Then the same model on the same node with the GPU switched off, four threads against the pod's four-core limit:

	Tokens per second	Notes
RTX A1000	about 30 (four runs: 29.9 to 31.6)	5,031 MiB of the 8,188 MiB used, 98-99% utilisation
CPU, 4 threads (i9-13900H)	about 4.2 (two runs: 4.1 and 4.3)	same model, same prompt, same seed

Mistral 7B on one RTX A1000 against the CPU beside it: about 30 tokens a second against about 4.2.

Seven times faster, on a 50-watt card that fits in a mini PC. A 400-token answer took 13 seconds on the GPU. At the CPU's rate it would take over a minute and a half. That is the difference between a tool you'd use and one you wouldn't.

Three things the run taught me that the spec sheet didn't:

It runs hot. The card idles at about 60 °C in the MS-01 and sat at 88 to 90 °C within a minute of load, a few degrees over its own 86 °C target, with the driver reporting a software thermal slowdown. Speed drifted from 31.6 to 29.9 tokens a second across the runs. It never reached the 96 °C hard slowdown, but a small case and a single-slot cooler have a ceiling, and sustained training will live at it.
It can't tell you its power draw. nvidia-smi reports power.draw as [N/A] on this card under passthrough. So the missing power metric isn't an exporter setting I forgot.
Threads matter more than cores. My first CPU run was left to pick its own thread count. It saw 20 cores, started 20 threads inside a four-core limit, and was still going ten minutes later. Pinned to four threads it produced 60 tokens in about fifteen seconds. If you benchmark CPU inside a container, set the thread count to the limit.

The manifest and the exact commands are in examples/llm-benchmark-pod.yaml. It's a benchmark, not a service: the pod holds a whole card while it exists, so I deleted it when the numbers were in, which is this article's own advice.

📸 SCREENSHOT: the Grafana DCGM dashboard for dc2 across the run: utilisation jumping from 0 to 98%, memory to 5 GiB, temperature from 60 to the high 80s. The first non-zero lines those panels have ever drawn.

Multi-GPU, and does a GPU change the result?

With more than one card, the internet will tell you to reach for tensor parallelism and model sharding. On three 8 GB cards over PCIe with no NVLink, most of that advice is wrong.

Strategy	Splits	Use when	On these cards
Data parallel (DDP)	the batch, full model per GPU	the model fits in 8 GB	✅ the default
Sharding (FSDP, ZeRO)	parameters and optimiser state	the model won't fit one GPU	⚠️ prefer offloading to CPU RAM
Tensor parallel	individual layers	needs NVLink	❌ slower over PCIe than not doing it
Pipeline parallel	layer stages	very deep models, many GPUs	⚠️ rarely worth it here

Use single-GPU tricks first (bf16, gradient checkpointing, smaller batches), then DDP for throughput, and only then sharding. For inference, don't shard: quantise. A 7B model in int4 fits on one card, and splitting it across two PCIe cards is strictly worse.

Does a GPU change the result, or just the speed? It changes how fast you get a result, not what the result is. With a fixed number of trials, a GPU gets you the same answer sooner. With a fixed amount of time, faster trials mean more trials, and more trials is how a search finds a better optimum. That gain comes from trial count, not from the GPU. And it can make results slightly worse: GPU reductions aren't always deterministic, so training often isn't bit-for-bit reproducible.

So putting a CPU-bound job on a GPU is theatre, and it costs a card something else needed.

Monitoring

The DCGM exporter publishes per-GPU metrics, and a ServiceMonitor puts them in Prometheus, with NVIDIA's dashboard in Grafana. The ones worth watching:

DCGM_FI_DEV_GPU_UTIL: utilisation
DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE: memory used and free
DCGM_FI_DEV_GPU_TEMP: temperature
DCGM_FI_DEV_PCIE_REPLAY_COUNTER: link errors on the PCIe slot

Two I want and don't have. DCGM_FI_DEV_XID_ERRORS isn't in my exporter's field list yet. DCGM_FI_DEV_POWER_USAGE isn't either, and adding it won't help: this card doesn't report power draw under passthrough at all.

The most useful panel is utilisation by pod:

sum(DCGM_FI_DEV_GPU_UTIL{Hostname="rke2-worker-dc2-gpu-a"}) by (exported_namespace, exported_pod)

Mind the label. On my setup pod is the exporter's own pod, and the workload is in exported_pod. Group by pod and every card looks like it's being used by the DCGM exporter.

What's missing is any alerting. The metrics are there, but nothing fires on a hot card, full memory or an XID error yet. And the metric I want most isn't on any dashboard: allocated versus used. A card that's 100% allocated and 0% used is the notebook problem, and utilisation alone makes it look exactly like a free card.

NVIDIA has since open-sourced something aimed at exactly that gap: the GPU Usage Monitor, an Apache 2.0 Helm chart that pairs the DCGM exporter with kube-state-metrics, Prometheus and Grafana, and ships dashboards for allocated against used, by namespace and by pod, plus GPU pods stuck Pending. I haven't installed it. I already run every one of those four pieces, so what I'd want from it is the dashboards, not a second Prometheus. And its platform filter is named for datacenter cards (Hopper, Blackwell), so I don't yet know how it treats an A1000. It's on the list.

The Grafana DCGM dashboard showing all three A1000s.

What it costs

Three used RTX A1000s for about £600, in slots the MS-01s already had. No extra power. The card is rated at 50 W; I can't give you a measured figure, because it doesn't report one. Cooling is the honest cost: nothing extra was fitted, and under load the card sits at its thermal limit.

The operational cost is mostly kernel upgrades on GPU nodes, which need the driver restart. Adding a GPU node is now Terraform the passthrough, label and taint it, and let the operator do the rest.

What's Next?

The sports car runs, survives a power cut and comes back on its own. What's left, roughly in order:

Try NVIDIA's GPU Usage Monitor dashboards against the Prometheus I already have, to get allocated versus used on a screen.
Export XID errors, then wire up GPU alerts: temperature (now that I know it reaches 90 °C), memory nearly full, XID errors, and a card allocated but unused for hours.
Turn on time-slicing on dc1 and dc2, and keep dc3 whole.
Put nvidia.com/gpu in the profile ResourceQuota, so one namespace can't take every card.
Run a real GPU TrainJob across two machines, and serve that LLM properly behind KServe with scale-to-zero, not from a benchmark pod.
Automate the driver restart after a GPU node's kernel upgrade, so it stops being a checklist item.

If you've got a GPU sitting in a box that Kubernetes can't see, I hope this saves you the thirteen traps on this list.

Follow me on LinkedIn

Author: George Ezejiofor

HA HashiCorp Vault: Single Source of Secrets on RKE2 Homelab (The secured Safe)

George Ezejiofor — Sat, 10 Oct 2026 18:22:09 +0000

Episode 6 of the homelab build series. The storage floor is down, the filing cabinets are full, and every one of them has a key. Those keys need somewhere secure to live, and this episode is that secured safe: a highly available, three-node HashiCorp Vault cluster, built and configured with Terraform, delivering secrets to pods through the Vault Secrets Operator with VaultAuth and VaultStaticSecret, and VaultDynamicSecret ready for short-lived credentials.

The Vault UI at vault.georgehomelab.com, signed in through Keycloak, on the Raft status page: one active node, two standbys, Seal Type: transit.
SSO Login with keycloak and Oauth2-proxy using google and Github social login.

Introduction

The cluster can survive a node failure. Could my secrets?

Everything in this homelab is built to survive. Five control-plane nodes. Three datacenters. Ceph keeping three copies of every byte. Postgres clusters that fail over on their own. Lose a node, a disk, even a whole datacenter, and the platform keeps running.

But every one of those systems is holding a key. Tekton Chains holds the private key that signs every image I build. Kyverno checks every image against its public half. Harbor hands out robot credentials. Every database has a password. So underneath the whole series sat a question I kept stepping around: where do the keys to the kingdom actually live, and what happens when that place goes down?

For a while, the honest answer was gitignored files and kubectl create secret run from my laptop. A pile of credentials wearing a security costume.

So I built a safe. A highly available HashiCorp Vault: three nodes on Raft, one active and two standbys, so any one of them can die and nobody notices. Terraform writes every secret. The Vault Secrets Operator delivers them to pods. No passwords in git, ever. I was proud of it.

Then I found out it had been dead for two months.

One token, the one the main Vault uses to fetch its unseal key from a second, tiny Vault, had quietly run out. A comment in my own playbook promised it renewed itself. It didn't. The main Vault couldn't unlock itself, and every pod went round in circles. Nothing visibly broke, because the secrets already handed out kept working. That's the scary part: a dead safe can look exactly like a healthy one.

So this episode isn't about installing Vault and declaring victory. It's about making the safe as reliable as everything it protects. Now, when the whole lab loses power, it opens itself in about two minutes, all three nodes healthy, with no human involved. If it ever stays shut, an alert tells me.

Because a secure platform isn't one where secrets are merely hidden. It's one where every app gets exactly the secrets it needs, without ever having to own them.

Why a safe at all

There are two kinds of homelab: the ones that have leaked a secret into a git repo, and the ones that are about to. I've been both.

Every credential starts life somewhere careless. A password in a values.yaml. An API key in a .env that's "definitely gitignored". A kubectl create secret run from a laptop at 1 a.m. that nobody can trace any more. It all works right up until you need to rotate something, rebuild the cluster, or work out how many copies of a password exist.

What I wanted fits in one sentence: every credential lives in one place, every workload fetches it the same way, and rotating one is a terraform apply.

Here's what that replaced:

Secrets in git, even encrypted ones, which always end up in someone's terminal history or a screenshot eventually.
Passwords in Helm values, which end up in the rendered manifests and the release history.
kubectl create secret from a laptop, which is fine until anyone else has to rebuild what you did.

Vault, or something lighter?

"Just use SOPS" is a fair question, so here's my fair answer. These tools don't all solve the same problem:

Encrypt and commit (Sealed Secrets, SOPS): the secret lives in git, encrypted. They answer "how do I put a secret in git safely?"
Sync from somewhere else (External Secrets Operator, Vault Secrets Operator): delivery trucks, not warehouses. They copy secrets from a store into Kubernetes.
A secrets engine (Vault, cloud secret managers): something that can store, issue, rotate, revoke and audit credentials, and do encryption as a service.

	Secret lives in	Short-lived creds	Rotation	Audit log	Encryption as a service	Runtime dependency
Vault	Vault	✅	`terraform apply`, then re-sync	✅	✅	Must be up and unsealed
Sealed Secrets	git, encrypted	❌	re-encrypt and commit	❌	❌	its controller
SOPS + age	git, encrypted	❌	re-encrypt and commit	❌	❌	none
External Secrets	whatever backend	backend's	backend's	backend's	❌	backend plus operator
Cloud secret manager	the cloud	✅	provider API	✅	✅	the cloud

If you've got a handful of static secrets and one GitOps loop, use Sealed Secrets or SOPS and stop there. If you live in one cloud, use its secret manager. I picked Vault for two reasons the git-based tools can't touch. Its transit engine is what lets one Vault unseal another, and the same Vault answers to pods, to Terraform and to me, with every read logged.

The cost is the one the rest of this article keeps coming back to. Vault is a live, stateful dependency. If it's sealed or down, nothing new gets a secret. SOPS can't fail that way, and I learned exactly how much that matters.

Two Vaults: the safe and the key-holder

Vault is sealed at rest. When a Vault pod starts, it can read its data from disk but can't decrypt any of it until something supplies the root key. In a cloud you'd hand that job to the cloud's key service. I don't have one, so I built a small one.

The two-Vault architecture: a three-pod main Vault that cannot unseal itself, and the one-pod transit Vault that holds the key. Links to the interactive diagram.

Click the image for the interactive version. Direct link: INTERACTIVE-DIAGRAM-URL

The main Vault runs three pods in the vault namespace (Vault 1.21.2, integrated Raft storage). Its config has a seal "transit" block. On start it sends its encrypted root key to the transit Vault, gets it back decrypted, and unseals itself.
The transit Vault is one pod in vault-transit. It holds a single transit key and nothing else, and it's sealed the old-fashioned way, with one Shamir share.

The transit Vault is the single point of failure for the whole secret tier, and I chose that on purpose. The alternative is Shamir on the main Vault, which means unsealing three pods by hand after every restart. One small Vault is easier to look after, as long as something actually looks after it. That turned out to be the hard part.

kubectl get pods -n vault next to kubectl get pods -n vault-transit: three main pods, the transit pod, and its unsealer.

The main Vault

Each of the three main pods has a 10Gi Raft data volume and a 5Gi audit volume, both on ceph-block. One pod is active and the other two are standbys that forward writes to it. If the active pod dies, Raft elects a new leader from the other two.

Today they're spread one per datacenter: dc1-app-d, dc2-app-c and dc3-app-d. Mind you, that's luck, not policy, and I'll come back to it at the end.

That's what HA buys you here. Kill the active pod and one of the standbys takes over the leadership, while the Service keeps sending requests to whichever pods are up. Reboot a node and its Vault pod comes back, unseals itself against transit, and rejoins the cluster from Raft. At no point does a human type a key.

The UI is at vault.georgehomelab.com behind Istio, and you log in through Keycloak. Yes, Vault is gated by Keycloak while Keycloak's database password lives in Vault. The first deploy breaks that circle with the root token, and only after Keycloak is up does the OIDC login get configured.

You can see the login page above. The Vault UI on the Raft page: one active node, two standbys, Seal Type: transit, Recovery Seal Type: shamir.

AUTOMATING HASHICORP VAULT CONFIGURATION WITH TERRAFORM

Vault Config — prod Environment
Calls the vault-config module against the homelab Vault. One module call, one terraform apply per environment.

terraform/vault-config/
├── modules/
│   └── vault-config/         # all resources (KV, PKI, policies, k8s auth, fleet, LLM)
│       ├── main.tf           # core: audit, kv mount, PKI, policies, k8s auth backend
│       ├── ai-agents-secrets.tf
│       ├── llm-secrets.tf
│       ├── gitops-secrets.tf
│       ├── keycloak-secrets.tf
│       ├── database-secrets.tf
│       ├── monitoring-secrets.tf
│       ├── storage-secrets.tf
│       ├── git-secrets.tf
│       ├── dns-secrets.tf
│       ├── vault-internal-secrets.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── versions.tf
└── environment/
    └── prod/                 ← you are here
        ├── main.tf           # module "vault_config" { source = "../../modules/vault-config" ... }
        ├── providers.tf      # provider "vault" { … }  (JWT auth via Keycloak)
        ├── variables.tf      # provider-level + module-passthrough variables
        ├── variables.secrets.tf
        ├── locals.secrets.tf
        ├── outputs.tf        # re-exports module outputs
        ├── moved.tf          # 62 state migrations (delete after first apply)
        ├── imports.tf        # HCL imports for Vault-auto-created mounts
        ├── terraform.auto.tfvars
        ├── .secrets.auto.tfvars.example
        ├── scripts/refresh-jwt.sh
        ├── vault-config-terraform-deploy.sh
        ├── terraform.tfstate
        └── README.md

Provider auth

JWT via Keycloak. scripts/refresh-jwt.sh fetches a fresh access token
from realm vault, client terraform-cicd, writes
.secrets.auto.tfvars. The provider's auth_login block exchanges that
JWT for a Vault token bound to policy terraform-cicd (TTL 1h, max 4h).
Every action attributable in the audit log to
service-account-terraform-cicd.

Why modules + environment

Re-use: drop a sister environment/staging/ to point the same module at a different Vault. The module is the source of truth; environment are just thin callers.
Per-environment overrides: each env has its own terraform.auto.tfvars, .secrets.auto.tfvars, terraform.tfstate. No cross-talk.
Maintenance: terraform plan shows you exactly what this env diverges from. The module change is one commit; rolling it across envs is one terraform apply per env.

Run flow

cd terraform/vault-config/environment/prod

# First time only
terraform init

# Every apply
./scripts/refresh-jwt.sh        # writes .secrets.auto.tfvars (mode 0600)
terraform plan
terraform apply

Or the one-shot wrapper that does init → validate → refresh-jwt → plan → apply:

./vault-config-terraform-deploy.sh

Getting secrets to pods: the Vault Secrets Operator

The main Vault is the safe. The Vault Secrets Operator (VSO 1.3.0) is the person who fetches things out of it and delivers them to the right room. It renders ordinary Kubernetes Secrets, so an unmodified Helm chart reads them with envFrom and never learns Vault exists.

VSO works with three resources. VaultAuth says how a workload proves who it is. VaultStaticSecret copies a KV secret into a Kubernetes Secret and keeps it fresh. VaultDynamicSecret asks Vault to mint a short-lived credential on demand; it's ready to use, but nothing here uses it yet, and that's on the list at the end.

Here's the real MLflow example, end to end. First, how MLflow's service account proves who it is:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata: { name: mlops-mlflow, namespace: mlops }
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: mlops-mlflow
    serviceAccount: mlops-mlflow
    audiences: [vault]
    tokenExpirationSeconds: 600

Then what to fetch and how to shape it:

{% raw %}

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: { name: mlops-mlflow-db, namespace: mlops }
spec:
  vaultAuthRef: mlops-mlflow
  mount: secret
  type: kv-v2
  path: homelab/mlops/mlflow-db
  refreshAfter: 1h
  destination:
    name: mlops-mlflow-db
    create: true
    overwrite: true
    type: kubernetes.io/basic-auth          # CloudNativePG insists on this type
    transformation:
      excludeRaw: true
      templates:
        username: { text: '{{ get .Secrets "username" }}' }
        password: { text: '{{ get .Secrets "password" }}' }
        uri:      { text: 'postgresql://{{ get .Secrets "username" }}:{{ get .Secrets "password" }}@postgres-mlops-cnpg-cluster-rw.mlops.svc.cluster.local:5432/mlflow_backend_db' }

That type: kubernetes.io/basic-auth line has a story of its own in the CNPG companion article: without it, the database role gets created with no password at all, and nothing complains.

The numbers are what turn "single source of truth" from a slogan into something you can count. Today there are 48 VaultStaticSecrets across 19 namespaces, 36 VaultAuths, and exactly one VaultConnection. Not one of those Secrets was made by hand with kubectl create secret.

Why my setup looks different from the tutorials

The tutorials teach the Agent Injector; I use VSO. The injector puts a sidecar in every pod and writes secrets to a file the app has to read. It's installed here (two pods), but no pod in the cluster uses it today.
The tutorials stop at Shamir. Five key shares, three needed, and you type them in after every restart. That's fine for a tutorial and miserable at 3 a.m.
Anti-affinity is required. I'd rather a Vault pod sit Pending than put two Raft members on one node. The catch is in the "What's Next" list.

Terraform is the only writer

No human writes to Vault by hand. Every secret starts in a gitignored .secrets.auto.tfvars file on my machine, and Terraform writes it to Vault:

1. edit .secrets.auto.tfvars              # gitignored, never committed
2. terraform apply                        # terraform/vault-config/environment/prod
3. Vault KV v2: secret/homelab/mlops/mlflow-db
4. VSO refreshes (hourly, or when nudged)
5. Secret mlops/mlops-mlflow-db
6. the pod's env, and CNPG's managed role password

Terraform logs in to Vault with a short-lived Keycloak JWT rather than a stored token, so even the writer has to prove who it is each time.

Today Terraform owns 47 secret paths, 55 policies and 65 Kubernetes auth roles. The paths are grouped by area under secret/homelab/. The biggest groups are monitoring, the AI agents and MLOps, each with somewhere between 15 and 19 path references.

The Vault UI's KV browser at secret/homelab/: one folder per area. Shoot the folder list only, never an opened secret.

Who can read what

Three layers keep each app in its own lane:

Paths. Each area owns its own folder under secret/homelab/.
Policies. Each role's policy grants read on exactly the paths it needs. For MLflow's database secret it boils down to this:

path "secret/data/homelab/mlops/mlflow-db" {
  capabilities = ["read"]
}

Roles bound to service accounts. The mlops-mlflow role only accepts a token from the mlops-mlflow service account in mlops.

So if someone took over MLflow's pod, they'd get MLflow's database password and nothing else. They couldn't read Keycloak's secrets or list other paths, and they couldn't reach the audit log.

The two months the safe stayed shut

Here's what actually happened.

The main Vault authenticates to the transit Vault with a token that Ansible created during the first install, with a 768-hour period. The playbook had a comment saying the token "auto-renews". It doesn't. A periodic token only resets its clock when something calls renew-self, and nothing did. When it ran out, every main Vault pod got permission denied from transit, couldn't unseal, and crash-looped.

The fix is a tiny daily CronJob, vault-transit-token-renewer, that renews the token. Daily against a 32-day period means it would have to fail about thirty times in a row before anything was at risk. That's deliberately generous, because the failure mode is the whole secret tier going dark.

That wasn't the only way the safe stayed shut:

Every reboot resealed the transit Vault. One Shamir share means someone has to type it in, and for months that someone was me, running an Ansible playbook. Now a small Deployment, vault-transit-unsealer, checks the transit Vault every 15 seconds and unseals it if it finds it sealed. It reads the key from a mounted Secret, so its service account needs no permission to read Secrets at all.
Power cuts left the transit pod stuck. When my hosts lose power, pods die without releasing their Ceph block-device locks, and on boot the transit pod sits in Unknown holding a stale one. Once, that took all of Vault down until I force-deleted pods by hand. A stuck-pod-reaper CronJob now does what I used to do.
Nobody was watching. There's now a PrometheusRule that alerts when the transit pod isn't ready. It uses pod readiness from kube-state-metrics rather than Vault's own metrics, because when Vault is sealed it's the worst possible thing to ask whether Vault is OK. The main Vault also has a ServiceMonitor scraping /v1/sys/metrics now.

The last power cut showed all of it working. The lab powered on, and the unsealer logged "cannot reach transit API" every 15 seconds while the transit pod came up. The moment transit answered, it logged vault-transit was SEALED -> unsealed OK. About two minutes later the main Vault pods were up and unsealed. I was making coffee.

kubectl -n vault-transit logs deploy/vault-transit-unsealer: the "cannot reach" lines, then SEALED -> unsealed OK. It's the most reassuring log line in the cluster.

How reliable is it now?

Here's what the safe survives, and the evidence for each:

What goes wrong	What happens	How I know
One Vault pod dies	The two standbys keep serving, and Raft elects a new leader if it was the active one	Three-node Raft, `HA Mode: active` plus two standbys
A node reboots	The pod comes back, auto-unseals against transit and rejoins Raft	After the last power cut, vault-1 and vault-2 came back and rejoined on their own, with no crash restarts
The whole lab loses power	Transit is unsealed within about 15 seconds of coming up, and the main Vault follows	Last power cut: transit unsealed the moment it answered, main Vault up about two minutes later, no human involved
The auto-unseal token ages	Renewed daily against a 32-day period	`vault-transit-token-renewer` runs every day
A pod is stuck after a power cut	Reaped within three minutes	`stuck-pod-reaper` runs every 3 minutes
The transit Vault isn't ready	An alert fires	PrometheusRule on transit readiness
Vault is down for a while	Running apps keep their existing Secrets; only new syncs wait	That's how the two-month outage went unnoticed

And the honest limits, all on the list at the end: the transit Vault is a single pod, the main Vault spreads across nodes rather than strictly across datacenters, and there are no scheduled Raft snapshots yet.

The traps that taught me

Istio makes a TLS listener look like plain HTTP

The pods came up green, then vault status said http: server gave HTTP response to HTTPS client. The listener was TLS. Istio's sidecar was grabbing the connection and forwarding plain text into it. Exclude Vault's ports in both directions:

server:
  annotations:
    traffic.sidecar.istio.io/excludeInboundPorts:  "8200,8201"
    traffic.sidecar.istio.io/excludeOutboundPorts: "8200,8201"

Everyone remembers inbound. Forgetting outbound breaks retry_join, because vault-1 calling vault-0 goes through the sidecar too. And the chart key is server.annotations, not server.podAnnotations, which the chart silently ignores.

Raft redirects to an IP your certificate doesn't cover

vault-1 and vault-2 joined, got redirected to the leader's pod IP, and died with x509: cannot validate certificate for 10.42.x.y because it doesn't contain any IP SANs. Pod IPs change on every reschedule, so they don't belong in a certificate. Pin the name instead:

retry_join {
  leader_api_addr       = "https://vault-0.vault-internal:8200"
  leader_tls_servername = "vault.vault.svc.cluster.local"
}

One letter in the seal block

Every pod crash-looped with seal init failed and nothing else. The transit Vault runs without TLS inside the cluster, so its address has to be http://. Every example online uses https://. I stared at that line for an hour before I saw it.

`vault operator init` fails one time in three

A fresh vault-0 spends its first few seconds in a busy retry_join loop, and init during that window returns a bare 500 internal error. Ansible's no_log hid even that. The fix is to wait until vault status settles, pause, then init with retries, and treat "already initialized" as success.

The unseal command that ignores your key

Older notes of mine, and the first version of this article, said to pipe the key in with vault operator unseal -. On Vault 1.21.2 that doesn't read stdin. It treats the - as the key itself, and the unseal quietly fails. The unsealer passes the key as an argument instead. If you're doing it by hand, read it into a variable first so it never lands in your shell history:

read -rs KEY
kubectl -n vault-transit exec vault-transit-0 -- vault operator unseal "$KEY"
unset KEY

The operator says 38 secrets are broken. They aren't.

After the last power-on, kubectl get vaultstaticsecrets -A showed 38 of 48 with SecretSynced=False. The message was a no route to host to an old pod IP, logged while Vault was still coming up. Every one of those 38 Secrets exists and has its keys. The condition just hadn't been updated since. Check the Secret itself, not the condition.

What I thought Vault was doing for my disks

An earlier draft of this article said Vault holds the encryption keys for my Ceph disks. It doesn't. The Ceph config points at Vault, a rook mount and a policy exist, and the disks really are encrypted. But the keys are in Ceph's own monitor store. Vault's rook mount is empty.

The cause is the Istio pattern again. A DestinationRule makes the sidecar wrap traffic to Vault in TLS, so meshed clients have to speak http://. Rook was set to https://, which is TLS inside TLS, and it never once connected:

TLSv1.3 (OUT), TLS alert, record overflow (534)
error:0A0000C6:SSL routines::packet length too long

Two things came out of finding that. The VAULT_SKIP_VERIFY setting I'd been apologising for never mattered. The thing actually skipping certificate checks is insecureSkipVerify: true in that DestinationRule, and it applies to every meshed client of Vault. And my storage doesn't depend on Vault at all, which is a happier failure mode than the one I thought I had. I wrote that Vault held the keys because the config said so. Config isn't evidence; ceph config-key ls is.

The CephCluster spec showing KMS_PROVIDER: vault, next to ceph config-key ls | grep dm-crypt listing six dm-crypt/osd/<fsid>/luks entries. Shoot the spec and the key names only, never a key, and never the rook/ KV path.

What it costs

Pods: 3 main Vault, 1 transit, 1 unsealer, 2 Agent Injector, the VSO controller, and a CSI provider on every node (29 on my cluster).
Storage: 3 × 10Gi Raft plus 3 × 5Gi audit on ceph-block, and 1Gi for transit.
Upkeep: the daily renewer, the reaper, and an alert. All three exist because of something that went wrong.

What it buys: one answer to "where does this password come from?", an audit log of every read, rotation by terraform apply, and per-app isolation enforced by Vault rather than by everyone being careful.

What's Next?

The safe now opens itself after a power cut, and I'll know if it doesn't. What's still on my list, roughly in order of how bad the failure would be:

Spread Vault by datacenter, not just by node. The anti-affinity uses kubernetes.io/hostname. Today the pods happen to sit one per datacenter, but nothing stops two landing in the same one. If that datacenter goes, Raft loses quorum and the safe stays shut. The CNPG clusters already do this properly with topology.kubernetes.io/zone.
Scheduled Raft snapshots. Ceph's three copies aren't a backup. vault operator raft snapshot save on a CronJob is.
A NetworkPolicy for the vault namespace. The transit Vault has one. The main Vault doesn't, so anything in the cluster can reach port 8200 and try its luck.
Close insecureSkipVerify by getting Vault's CA to the sidecars, then point Rook at http:// so disk keys can finally reach Vault.
Dynamic database credentials, so apps get short-lived Postgres logins instead of static passwords.

Next episode, the secrets get put to work. The NVIDIA GPU Operator brings the cluster's GPUs online, and everything that trains a model starts asking the safe for its keys.

If you're still keeping secrets in a .env you swear is gitignored, this is the episode to borrow from. Just set an alarm for your unseal token.

Follow me on LinkedIn

Author: George Ezejiofor

RKE2 Homelab Storage: Rook-Ceph & CloudNativePG (The Garage & Cabinets )

George Ezejiofor — Wed, 07 Oct 2026 09:31:29 +0000

Episode 5 of the homelab build series. Terraform poured the foundation, Ansible and RKE2 framed the walls, Kyverno and friends wired the alarms, and Tekton and Harbor built the workshop out back. This one is about the floor everything else stands on: Rook-Ceph as the garage that holds whatever you throw at it, and CloudNativePG as the filing cabinets for the stuff that needs a proper label.

The Ceph dashboard landing page: HEALTH_OK, 22 TiB raw, 6 OSDs up and in across three zones. SSO Login with keycloak and Oauth2-proxy using google and Github social login.

Grafana Dashboard for CEPH `signed in with google SSO and keycloak`

Introduction

I was proud of a cluster that was lying to me

For a few months I told anyone who'd listen that my data was safe. Six Ceph OSDs, three datacenters, every object stored three times. It sounded great when I said it out loud.

Then I actually looked at where those OSDs lived. Each one was a 100 GiB slice of the same local disk that held the VM boot volumes on that host, etcd's included. I'd built three copies of my data on three copies of the same weak spot, and I'd been bragging about it.

Ceph kept filling up, and I kept blaming Ceph. It wasn't Ceph. I'd put the garage, the filing cabinets and the house wiring on one shelf and hoped nobody leaned on it.

So this episode is the fix, and the story of what the fix taught me. It's the least exciting room in the house. Nobody comes round to admire your PVCs. But every episode after this one (the MLOps stack, the lakehouse, Kubeflow) is a tenant that needs somewhere to put things, and tenants don't move into a building without floors.

Two floors, in the end. Rook-Ceph at the bottom gives the cluster block, file and object storage from one system. CloudNativePG sits on top of it for the one kind of data that deserves its own filing system: relational state. Here's how both are built, what moving onto real NVMe took, and the traps that taught me the difference between "replicated" and "safe".

Why storage had to come now

If you've been following along, the cluster so far:

Foundation. Terraform stood the VMs up on Proxmox.
Framing. Ansible turned them into a five-master HA RKE2 cluster.
Security. Kyverno at the door, KubeArmor on the inside locks, Falco on the cameras, Trivy doing the rounds.
Workshop. Tekton, SonarQube, Harbor and Chains: the supply chain that builds and signs everything that lands here.

Storage had to come next because everything after this point wants somewhere to keep things. The MLOps stack alone wanted Postgres for Hive Metastore, MLflow and Airflow plus buckets for artifacts. Harbor wanted a database, a Redis and a blob store. Keycloak, SonarQube and Vault each wanted their own.

Without a real storage layer, each of those apps either drags in its own bundled database (another StatefulSet I'd have to babysit) or writes to hostPath and dies with whatever node it happened to land on. I've run the hostPath version before. Never again.

The repo side is two folders that do very different jobs but behave like one layer:

0-boostrap/
  rook-ceph/                    # the garage: block, file and object
    values-operator.yaml
    values-cluster.yaml         # OSD devices, pools, storage classes
    rgw-tls.yaml                # RGW is TLS-only
    rgw-s3-tls-origination.yaml # how Istio talks HTTPS to it
    rgw-keycloak-oidc/          # S3 access via Keycloak tokens
    local-ceph-pvs.yaml         # LEGACY: the old 100 GiB slices, unused
  cnpg/                         # the filing cabinets
    cnpg-operator/
    cnpg-plugin-barman-cloud/
    cnpg-clusters/              # Cluster + Database CRs per app

The two floors

The mental model I settled on is simple:

Rook-Ceph is the garage. It doesn't care what you put in it. It hands Kubernetes three storage classes and sits underneath everything.
CloudNativePG is the filing cabinets inside the garage. It's for one specific thing, relational data, and it lives on Ceph. Every Postgres data directory is itself a ceph-block volume.

That second point matters more than it sounds. CNPG isn't a second storage system. It's a very opinionated tenant of the first one.

The two-tier storage stack: applications on top, CloudNativePG in the middle, Rook-Ceph and six NVMe drives underneath. Links to the interactive diagram.

Click the image for the interactive version. You can pan, zoom and step through five guided views. Direct link: INTERACTIVE-DIAGRAM-URL

The garage: Rook-Ceph

Rook is the operator that runs Ceph on Kubernetes. It turns MONs, MGRs, OSDs, metadata servers and S3 gateways into ordinary Kubernetes objects and keeps them alive. Underneath, it's plain Ceph: version 19.2.4, "Squid", deployed with the rook-ceph and rook-ceph-cluster charts at v1.16.2.

What's actually running:

3 MONs keep the cluster map and agree on it by quorum.
2 MGRs, one active and one standby, run the dashboard, metrics and balancer.
6 OSDs, one per NVMe drive, two in each datacenter.
2 MDS daemons for CephFS, one active and one hot standby.
2 RGW gateways speaking S3, spread across zones.

The OSDs are the part that changed. Each box has two Samsung 990 PRO 4TB drives, both handed to that box's single storage VM by PCI passthrough, and Rook claims it as a raw device by its stable path:

storage:
  useAllNodes: false
  useAllDevices: false
  nodes:
    - name: rke2-worker-dc1-db-b
      devices:
        - name: /dev/disk/by-id/nvme-Samsung_SSD_990_PRO_4TB_<drive-serial>

The by-id path isn't me being fussy. /dev/sdc is just whatever the kernel found third this boot. After a reboot it can be a different disk, and that's how you hand a live OSD to the wrong drive. I learned that one properly, and it's in the traps section below.

The last time I checked, the whole lab had just come back up from a power cut. The monitors had been running for twelve minutes, and Ceph had already sorted itself out:

$ ceph -s
health: HEALTH_OK
mon: 3 daemons, quorum a,c,d (age 12m)
mgr: a(active, since 12m), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 6 osds: 6 up (since 11m), 6 in (since 2w)
rgw: 2 daemons active (2 hosts, 1 zones)
pgs: 169 active+clean

That's the part I still enjoy. I didn't touch anything. The drives came back, the placement groups peered, and it went green on its own.

ceph osd tree: six OSDs, two per datacenter, each a whole 3.64 TiB drive.

Why Ceph and not something lighter

People ask me this a lot, usually with MinIO or Longhorn in mind. The thing that took me a while to see is that most of Ceph's "competitors" are only competing with a third of it. Storage on Kubernetes comes in three shapes, and they aren't interchangeable:

Block (RWO). One writer, low latency. What a database's data directory wants.
File (RWX). Lots of pods mounting the same filesystem at once. Shared datasets, shared scratch space.
Object (S3). HTTP put and get of whole blobs. Backups, artifacts, data lakes.

MinIO does object. Longhorn does block. SeaweedFS does object and file. HDFS is a batch analytics filesystem from another era. Ceph does all three from one cluster, and that's the whole argument. When you compare Ceph to MinIO, the real question is "do I want to run one storage system or three?"

I didn't arrive here from theory. I ran Longhorn first and liked it, right up until I started pulling nodes out on purpose. A node crash kicked off a rebuild that ate enough CPU and memory to get unrelated apps OOM-killed. RWX meant bolting NFS on top, which is its own single point of failure. And I never felt confident about point-in-time backups with it.

Here's how I'd sum the options up for a friend:

System	Block	File	Object	Runs as an operator	Best at	Watch out for
Rook-Ceph	✅ RBD	✅ CephFS	✅ RGW	✅	All three from one self-healing cluster	Heavy, and a real learning curve
MinIO	❌	❌	✅	⚠️ Helm	Fast, simple S3	No volumes for your databases
SeaweedFS	⚠️ CSI	✅	✅	⚠️	Huge numbers of small files	Smaller community, block isn't its thing
Longhorn	✅	⚠️ via NFS	❌	✅	Easiest block on a small cluster	Rebuild storms, no object
OpenEBS Mayastor	✅ NVMe-oF	❌	❌	✅	Lowest-latency block	Block only, younger
HDFS	❌	⚠️ not POSIX	❌	❌	Old-school batch analytics	Wrong tool for pod volumes

If you only need one of those things, use the specialist. Pure S3 is MinIO. Simple block on a small cluster is Longhorn. Mind you, MinIO's catch shows up the day you also need a block volume for Postgres, because now you're running two systems. That's exactly why my KFP and MLflow artifact stores point at Ceph's S3 gateway instead of the MinIO their manifests ship with.

HDFS isn't really in this race. Object storage plus open table formats like Iceberg replaced that whole way of working, and in my lakehouse that's literally what happened: Trino and Iceberg on Ceph's S3 retired the Hive and HDFS setup. Ceph didn't beat HDFS. Object storage did, and Ceph is how I serve it.

The cost is real, though, so let me be straight about it. The rook-ceph namespace runs 115 pods. Only 17 of those are Ceph itself; the rest are per-node CSI drivers, metrics exporters and crash collectors. And on the night something breaks, you'll be reading about placement groups and CRUSH rules at an hour you'd rather be asleep. With one machine and one disk, don't bother: use Longhorn or hostPath. Ceph starts paying off around three machines, more than one failure domain, and needing more than one kind of storage.

Three doors into the garage

Ceph gives Kubernetes three storage classes, and each one behaves very differently.

`ceph-block`: the locked cabinet

Provisioner: rook-ceph.rbd.csi.ceph.com
Access: ReadWriteOnce, and it's the cluster default
For: database data directories, Redis, anything one pod owns

One pod gets a block device, formats it and writes to it like a normal disk. Postgres can fsync straight to it. Every database in the cluster lives behind this door: the CNPG clusters, Harbor's database and Redis, SonarQube's data, Vault's storage.

`ceph-filesystem`: the shared shed

Provisioner: rook-ceph.cephfs.csi.ceph.com
Access: ReadWriteMany
For: anything several pods need to read and write together

My rule is to avoid it unless a workload genuinely needs it, and the numbers back that up. There are 52 volumes on ceph-block and 3 on ceph-filesystem. The three are dbt's shared artifacts directory in mlops, the technical-writer agents' shared workspace, and a test volume. Everything else is one-writer storage.

`ceph-bucket`: the loading bay

Provisioner: rook-ceph.ceph.rook.io/bucket
Access: S3 over HTTPS, not a mounted volume
For: ML artifacts, pipeline outputs, the data lake, backups

This door has changed since I first built it. The S3 gateway used to answer on plain HTTP port 80 inside the cluster. Now it's TLS-only: the in-cluster Service is rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:443, and everything outside the gateway uses https://s3.georgehomelab.com. Getting Istio to talk HTTPS to it properly was its own little adventure, which I've put in the traps below.

kubectl get sc: the three doors with ceph-block (default). You'll also see otelcol-premium-retain, which is the same RBD provisioner with Retain instead of Delete for telemetry that must outlive its PVC. Same door, different lock.

Buckets with one `kubectl apply`

This is the bit of Rook that surprised me most. A new bucket is just a claim:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: mlflow-artifacts
  namespace: mlops
spec:
  bucketName: mlflow-artifacts
  storageClassName: ceph-bucket

Apply that and Rook creates an S3 user and the bucket, then drops a ConfigMap with the bucket name and endpoint, plus a Secret with that user's keys, into the same namespace. The app reads both with envFrom and gets on with its life. No console, no clicking around, and each bucket has its own keys instead of everyone sharing a root key.

Four buckets come from claims:

kfp-artifacts:        kubeflow   # pipeline artifacts, instead of bundled MinIO
kubeflow-db-backups:  kubeflow
mlflow-artifacts:     mlops      # model files
mlops-data-lake:      mlops      # raw and curated data, the Iceberg warehouse

There's a fifth bucket, cnpg-backups, that deliberately isn't a claim. It belongs to a dedicated cnpg-backup S3 user whose keys live in Vault, so the database backups don't depend on Rook's claim machinery. That choice has a catch, which you'll find in the traps.

The filing cabinets: CloudNativePG

CNPG is a Postgres operator. You write one Cluster resource saying "three instances, this much storage, back up here", and it does the rest: creates the pods, sets up replication, picks a primary, fails over when a pod dies, ships WAL to S3, and restores to a point in time when you ask.

It runs in cnpg-system at version 1.29.0. Here's the MLOps cluster, trimmed to the parts that matter:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-mlops-cnpg-cluster
  namespace: mlops
spec:
  instances: 3
  storage:
    size: 40Gi
    storageClass: ceph-block-pg-local        # the filing cabinet sits on the garage floor
  affinity:
    topologyKey: topology.kubernetes.io/zone
    podAntiAffinityType: required   # one instance per datacenter, no exceptions
  backup:
    retentionPolicy: 7d
    barmanObjectStore:
      destinationPath: s3://cnpg-backups/postgres-mlops-cnpg-cluster
      endpointURL: https://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:443

A few things worth pointing out:

Three instances, one per datacenter. The anti-affinity is required, so the scheduler would rather leave a pod Pending than put two members in the same building.
Replication is asynchronous (maxSyncReplicas: 0). That gives great availability, but a failover can lose the last few transactions. For Keycloak sessions and ML metadata that's fine. For anything involving money it wouldn't be.
Backups are continuous, plus a nightly base backup. WAL streams to s3://cnpg-backups/ all the time, and a ScheduledBackup takes a full base backup every day. That's why "restore MLflow's database to last Tuesday at 4pm" is a field in a resource, not a weekend.

The Barman Cloud plugin is installed too (v0.12.0), but the clusters still use the built-in barmanObjectStore method. Moving to the plugin is on the list, not done.

One cluster at 3/3, naming its own primary:

kubectl get clusters.postgresql.cnpg.io -A

kubectl -n keycloak get clusters.postgresql.cnpg.io postgres-keycloak-cnpg-cluster \
  -o custom-columns='NAME:.metadata.name,INSTANCES:.spec.instances,READY:.status.readyInstances,PRIMARY:.status.currentPrimary'

What "highly available" actually buys here

"HA" gets said so often it stops meaning anything, so I went and checked what mine really survives.

Ceph survives losing a whole datacenter. Not a disk or a node: a building. That comes from how the pools are set up:

ceph-blockpool:                     { size: 3, min_size: 2, failure_domain: host }
ceph-objectstore.rgw.buckets.data:  { size: 3, min_size: 2, failure_domain: host }
ceph-filesystem-data0:              { size: 3, min_size: 2, failure_domain: host }

Three copies in three different datacenters, and min_size: 2 means the pools stay writable with one datacenter gone, not just readable.

CNPG survives it too, because each cluster has exactly one member per datacenter.

Two things I only found by looking:

Two pools were quietly keeping two copies in one box. Both are fixed; every pool now spans three boxes. It's the only pool that does. Lose a whole datacenter and CephFS metadata could drop below min_size even though the data is fine. Only three volumes use CephFS, so the blast radius is small, but it's inconsistent and I want it fixed.
Async replication, as above. Great uptime, not zero data loss.

Click the image for the interactive version. Direct link: INTERACTIVE-DIAGRAM-URL

Three datacenters, two OSDs each, one object stored in all three, and the one pool that uses host instead of zone. Links to the interactive diagram.

One cluster per app, lots of databases inside

The pattern that mattered most in the end: one CNPG cluster per trust boundary, with as many logical databases inside it as that boundary needs.

Today that's three clusters and nine Postgres pods:

Cluster	Namespace	Instances	Size	Databases	Used by
`postgres-keycloak-cnpg-cluster`	`keycloak`	3	25Gi	`keycloak_db`	Keycloak
`postgres-sonarqube-cnpg-cluster`	`sonarqube`	3	10Gi	`sonar_db`	SonarQube
`postgres-mlops-cnpg-cluster`	`mlops`	3	40Gi	six, below	The MLOps platform

The MLOps cluster is the one that sold me on this. It holds six databases: hive_metastore_db, mlflow_backend_db, airflow_metadata_db, kubeflow_metadata_db, nessie_db and openmetadata_db. It started with four. Nessie and OpenMetadata arrived with the lakehouse work, and adding them was one small resource each, not a new Postgres to look after:

apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
  name: mlflow
  namespace: mlops
spec:
  name: mlflow_backend_db
  owner: mlflow
  cluster:
    name: postgres-mlops-cnpg-cluster

Each database has its own owner role, and each role's password lives at its own Vault path, written by Terraform and synced into the namespace by the Vault Secrets Operator:

secret/homelab/mlops/cnpg-bootstrap:  # the cluster's superuser
secret/homelab/mlops/hive-db:         # hive role
secret/homelab/mlops/mlflow-db:       # mlflow role
secret/homelab/mlops/airflow-db:      # airflow role
secret/homelab/mlops/nessie-db:       # nessie role
secret/homelab/mlops/openmetadata-db: # openmetadata role
secret/homelab/kubeflow/kubeflow-metadata-db: # kfp role, same cluster

Why not one cluster per database? That would be eight databases times three pods, so 24 Postgres pods instead of 9, eight backup streams to manage, and a control plane spending more time on Postgres than on actual work.

Why not one giant cluster for everything? Because then one bad migration or one operator bug hits every app at once. Keycloak is security-critical and shouldn't share a database server with my ML experiments. The MLOps apps already read each other's data, so sharing a Postgres among them costs nothing.

kubectl get database.postgresql.cnpg.io -A: six Database resources inside the one MLOps cluster.

The traps that taught me

Every one of these cost me at least an evening, and every one is now written down so it only costs me once.

Everyone types "longhorn" first. Older docs in the repo still mention Longhorn, so people reach for it in their first PVC. The default storage class is ceph-block. If you don't name a class, you get the right one.

CNPG stuck on "Setting up primary" with no pods. The initdb Job was getting FailedCreate and nothing said why. The namespace was on Pod Security baseline, and the Istio init container needs NET_ADMIN and NET_RAW, so admission quietly refused it. Every namespace that hosts a CNPG cluster is now labelled pod-security.kubernetes.io/enforce=privileged:

kubectl label ns my-app pod-security.kubernetes.io/enforce=privileged --overwrite

"Degraded" clusters that weren't degraded. CNPG pods showing 1/2 Running looked like a Postgres problem. It was the Istio sidecar restarting. Postgres was fine the whole time. Check which container is unready before you start debugging the database.

The old plain-HTTP S3 endpoint vanished. When RGW went TLS-only, anything still pointing at the old :80 address broke. The Kubeflow Pipelines swap from MinIO had been done with an ExternalName Service aimed at RGW's in-cluster port 80, and that died with it. Every KFP S3 client now points straight at https://s3.georgehomelab.com:443, with TLS switched on explicitly in its env. The ExternalName is still there but nothing relies on it.

Istio can't route HTTP to a port called https. Moving the gateway to TLS wasn't just a port change. Istio decides a port's protocol from its name, and Rook names its 443 port https, so Istio treats it as plain TCP and an HTTP route can't target it. The fix in rgw-s3-tls-origination.yaml is a separate Service whose port Istio sees as HTTP, plus a DestinationRule that does the TLS to RGW itself. The file's comments have the whole story, including the config dump that finally made it click.

Bucket ConfigMaps freezing. A Kyverno policy marks new ConfigMaps immutable, and the ConfigMaps Rook writes for bucket claims would get caught. If the endpoint ever changes, an immutable ConfigMap silently refuses the update. rook-ceph, mlops and kubeflow are on the policy's exclusion list. Add any new namespace that creates bucket claims to it.

Moving the OSDs onto the 990 PROs. Three lessons in the order they hit me:

Name disks by by-id, never /dev/sdX. Letters are handed out in discovery order and move between boots.
A purged OSD comes back from the dead. ceph osd purge removes it from the cluster map but leaves the LVM and BlueStore labels on the disk, so Rook helpfully recreates it on the next reconcile. Scale the operator to zero, wipe the device, then scale it back.
HEALTH_OK doesn't mean nothing's wrong. After the swap, ceph -s was green while orphaned OSD deployments crash-looped against disks that no longer existed. Ceph had just stopped counting them. Compare kubectl -n rook-ceph get deploy -l app=rook-ceph-osd with ceph osd tree yourself.

Backups that silently stopped. After a switchover, a former CNPG primary sat at 1/2 with a failing startup probe. It has to archive its pending WAL before it can rejoin, and the archive was getting 403 Forbidden. A Ceph rebuild had wiped the cnpg-backup S3 user, while Vault still held its keys. That broke WAL archiving for every cluster, not just the stuck one. The fix was recreating the user in the Rook toolbox with the exact keys from Vault, because Vault is the source of truth. If your backups go quiet after any Ceph rebuild, check that the user still exists before anything else.

The Rook values aren't GitOps. The Rook charts are applied by hand with Helm, not by ArgoCD. Change the device list in Git and nothing happens until someone runs the upgrade. I lost an afternoon wondering why a new disk never became an OSD.

Conclusion

Here's what it costs and what it gives back.

On capacity: 22 TiB raw, 854 GiB used, 3.82%. Everything is stored three times, so the number to plan against is MAX AVAIL, which is 6.6 TiB per pool. I forgot that gap between raw and usable the first time I sized this, and I won't again.

Before the move, the OSDs were about 600 GiB of slices on the boot disks. Dedicated NVMe gave me roughly 36 times the raw space, and more importantly it got Ceph off the drive etcd lives on.

What I got for it:

One storage system for block, file and object. No MinIO plus Longhorn plus an NFS server glued together.
A data layer that fixes itself. After a whole-lab power cut, it was green before I'd made coffee.
Buckets from a YAML file, each with its own keys.
Postgres per trust boundary. Keycloak's database can't see SonarQube's, and the MLOps apps share one cluster without stepping on each other.
Point-in-time recovery that's actually there. WAL to S3 all day, a base backup every night, seven days kept.

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df

ceph df: 22 TiB raw, 854 GiB used, MAX AVAIL 6.6 TiB. The gap between stored and raw is the replication factor.

What's Next?

Storage is the body of the house. The next room is the one that holds the keys to all of it: HashiCorp Vault. Every database password and bucket credential in this article lives there, gets written by Terraform and reaches its pod through the Vault Secrets Operator. That's Episode 6.

On the storage side, my own to-do list is short:

Move backups out of the cluster and Drives with power-loss protection so CephFS survives losing a datacenter like everything else does.
Switch the CNPG backups to the Barman Cloud plugin, which is installed and waiting.
Put the Rook charts under ArgoCD, so a change to the device list in Git actually does something.

If you're building something similar and want to compare notes, or you've got a Ceph horror story that beats mine, I'd love to hear it.

Follow me on LinkedIn

Author: George Ezejiofor

Tekton, Cosign, Kyverno: A Signed CI/CD Supply Chain on RKE2

George Ezejiofor — Wed, 23 Sep 2026 17:41:39 +0000

Introduction

The Workshop Where Trust Is Manufactured

Here's a confession I'll make out loud: I was running code I'd never read, built by people I've never met, pulled from a registry I don't control—and deploying it with a straight face. docker pull something:latest, kubectl apply, ship it. I called it a supply chain. It was really a leap of faith with extra steps.

Not in this cluster.

By Episode 3, the house had its locks, its cameras, and its inspector—a four-layer security system that refuses a bad manifest at the door and a bad syscall at the kernel. But a guarded house is only ever as safe as what you carry into it. And the single most dangerous thing you can carry into a Kubernetes cluster is an image you cannot prove you built.

So before anything ships, it goes through the workshop.

This is Episode 4 — the CI/CD stack, and it runs on one non-negotiable rule: nothing executes in this cluster that wasn't built here, scanned here, and signed here. Every image is assembled on my own line, put through a quality gate, stamped with a cryptographic maker's mark, and stored in my own warehouse. And the security system from Episode 3? Its front door—Kyverno—refuses to admit anything that doesn't carry that mark. The workshop and the security system are two ends of the same handshake.

the supply-chain Interactive diagram: (Git → Tekton → SonarQube → kaniko → Chains → Harbor → Kyverno), or a Harbor repo showing an image tag with its .sig + .att artifacts.

The Stations

Here's what's on the workshop floor:

Station	Engine	Job
🏭 The Assembly Line	Tekton	Builds the image, in-cluster, from source
🔬 The Quality Inspector	SonarQube	Fails the build if the code doesn't pass the gate
🧪 The Safety Test	Trivy	Fails the build if the pushed image carries a CRITICAL or HIGH CVE
✍️ The Maker's Mark	Tekton Chains	Signs every artifact (cosign) + attaches SLSA provenance
📦 The Warehouse	Harbor	Stores the image, its signature, its provenance, its CVE scan and its SBOM
🚪 The Checkpoint	Kyverno `verifyImages`	Admission rejects anything unsigned — and pins what it admits to the signed digest

Each is independent. But wired in sequence they produce something a public registry never can: an image with a provable birth certificate.

Where I Am in the Build

This is the fourth episode of a homelab that stopped being a project a while ago:

Episode 1 — The Foundation. Terraform across three Proxmox datacenters.
Episode 2 — The Framing. HA RKE2 control plane via Ansible, kube-vip, Cilium.
Episode 3 — The Security System. Kyverno, KubeArmor, Falco, Trivy.
Episode 4 — The Workshop. Tekton, SonarQube, Tekton Chains, Harbor — the subject of this piece.

Two tools got removed on the way here, and the removals are as deliberate as the additions: Jenkins (Tekton + Chains + Argo Events cover every trigger without a snowflake controller) and Nexus (Harbor is a purpose-built OCI registry with native cosign + Trivy baked in). Fewer moving parts, every one of them cloud-native.

The stack lives in kubernetes-addons/cicd-stack/, deployed as a single ArgoCD ApplicationSet:

cicd-stack/
├── README.md
├── tekton/
│   ├── tekton-operator/         ← the Operator + TektonConfig (one CR, whole estate)
│   ├── tekton-pipeline/         ← reference build-and-push Pipeline + PipelineRuns
│   └── tekton-chains/           ← cosign keypair + cosign.pub for verifiers
├── harbor/                      ← OCI registry (+ Trivy scan, native Keycloak OIDC)
│   ├── charts/
│   └── vault-integration/       ← admin + registry creds via Vault → VSO
├── sonarqube/                   ← code-quality gate (SAML2 SSO)
│   ├── charts/
│   └── vault-integration/
├── github-action-runner/        ← self-hosted GitHub runners (ARC)
│   ├── controller/
│   └── runner-set/
├── cicd-pipeline-e2e-example/   ← the whole chain, end to end, as one example
│   ├── app/                     ← trivial Go service + its Dockerfile
│   ├── pipeline/                ← e2e-build-scan-sign + a PipelineRun to trigger it
│   ├── policy/                  ← the Kyverno verifyImages gate (Enforce)
│   └── deploy/                  ← what ArgoCD syncs once the image is signed
└── external-dns/                ← Cloudflare DNS sync (scaffolded)

A quick word on SSO — three tools, three patterns

One detail worth calling out, because it trips people up: the three UIs each authenticate differently, dictated by what each tool natively supports.

Tekton Dashboard → oauth2-proxy reverse-proxy (no native auth).
Harbor → native OIDC (Harbor speaks it directly).
SonarQube → native SAML2 (Community edition has no OIDC).

All three federate to Keycloak (with Google + GitHub behind it), but you don't get to pick one pattern and reuse it—you meet each tool where it lives. (And if the SonarQube/Harbor SAML login ever starts redirect-looping, it's almost always a stale IdP cert after a Keycloak key rotation—re-sync it, don't rebuild the realm.)

Three UIs with SSO login with Google and Github — the Tekton Dashboard, Harbor, and SonarQube — nine ArgoCD child applications, and a supply chain that ends with a cryptographic proof. I'll walk the floor.

Why Build Your Own

Let's be blunt about the threat, because it's the reason the whole workshop exists.

When you pull nginx:latest or some-vendor/app:v2, you are trusting: the author, every dependency the author pulled, the CI system that built it, the registry that stored it, and the network path in between—none of which you can see, and any of which can be compromised without you ever knowing. The famous supply-chain attacks of the last few years didn't break in through the front door. They walked in inside a trusted image.

Microsoft Supply Chain Attack (February 2023)
In February 2023, a software supply chain attack also affected Microsoft. The attack exploited a vulnerability in the Jfrog Artifactory, a binary repository manager that Microsoft uses to distribute and store its software components.
The attackers accessed Jfrog Artifactory and injected malicious code into some of Microsoft’s software components, allowing them to access Microsoft’s network while stealing source code and other confidential information.

A homelab that hosts Vault, Keycloak, private model registries, and real data pipelines can't hand-wave that away. So the answer isn't "scan harder." The answer is provenance: don't run what you can't trace. Build the image yourself, from source you control, on a line you can audit. Prove—cryptographically—that the thing running in production is the exact thing your pipeline produced, and reject everything else at the door.

That's a chain of custody. And a chain is only as strong as its weakest link, so every station below hardens one link:

The Assembly Line proves where it was built (in-cluster, from your Git).
The Inspector proves the code met a bar (quality gate).
The Maker's Mark proves it wasn't tampered with after the build (signature + SLSA attestation).
The Warehouse proves it hasn't rotted (continuous CVE scan).
The Checkpoint enforces all of the above at runtime (unsigned → rejected).

The honest head-to-head — assemble best-of-breed, or buy the integrated platform?

There's a fair objection to everything above: GitLab already does all of this in one product. So do GitHub, Azure DevOps, and JFrog. Why bolt together seven tools when a single platform ships SCM + CI + registry + scanning + deploy pre-wired, one login, one vendor to call? That's the real decision at the workshop level — not "Tekton vs Jenkins" (that's the Assembly-Line post), but assemble a best-of-breed supply chain, or buy an all-inclusive one. The field splits three ways:

Integrated DevOps platforms — GitLab (SCM + CI + registry + SAST/DAST/container scan + deploy in one product), GitHub (Actions + Packages + Advanced Security + Environments), Azure DevOps, JFrog Platform (Artifactory + Xray). One product, one UI, one auth, batteries included.
Managed cloud-native chains — AWS (CodePipeline + ECR), GCP (Cloud Build + Artifact Registry + Binary Authorization). Wired into cloud IAM.
Assembled best-of-breed (this workshop) — Tekton + SonarQube + Trivy + Harbor + Chains + Kyverno + ArgoCD. Best-in-class per stage, K8s-native, self-hosted, composable — but you are the integrator.

	Integration	Best-in-class / stage	Self-host on your K8s	Supply-chain depth (sign→verify)	Lock-in	You integrate
This workshop	Self-hosted	✅ each	✅	✅✅ cosign + SLSA + admission verify	none	✅ (the tax)
GitLab	✅✅ pre-wired	⚠️ "good enough"	✅ (CE/EE)	⚠️ signs, weaker on-cluster admission	medium	❌
GitHub	✅✅	✅ SCM	❌ (GHES $$)	⚠️ attestations, off-cluster	medium	❌
Azure DevOps	✅	⚠️	❌	⚠️	medium	❌
JFrog	✅ artifact-centric	✅✅ registry/scan	✅	✅ Xray	high ($)	partial

Where the integrated platforms genuinely win:

GitLab is the strongest all-in-one, full stop. One product, one UI, one auth, SCM through deploy — and it self-hosts. For a team that wants CI/CD to just work without becoming systems integrators, GitLab is the right answer and I'd recommend it without flinching. My only counter: it's a monolith where each stage is "good enough" rather than best-in-class, and you live on GitLab's rails.
GitHub (Actions + GHAS) — unmatched SCM, marketplace, Dependabot, and native build attestations. If your code already lives on GitHub, the integrated path is the path of least resistance. It leans cloud (self-hosted GHES is a bill), and the trust story lives off your cluster.
Azure DevOps — mature and enterprise-solid in Microsoft shops (Boards + Repos + Pipelines + Artifacts); aging, but cohesive.
JFrog Platform — if artifacts are your center of gravity, Artifactory + Xray is the gold standard for binary management and deep dependency scanning. Commercial, and heavier than Harbor for a homelab.

Why assemble for this cluster: every station is best-in-class and Kubernetes-native — the whole thing runs on my own RKE2 with no external runners, no per-seat SaaS bill, and no cloud — it's composable (swap SonarQube out without touching the rest), there's zero lock-in, and — the part integrated platforms rarely match on-prem — the supply-chain depth: cosign signatures + SLSA provenance from Tekton Chains, enforced at admission by Kyverno verifyImages. The cluster refuses to run an image it can't prove it built. That last, load-bearing link is the one most platforms don't give you as rigorously inside your own cluster.

The honest cost is precisely what GitLab sells against: I am the systems integrator. Seven tools is seven things to deploy, wire, and keep talking — Tekton's robot creds into Harbor, Chains' signing key, Kyverno's cosign.pub, ArgoCD watching all of it. GitLab hands you that pre-wired; I hand it to myself, and that assembly is the tax. I pay it for best-of-breed, self-hosting, and a provenance story I control end to end — which, for a homelab hosting Vault, Keycloak, and real data pipelines, is the entire point. If I were shipping a product on a deadline with a small team, I'd probably just buy GitLab and move on.

The Supply Chain — seven stages

The diagram for this one lives in
CICD-ARCHITECTURE-DIAGRAM-AI-DESIGN-PROMPT.md next to this article (paste it
into Gemini or ChatGPT for the rendered version). In words, the line runs:

#	Stage	Engine	Produces	Fails the run if…
1	Trigger	GitHub Actions on a self-hosted ARC runner	a `PipelineRun`	—
2	Quality gate	SonarQube, in-pipeline before the build	pass/fail	the quality gate fails (proven: exit 3 on planted violations)
3	Build	Tekton + kaniko `v1.23.2`	image + digest	the build fails
4	Vulnerability gate	Trivy `0.58.0` against the pushed digest	pass/fail	a CRITICAL or HIGH CVE is present
5	Sign + attest	Tekton Chains (x509, `simplesigning` + in-toto)	`.sig` + `.att` in Harbor	—
6	Label	the pipeline's own `finally` task	Harbor `signed` label	Chains never published a signature
7	Admission verify	Kyverno `verifyImages`	admit / reject	the image is unsigned or signed by the wrong key

Notice the shape: it's a loop that closes on itself. The workshop produces a signed image; GitOps deploys it; the security system checks the workshop's own signature before it runs. Nothing about that trust is imported.

And every one of those gates has been shown to say no, not just yes — which
is the only evidence that any of them are real. A gate you've never seen refuse
something is indistinguishable from a gate that admits everything. The two
negative tests are in the walkthrough below.

The Assembly Line — Tekton :

Tekton is the line itself—Kubernetes-native CI, where every build step is a container and every pipeline is a CRD. There's no long-lived build server to patch, no plugin hell; a PipelineRun spins up pods, does its work, and disappears.

I run it through the Tekton Operator: a single TektonConfig CR is the whole control surface. From that one resource the operator reconciles Pipelines, Triggers, the Dashboard, Chains, the Pruner, and the ManualApprovalGate into the tekton-pipelines namespace. One CR, the entire Tekton estate—declarative, versioned, self-healing.

The working pipeline is e2e-build-scan-sign, and it runs in supply-chain
order:

prepare-source — stages main.go + Dockerfile into the build workspace. Source arrives as a content-hashed ConfigMap, not a git-clone: this repo is private, and an unauthenticated clone 404s.
sonar-scan — the SonarQube quality gate, with qualitygate.wait=true so the scanner blocks on the server's verdict instead of posting an analysis and exiting 0.
trivy-fs-scan — a filesystem pre-check for secrets and Dockerfile misconfiguration. This is not the vulnerability gate. More on that in a moment, because it is the most instructive trap in the whole pipeline.
kaniko — builds the OCI image without a Docker daemon (no privileged socket to escape from) and pushes harbor.georgehomelab.com/library/<app>:<tag>, writing the digest to IMAGE_DIGEST.
trivy-image-scan — the real gate: trivy image --severity CRITICAL,HIGH --exit-code 1 against the pushed digest.
write-image-url — emits IMAGE_URL alongside IMAGE_DIGEST, so Chains knows exactly which artifact to sign.
finally: label-signed — waits for Chains to publish the signature, then labels the artifact in Harbor. It is a finally task for a reason, covered in the Maker's Mark section.

kubectl create -f cicd-pipeline-e2e-example/pipeline/pipelinerun.yaml
kubectl -n tekton-builds get pipelinerun -w

Why the filesystem scan is not the gate. Both steps are Trivy, both carry
--exit-code 1, and only one of them can actually fail a build. I proved it by
pinning an EOL golang:1.23-alpine base image and running it:

prepare-source     → exit 0
trivy-fs-scan      → exit 0        # ← blind to it
kaniko             → exit 0        # vulnerable image is pushed, by design
trivy-image-scan   → exit 1        # Total: 22 (HIGH: 21, CRITICAL: 1)
write-image-url    → never ran

trivy filesystem reads dependency manifests and lockfiles. A stdlib-only Go
program has none, so it reports num=0, exits 0, and looks exactly like a pass
— while 22 CVEs sit compiled into the binary. Only trivy image against the
pushed digest sees them. A pipeline carrying just the filesystem scan would
look identical to this one on every green run and gate nothing.

The CRITICAL was CVE-2025-68121 in crypto/tls. Putting the toolchain back
to golang:1.27-alpine went green again with a byte-identical digest —
kaniko --reproducible means the same source rebuilds to the same bytes, which
is what makes a signature worth anything.

Scanning after the push is deliberate, and worth being explicit about: a
failing image does land in Harbor. But the step still gates deployment, because
a failed step fails the TaskRun, so IMAGE_URL is never written, Chains never
signs it, and the digest never reaches a manifest. The unsigned failure then
gets refused a second time at admission. Two independent gates catch it.

📸 SCREENSHOT: the Tekton Dashboard mid-run — a PipelineRun with its Tasks going green, or the kaniko step streaming a build.

Two traps I paid for:

The /workspace/source permission wall. Tekton's prepare initContainer pre-creates the shared workspace as uid 65532. Your Task containers run non-root and then can't write into their own workspace—permission denied, mid-build, for no obvious reason. The fix is to grant the one capability that unblocks it (or match the uid):

securityContext:
  capabilities:
    add: ["DAC_OVERRIDE"]     # or: runAsUser: 65532

The Dashboard's login lie (feedback_oauth2_proxy_authheader_breaks_kube_passthrough). The Tekton Dashboard has no native auth, so it sits behind an oauth2-proxy (Keycloak realm tekton-dashboard). The overlay must set PASS_AUTHORIZATION_HEADER=false and SET_AUTHORIZATION_HEADER=false. Flip them to true and oauth2-proxy forwards the Keycloak JWT to the Dashboard, which hands it to the kube-apiserver as a bearer token—the apiserver rejects it (Keycloak isn't a registered OIDC provider at the apiserver), and the UI shows 401 Error loading <kind> after a visibly successful login. The meanest kind of bug: the one that looks like it worked.

The Quality Inspector — SonarQube :

A build that compiles isn't a build worth shipping. SonarQube is the inspector standing between "it built" and "it's allowed to leave the workshop": if the code fails the quality gate—coverage, bugs, security hotspots, code smells past a threshold—the run fails and no image gets signed. The gate isn't advisory; it's wired into the line.

Writing this section is what got the gate fixed. The draft originally said the
scan ran inline, then that it ran from the workflow — both were wrong, and the
truth was that e2e-build-scan-sign simply had no SonarQube step at all. It had
been dropped because it failed both PipelineRuns this cluster had ever had. The
failures turned out to be a usage error, not a gate saying no:
sonar-scanner ran without sonar.projectKey, which exits 1 or 2 before kaniko
is reached. That reads identically to a quality gate rejection in the
PipelineRun view, which is why it stood for three months.

It is back now, as step 2, before the build — there is no reason to compile and
push code that has already failed review.

And the second attempt failed more quietly than the first. My step passed
-Dsonar.sources=/workspace/build, an absolute path. sonar.sources resolves
relative to sonar.projectBaseDir, so the scanner matched nothing and said so
in a line nobody reads:

0 languages detected in 0 preprocessed files
0 files indexed
QUALITY GATE STATUS: PASSED

Green build, gate passed, nothing analysed. I only caught it because I
planted deliberate violations in main.go and watched them sail through. With
-Dsonar.projectBaseDir=/workspace/build -Dsonar.sources=. the same source
indexes 2 files, the gate returns FAILED, the step exits 3, and kaniko never
runs. Reverting the violations puts it back to green with a byte-identical
digest.

Note which failure was better: the original exited 2 and stopped the line. Mine
exited 0 and shipped. A gate that fails loudly is a working gate; a gate that
passes without looking is worse than no gate at all, because it buys you
confidence you have not earned. The only way to tell them apart is to make the
thing fail on purpose.

The scan Task talks to SonarQube over the cluster-internal URL (http://sonarqube-sonarqube.sonarqube.svc:9000) with a scan-only token from Vault—it never touches the SAML2 SSO that gates the human UI. Machines use tokens; people use Keycloak. (SonarQube Community has no native OIDC, so the human path is SAML2, which is its own small saga—see the SSO note below.)

Two traps here that cost real time:

The CrashLoop with no error (feedback_sonarqube_chart_gotchas). SonarQube's most baffling failure mode is a pod that restarts forever with liveness-probe failures and nothing useful in the logs. The cause is almost always a missing monitoringPasscode (SONAR_WEB_SYSTEMPASSCODE)—without it the liveness endpoint never returns healthy, so the pod is killed and restarted in perpetuity. The Vault path secret/homelab/sonarqube carries a monitoring_passcode field for exactly this.

The migration dirty-flag deadlock (feedback_golang_migrate_dirty_trap). On an interrupted first boot, the schema-migration table can be left with dirty=true, and every subsequent start refuses to proceed. The release valve is holdApplicationUntilProxyStarts (so Istio's sidecar is up before the migration runs) plus a one-time UPDATE schema_migrations SET dirty=false. Also: don't double-declare SonarQube's ServiceAccount—the chart already makes one, and declaring it again in the kustomization gives ArgoCD a render conflict.

A SonarQube project's Quality Gate — the green "Passed" (or a red "Failed" that killed a pipeline), with the coverage/bugs/hotspots breakdown.

The Maker's Mark — Tekton Chains :

This is the station that turns "I built it" into "I can prove I built it, and prove it hasn't changed since." Tekton Chains watches every TaskRun, and the moment a build produces an image, it:

Signs it with cosign — a .sig artifact, keyed to the private half of a keypair that lives only in the signing-secrets Secret inside the cluster.
Attaches an in-toto SLSA provenance attestation — a .att artifact that records what was built, from what, by which pipeline. A machine-readable birth certificate.

The keypair is owned by Vault, not by a bootstrap script. It lives at
secret/homelab/tekton/cosign and VSO projects it into the signing-secrets
Secret that the Chains controller mounts. The Vault role is bound to the
tekton-chains-controller ServiceAccount alone — deliberately not to
tekton-builder, the identity kaniko runs as. Folding the key into the robot
credential that already has push rights would collapse "can push" and "can
sign" into one blast radius, which is the exact property signing exists to keep
apart. The public half is committed so anyone — including the admission
controller — can verify without holding the secret.

Verifying offline is a two-liner:

cosign verify --insecure-ignore-tlog \
  --key tekton/tekton-chains/cosign.pub \
  harbor.georgehomelab.com/library/hello-supply-chain:v1

cosign verify-attestation --insecure-ignore-tlog --type slsaprovenance \
  --key tekton/tekton-chains/cosign.pub \
  harbor.georgehomelab.com/library/hello-supply-chain:v1

The trap that cost me 101 days: chains.tekton.dev/signed: "true" is not
evidence of anything. The signing-secrets Secret shipped by the Tekton
operator is created empty, and Chains stamps that annotation on completed
runs regardless. Every run for over three months carried signed=true while
the signer was failing with no valid private key found, looked for: [x509.pem, cosign.key] — a warning in the controller log and nowhere else. The
run is green, the annotation says signed, and there is no signature.

Only cosign verify against the public key counts. A related detail: because
the key is a volume mount read once at signer startup, and the Tekton
operator reverts kubectl rollout restart on its own Deployments within about
two seconds, the only way to load a rotated key is to delete the pod.

That lesson is also why the finally task exists. Chains signs a TaskRun only
after it completes, so a step inside the build cannot see the signature yet —
labelling there would assert something unverified, which is the same bug in a
new costume. Instead the finally task polls Harbor for the signature
accessory and only then writes the label. No signature inside five minutes and
it fails the run: the build, scan and push all succeeded, but an artifact
admission will refuse is not a green pipeline.

(--insecure-ignore-tlog because Chains' transparency-log integration isn't wired to a public Rekor here—signatures are verified against the key, not a public log.)

The trap (feedback_harbor_chart_gotchas, last one): Chains' k8schain reads imagePullSecrets from the TaskRun's ServiceAccount, not the Chains controller's. If the controller logs UNAUTHORIZED reading a manifest it's trying to sign, it's not a Chains bug—the tekton-builder SA in tekton-builds needs the Harbor robot secret attached:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-builder
  namespace: tekton-builds
imagePullSecrets:
  - name: harbor-tekton-builder-creds

📸 SCREENSHOT: the cosign verify output returning Verified OK, and/or a Harbor tag's "Signatures" column showing the .sig + .att sitting next to the manifest.

The Warehouse — Harbor :

Every workshop needs a warehouse it controls, not a public shelf anyone can restock. Harbor (harbor.georgehomelab.com) is that warehouse: a purpose-built OCI registry that stores the image, displays its cosign signature and SLSA attestation inline, runs a native Trivy scan on every push, and gates human access with native Keycloak OIDC (callback /c/oidc/callback; the harbor-admin Keycloak group maps to Harbor system admin). Machines authenticate as robot accounts—tekton-builder in the library project, push+pull, one-shot secret vaulted.

A warehouse only shows you what you've asked it to look for. My artifact
page displayed a signed, scanned, pipeline-built image as: not scanned, no
SBOM, no labels. Nothing was broken. The library project carried exactly one
setting — public: true — and both auto_scan and auto_sbom_generation
default to off, so Harbor's own Trivy had never run on a single artifact in
it, despite the chart shipping that scanner and registering it as default.

Two API calls fixed it permanently, and they now live in a PostSync hook Job
beside the OIDC one, because this configuration lives in Harbor's database,
not the chart — a rebuilt Harbor would silently regress to the same blank page:

curl -u admin:$PW -X PUT -H 'Content-Type: application/json' \
  -d '{"metadata":{"auto_scan":"true","auto_sbom_generation":"true"}}' \
  "$H/api/v2.0/projects/1"

Worth saying plainly: Harbor's scan is not a duplicate of the pipeline's.
The pipeline gate proves an image was clean when it was built. Harbor
re-scans on its own schedule and catches CVEs disclosed after publication —
something a build-time gate structurally cannot do.

The signature, meanwhile, had been there the whole time. Harbor 2.x renders
cosign signatures as accessories, behind the expand arrow on the artifact
row — there is no "signed" column any more; that was Notary v1. An artifact
that looks unsigned in the list may be perfectly signed, and
?with_accessory=true on the API is how you find out.

Harbor is also, hands down, the fussiest chart in this whole stack. Three traps enforced by the chart itself (feedback_harbor_chart_gotchas):

secret_key must be EXACTLY 16 characters — Harbor core has a hardcoded length check, and the same value has to appear under three Secret keys (secret, secretKey, HARBOR_SECRET_KEY). Sixteen. Not fifteen, not seventeen.
The htpasswd value must keep its username: prefix — the chart strips trailing newlines but does not add the prefix, so you generate it with htpasswd -nbBC 10 harbor_registry_user <password> and leave the user: on the front.
The PostgreSQL StatefulSet needs updateStrategy: Recreate — leave it on the default RollingUpdate and every upgrade deadlocks, because the new pod can't bind the RWO PVC the old pod still holds.

None of these are in the quickstart. All three will cost you an evening if you don't know them going in.

A Harbor repo view — an image tag with its Trivy CVE summary and the Signatures column populated. The warehouse shelf, labelled and sealed.

The Signed Blue/Green images deployed via Argocd & exposed with istio service-mesh

Closing the Loop — where the Workshop meets the Security System

Here's the payoff, and the reason Episodes 3 and 4 are really one system.

Everything above produces a signed image. But a signature nobody checks is just decoration. So the last link isn't in the workshop at all—it's the front door from Episode 3: a Kyverno verifyImages ClusterPolicy that, at admission, cryptographically checks every image against cosign.pub and rejects anything unsigned or signed by the wrong key.

git push → Tekton builds → SonarQube gate → Trivy gate → Chains signs →
Harbor stores → ArgoCD deploys → Kyverno verifies the signature → runs
                                                     │
                                        unsigned? ───┴──▶ REJECTED at the door

The policy does two things at admission, and they have to be switched on
together: it verifies the signature, and with mutateDigest: true it
rewrites :v1 to the verified @sha256:…. Git still says :v1; what actually
runs is the exact bytes that were signed. I learned the ordering the hard way —
verifyDigest: true demands a digest on the reference, and with mutateDigest
still false a tag never acquires one, so Enforce rejected my own signed
image with missing digest. The signature was fine the entire time; the
report message just reads almost identically to a signature failure.

One more line earns its place, for anyone running this under GitOps:

annotations:
  pod-policies.kyverno.io/autogen-controllers: none

Without it Kyverno autogenerates a Deployment-scoped copy of the rule that
validates the pod template, where the reference is still a tag. Under Enforce
that denies every write to the Deployment — including ArgoCD's server-side
apply dry run — so the Application parks in ComparisonError while the pods
stay happily Healthy, and the error names validate.kyverno.svc-ignore rather
than anything that sounds like ArgoCD. Pods are the right enforcement point
anyway: an unsigned Deployment rolls out zero of them, because every Pod the
ReplicaSet creates is refused.

And the assertion that matters, the one that proves the door is a door — I
pushed an image to the same repository outside the pipeline, so Chains never
saw it:

Error from server: admission webhook "validate.kyverno.svc-ignore" denied the request:
  verify-hello-supply-chain-signature:
    verify-cosign-signature: 'failed to verify image
      harbor.georgehomelab.com/library/hello-supply-chain:unsigned:
      .attestors[0].entries[0].keys: no signatures found'

That's the whole thesis made real: the cluster doesn't trust an image because it came from a registry. It trusts it because it can prove the image came from this workshop. Provenance in, provenance enforced—a closed loop with no imported faith.

A kubectl apply of an unsigned public image being rejected by the Kyverno verifyImages policy — the checkpoint turning away an artifact with no maker's mark.

What's Next? (Don't Miss Out!)

The workshop is running. Source goes in one end; a scanned, signed, provenance-stamped image comes out the other—and the front door won't admit anything that didn't. The cluster now manufactures its own trust instead of importing it.

Which means the platform is finally ready for the thing all of this was for: real workloads. With a security system that guards the house and a workshop that proves everything built for it, the next episodes move the tenants in—the MLOps platform, the data lakehouse, the private services—every one of them running on images this cluster built and signed itself.

That's the next chapter: from a secured, self-provisioning platform to a busy one. Follow along so you don't miss how the empty, hardened house finally fills up. 🏭

The ArgoCD cicd-stack ApplicationSet — Tekton, Harbor, and SonarQube all Synced + Healthy. The whole workshop, GitOps-reconciled, in one frame.

Subscribe and follow me on [LinkedIn] to catch my upcoming articles.

Harden & Lockdown RKE2 Cluster with a 4-Layer DevSecOps Stack

George Ezejiofor — Sun, 13 Sep 2026 23:56:31 +0000

Introduction

Before you move furniture into a new house, you install the locks, the cameras, and the gates. Kubernetes is no different, yet engineers constantly build clusters inside-out, treating security as a cosmetic bolt-on to be handled "later."

Not this time. The order here is entirely deliberate. You don't move into a finished house and then start wondering about the security system—you wire the alarms while the building is still empty.

With my 29-node RKE2 infrastructure fully framed across three distributed Proxmox datacenters, the platform is technically alive. But it is also completely vacant. Before a single tenant workload steps foot inside—long before MLflow, Kubeflow, Tekton, or Harbor’s first private project—the security stack goes live.

When a homelab scales past a handful of nodes and begins hosting your actual identity providers, encrypted secrets engines, private registries, and core data pipelines, the blast radius of a compromise becomes real. Security transforms from a theoretical checklist into an absolute prerequisite.

This marks the definitive boundary where this project stops behaving like a handcrafted science project and starts operating like a zero-trust production platform. It also marks the moment I transition completely to a 100% GitOps workflow via ArgoCD. The days of ad-hoc terminal manipulation and local kubectl apply commands are over. From here on out, every single policy configuration is a version-controlled commit.

This article documents the exact mechanics of how I wired four distinct, cloud-native security engines into a unified defensive front covering the full workload lifecycle:

The Four Guards

Here's what I wired together:

Guard	Engine	Job
🚪 The Bouncer	Kyverno	Blocks bad manifests at admission
🔒 The Locks	KubeArmor	Watches syscalls at the kernel — and can block them
📹 The CCTV	Falco	Detects and alerts on abnormal behavior
🔍 The Inspector	Trivy	Continuously scans for CVEs and misconfigurations

Each works independently. If one fails, the others don't notice. That's the point.

What Makes This Different

This episode covers the full workload lifecycle:

Before workloads enter the API (admission control)
While workloads run at the kernel tier (runtime audit, enforcement on demand)
When workloads behave anomalously (behavioural detection)
Long after workloads were deployed (continuous scanning)

This isn’t a pristine reference diagram copied from a vendor's whitepaper. These are real, unvarnished operational lessons, architectural deadlocks, and configuration missteps encountered while making these engines work together cleanly. My hope is that sharing these traps saves you time on your own infrastructure journey.

Where I Am in the Build

This security layer is part of a larger infrastructure series documenting the evolution of the cluster:

Episode 1 — The Foundation

Infrastructure provisioning with Terraform across multiple Proxmox datacenters
Episode 2 — The Framing

Building the HA RKE2 control plane using Ansible, kube-vip, and Cilium
Episode 3 — The Security System

Hardening the platform using Kyverno, KubeArmor, Falco, and Trivy Operator

The infrastructure exists.

Now the platform gets secured before anything important is allowed to run inside it.

The repository layout for this stack lives inside kubernetes-addons/security-stack/ with an intentionally clean directory structure:

security-stack/
├── README.md                                   ← Walkthrough + disaster recovery runbook
│
├── kyverno/                                    ← The Bouncer: admission control
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── kyverno-kubearmor-falco-trivy-values.yaml
│   └── policies/                               ← shipped as its OWN ArgoCD app
│       ├── kustomization.yaml
│       ├── charts/kyverno-policies-3.8.0/      ← vendored upstream PSS baseline
│       ├── security-policies.yaml              ← my custom hardening rules
│       ├── compliance-policies.yaml
│       ├── kubeflow-user-namespace-governance.yaml
│       ├── gpu-node-scheduling.yaml
│       ├── job-ttl-after-finished.yaml
│       ├── cleanup-finished-jobs.yaml
│       ├── cleanup-stale-machineconfig-jobs.yaml
│       └── rook-csi-apparmor-unconfined.yaml   ← the Ceph CSI scar, as code
│
├── kubearmor/                                  ← The Locks: runtime LSM (audit today)
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── kubearmor-falco-trivy-values.yaml
│   ├── example-policies.yaml
│   ├── webhook-skip-terminating-hook.yaml      ← the wedged-pod workaround
│   └── charts/kubearmor-operator-v1.6.18/      ← vendored operator chart
│
├── falco/                                      ← The CCTV: runtime detection
│   ├── kustomization.yaml
│   ├── namespace.yaml
│   ├── falco-trivy-values.yaml
│   ├── custom-rules.yaml                       ← scoped exceptions live here
│   └── servicemonitor.yaml
│
└── trivy/                                      ← The Inspector: continuous scanning
    ├── kustomization.yaml
    ├── namespace.yaml
    ├── trivy-values.yaml
    └── servicemonitor.yaml

Two things in that tree are worth pointing at before I go further, because both are scars rather than design:
kubearmor/webhook-skip-terminating-hook.yaml and
kyverno/policies/rook-csi-apparmor-unconfined.yaml only exist because KubeArmor's
AppArmor annotations wedged terminating pods and confined my Ceph CSI plugins. I
get to those later. Each stack also carries its own README.md.

Five ArgoCD child Applications, one single master ApplicationSet, and two distinct sync waves. The structural walls of my cluster haven't shifted an inch since Episode 2; what is about to change is what I allow to get inside, and what gets flagged the second it steps out of line.

The Threat Surface

Let’s be completely transparent: a modern homelab is no longer a casual testing environment where safety defaults can be skipped. This 29-node environment hosts HashiCorp Vault instances, enterprise Keycloak identity providers, MLflow registries, Kubeflow modeling pipelines, and private Harbor registries. Every single one of these software components holds explicit corporate identities, cryptographic keys, machine learning models, or proprietary container images.

The blast radius of a single container escape or internal compromise is incredibly high. More importantly, this lab is my production testing sandbox; if I cut corners on security configurations here, I build bad architectural habits that will inevitably leak into the enterprise architectures I design for clients.

The cluster threat vectors break down into four critical layers:

Misconfigured Manifests (The Illegal Entry): Workloads containing unsafe specifications that should never be written to etcd. Think of a deployment requesting runAsUser: 0, a pod requesting host namespaces hostNetwork: true, images pointing to mutable tags like :latest, or a generic network policy leaving wide-open public egress. These should be caught and stopped dead at the API gateway during kubectl apply.

Malicious Post-Admission Execution (The Broken Lock): Pods that legitimately pass validation checks but once scheduled onto a node, attempt to violate boundaries—such as writing directly to /etc/shadow, attempting to run an unexpected exec /bin/sh inside a production database pod, or mapping block devices. The local Linux kernel needs to refuse these operations natively.

Subtle Runtime Anomalies (The Stealth Intruder): Exploits that bypass standard rule boundaries entirely. Fileless malware executions, unauthorized sudo executions inside an active application container, or an application process spawning unexpected binary payloads. These system calls must be observed, parsed, and logged instantly.

The Zero-Day Decay (The Expired Clearance): Images that are fully verified, pristine, and compliant when pushed to the cluster on Friday, but become highly vulnerable on Monday morning because a new critical CVE was disclosed over the weekend. The cluster must continuously audit its own passive inventory.

No single security engine can mitigate this entire lifecycle. That is the core tenet of the defense-in-depth model: I install four distinct guards that share zero dependency lines, hook into entirely different layers of the OS and Kubernetes lifecycle, and run side-by-side without interference.

The Four Guards

Architecture Here is the exact layout of how my security tooling intercepts threat vectors across the entire deployment and runtime lifecycle:

The dotted line is deliberate. Everything drawn with a solid arrow is running in my cluster
today; the alert fan-out is one chart flag away and is the one piece I have not turned on yet.
I would rather draw the gap than pretend the box is filled.

By segregating these duties, I protect against systemic failure. If the admission webhook is temporarily bypassed or down during a system upgrade, the kernel layer is still watching — and, once policies land, still enforcing. If the kernel layer has no rule for a specific zero-day vector, the behavioural CCTV logging system still records and exposes the activity.

The Bouncer at the Door — Kyverno :

Kyverno runs as my primary admission controller. Every single API interaction targeting my cluster passes directly through its webhook filters before the specification can be saved to etcd. If a defined policy evaluates to a failure, the request is outright denied. If a policy specifies an auto-remediation rule, the specification is seamlessly mutated in flight.

My architecture runs Kyverno Chart 3.8.0 isolated completely within the kyverno namespace, consisting of five core deployments:

kyverno-admission-controller — The primary engine processing real-time validation and mutation webhooks.

kyverno-background-controller — Manages asynchronous generate policy reconciliation loops.

kyverno-cleanup-controller — Garbage cycles dead PolicyReport and EphemeralReport metrics.

kyverno-reports-controller — Aggregates policy data cluster-wide.

kyverno-policy-reporter — The reporting visualization plane (disabled in my default values to preserve RAM).

While clean on paper, syncing this setup via GitOps under ArgoCD presented a massive hurdle. Out of the box, the 3.8.0 chart immediately breaks synchronization pipelines due to mutating resource values. To establish an entirely hands-off automated sync loop, I had to introduce four specific structural fixes directly into the ArgoCD configuration:

# Inside my master ApplicationSet / Application definitions:
spec:
  ignoreDifferences:
    # 1. Neutralize the Kustomize CRD naming hash mutation
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      jsonPointers:
        - /metadata/annotations/kustomize.config.k8s.io~1needs-hash
    # 2. Prevent permanent diffs on ClusterPolicies defaulting validation modes
    - group: kyverno.io
      kind: ClusterPolicy
      jsonPointers:
        - /spec/validationFailureAction
    # 3. Strip dynamic API mutations from webhook manifests
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      jsonPointers:
        - /spec/conversion/webhook/clientConfig/caBundle

Additionally, I forced forceFailurePolicyIgnore: true across all helm configurations. This ensures that during a major cluster upgrade or node restart where the Kyverno pods are initializing, the Kubernetes control plane switches to an Ignore mode rather than locking up the API server entirely.

One major day-two operational trap to memorize (feedback_kyverno_chart_upgrade_job_leftovers): during every chart upgrade, a temporary hook job named Job/kyverno-migrate-resources is run alongside associated service accounts and cluster roles. ArgoCD marks these resources as unmanaged orphans (sync=None requiresPruning=True). You can manually execute a safe kubectl delete once you confirm the job completed its task; however, never trigger a blind global argocd sync --prune, as it will inadvertently purge active sibling metadata from unassociated applications.

Here is the Bouncer's actual roster, and what happens to a walk-in who ignores the dress code:

kubectl get clusterpolicy
kubectl run nginx --image=nginx

the output of both commands — 36 ClusterPolicies all READY=True, then the plain kubectl run nginx bounced before it ever reaches a node.

Worth reading that second command's output carefully, because it shows two guards, not one:

Error from server (Forbidden): pods "nginx" is forbidden:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false ...
runAsNonRoot != true ... seccompProfile ...

That rejection says PodSecurity, not Kyverno — it is Kubernetes' native Pod Security Admission, the free floor built into the API server, catching the most naive possible mistake before Kyverno is even consulted. The 36 ClusterPolicies sitting above it are what PSA can't do: mutate a spec in flight, generate a NetworkPolicy into every new namespace, verify a cosign signature, pin GPU pods to GPU nodes.

I keep both deliberately, and the layering is the point — the cheapest guard catches the cheapest mistake. A plain kubectl run is exactly the kind of thing a tired operator does at 2am, and it never gets far enough to need the expensive machinery.

The Locks on Every Door — KubeArmor :

Kyverno decides who gets in. KubeArmor governs what they can do once they're inside. This is the distinction most homelabs miss: a pod can pass every admission check — correct securityContext, no host namespaces, a pinned image digest — and still, the moment it's running, try to read /etc/shadow, spawn /bin/sh inside a production database, or map a block device. Admission control is blind to all of it, because admission already happened.

KubeArmor closes that gap at the only place that can't be argued with: the Linux kernel itself. It enforces per-pod policies through the kernel's Linux Security Modules (AppArmor + BPF-LSM), so a blocked action isn't reported — it's refused, natively, before the syscall completes. That is the capability. What I actually have turned on is a step short of it, and I'll show you exactly where I stopped.

My architecture runs the KubeArmor operator v1.6.18 in the kubearmor namespace, which stands up three pieces:

kubearmor (DaemonSet) — the per-node enforcement engine that talks to the kernel LSM.

kubearmor-relay — aggregates the realtime block/audit event stream from every node into one place.

kubearmor-controller — the admission + annotation controller that wires each pod to its AppArmor profile.

Policies are per-workload and live right next to the thing they protect — KubeArmor watches every namespace. Locking a pod down is deliberately boring:

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: no-shell-in-my-app
  namespace: my-ns
spec:
  selector:
    matchLabels: { app: my-app }
  process:
    matchPaths:
      - path: /bin/sh
        action: Block
      - path: /bin/bash
        action: Block

With that applied, an attacker who gets code execution inside my-app reaches for a shell and the kernel hands them nothing.

Except I haven't applied it. Here is the honest state of my cluster, and it is the most useful thing in this section:

kubectl get kubearmorpolicy -A
No resources found

Zero policies. Not one. And the operator's default posture backs that up:

kubectl -n kubearmor get kubearmorconfig -o yaml | grep Posture
  defaultCapabilitiesPosture: audit
  defaultFilePosture: audit
  defaultNetworkPosture: audit

So what is KubeArmor doing on my 29 nodes right now? Watching, in enormous detail. The relay aggregates a live stream of every process exec, file access and network connection, resolved to pod and namespace:

kubectl -n kubearmor logs -l kubearmor-app=kubearmor-relay

the kubearmor-relay stream — a live feed of resolved connections (remotehost=pod/harbor/harbor-nginx-..., svc/rook-ceph/rook-ceph-rgw-...), alongside kubectl get kubearmorpolicy -A returning No resources found. Visibility without enforcement, which is exactly what audit posture looks like.

That gap is deliberate, and it is the correct order of operations. You do not deploy an LSM in enforce mode on a cluster you have not first watched. Audit posture tells you what normal looks like — which binaries actually exec, which paths actually get written, which pods actually talk to which — and that is what a least-privilege policy is written from. Ship action: Block on day one and you will spend your week explaining to yourself why CNPG can't start.

I know this because KubeArmor has already blocked things on this cluster — just not attackers. The baseline AppArmor profile it generates for every pod carries its own deny-list (deny mount, deny /sys/... writes), and that alone was enough to break two critical subsystems. Which brings me to the traps.

Here's the honest part. KubeArmor is the guard that will bite you before it ever bites an attacker, because it doesn't only lock down your workloads — by default it generates a restrictive AppArmor profile for every pod, including the critical infrastructure DaemonSets that keep the cluster alive. Three traps cost me real time:

Trap 1 — KubeArmor vs. Cilium's eBPF (feedback_cilium_kubearmor_apparmor_block). After any change that rolls a BPF-using DaemonSet, some nodes wedged with:

mkdir /sys/fs/bpf/tc: permission denied

KubeArmor's default-deny profile was refusing Cilium's own eBPF mount. And it survives reboots — the restrictive profile is reapplied on every pod start, so a reboot "fix" lasts exactly until the next cilium-agent restart. The fix is to opt the DaemonSet out with a pod-template annotation:

spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/cilium-agent: unconfined

For Cilium this is now baked into the Ansible RKE2 bootstrap manifest (rke2-cilium-config.yaml) so a fresh install never rediscovers it. Any other BPF-touching DaemonSet (kube-vip, NFD) can need the same treatment.

Trap 2 — the terminating-pod deadlock (feedback_kubearmor_apparmor_annotation_stuck_pod_k8s134). On newer Kubernetes, the controller's AppArmor-annotation mutation can leave pods stuck Terminating indefinitely — the annotation it wants to apply races the pod's own teardown. The release valve is blunt but reliable: scale kubearmor-controller to 0, let the stuck pods drain, then scale it back. A PostSync hook that skips terminating pods keeps it from recurring.

Trap 3 — don't confine the storage plumbing (feedback_ceph_csi_rook_apparmor_rbd_module). KubeArmor's profile will happily confine the Rook-Ceph CSI plugin pods, and when it does, RBD volume mounts start failing cluster-wide — a storage outage wearing a security costume. The CSI plugins get the unconfined treatment for the same reason Cilium does: infrastructure that manipulates the kernel directly cannot run under a default-deny LSM profile.

The lesson across all three: KubeArmor is a scalpel, not a blanket. Lock down tenant workloads aggressively; leave the infrastructure that talks to the kernel alone.

The CCTV on the Ceiling — Falco :

Locks stop the break-ins you anticipated. But security's oldest truth is that the attack you didn't write a rule for is the one that gets you. That's what the cameras are for. Falco doesn't block anything — it watches everything, and the second something moves that shouldn't, it shouts.

Where KubeArmor enforces a known-bad list at the kernel, Falco taps kernel tracepoints via eBPF and pattern-matches the entire syscall firehose against a ruleset. A fileless payload executing from memory, an unexpected outbound connection from a pod that should only ever talk to Postgres, a sudo inside a container that has no business having one — none of those may trip a specific KubeArmor lock, but all of them light up the CCTV.

I run the Falco chart 8.0.2 as a DaemonSet in the falco namespace on the modern_ebpf driver — one sensor per node, no kernel module to compile. Today, detections go exactly one place: stdout, which means the camera feed is kubectl logs:

kubectl -n falco logs falco-2289j --tail=5 --max-log-requests=5 | grep -i warning

And that is the honest weak point in this stack. A camera nobody watches is a camera that catches nothing after the fact. stdout is fine while I'm actively tailing it and useless at 3am — the detection fires, scrolls past, and ages out of the log buffer. Falco is the one guard here whose value collapses without somewhere to send the alert.

Routing it is a chart flag, not a rebuild — Falco ships Falcosidekick as a subchart, so there is no second Helm install:

# falco/falco-trivy-values.yaml
falcosidekick:
  enabled: false        # ← the honest current state

Flip that to true, add a backend (Slack, Discord, Loki, PagerDuty), and the fan-out is done. The only reason it isn't already on is that the webhook is a credential, and I am not putting one in Git — so it waits for the same treatment everything else in this homelab gets: Terraform writes it to Vault, VSO syncs it in as a Secret. That is the next commit on this stack, and I would rather publish the gap than describe a pipeline I have not wired.

a live Falco detection in kubectl -n falco logs falco-2289j --tail=5 --max-log-requests=5 | grep -i warning -f — rule name, pod, namespace, the offending command — the CCTV catching someone mid-motion.

Two traps here, both about the sensor itself:

The kernel floor. The modern_ebpf driver needs CAP_PERFMON + CAP_BPF and a kernel ≥ 5.8. On a compliant node it just works; drop in an older distro and Falco CrashLoopBackOffs on startup with a driver-load failure. The fallback is the legacy ebpf driver (or the kmod driver, which then wants kernel headers on the node) — set via values. The homelab's Ubuntu 24.04 image is well clear of the floor, but it's the first thing to check when a new node's Falco pod won't come up.

Tuning the sensitivity, not silencing the camera. A fresh Falco install is noisy — legitimate operations trip generic rules. The wrong reflex is to delete the rule. The right one is to add a scoped exception in falco/custom-rules.yaml, commit, and push; ArgoCD reconciles within 30 seconds and Falco hot-reloads its config with no pod restart. You narrow the camera's blind spot deliberately, in version control — you never unplug it.

The Inspector on Patrol — Trivy :

The first three guards all police the present: what's entering, what's running, what's misbehaving right now. Trivy polices the thing none of them can see — time. An image can be pristine, signed, and fully compliant when it's pushed on Friday, and be a critical CVE waiting to happen by Monday morning, because a vulnerability was disclosed over the weekend and nothing about the image changed. The lock is fine. The camera sees nothing. The image just quietly went bad.

So the Trivy operator (0.32.1), in the trivy-system namespace, walks the beat: it continuously re-scans the cluster's entire passive inventory and writes every finding as a Kubernetes custom resource — no database, no PVC, no UI. Six report types cover the surface:

Report CR	What it audits
`VulnerabilityReport`	Image CVEs, per workload, auto-rescanned
`ConfigAuditReport`	Workload misconfiguration (Pod security)
`RbacAssessmentReport`	Over-permissioned RBAC
`InfraAssessmentReport`	Node / control-plane CIS checks
`SbomReport`	A full SBOM for every pulled image
`ClusterComplianceReport`	NSA / CIS Benchmark posture, cluster-wide

Because it's all just CRs, your SIEM is kubectl:

# Every workload with a CRITICAL CVE, right now
kubectl get vulnerabilityreport -A -o json | \
  jq '.items[] | select(.report.summary.criticalCount > 0) |
      {ns:.metadata.namespace, name:.metadata.name, crit:.report.summary.criticalCount}'

# Cluster-wide compliance posture — note the REAL report names
kubectl get clustercompliancereport
#   k8s-cis-1.23
#   k8s-nsa-1.0
#   k8s-pss-baseline-0.1
#   k8s-pss-restricted-0.1
kubectl describe clustercompliancereport k8s-nsa-1.0

the jq output above with its CRITICAL counts, and kubectl describe clustercompliancereport k8s-nsa-1.0 — the Inspector's findings board.

And since the whole point of this stack is that I report what it actually finds rather than what I wish it found: the worst offender on my cluster is my own registry. Harbor's components come back at 26–33 CRITICAL apiece — harbor-portal at 33, harbor-core, registry and jobservice at 26 each — followed by my CNPG Postgres pods at 14, Elasticsearch at 8, and vault-transit at 6.

That is an uncomfortable result to publish, and it is exactly why the Inspector earns its place. Harbor is the thing that scans everything else on the way in, and it is carrying the heaviest CVE load in the cluster. Nothing in the first three guards would ever have told me that — the manifests are valid, the syscalls are ordinary, the behaviour is normal. Only the thing that polices time finds it.

Worth being precise about what those numbers mean: a CRITICAL count is not a breach, and most of these are unreachable code paths in base images. The signal is the trend and the ranking, not the absolute number — and the ranking is telling me which upstream images to pin, rebuild, or replace first.

Pair the SbomReport output with a Kyverno verifyImages policy and the Bouncer and the Inspector start working together: Trivy tells you what's actually in every image, Kyverno refuses anything unsigned at the door. That's a real supply-chain story, not a checkbox.

The one trap worth naming: private images silently fail to scan. Trivy's k8schain reads imagePullSecrets from the workload's ServiceAccount — so anything in an auth-required registry (some of Harbor's projects) won't scan until the namespace SA carries the pull secret:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: my-ns
imagePullSecrets:
  - name: harbor-tekton-builder-creds

Miss it and you get the most dangerous kind of failure — a silent one. The reports look clean not because the images are clean, but because Trivy never got to look.

The Honest Head-to-Head — Why These Four, and Not the Alternatives

First, the framing most security comparisons get backwards: these four tools are not competing with each other. "Should I use Falco or Kyverno?" is the wrong question — it's like asking whether a building needs a door lock or a smoke alarm. Security isn't a tool, it's a pipeline, and each guard owns a different stage of it:

Build time → Trivy scans the image for CVEs before it ships.
Admission time → Kyverno refuses the manifest (and cosign-verifies the signature) before it ever reaches etcd.
Runtime, detect → Falco watches syscalls and alerts on the abnormal.
Runtime, enforce → KubeArmor refuses the malicious syscall at the kernel.

So the real question is never "which one" — it's "for this layer, which tool?" Here's that answer, honestly, layer by layer:

Layer	My pick	The alternatives	Why I picked mine
Admission / policy-as-code	Kyverno	OPA/Gatekeeper, Kubewarden, native PSA	YAML-native policies (no Rego to learn) + `mutate` + `generate`
Runtime enforcement (LSM)	KubeArmor	raw AppArmor/SELinux, Tetragon	K8s-native CRDs over the kernel LSM; per-workload least-privilege
Runtime detection	Falco	Tetragon, Tracee, Sysdig Secure	CNCF-graduated; biggest rule ecosystem; plugin model
Vuln / misconfig scanning	Trivy	Grype, Clair, Snyk, Docker Scout	All-in-one (CVE + IaC + secrets + SBOM); free; already in Harbor

Where the alternatives genuinely win — because every one of my picks has a real rival:

OPA/Gatekeeper vs Kyverno. Gatekeeper's Rego is a genuine policy language — more expressive than Kyverno's YAML for complex cross-resource logic, and the incumbent in heavily-regulated enterprises. I chose Kyverno because I'd rather write policies in the same YAML as everything else, and because Kyverno's generate (auto-stamp a NetworkPolicy / ResourceQuota / alias Service into every new namespace) and verifyImages (cosign — the same signatures my Tekton Chains pipeline produces) are first-class. That's load-bearing in my Kubeflow multi-tenancy. If I needed Rego's raw expressiveness, Gatekeeper would win. Kubewarden (WASM policies in any language) is the elegant dark horse; a smaller community kept it off my cluster.
Native Pod Security Admission is free and built in — three levels, zero install. If all you need is "no privileged pods," use PSA and skip a policy engine entirely. I run both: PSA sets the floor, Kyverno does everything PSA can't (mutate, generate, image verification, custom logic).
Tetragon vs Falco (and KubeArmor). This is the sharpest call in the stack, and the most honest thing I can say is: I run Cilium, so Tetragon is already right there — eBPF-based, and it can both detect and enforce in one tool, potentially collapsing two of my guards into one. I stayed with Falco + KubeArmor because Falco's rule ecosystem and CNCF maturity are unmatched for detection, and KubeArmor's CRD model is friendlier than hand-driving Tetragon TracingPolicies for enforcement. A leaner cluster could genuinely do this with Tetragon alone — it's on my honest "re-evaluate" list.
Grype / Clair / Snyk / Docker Scout vs Trivy. All fine scanners. Snyk has the best developer DX (and a bill); Grype is fast and SBOM-native; Clair is the registry classic. Trivy won because it's all-in-one — CVEs and IaC misconfiguration and leaked secrets and SBOM in one binary — it's free, and my Harbor already scans every push with Trivy natively. One scanner, build-time and registry-time, no new tool to learn.

The honest cost of running all four, stated plainly: it's real overhead — two DaemonSets (KubeArmor + Falco), Kyverno's five controllers, and the Trivy operator — and KubeArmor is the finickiest thing in my whole cluster. Its AppArmor annotations have wedged terminating pods, blocked Cilium's own BPF mounts, and confined CSI plugins — each of those is a scar in my notes. Falco will drown you in alerts until it's tuned. Kyverno's immutable-ConfigMap policy has bitten me more than once. If you want one layer to start, start with Kyverno — admission is the highest-leverage gate — and add the others as your threat model demands. Defense-in-depth is a spectrum, not all-or-nothing. But the four together are what turns "I hope nothing's wrong" into "I can prove nothing's wrong."

Bringing the Four Guards Together

Step back and look at what's now standing watch over an empty house:

Layer	Guard	Mode	When it acts
Admission	Kyverno	gate / mutate	before the workload enters etcd
Runtime audit	KubeArmor	LSM, `audit` posture	while it runs, at the kernel
Runtime detect	Falco	tracepoint match	when it behaves anomalously
Continuous	Trivy	scheduled scan	long after it was deployed

Four engines, zero shared dependencies — no common operator, no shared CRDs, no single point of failure. If Kyverno's webhook is down mid-upgrade, the kernel layer is still watching. If a zero-day slips past every lock, the cameras still record it. If everything passes today, the Inspector still finds the CVE that lands tomorrow. Defense-in-depth isn't four tools; it's four tools that don't know about each other, so no single failure blinds the whole system.

And notice what isn't here: a dashboard to log into, an SSO redirect to configure, a UI to gate. The entire security posture is kubectl get against custom resources. Security you query, not security you click. For a GitOps platform, that's the point — every policy, every rule, every exception is a commit, reviewable and revertible, with no console drift.

What's Next? (Don't Miss Out!)

The locks are set. The cameras are live. The inspector is on patrol. The house is finally safe to move into — and that's exactly what happens next.

With the security system armed first, the tenants can finally arrive: the private Harbor registry, Tekton pipelines, MLflow, Kubeflow, the data platform — every one of them landing in a cluster that rejects a bad manifest at the door, records every syscall at the kernel, alerts on anything anomalous, and never stops scanning its own inventory. That was the whole point of doing this in the empty house.

And the two honest gaps stay on the board where I can see them: Falco's alerts still only reach stdout, and KubeArmor is still in audit posture with zero policies. Neither is a surprise and neither is an accident — one needs a webhook in Vault, the other needs a few weeks of audit data before enforcement is anything other than an outage generator. Writing them down is how they stay next actions instead of quietly becoming the permanent state.

That's the next episode: moving the workloads in — safely. Follow along so you don't miss how the platform goes from secured and vacant to secured and busy. 🔐

the ArgoCD security-stack ApplicationSet — all five child apps across two sync waves, Synced + Healthy. The whole security system, GitOps-reconciled, in one view.

kyverno #kubearmor #falco #trivy #devsecops

Subscribe and follow me on [LinkedIn] to catch my upcoming articles.

Kubeadm to RKE2: Transformed My K8s Homelab Into Production‑Grade Infra

George Ezejiofor — Wed, 29 Apr 2026 19:05:09 +0000

Scaling to 21 Nodes Across 3 Datacenters With Terraform and Ansible

INTRODUCTION:

The Evolution of an Empire

Most homelabs start as experiments — a single node, a few containers, and a steep learning curve. Mine stopped being a project the moment it evolved into a 21‑node, multi‑datacenter powerhouse. With control‑plane nodes stretched across physical zones and specialised worker pools for high‑IOPS databases, GPU‑accelerated AI workloads, and latency‑sensitive apps, the stakes changed. I wasn’t just “running services” anymore; I was operating real infrastructure.

That was the turning point.

For years, kubeadm was my gateway into Kubernetes. It taught me the fundamentals, the internals, and the discipline. But at 21 nodes across three physical zones, the learning phase ended — and the limitations began.

High availability felt like fragile choreography.
Upgrades demanded a mix of surgical precision and sheer luck.
Security hardening meant manually chasing CIS Benchmark alignment across dozens of nodes.
Automation became a battle against the tool instead of a path to progress.

I didn’t want a cluster I had to babysit.
I wanted one that was deterministic, repeatable, and rebuildable — a platform that behaved like production, not a handcrafted science project.

The “Boring” Power of RKE2

RKE2 — Rancher’s next‑generation Kubernetes distribution, often known as “RKE Government.” It blends the simplicity of K3s with the enterprise‑grade security of RKE1. Unlike kubeadm, which hands you a toolbox and expects you to assemble the cluster yourself, RKE2 is a secure‑by‑default engine. It replaces Docker with containerd, launches control‑plane components as static pods, and enforces hardened defaults from the moment the node boots.

But the real reason I moved wasn’t the binary.
It was the operational excellence.

RKE2 didn’t just simplify my cluster — it transformed it. My 21‑node migration proved what the documentation promises: RKE2 is built for environments where security, consistency, and automation matter.

It gave me:

Security‑first architecture — CIS‑hardened by default and FIPS 140‑2 readiness out of the box.
True HA without the headache — native etcd quorum management that thrives across multi‑zone topologies.
GitOps‑ready automation — a single declarative config that makes Terraform and Ansible work with the cluster, not against it.

This wasn’t just a software upgrade.
It was the moment my homelab crossed the line from a weekend project into a self‑healing, production‑grade, automated empire.

The Blueprint - Architecture at Scale

The 3‑Zone Topology - Why Three Datacenters
Distributed systems love odd numbers, and etcd is the perfect example.
Quorum is everything - and three nodes is the sweet spot.

Nodes	Quorum Required	Fault Tolerance
1	1	0 failures
2	2	0 failures
3	2	1 failure
4	3	1 failure
5	3	2 failures

Each zone is:

A separate Proxmox host
A separate subnet
A separate failure domain

If one rack loses power, or a host crashes, the system degrades — but does not fail.

💡 Pro Tip: Network Reality

You don’t need 10GbE for this to work.

etcd traffic is lightweight (~1–2 Mbps)
What matters is latency, not bandwidth
Keep inter-zone latency under ~5ms for stable consensus

Cilium’s native routing mode handles cross-zone traffic cleanly even on 1GbE.

Node Categorisation — Designing for Workload Isolation

At small scale, all nodes look the same.

At 21 nodes, that approach breaks.

Different workloads have fundamentally different resource profiles:

Databases need consistent IOPS and memory
Applications need horizontal scalability
GPU workloads need isolation and scheduling guarantees

Mixing them leads to contention, unpredictability, and performance degradation.

So the cluster is split into specialised pools.

The Quorum Trap (Critical for 3‑DC Clusters)

Even though RKE2 simplifies HA dramatically, etcd still requires an odd number of masters.

3 masters = safe
2 masters = dangerous
4 masters = pointles
5 masters = overkill for a homelab

RKE2 handles the etcd cluster automatically, but kubeadm would require:

Manual etcd initialization
Manual peer configuration
Manual certificate distribution
Manual recovery steps

This is one of the biggest reasons kubeadm collapses at scale - and RKE2 thrives.

My Physical Layout

Zone	Proxmox Host	Network	Control Plane	Workers
dc1	pve‑01 (Rack A)	10.0.1.0/24	rke2-master-dc1-a	2 app + 3 db + 1 gpu
dc2	pve‑02 (Rack B)	10.0.2.0/24	rke2-master-dc2-a	2 app + 3 db + 1 gpu
dc3	pve‑03 (Rack C)	10.0.3.0/24	rke2-master-dc3-a	2 app + 3 db + 1 gpu

Node Categorisation — The Power of Specialised Pools

Mixed workloads are dangerous without isolation.
A single noisy ML job can starve your database or API layer.

That’s why I built dedicated pools, each with its own CPU, GPU, RAM, storage, taints, and scheduling rules.

Node Pools

Pool	Count	vCPU	RAM	Storage	Taint	Workloads
Control-plane	3	8	16GB	50GB SSD	CriticalAddonsOnly=true:NoSchedule	etcd, API server
App	6	4	8GB	30GB SSD	None	APIs, web apps
DB	9	8	16GB	100GB NVMe	database=true:NoSchedule	PostgreSQL, Redis
GPU	3	16	64GB	SSD + NVIDIA	nvidia.com/gpu=true:NoSchedule	ML workloads

Taint Strategy

# DB nodes: Only workloads that tolerate database=true
kubectl taint nodes rke2-worker-dc1-db-a database=true:NoSchedule

# GPU nodes: Only workloads that request GPU resources
kubectl taint nodes rke2-worker-dc1-gpu-a nvidia.com/gpu=true:NoSchedule

Label Strategy

topology.kubernetes.io/zone: dc1
node-type: app|db|gpu
pool: general|database|ml

This ensures:

Databases stay on DB nodes
ML workloads stay on GPU nodes
Apps stay on app nodes
And nothing fights for resources it shouldn't

This is how you prevent noisy-neighbour problems at scale.

The Networking Layer - Cilium + kube‑vip

Why Cilium Instead of Calico?

Feature	Calico	Cilium
eBPF-based	❌	✅
Multi-zone routing	✅	✅ (faster)
Network policies	✅	✅
Observability	❌	✅ (Hubble)
Service mesh	⚠️	✅ (built-in)

Building the Foundation (Terraform)

If Proxmox is the cleared land for my estate, Terraform is used to work on the foundation. At a 21-node scale, consistency is everything. If the foundation is crooked, the rest of the house will eventually lean. Ansible will be used to elevate the building (Kubernetes setup), then Gitops (ArgoCD) will be used for decoration (Furniture).

By using Terraform, I ensure that every one of my 21 nodes is a "perfect twin" of its peers in the same pool.

The Hybrid Workflow: Local Execution, Cloud Intelligence

I execute my Terraform runs locally for speed, but I offload the "brains" of the operation to Terraform Cloud (TFC).

The Source of Truth: TFC stores my state files securely. This ensures that no matter where I run the code from, the infrastructure state is locked and versioned.
Secret Vaulting: Sensitive data - like my Proxmox API tokens and SSH keys - never touch my local machine or GitHub. They live as encrypted variables within the Terraform Cloud workspace.

Constructing the Footprint

Instead of manually clicking through the Proxmox GUI 21 times, I use a modular approach. I've defined a "Blueprint" (module) for a generic Ubuntu 24.04 VM, which I then deploy across my three zones:

The Strategy: I manage pools, not individual nodes. I tell Terraform I want 9 Database nodes, and it calculates the distribution across the three datacenters automatically.
Pool-Specific Hardware: Terraform provisions each pool with its unique resource requirements - NVMe storage for database nodes, GPU passthrough for ML nodes, and learner specs for the general app pool.
The Identity: Terraform handles the static IP assignment and Cloud-Init injection. When a node boots, it already knows its name, its IP, and which Kubernetes pool it belongs to.
Remote state management: Terraform state file is managed in Terraform Cloud. But I run the command locally. I might use remote run too if I create a self-hosted runner later in future with kubernetes pod.

The foundation is poured. We have 21 heartbeating Ubuntu 24.04 nodes, perfectly distributed across three zones, with static IPs and pool-specific hardware.

Now, we hand the baton to Ansible to take us from raw VMs to a security-hardened Kubernetes powerhouse.

Building to the Roof (Ansible)

With the foundation laid by Terraform, Ansible is the construction crew that turns bare VMs into a hardened, HA RKE2 cluster. Think of Terraform as the concrete slab and Ansible as the framing, insulation, and roof - everything that makes the house livable and resilient.

The Role of lab-lab-rke2

lab-lab-rke2 is the single Ansible role that codifies your cluster build rules.

What it does, at a glance:

OS hardening for Ubuntu 24.04 (kernel tuning, sysctl, package hygiene)
RKE2 installation and configuration (server vs agent)
kube‑vip and Cilium configuration hooks for HA and networking
Node labeling and tainting for pool enforcement
GPU runtime setup and driver validation on GPU nodes
Idempotent checks so repeated runs are safe

This role is the repeatable recipe that guarantees every node is built the same way.

The Bootstrap Sequence (Deterministic, Step‑by‑Step)

OS Prep - apply security hardening, install prerequisites, configure SSH and time sync.
First Master Init - install RKE2 server, start the first control‑plane, generate initial certificates.
HA Master Join - sequentially join the remaining masters to form etcd quorum and control plane.
Worker Join - register app, db, and GPU workers; apply labels and taints.
Post‑Install Validation - verify control‑plane health, etcd quorum, Cilium status, and kube‑vip VIP ownership.

Why sequential matters: the first master must be healthy before others join to avoid split‑brain and to ensure deterministic certificate distribution.

Idempotency and Safety

Ansible's idempotency is the operational superpower here.

One variable change in the role updates only what's necessary.
Re-running the playbook is safe and becomes the standard maintenance operation.
Playbook checks validate preconditions and abort safely if something is off.

This turns maintenance into a predictable, low‑risk activity rather than a manual firefight.

Integration with Terraform Outputs

Ansible consumes Terraform outputs as its inventory and input source.

How it flows: Terraform outputs node IPs, hostnames, and tags → Ansible reads terraform output -json or the TFC API → lab-lab-rke2 targets nodes by role and zone.
Result: No guesswork. No manual SSH lists. The bootstrap is fully data‑driven.

This tight coupling is what makes the build deterministic and reproducible.

Rolling Upgrades and Day‑2 Operations

Your Ansible role also codifies safe upgrades and routine ops.

Rolling upgrades: drain → upgrade RKE2 binary → restart → uncordon.
Multi‑DC awareness: upgrade order preserves quorum and minimizes risk by upgrading workers per DC and masters with quorum safety.
Nuke & Pave: destroy a node, let Terraform recreate it, and let Ansible rejoin it - zero drift guaranteed.

Upgrades become a scripted, auditable operation rather than an ad‑hoc event.

The Control Room (Rancher UI and Unified SSO)

Rancher is the single pane of glass for cluster operations, and Keycloak is the identity brain that powers seamless Single Sign‑On across the platform. I configured Rancher to use OIDC via Keycloak, and Keycloak federates authentication to Google and GitHub, so users sign in with familiar accounts.

Security: No more "admin/admin" credentials. Access is governed by GitHub Organisation membership or Google Workspace groups.
Auditability: Every action taken in the UI is tied to a real identity.
Centralisation: One login to rule them all. Once you are in via Keycloak, you have access to the entire multi-DC dashboard.

Redirects me to keycloak

Conclusion: From Project to Production

Moving from kubeadm to RKE2 wasn't just about changing a tool; it was about adopting a "Production-First" mindset. By treating my homelab like a professional datacenter using Terraform for the foundation, Ansible for the build, and RKE2 for the hardened engine I've created a 21-node environment that is self-healing, deterministic, and secure.

If Proxmox is my land and RKE2 is my building, I am now ready to "move in" the furniture. The cluster is rock-solid, the foundation is level, and the lights are on.

What's Next? (Don't Miss Out!)

This is just the beginning of the "Empire" series. I will be deep-diving into every layer of this stack, showing you how to turn raw compute into a fully automated platform.
Subscribe and follow me on [LinkedIn] to catch my upcoming articles on:

Advanced Rancher UI: Deep-dives into SSO with SAML, Keycloak, and Google/GitHub providers.
The Storage Stack: Deploying Ceph (Operator and Cluster) for distributed persistent data.
Cloud Native Postgres (CNPG): Running enterprise-grade databases.
GitOps Mastery: Automating everything with ArgoCD.
The Security Vault: Secrets management with HashiCorp Vault.
AI Readiness: Installing the NVIDIA GPU-Operator.
External Access: Securing apps with OAuth2-Proxy and SSO.

Let's connect and build something massive. See you in the next one!

Homelab HA Kubernetes Cluster Upgrade: My New Shrine / Altar

George Ezejiofor — Fri, 03 Apr 2026 21:24:36 +0000

INTRODUCTION

In the beginning, there was MicroK8s on a Mac Studio. It was fast, it was ARM64, but it was lonely. Today, I stand before a high-availability monument built on Proxmox, orchestrated by Terraform, and kept in holy alignment by FluxCD.

Not long ago, my entire Kubernetes universe lived inside a humble Mac Studio - a single microk8s cluster with 6 nodes running on ARM64. It was cute, quiet, and completely unfit for the kind of multi‑DC, production‑grade nonsense I wanted to learn.

So I burned it down. And built this new place of worship.

Today, I run a high‑availability kubeadm cluster across three bare‑metal Proxmox Datacenters, all managed with Terraform, Ansible, and FluxCD. No cloud vendor lock‑in. No magic. Just a rack full of metal, a bunch of cables, and a lot of terminal time.

This is the story of my shrine - and how you can build one too.

UGLY WIRING:

MAJOR REASON WHY I CALLED IT SHRINE 😂

Traffic Flow at a Glance

Before we dive into the layers, here's how the traffic moves from my "pulpit" (Mac Studio) to the "shrine" (the cluster):

No inbound holes – all management traffic originates from my Mac or the cluster itself (GitOps pulls). This is how real datacenters work.

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🖥️ macOS COMMAND CENTER (The Pulpit)                         │
│                                                                                      │
│              kubectl  │  Terraform  │  Ansible  │  Flux CLI  │  Git                  │
│                                                                                      │
│                        (All management tools installed locally)                      │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                          SSH │ API (HTTPS) │ Git (SSH/HTTPS)
                                          │
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🛡️ OPNsense Firewall (10.0.1.1)                              │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ DHCP Server     │  │ Static DHCP     │  │ WireGuard VPN   │                      │
│   │ 10.0.1.100-200  │  │ MAC → IP Pinning │  │ Remote Access   │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   • Split-Horizon DNS: *.georgehomelab.com → 10.0.1.x                               │
│   • Gateway for all Proxmox + Kubernetes traffic                                     │
│   • Firewall rules: WAN → LAN passes for management                                  │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ LAN (10.0.1.0/16)
                                          │ 2.5GbE Links
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🔌 Zyxel XMG1915-10E Switch                                  │
│                                                                                      │
│                     Star topology │ 8× 2.5GbE + 2× SFP+                              │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
              ┌───────────────────────────┼───────────────────────────┐
              │                           │                           │
              ▼                           ▼                           ▼
┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│                         │ │                         │ │                         │
│  🏗️ Proxmox Node 1      │ │  🏗️ Proxmox Node 2      │ │  🏗️ Proxmox Node 3      │
│  (proxmox-dc-1)         │ │  (proxmox-dc-2)         │ │  (proxmox-dc-3)         │
│  10.0.1.10              │ │  10.0.1.11              │ │  10.0.1.12              │
│                         │ │                         │ │                         │
│  • Local ZFS Storage    │ │  • Local ZFS Storage    │ │  • Local ZFS Storage    │
│  • vmbr0 Bridge         │ │  • vmbr0 Bridge         │ │  • vmbr0 Bridge         │
│  • NFS Client (Backups) │ │  • NFS Client (Backups) │ │  • NFS Client (Backups) │
│                         │ │                         │ │                         │
│  Terraform → VM Creation via Proxmox API (telmate/provider)                          │
│  Packer → Ubuntu Cloud-Init Templates                                                │
│                                                                                      │
└─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
              │                           │                           │
              │ Cloud-Init DHCP (Static Reservations → Predictable IPs)                │
              │                           │                           │
              └───────────────────────────┼───────────────────────────┘
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         ☸️ HA Kubernetes Cluster (kubeadm)                           │
│                                                                                      │
│   ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│   │ Control Plane Node 1    │ │ Control Plane Node 2    │ │ Control Plane Node 3    │
│   │ k8s-cp-1                │ │ k8s-cp-2                │ │ k8s-cp-3                │
│   │ 10.0.1.110              │ │ 10.0.1.111              │ │ 10.0.1.112              │
│   │                         │ │                         │ │                         │
│   │ • etcd (stacked)        │ │ • etcd (stacked)        │ │ • etcd (stacked)        │
│   │ • kube-apiserver        │ │ • kube-apiserver        │ │ • kube-apiserver        │
│   │ • kube-vip (VIP)        │ │ • kube-vip (VIP)        │ │ • kube-vip (VIP)        │
│   └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
│                                                                                      │
│   ┌─────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐
│   │ Worker Node 1           │ │ Worker Node 2           │ │ Worker Node 3           │
│   │ k8s-worker-1            │ │ k8s-worker-2            │ │ k8s-worker-3            │
│   │ 10.0.1.120              │ │ 10.0.1.121              │ │ 10.0.1.122              │
│   │                         │ │                         │ │                         │
│   │ • Calico CNI (BGP)      │ │ • Calico CNI (BGP)      │ │ • Calico CNI (BGP)      │
│   │ • kube-proxy            │ │ • kube-proxy            │ │ • kube-proxy            │
│   │ • Workload Pods         │ │ • Workload Pods         │ │ • Workload Pods         │
│   └─────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘
│                                                                                      │
│   Pod CIDR: 10.244.0.0/16 │ Service CIDR: 10.245.0.0/16 │ MetalLB: 10.0.1.200-210  │
│                                                                                      │
│   🔧 Bootstrapped entirely by Ansible (kubeadm playbook)                            │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ GitOps Sync (Outbound Only)
                                          │ FluxCD pulls from GitHub (no inbound!)
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                         🔄 FluxCD System (Inside Cluster)                            │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ source-         │  │ kustomize-      │  │ helm-           │                      │
│   │ controller      │  │ controller      │  │ controller      │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐                      │
│   │ notification-   │  │ image-reflector-│  │ image-          │                      │
│   │ controller      │  │ controller      │  │ automation-     │                      │
│   └─────────────────┘  └─────────────────┘  └─────────────────┘                      │
│                                                                                      │
│   • Deployed as part of Ansible playbook (not a separate step)                       │
│   • Continuously reconciles cluster state with Git                                   │
│   • Auto-heals configuration drift                                                   │
│                                                                                      │
└─────────────────────────────────────────┬────────────────────────────────────────────┘
                                          │
                                          │ HTTPS/SSH (Outbound Pull)
                                          │
                                          ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                      │
│                              📦 GitHub Private Repository                            │
│                                                                                      │
│   ┌─────────────────────────────────────────────────────────────────────────────┐    │
│   │  clusters/prod/                                                             │    │
│   │  ├── flux-system/          # Flux bootstrapping config                      │    │
│   │  │   ├── gotk-components.yaml                                               │    │
│   │  │   └── gotk-sync.yaml                                                     │    │
│   │  ├── apps/                  # Application deployments                       │    │
│   │  │   ├── metallb/                                                          │    │
│   │  │   ├── istio-ingress/                                                    │    │
│   │  │   └── prometheus-stack/                                                 │    │
│   │  └── infrastructure/        # Cluster-wide config                          │    │
│   │      ├── namespaces.yaml                                                   │    │
│   │      └── storage-class.yaml                                                │    │
│   └─────────────────────────────────────────────────────────────────────────────┘    │
│                                                                                      │
│   🔑 Source of Truth: Every change starts as a PR, reviewed, merged, then applied   │
│                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────┘

Level 1: The Physical Layer (The Foundations)

Every altar begins with something tangible.

Hardware: A fleet of Minisforum MS-01 machines acting as compute Datacenter (96GB RAM and 8GB nvidia GPU on each machine) . That's Total of 288GB RAM and 24GB nvidia GPU in the three _Minisforum MS-01
_

Network Entry Point: My Mac Studio (the “pulpit”), connected via Wi-Fi

Firewall: OPNsense bridging external (192.168.1.x) to internal lab network (10.0.1.x)

Out-of-Band Access: TinyPilot Voyager 2a and TESmart 4‑port HDMI KVM — BIOS-level control even when the OS is down

Switch: Zyxel XMG1915-10E (2.5GbE + SFP+) is the Central Nervous System. With High-Velocity East-West Traffic ( Low Latency / High Throughput for etcd and storage )

Why I worship here:

Physical simplicity enables logical complexity.

No mystery cables. Everything is deliberate. This playground makes me the opportunity to play with any cloud-native tool with ease.

Level 2: The Infrastructure Layer (Proxmox Datacenter)

Before automation, there must be a foundation.

Proxmox VE installed manually on all three Minisforum MS-01 machine

Clustered into a single datacenter abstraction

Networking:

vmbr0 → Kubernetes network

Static host IPs:

10.0.1.1x
10.0.1.1x
10.0.1.1x

Gateway: 10.0.1.x (OPNsense)

Storage:

Local ZFS (NVMe)
NFS for shared ISO + backups

The ritual:

I installed the first Proxmox VE manually on each machine via TinyPilot’s virtual media from my MAC-Studio browser over wifi.

No HDMI cable ever touched my desk.

Level 3: The Node Layer (Terraform Automation)

I no longer click buttons to create infrastructure.

I declare it.

Using the Proxmox Terraform provider, I define:

VM CPU, memory, disk
Network interfaces
Clone source (Ubuntu template from Packer)

resource "proxmox_vm_qemu" "k8s_node" {
  for_each = var.nodes

  name        = each.value.name
  target_node = each.value.proxmox_node
  clone       = "ubuntu-24-04-template"
  cores       = each.value.cores
  memory      = each.value.memory

  network {
    model     = "virtio"
    bridge    = "vmbr0"
    ipconfig0 = "ip=dhcp"
  }
}

The DHCP Decision (And Why It Matters)

This was one of the most important lessons in my journey.

In my old Mac Studio setup, I used pure DHCP for Kubernetes nodes.

It worked… until every restart broke my cluster access.

What went wrong?

Control plane nodes changed IPs
kubeconfig became invalid
API server endpoints broke
etcd stability was at risk

Even with 3 control planes, the cluster wasn’t truly stable.

Why Not Static IPs?

Because static IPs inside the OS mean:

Manual netplan configuration
Hardcoding network logic into templates
Reduced rebuild flexibility

That’s not how cloud-native systems behave.

The Solution: DHCP + Reservations

I used DHCP everywhere — but configured static reservations in OPNsense.

✔ Nodes auto-configure
✔ IPs never change
✔ Rebuilds are seamless
✔ etcd remains stable

💡 The Real Insight

Kubernetes doesn’t care how IPs are assigned — only that they don’t change.

Level 4: The Cluster Layer (Ansible + Kubeadm)

Once the infrastructure exists, it must be transformed.

Using Ansible:

- OS hardening
- Swap disabled
- containerd installed
kubeadm, kubelet, kubectl configured

HA Control Plane

3 control plane nodes
Stacked etcd (homelab-friendly)
kube-vip for API virtual IP

Level 5: The Application Layer (GitOps with FluxCD)

This is where everything changes.

Instead of imperative deployments or declarative deployment with kubectl, I use GitOps FluxCD.

GitOps From Day One

FluxCD is not an add-on.

It is deployed during cluster creation via Ansible.

That means:

Cluster is GitOps-ready immediately
No manual bootstrap later
No drift from day one

The Pull Model

Flux runs inside the cluster
Watches Git repository
Pulls changes automatically

No inbound access required.

Traffic Flow

Mac Studio (192.168.1.x)
        │
        ▼
OPNsense Firewall (10.0.1.x)
        │
        ▼
Proxmox Cluster (10.0.1.1x–1x)
        │
        ▼
Kubernetes Nodes (DHCP → Reserved IPs)
        │
        ▼
FluxCD Controllers (inside cluster)
        │
        ▼
GitHub (OUTBOUND pull model)

Key Insight:

❌ GitHub never connects to your cluster
❌ No firewall holes needed
✅ Flux initiates outbound sync

** Current State of the Shrine**

3 control plane nodes ✅
3 worker nodes ✅
etcd cluster healthy ✅
Flux controllers distributed across nodes ✅
Calico networking active ✅

This is no longer a lab.

It is a self-healing platform.

What I Learned

DHCP + reservations is the sweet spot
etcd requires stable identity, not static config
GitOps removes human drift completely
Terraform + Ansible + FluxCD = powerful combination
Firewalls must allow internal routing for automation
Never use root API for automation — use scoped tokens

What’s Next on the Altar

Ceph or Longhorn for HA storage
Velero for cluster backups
External Secrets + Vault
Cluster autoscaler experiments

Final Words

This homelab is more than a project.

It is a practice ground for real-world platform engineering.

The move from a single ARM node to a distributed HA cluster wasn’t just an upgrade in hardware — it was an upgrade in mindset.

My Mac Studio is no longer the host.

It is the pulpit.

The Shrine runs independently.

If you’re thinking of building something like this — do it.
Start small. Break things. Rebuild them better.

Now go build your own altar. 🛐

🤝 Stay Connected

Found this guide helpful? Follow my journey into Homelabing on LinkedIn! Click the blue LinkedIn button to connect: George Ezejiofor . Let’s keep building scalable, secure cloud-native systems, one project at a time! 🌐🔧