George Ezejiofor

Posted on Oct 17 Originally published at georgeezejiofor.com

Brownfield Kubeflow on RKE2: Swapping Dex & SeaweedFS for Keycloak & Ceph RGW

#kubernetes #kubeflow #mlops #devops

Upstream Kubeflow assumes a greenfield cluster and ships its own Dex, SeaweedFS and MySQL, with its own oauth2-proxy in front. My homelab already had better versions of most of those. This is the **brownfield* deploy on an HA RKE2 cluster: every reference component I ripped out (and the one I haven't yet), what I wired in its place, and the auth chain that ate most of the debugging time.*

The Kubeflow Central Dashboard, logged in through Keycloak SSO (not Dex), with the whole platform iframed into one tab.

Introduction

When the platform assumes it's the only thing in the cluster

Kubeflow was designed as if it would have the cluster to itself, and it still shows. Point the reference installer at an empty cluster and it works well: nine controllers, four web UIs, two workflow engines, an inference layer and a per-user namespace system, all iframed into one dashboard and running in twenty minutes.

Point it at a cluster that already has an identity provider, an object store, a database fleet, a service mesh and an SSO layer guarding ten other apps, and you're in for a fight. Kubeflow never asks what you already run. The v1.11 multi-user install brings Dex for identity (behind its own oauth2-proxy, wired into Istio as an external authorizer), SeaweedFS for object storage, MySQL for state, and a mesh-wide trust model that doesn't match any real cluster I know of.

That's the gap between a greenfield deploy and a brownfield one. Greenfield, you accept whatever the platform brings. Brownfield, the platform moves into a neighbourhood that was there first. Mine already had Keycloak issuing every realm, Rook-Ceph serving S3 to half the stack, CloudNativePG holding everyone's relational state, and oauth2-proxy in front of ten apps.

I could run two of every backend, or rewrite Kubeflow's integration layer to talk to what was already in the building. I picked the rewrite. It cost roughly six weekends and a kustomize overlay per component, and the platform now fits the cluster it runs on.

Here's the full list: every reference component upstream Kubeflow ships, what I put in its place, and why the stock one didn't fit.

Stock Kubeflow manifests ship…	I substituted…	Why the stock one didn't fit
Dex (bundled OIDC identity provider)	Keycloak, central realm `kubeflow`	One IdP already issues every app's realm. A Kubeflow-only Dex is a second identity island to run, patch and secure.
oauth2-proxy + Dex as Istio ext_authz (every request checked by an external authorizer at the gateway)	My own `oauth2-proxy-kubeflow` in reverse-proxy mode against Keycloak, with ext_authz kept only for `/notebook/*`	It's the same OIDC pattern that already gates ~10 apps on per-app cookie scopes, and proxy mode lets one instance route all the static-URL UIs.
SeaweedFS (KFP artifact store, exposed behind the `minio-service` name; MinIO in older releases)	Rook-Ceph RGW (S3) via an `ObjectBucketClaim`	One storage floor already serves S3 to MLflow and the rest of the data stack. Every KFP S3 client points straight at `https://s3.georgehomelab.com:443` (RGW is TLS-only); VSO maps `AWS_*` → `accesskey`/`secretkey`.
MySQL (KFP metadata DB)	Not swapped yet: still the bundled MySQL. CloudNativePG `kubeflow_metadata_db` is provisioned and waiting	CNPG already hosts everyone's relational state with backups to Ceph RGW. The database and its Vault credentials exist; the KFP Deployments don't point at them yet (I checked live).
Principal-scoped AuthorizationPolicies (allow only the ingress-gateway service account)	Path-based AP rules (`paths: ["/*"]`)	A PERMISSIVE mesh doesn't populate the SPIFFE principal reliably, so principal-scoped rules silently deny everything.
Bundled cert-manager + self-signed Issuer	Existing cert-manager + Let's Encrypt (Cloudflare DNS-01)	Real, browser-trusted certs are already automated cluster-wide; no self-signed CA to hand out.
Kubeflow's Spark-on-K8s overlay	Existing mlops-spark-operator	The MLOps stack already runs a Spark operator; two of them fight over the same CRDs.
Istio bundled by the installer	Existing RKE2 Istio 1.30 mesh (PERMISSIVE)	The mesh predates Kubeflow and runs everything else. Kubeflow gets injected into it; an ML installer doesn't get to replace the service mesh.

The rest of this piece walks through those swaps, the "huge CRD" trap that breaks client-side apply, the per-Profile governance Kyverno generates, and the Auth Chain Saga, where RBAC: access denied turned up at five different layers over several days before login finally worked.

Why Kubeflow, and which release

I pinned to v1.11.0 of the kubeflow/manifests repo, the most recent immutable release tag when I started. Don't be fooled by releases/release-1.11/ on master: it's a release-process tracker (a README about dates and the team). The real installer entry point is example/kustomization.yaml, and that differs between master HEAD (a moving target) and v1.11.0 (immutable). I wanted a build I could reproduce, so I took the tag.

Most of that time went on the auth chain (the Auth Chain Saga and the notebook routing pivot below). The heavy KFP overlay and the Model Registry integration were the runners-up.

The honest head-to-head: Kubeflow vs SageMaker/Vertex, Flyte, Metaflow, Ray, and MLflow

People ask me "should I use Kubeflow or MLflow?" a lot, and the question doesn't quite work. It's like asking whether to buy a kitchen or a knife. Kubeflow is a nine-component platform, and almost everything people compare it to replaces exactly one of those components. To compare fairly you have to sort the field by the layer each tool occupies:

Whole self-hosted platform (notebooks + pipelines + training + HPO + serving + registry + multi-tenant portal): Kubeflow, plus the open all-in-ones MLRun and Determined AI.
Managed cloud platform (all of the above, on someone else's hardware): AWS SageMaker, GCP Vertex AI, Azure ML.
Pipeline / orchestration only: Flyte, Metaflow, Airflow, Prefect, Dagster. These compete with KFP, which is one of the nine boxes.
Experiment tracking + registry only: MLflow, Weights & Biases, ClearML. A component, not a platform (I run MLflow inside this Kubeflow).
Serving only: KServe (which is part of Kubeflow), Seldon, BentoML, Triton.
Distributed compute only: Ray (KubeRay), which runs alongside Kubeflow rather than replacing it.

In other words, MLflow, Seldon and Ray are things you run inside Kubeflow, not alternatives to it. The only real peers to the whole platform are the managed clouds and a couple of open all-in-ones:

	Notebooks	Pipelines	HPO	Serving	Registry	Multi-tenant	Self-hosted	Best at
Kubeflow	✅	✅ KFP	✅ Katib	✅ KServe	✅	✅ Profiles	✅	Breadth, on your own hardware, no lock-in
SageMaker / Vertex	✅	✅	✅	✅	✅	✅	❌ cloud	Zero-ops DX, if you'll pay + go off-prem
Flyte	❌	✅✅ typed	⚠️	❌	⚠️	✅	✅	Best pipeline authoring: typed, reproducible
Metaflow	❌	✅	⚠️	❌	✅ tracking	⚠️	✅	Data-scientist Python DX; hide the infra
Ray (KubeRay)	⚠️	✅ tasks	✅ Tune	✅ Serve	❌	⚠️	✅	Distributed compute (train/tune at scale)
MLflow	❌	❌	❌	⚠️	✅✅	❌	✅	Experiment tracking + model registry

Where each one wins:

SageMaker / Vertex / Azure ML: the best managed DX, zero platform ops, elastic GPU infra, wired into cloud data and IAM. If you already live in a cloud and care more about time-to-value than control, cost and data sovereignty, managed wins clearly. Kubeflow's answer is the flip side: no cloud lock-in, no per-GPU-hour rental markup, data never leaves my rack, and it runs on hardware I own. Think of it as a self-hosted SageMaker for people who want to own the platform.
Flyte: honestly, a better pipeline engine than KFP. Strongly-typed Python, first-class versioning and caching, cleaner local dev. If pipelines are the whole job and you don't need notebooks, serving, HPO or a multi-tenant portal, Flyte is the nicer tool. Kubeflow wins on breadth: the nine boxes plus the portal that puts them in one tab.
Metaflow: a pleasant Python DX from Netflix, aimed at data scientists who don't want to think about infrastructure. It covers orchestration and tracking, not serving, HPO or multi-tenancy, and its Kubernetes support is thinner than its AWS support.
Ray (KubeRay): the best tool for distributed compute: training big models, hyperparameter tuning, RL, distributed data. It doesn't replace Kubeflow, it sits under it (Katib overlaps Ray Tune; KServe can serve Ray). If your bottleneck is Python-native distributed compute, run Ray alongside Kubeflow.
MLflow / W&B: the best experiment tracking and registry around. That's why MLflow is a tile in my Central Dashboard rather than something I picked instead of Kubeflow.
Seldon / BentoML / Triton: serving specialists. KServe covers most cases, but for advanced inference graphs or GPU-optimized Triton runtimes they can beat it on serving alone.

Why Kubeflow here, then? It's the one open-source platform that gives me all nine boxes (Jupyter notebooks, KFP pipelines, Trainer V2, Katib HPO, KServe serving, Model Registry and per-user Profiles) on my own RKE2 + GPU hardware, integrated with the stacks I already run (Keycloak, Ceph, CNPG, Istio), with no cloud lock-in. I couldn't find anything else in open source that complete and also self-hosted.

The catch is complexity. Kubeflow has a lot of moving parts, an auth chain that took several days to untangle, and an installer that assumes a greenfield cluster. If all you need is pipelines, Flyte has a fraction of the operational surface. If you'll pay for managed, SageMaker takes the ops away entirely.

Kubeflow only makes sense when you want the full platform, self-hosted, on your own hardware, and you're prepared to pay the complexity cost once. If you don't need all that, a specialist tool or a managed service on a credit card is the better call.

The cluster Kubeflow is landing on

The facts that matter:

RKE2 with five masters and a mix of worker pools (app, data, GPU). HA control plane via kube-vip. The whole lab powers off every night and cold-boots in the morning, which matters more than I expected (see the cold-boot race below).
Cilium in eBPF mode, with Hubble enabled.
Istio 1.30 as the service mesh. The default mode is PERMISSIVE, and per-namespace PeerAuthentication CRs flip to STRICT where it matters. istio-system's default stays PERMISSIVE because the API server has to reach admission webhooks in injected namespaces without holding a client cert.
Rook-Ceph for storage, with three SCs: ceph-block (RBD, RWO), ceph-filesystem (CephFS, RWX), ceph-bucket (RGW, S3 via OBC).
CloudNativePG for relational state. The MLOps cluster postgres-mlops-cnpg-cluster hosts kubeflow_metadata_db, ready for Kubeflow Pipelines but not in use yet.
Vault + Vault Secrets Operator, auto-unsealed by a separate transit Vault in its own vault-transit namespace (Shamir, 1 share).
Keycloak as the OIDC provider for every app. Realms and clients are managed by Terraform; there are no Keycloak operator CRDs on this cluster.
ArgoCD, driven by AppSets. Each major area (mlops-stack, kubeflow-stack, cicd-stack, security-stack, monitoring-stack) has its own AppSet generating a child App per component.
Kyverno for admission policy and generate rules. At the time, the cluster-wide make-configmaps-immutable policy was the one that bit hardest on this build.

The kubeflow-stack.yaml AppSet in kubernetes-addons/0-georgehomelab-argocd-projects/2-argocd-applicationsets/ phases the rollout. Sync waves run from 1 (foundation) to 8 (the SSO front door), across 24 child Applications. Each component is its own Application, so I can debug it in isolation.

Kubeflow on RKE2, what I replaced from upstream: the cluster's Istio gateway, oauth2-proxy as a reverse proxy and Keycloak in front of the Central Dashboard, its six apps, and a foundation of GPUs, bundled MySQL, Ceph object storage and CloudNativePG, each marked replaced, upstream or added

Kubeflow as deployed: oauth2-proxy → Central dashboard → apps, MLflow beside it, and every swap from upstream tagged.

The ArgoCD UI filtered to kubeflow-*: the 24 child Applications from the kubeflow-stack AppSet, all Synced/Healthy, in sync-wave order.

Prerequisites first

Phase 0 is everything that has to exist before Phase 1's AppSet even sees the cluster. Five things:

The CNPG kubeflow_metadata_db Database CR. It lives in kubernetes-addons/Data-platform-stacks/mlops-cnpg/postgres-mlops-cnpg-cluster.yaml next to the other MLOps databases. The owner is the kubeflow Postgres role, with its password in Vault. Applying the Database CR creates the database on the already-running CNPG cluster. It's there for the MySQL → CNPG move, which hasn't happened yet (see the KFP overlay section).
Vault paths, populated by Terraform. Three matter:
- secret/homelab/kubeflow/kubeflow-metadata-db: {username: kubeflow, password: ...}
- secret/homelab/kubeflow/oauth2-proxy: {client-id, client-secret, cookie-secret} for the Keycloak kubeflow realm client.
- secret/homelab/kubeflow/centraldashboard-session (optional): the Central Dashboard session secret. Terraform writes these from .secrets.auto.tfvars. No human writes Vault paths by hand.
The Keycloak kubeflow realm. Terraform creates the realm, its Google and GitHub identity providers and the oauth2-proxy client, and it reads that client's secret from the same Vault path oauth2-proxy uses, so the two can't drift. Before the SSO front door goes in, I check two things directly: the realm's .well-known/openid-configuration returns 200 with a valid issuer, and the Vault path holds all three keys.
Google OAuth client. Add kubeflow.georgehomelab.com/oauth2/callback to the authorized redirect URIs on the existing OAuth client.
DNS and a dedicated certificate. kubeflow.georgehomelab.com → 10.0.1.21 (the kube-vip ingress VIP), managed in Cloudflare. Kubeflow gets its own cert-manager Certificate (kubeflow-georgehomelab, issued via Let's Encrypt DNS-01), which writes the kubeflow-georgehomelab-tls-cert Secret the Kubeflow Gateway serves.

With those five in place, Phase 1 can fire.

Laying the foundation

Phase 1 is five Apps at sync waves 1 and 2.

kubeflow-namespace creates the kubeflow and kubeflow-system namespaces with pod-security.kubernetes.io/enforce: privileged. Upstream uses restricted, and my cluster default is also restricted. But every istio-injected namespace needs privileged, because istio-init runs with CAP_NET_ADMIN + CAP_NET_RAW to program iptables. Without privileged, istio-init fails admission and every pod in the namespace hits FailedCreate with no obvious cause. CNPG's initdb Job hits the same trap.

kubeflow-cert-issuer installs a selfSigned ClusterIssuer for Kubeflow's internal admission-webhook certs. Public TLS doesn't use it; that comes from the dedicated Let's Encrypt certificate.

kubeflow-roles holds the kubeflow-admin / kubeflow-edit / kubeflow-view ClusterRole aggregates. The Profiles controller binds them in per-user namespaces when it reconciles a Profile CR.

kubeflow-istio-resources is the dedicated kubeflow-gateway Gateway, which I put in istio-system rather than in the kubeflow namespace where upstream puts it. It binds kubeflow.georgehomelab.com specifically and serves the kubeflow-georgehomelab-tls-cert Secret, so I can rotate the Kubeflow cert without touching anything else.

The plan was failure-domain isolation from the shared homelab-gateway, and that only half worked. homelab-gateway's *.georgehomelab.com server overlaps this host on the same ingress gateway port 443, and Istio picks one per ingress replica. It showed up as intermittent 404s: curl worked, the browser didn't. The front-door VirtualService now binds both gateways, so the route exists whichever one Istio picks.

The same App later picked up the notebook ext_authz policy and its redirect EnvoyFilter (see the notebook routing section), but in Phase 1 it's just the Gateway and the RBAC aggregates.

kubeflow-kfp-infrastructure holds KFP's prerequisites. It creates the kfp-artifacts ObjectBucketClaim (Rook-Ceph provisions a bucket on RGW with its own access keys), plus the VaultAuth + VaultStaticSecret that sync the CNPG kubeflow user's credentials into a kubeflow-metadata-db Secret in the kubeflow namespace.

Those credentials are ready and unused. The ml-pipeline Deployment that lands in Phase 3 pulls the OBC-produced kfp-artifacts-bucket Secret via envFrom, but still reads its database credentials from the bundled MySQL's mysql-secret (I checked live). The same App also carries a daily backup CronJob that dumps both bundled MySQL servers (KFP and Katib) and ships them to a kubeflow-db-backups bucket on RGW. I only added it after noticing that neither database had a backup at all.

After Phase 1 there are no application pods yet, just CRDs, OBCs, Secrets and the gateway. That's on purpose. Phase 1 only lays the substrate.

Knative Serving and Eventing

Phase 2 is Knative. Serving is a KServe dependency (KServe's InferenceService Pod is built on Knative's Service CR). Eventing is in for completeness, in case a later workload wants Trigger / Broker to fan events out.

The deploy itself is uneventful; upstream's overlays land cleanly. The trouble is the CRD spec.conversion blocks. Some Knative CRDs (pingsources.sources.knative.dev, eventtypes.eventing.knative.dev) ship with webhookClientConfig set but spec.conversion.strategy missing. ArgoCD's ServerSideDiff dry-run rejects that with "Required value / Forbidden", and the App sits in ComparisonError, stuck on Unknown.

The fix goes in the AppSet template: argocd.argoproj.io/compare-options: ServerSideDiff=false. Falling back to the legacy 3-way merge skips the dry-run. You lose mutation-webhook injection in the diff, which I can live with because those webhooks are already in the broader ignoreDifferences block.

The second Knative trap: the controller rewrites the entire .webhooks[].rules[] array at runtime. It consolidates per-resource entries into wildcards, adds status subresources and uses apiVersion *, while the source manifest's rules block is a stub. Result: permanent OutOfSync. The fix is ignoreDifferences on the whole .webhooks array, scoped to the named Knative webhooks (webhook.serving.knative.dev, validation.webhook.serving.knative.dev).

A third one was on the risk register but never bit: the cluster-local-gateway. Knative needs an internal Istio gateway for in-cluster routing, and the homelab already had one in istio-system from an earlier deploy (30+ day uptime, helm-managed). Installing it again from the Knative manifests would fight the existing helm field-manager, so the Knative phase leaves the cluster-local-gateway out and relies on the one that's there.

The operators

Phase 3 is seven Apps at sync waves 4 and 5: admission-webhook, trainer, katib, pipelines (KFP), kserve, kserve-models-web-app and model-registry.

admission-webhook is the PodDefaults admission controller. It mutates pods at admission to inject volumes, env vars and secrets. Tiny footprint, no surprises, and it turns out to be the piece users notice most (more on that in the governance section).

trainer is Kubeflow Trainer V2 (formerly Training Operator V2). In 1.11 it's pure backend with no UI: the TrainJob / TrainingRuntime / ClusterTrainingRuntime CRDs and a controller manager. Users create TrainJob CRs from notebooks with the Python SDK (kubeflow.trainer).

Trainer has a kustomize trap. The upstream overlay sets app.kubernetes.io/part-of: kubeflow-training on both the Deployment's selector and the pod template. My overlay's labels: block (which stamps app.kubernetes.io/part-of: kubeflow-stack) with includeSelectors:false includeTemplates:true only touches the template, so selector and template disagree and the apiserver rejects the Deployment. The fix was to drop the labels: block from the trainer overlay.

katib is the hyperparameter optimization service: Experiment / Suggestion / Trial CRDs, a controller, a UI and a bundled MySQL backend. I left that MySQL alone. Katib only stores short-lived experiment metadata, and I saw no reason to put it on CNPG.

kserve and kserve-models-web-app are the inference layer. KServe's controller ships ~30 CRDs (InferenceService, InferenceGraph, ClusterServingRuntime and so on), and the Models Web App is a UI for listing and configuring InferenceService CRs. model-registry gets its own section further down.

The KFP heavy overlay

This is the most invasive overlay in the whole deploy. Upstream's applications/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user overlay assumes:

SeaweedFS for KFP artifacts: a bundled S3-compatible store that upstream exposes under the old minio-service Service name, so KFP's MinIO-era config keeps working. (Older releases shipped MinIO itself.)
MySQL for the ml-pipeline metadata database: bundled Deployment, Service and PVC.
A mlpipeline-minio-artifact Secret with hardcoded minio / minio123 creds.

I deleted the SeaweedFS workloads and replaced the Secret. MySQL is still running (more on that below).

SeaweedFS → RGW. Every KFP component that talks S3 now goes straight to RGW's public TLS endpoint, s3.georgehomelab.com:443. There are five of them:

ml-pipeline (the API server) and cache-server, through env vars;
pipeline-install-config (minioServiceHost);
the Argo artifactRepository;
the pipelines profile-controller's sync.py, through AWS_ENDPOINT_URL.

The first version of this overlay worked differently. It turned upstream's minio-service into an ExternalName pointing at the in-cluster RGW Service on plain port 80. That broke the day RGW went TLS-only and port 80 disappeared.

The ExternalName is still there, now pointing at s3.georgehomelab.com:443, but it's vestigial. Its only job is to stop upstream's minio-service, whose selector targets the SeaweedFS pod I deleted, from dangling. Nothing should depend on it. The minio-service.kubeflow* names were dropped from RGW's hosting.dnsNames, so a path-style request through that alias gets a 403.

Host and port alone aren't enough. ml-pipeline and cache-server read OBJECTSTORECONFIG_* straight from their env and ignore the ConfigMap's minioServicePort. Point them at port 443 without turning TLS on and they speak plain HTTP to a TLS port, and ensureBucketExists dies with upstream connect error or disconnect/reset before headers. The patch sets all three:

- name: OBJECTSTORECONFIG_HOST
  value: "s3.georgehomelab.com"
- name: OBJECTSTORECONFIG_PORT
  value: "443"
- name: OBJECTSTORECONFIG_SECURE
  value: "true"

The artifact Secret. Upstream ships mlpipeline-minio-artifact with minio/minio123 in it. If you leave that Secret in the kustomize render, VSO's overwrite: true only merges. The placeholder keys survive, KFP reads accesskey=minio, and RGW answers "access key ID does not exist".

So the overlay deletes the upstream Secret with $patch: delete, and VSO creates it fresh from Vault with only the two keys KFP reads, accesskey and secretkey. The KFP Deployments reference the Secret by name, so they don't care who created it.

MySQL → CNPG (planned, not live). The cluster still runs the bundled mysql Deployment, and pipeline-install-config still says dbType=mysql. Upstream ships no multi-user Postgres overlay and the data is small, so the daily dump to RGW closed the real gap (no backups at all) first.

The CNPG database and its credentials are provisioned, and the patch below is the plan. The ml-pipeline, cache-server, metadata-grpc-deployment, ml-pipeline-persistenceagent, ml-pipeline-scheduledworkflow, ml-pipeline-ui and ml-pipeline-viewer-crd Deployments all take the database from env vars, and each gets patched to use the CNPG-provisioned creds:

env:
  - name: DBCONFIG_USER
    value: kubeflow
  - name: DBCONFIG_PASSWORD
    valueFrom:
      secretKeyRef:
        name: kubeflow-metadata-db
        key: password
  - name: DBCONFIG_HOST
    value: postgres-mlops-cnpg-cluster-rw.mlops.svc.cluster.local
  - name: DBCONFIG_PORT
    value: "5432"
  - name: DBCONFIG_NAME
    value: kubeflow_metadata_db

The kubeflow-metadata-db Secret is VSO-synced from secret/homelab/kubeflow/kubeflow-metadata-db. Once this lands, the bundled MySQL Deployment goes via $patch: delete.

The istio init race. ml-pipeline-api-server calls glog.Fatalf on its first S3 connection failure, with no retry. Without proxy.istio.io/config: holdApplicationUntilProxyStarts: true, the apiserver gets "connection refused" because istio-proxy hasn't opened its outbound listeners yet, the app exits, and you have a CrashLoop. The annotation makes the main container wait for the sidecar's readiness probe. The same trap comes back with Model Registry's golang-migrate flow.

The cold-boot race. The lab powers off every night, so every morning Kubeflow starts from nothing. KFP's scheduler came up before ML Metadata, saw that the daily run's slot had been missed, and fired it straight away. The run's first step needs ML Metadata, so it failed before doing any work. I added an init container to the scheduler that waits for ML Metadata, MySQL, the cache server, Kyverno and a healthy API server, then gives everything two more minutes.

The first version of that wait didn't wait at all. It opened a TCP connection to each Service, and inside the mesh that always succeeds: the pod's own Envoy sidecar accepts the connection, whether or not anything is behind the Service. I only caught it by pointing the check at a Service with no endpoints and watching it pass. The fix is traffic.sidecar.istio.io/excludeOutboundPorts on the scheduler pod, so those checks go straight to the real Service. On the next cold boot it held the scheduler for 220 seconds while ML Metadata came up, and the run that followed succeeded.

template:
  metadata:
    annotations:
      traffic.sidecar.istio.io/excludeOutboundPorts: "8080,3306,443"

A KFP run's artifact panel showing an s3://kfp-artifacts/... URI (served by Ceph RGW, not SeaweedFS), or the Ceph dashboard → Object Gateway → Buckets view with kfp-artifacts listed.

The "huge CRD" trap

Kubeflow CRDs are big. clustertrainingruntimes.trainer.kubeflow.org weighs ~280KB, trainjobs.trainer.kubeflow.org ~220KB, and the InferenceService and Knative ones aren't far behind.

The Kubernetes apiserver caps metadata.annotations at 262144 bytes (256KB). ArgoCD's default CSA-to-SSA migration writes a kubectl.kubernetes.io/last-applied-configuration annotation during the handover, and on these CRDs that annotation alone is >256KB. You get metadata.annotations: Too long: may not be more than 262144 bytes.

I stacked two fixes:

Per-CRD SSA via a kustomize patch. Put the argocd.argoproj.io/sync-options: ServerSideApply=true annotation on each problem CRD. ArgoCD then skips CSA and goes straight to SSA, which doesn't write the annotation.
App-level ClientSideApplyMigration=false. This turns the migration off entirely; it's set in the AppSet template's syncOptions. The CRD becomes argocd-controller-owned on first sync and stays SSA-managed.

Workspace UIs

Phase 4 is six Apps at sync wave 6: notebook-controller, jupyter-web-app, volumes-web-app, tensorboard-controller, tensorboards-web-app and pvcviewer-controller.

The web apps are SPAs and the controllers are headless. Between them, users can:

spawn JupyterLab notebooks from the Jupyter UI;
browse and create PVCs in the Volumes UI;
launch TensorBoard instances against TensorBoard log dirs;
view files in a PVC through the PVCViewer's per-PVC file browser.

Two integration points matter:

Every UI reads a USERID_HEADER env to find out who's making the request. Upstream's default is kubeflow-userid; I override it to X-Forwarded-Email (the Auth Chain Saga explains why).
Each UI's AuthorizationPolicy originally allowed requests only from cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account. Under a PERMISSIVE mesh that principal-based ALLOW matches nothing, so I replaced it with a path-based ALLOW.

The notebook-controller's ISTIO_GATEWAY env defaults to kubeflow/kubeflow-gateway, a stub Gateway in the kubeflow ns that isn't bound to any ingress IP in my setup. I override it to istio-system/kubeflow-gateway with a kustomize configMapGenerator using behavior:merge. The per-notebook VirtualService the controller emits then binds to the real ingress, and traffic to /notebook/<ns>/<name>/ routes correctly.

Ten surfaces read the user header: centraldashboard, kfam, manager, jupyter-web-app, katib-ui, kserve-models-web-app, volumes-web-app, tensorboards-web-app, ml-pipeline-ui and ml-pipeline-api-server. Most use USERID_HEADER; ml-pipeline-ui and ml-pipeline-api-server use KUBEFLOW_USERID_HEADER. Get it wrong on any single component and that UI answers "no user identity in header" on every request, even though oauth2-proxy has "authenticated" the user.

One more quirk. jupyter-web-app's /api/namespaces/<ns>/notebooks listing endpoint returns a 500 for any Notebook CR with a container that has no volumeMounts field. The web UI always adds a workspace volume to the notebooks it spawns, so you only see this with directly-applied CRs with bare containers. The bug is at utils.py:145, which does cntr["volumeMounts"] without a key check. Workaround: always add at least one volumeMount, even an emptyDir.

The Jupyter web app's notebook list in a Profile namespace, with a running GPU notebook (nvidia.com/gpu: 1) and its Connect button.

Two notebooks, one shared volume

A notebook's storage is only as useful as your ability to get at it from somewhere else. A teammate's notebook, a second kernel, a training job that needs the same dataset. So I tested the thing the Jupyter form quietly offers: a volume that more than one notebook can mount and write to at the same time.

The storage underneath is CephFS, through the ceph-filesystem storage class with ReadWriteMany. The default block class is ReadWriteOnce, which gives one pod its own disk and nobody else.

Create the first notebook with two shared volumes. In the "New notebook" form, untick "Use default class" on the workspace volume, pick ceph-filesystem and ReadWriteMany, then add one data volume the same way:

Volume	Storage class	Access mode	Size	Mounted at
`pod-nfs-shared-nb-workspace`	`ceph-filesystem`	ReadWriteMany	5Gi	`/home/jovyan`
`pod-nfs-shared-nb-shared`	`ceph-filesystem`	ReadWriteMany	10Gi	`/home/jovyan/pod-nfs-shared-nb-shared`

Ignore the "nfs" in the name. I named the notebook before I'd settled on the backend, and there is no NFS server in this cluster.

The "New notebook" form with the workspace volume set to ceph-filesystem and ReadWriteMany.

Check that both volumes are mounted, and that they really are two volumes. Open a terminal in JupyterLab:

df -hT /home/jovyan /home/jovyan/pod-nfs-shared-nb-shared

Filesystem                                                       Type  Size  Used Avail Use% Mounted on
csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-ebf4d59a-...  ceph  5.0G     0  5.0G   0% /home/jovyan
csi-cephfs-node@<cluster-id>.ceph-filesystem=/volumes/csi/csi-vol-198d5e31-...  ceph   10G     0   10G   0% /home/jovyan/pod-nfs-shared-nb-shared

Two different csi-vol IDs and two different sizes. It isn't one volume with a subfolder in it.

Write a marker file from the first notebook:

echo "written by $(hostname) at $(date)" > /home/jovyan/pod-nfs-shared-nb-shared/proof.txt
cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>

Create a second notebook and attach the same volume. I called it share-test. Under Data Volumes, choose "Attach existing volume" and pick pod-nfs-shared-nb-shared. The dropdown lists every claim in the namespace next to its access mode, which is a quiet but useful piece of UI: you can see which volumes are shareable before you try.

Read the file from the second notebook, then reply. Both notebooks are running:

cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt
echo "reply from $(hostname)" >> /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>

Back in the first notebook, read it again:

cat /home/jovyan/pod-nfs-shared-nb-shared/proof.txt

written by pod-nfs-shared-nb-0 at <timestamp>
reply from share-test-0

Two lines, two hostnames, one file, and neither notebook was restarted in between.

The two JupyterLab terminals side by side, each showing both lines of proof.txt. Crop out the browser tabs and bookmarks bar.

Ask Kubernetes who is holding the claim:

kubectl -n terranetes describe pvc pod-nfs-shared-nb-shared | grep -A2 "Used By"

Used By:       pod-nfs-shared-nb-0
               share-test-0

Three things worth knowing before you hand this to a team:

You don't need to own the namespace. The kubeflow-edit role, which is what a contributor gets, can create notebooks and create and delete volume claims. kubeflow-view can only look.
The quota is what stops you. My Kyverno-generated quota gives each user namespace 10 claims and 100Gi. Every notebook with its own workspace eats into that.
Every notebook here asks for a GPU. With three cards in the cluster, a forgotten share-test notebook is a third of the GPU capacity doing nothing. Stop it when you're done.

Central Dashboard, Profiles and Kyverno governance

Phase 5 brings the portal and multi-user setup. Central Dashboard is the iframe-style UI portal. Its sidebar tile list lives in a ConfigMap that the centraldashboard pod reads at runtime through the K8s API. Tiles come in four kinds: menuLinks (iframed sub-UIs), externalLinks (opened in new tabs), documentationItems and quickLinks.

Profiles handle multi-user, run by the Profiles controller and KFAM (Kubeflow Access Manager). A Profile CR defines a user. The controller creates a namespace with the same name as the Profile's metadata.name, binds the appropriate ClusterRoles, generates a ns-owner-access-istio AuthorizationPolicy and registers the namespace with KFAM.

My Kyverno governance policies fire on app.kubernetes.io/part-of: kubeflow-profile namespaces and generate:

ResourceQuota (kubeflow-user-quota): cpu 4 requested / 16 limit, memory 8/16 Gi, storage 100 Gi, 20 pods, 10 PVCs. The CPU limit started at 8 and that was too tight: a running notebook holds about 2 CPUs of limit, and the pipeline's training Jobs were refused for the rest. A refused Job gets no pod and no error, only a FailedCreate event, so the step simply waited until it timed out. The pipeline step now reads those events and fails within five minutes with the quota's numbers in the message.
LimitRange (kubeflow-user-limits): default cpu 500m / mem 512Mi, max cpu 4 / mem 8 Gi.
Two NetworkPolicies: default-deny-all as the baseline, then allow-in-ns-and-kubeflow, which lets traffic in from the namespace itself and the Kubeflow control plane (kubeflow, kubeflow-system, knative-serving, istio-system), and out to that same control plane plus mlops (MLflow), rook-ceph (RGW) and DNS in kube-system.
Two CiliumNetworkPolicies: one allowing egress to the kube-apiserver (a plain NetworkPolicy can't express that reliably under Cilium) and one allowing egress to PyPI by FQDN, so pip install works in a notebook and nothing else on the internet does.

A separate Kyverno mutate policy, mutate-pss-privileged-kubeflow-user-ns, stamps every Profile namespace with PSS privileged at admission. User namespaces are istio-injected too, so istio-init needs the same capabilities there as in the control plane.

So user namespaces are bounded by default. A user can't reach anyone else's namespace, oversubscribe the cluster, or deploy anything bigger than the LimitRange allows. They can talk to the Kubeflow platform services, MLflow, Ceph RGW and PyPI.

The same policy file also generates three PodDefaults in every Profile namespace, and they're the best user-facing feature of the whole build. A PodDefault is a tick-box in the Jupyter spawn form: one for KFP access (a projected service-account token so the KFP SDK can call the API from a notebook), one for S3 (the Ceph RGW endpoint and bucket credentials as AWS_* env vars), and one for MLflow (MLFLOW_TRACKING_URI and the RGW endpoint). A user ticks "MLflow" and their notebook can log runs without anyone pasting a URL or a key into a cell.

Central Dashboard runs with REGISTRATION_FLOW=true, so a first-time user with no Profile gets a "pick a namespace" form instead of RBAC: access denied. I still pre-create Profiles in Git (22-user-namespace/profile-<alias>.yaml), because a Git-managed Profile carries the kubeflow-profile label the governance policies key on, and it's reviewed like any other change. KFAM provisions the namespace and RBAC, and Kyverno fires the generate policies.

The one in the repo looks like this (owner changed for the article):

apiVersion: kubeflow.org/v1
kind: Profile
metadata:
  name: kubeflow-user-george
  labels:
    app.kubernetes.io/part-of: kubeflow-profile
spec:
  owner:
    kind: User
    name: user@example.com
  resourceQuotaSpec: {}   # Kyverno generates this; leave empty here

That's the whole onboarding flow: add the user in Keycloak, add a Profile YAML, commit and push. ArgoCD reconciles, and by the next refresh the user has a namespace with quotas, limits, network isolation and PodDefaults.

Live, there are three Profiles: upstream's default kubeflow-user-example-com, the Git-managed kubeflow-user-george, and terranetes, which I created by hand early on and which isn't in Git yet. That last one is where the real ML work runs.

The Kubeflow login redirecting to the Keycloak realm login page (proof Dex is gone), then landing on the Central Dashboard.

The SSO front door

Phase 6, the last piece. The oauth2-proxy that gates the whole platform lives in the kubeflow-sso namespace and is an ordinary AppSet element, central-dashboard-sso, at sync wave 8. It was briefly parked while I re-verified the Keycloak realm and the Vault secret, and re-enabled a day later.

The overlay sits at kubernetes-addons/0-boostrap/oauth2-proxy/overlays/kubeflow-centraldashboard/. It builds on the base oauth2-proxy chart and adds:

a VirtualService for host kubeflow.georgehomelab.com, bound to both istio-system/kubeflow-gateway and istio-system/homelab-gateway (the overlapping-SNI story from Phase 1);
a multi-upstream config (below);
the cookie domain set to kubeflow.georgehomelab.com, with cookie name _oauth2_proxy_kubeflow;
a Redis session store, so the browser cookie is a small session ID rather than the whole token set (Failure 9 below explains why that matters);
a VaultStaticSecret syncing the client secret and cookie secret from secret/homelab/kubeflow/oauth2-proxy;
the oauth2-proxy-kubeflow-userid-header EnvoyFilter (Lua), which aliases X-Forwarded-Email → kubeflow-userid on outbound (see the Auth Chain Saga);
the oauth2-proxy-kubeflow-authz-path-normalize EnvoyFilter, which only matters for notebooks (see the notebook routing section).

The multi-upstream config is the heart of it. Central Dashboard iframes every sub-UI at a fixed path: /jupyter/, /katib/, /pipeline/, /kserve-endpoints/, /tensorboards/, /volumes/, /model-registry/. Each one is a separate backend Service, and oauth2-proxy has to know about all of them:

args:
  - --upstream=http://centraldashboard.kubeflow.svc.cluster.local:80/
  - --upstream=http://jupyter-web-app-service.kubeflow.svc.cluster.local:80/jupyter/
  - --upstream=http://katib-ui.kubeflow.svc.cluster.local:80/katib/
  - --upstream=http://ml-pipeline-ui.kubeflow.svc.cluster.local:80/pipeline/
  - --upstream=http://kserve-models-web-app.kubeflow.svc.cluster.local:80/kserve-endpoints/
  - --upstream=http://tensorboards-web-app-service.kubeflow.svc.cluster.local:80/tensorboards/
  - --upstream=http://volumes-web-app-service.kubeflow.svc.cluster.local:80/volumes/
  - --upstream=http://profiles-kfam.kubeflow.svc.cluster.local:8081/kfam/
  - --upstream=http://metadata-envoy-service.kubeflow.svc.cluster.local:9090/ml_metadata.MetadataStoreService/
  - --upstream=http://model-registry-ui-service.kubeflow.svc.cluster.local:8080/model-registry/
  - --pass-host-header=false

Upstream's per-component overlays each ship a VirtualService bound to kubeflow-gateway with hosts: ['*'] and prefix: /jupyter/ (and the equivalent for every other component). With my kubeflow-gateway bound to kubeflow.georgehomelab.com, those wildcard-host VSes collide with my specific-host oauth2-proxy VS. Istio's route selection picks the wildcard route, requests skip oauth2-proxy and hit the components directly with no auth headers, and you get an "RBAC: access denied" body with HTTP 403.

The fix is to delete every upstream hosts: ['*'] VS with a kustomize $patch: delete and let oauth2-proxy do the path-based routing.

Login (no Dex) view: People → kubeflow-gateway → oauth2-proxy → Keycloak → Google, then oauth2-proxy → Central dashboard → each app with X-Forwarded-Email.

The Auth Chain Saga

I'll be re-reading this section every time I deploy another OIDC-fronted, multi-component platform. It's the longest in the article because the debugging took the longest.

Here's the auth flow end to end. Every step is a place it can break:

One request from the browser to jupyter-web-app, with the failure points pinned to the stage each one hits.

The browser asks for kubeflow.georgehomelab.com/_/jupyter/?ns=..., and TLS terminates at the istio-ingressgateway.
Istio routes it to the oauth2-proxy-kubeflow VirtualService (catch-all prefix /).
oauth2-proxy checks the _oauth2_proxy_kubeflow cookie. No cookie means a 302 to /oauth2/start, the Keycloak login, the callback, and a fresh cookie. With a cookie, the request goes upstream.
With --pass-host-header=false, oauth2-proxy rewrites Host to the upstream service FQDN.
Istio sends the outbound request through an internal rewrite VS that strips /jupyter/ to /.
An EnvoyFilter Lua script reads X-Forwarded-Email and adds the kubeflow-userid header.
The request reaches the jupyter-web-app pod, which reads the USERID_HEADER env to find the identity header.
An Istio AuthorizationPolicy checks the path, and the request is handled.

That's eight steps, and between them they produced nine distinct failures.

Failure 1: the wrong upstream Host header. With the default --pass-host-header=true, oauth2-proxy forwards Host: kubeflow.georgehomelab.com to every backend. Istio's outbound listener looks up the vhost by Host header, and the public FQDN matches no in-cluster cluster, so traffic falls through to PassthroughCluster (raw TCP, no HTTP filters).

The rewrite VS never fires, the backend receives /jupyter/static/main.js unrewritten, and the SPA serves index.html (text/html) as a 200 OK. The browser gets HTML where it expected JS and shows a blank page. Fix: --pass-host-header=false, so Host gets rewritten to the upstream service FQDN.

Failure 2: the wrong header name (X-Forwarded-Email vs X-Auth-Request-Email). I first patched the Kubeflow components to read USERID_HEADER=X-Auth-Request-Email. That's the response header oauth2-proxy sets when it's answering an auth check. In proxy mode, which is what the dashboard path runs, the request going upstream carries X-Forwarded-Email (set by --pass-user-headers=true). A component reading the wrong header logs no user identity in request and every API call 401s. Login looks fine; the UI is blank.

If you spot X-Auth-Request-Email later in the notebook section, that isn't a contradiction. Proxy mode passes X-Forwarded-Email upstream; the ext_authz path only gets X-Auth-Request-Email, because that's the header the Istio provider copies onto the allowed request through headersToUpstreamOnAllow.

Failure 3: principal-based APs under a PERMISSIVE mesh. The upstream AuthorizationPolicies require principals: cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account. Under my PERMISSIVE mesh the principal isn't reliably populated, the AP rule doesn't match, and you get 403 RBAC: access denied. Fix: replace it with a path-based ALLOW (paths: ["/*"]).

Failure 4: KUBEFLOW_USERID_HEADER vs USERID_HEADER. Most components read USERID_HEADER. ml-pipeline-ui reads KUBEFLOW_USERID_HEADER, a different env name. And ml-pipeline-api (the backend) has its own KUBEFLOW_USERID_HEADER to override, which is the easy one to miss. The symptom: the KFP UI loads, but /apis/v1beta1/{pipelines,experiments,runs,recurringruns} returns "Request header error: there is no user identity header."

Failure 5: kustomize vs VSO on the secret. The oauth2-proxy base used to ship the Secret with placeholder values (REPLACE_VIA_SETUP_SECRET) for the client secret and cookie secret. Every sync put the placeholders back, VSO's next reconcile put the real values back, and in between oauth2-proxy was reading a placeholder. Keycloak answered Invalid client credentials on the callback.

The fix was to stop shipping the placeholders. The base generator now creates the Secret with client-id only, and VSO (create: false, overwrite: true) is the sole writer of client-secret and cookie-secret. With nothing in Git to revert to, the tug-of-war between ArgoCD and VSO stopped.

Failure 6: Vault sealed, so every secret is stale. The transit Vault was sealed, which left main Vault crashlooping and VSO unable to sync. Every Secret stayed at its last-known value, which at the time meant the placeholders from Failure 5. Unseal transit, main Vault recovers, VSO catches up. There's a second trap inside this one: kubectl exec -it ... vault operator unseal quietly sends empty stdin, so use the positional-argument form.

Failure 7: Katib only serves at /katib/. Most Kubeflow web apps serve at both / and /<prefix>/. Katib serves only at /katib/, and its index.html emits absolute asset paths (src="/katib/static/main.js"). The Istio rewrite VS that strips /katib/ → / broke it (404 at /). Fix: no rewrite VS for katib. oauth2-proxy forwards /katib/ to the backend unchanged.

Failure 8: the EnvoyFilter Lua reading the wrong header. The Lua filter that aliases the user email to kubeflow-userid (to satisfy Model Registry's hardcoded header) originally read x-auth-request-email. Same bug as Failure 2, in EnvoyFilter form. model-registry-ui returned 400 "missing required kubeflow-userid header" on every API call and the SPA showed "Error communicating with server". Fix: read x-forwarded-email.

Failure 9: HTTP 431, the cookie that was too big. This one hid for a long time because the dashboard shell always loaded. Notebooks, Endpoints, Tensorboards and Volumes didn't. Those four are Tornado-based web apps that cap request headers at about 8 KB, and a Keycloak session stored in the cookie (access, ID and refresh tokens) came to 12–16 KB. They answered 431 Request Header Fields Too Large before ever looking at auth.

I measured it by sending each upstream Service the same request with different cookie sizes: those four flipped to 431 above 8 KB, while the dashboard, Katib, KFP, KFAM and Model Registry stayed at 200. The fix was moving oauth2-proxy's session store to Redis, so the browser only carries a short session ID. The overlay's deployment-patch.yaml keeps the measurement table as evidence.

Each of those took me somewhere between a few hours and half a day to find. I wrote every one down as a trap note the day it was fixed, which is the only reason this section exists in this much detail.

The Model Registry UI listing a RegisteredModel + its versions, reached from inside a Profile namespace via the Kyverno-generated alias Service.

Adding Model Registry

Model Registry is five pieces, all in the kubeflow namespace: model-registry-deployment (REST + gRPC API server), model-registry-db (bundled postgres 16), model-registry-ui (SPA + BFF), model-catalog-server (catalog API) and model-catalog-postgres (catalog DB). It's a Wave-5 App, registered in the AppSet.

The deploy is three upstream overlays (overlays/postgres for core + postgres, options/ui/overlays/kubeflow for the UI in kubeflow mode, options/catalog/base for the catalog) plus kustomize patches for:

appProtocol: http on three Services (model-registry-ui-service, model-registry-service, model-catalog). Without it Istio routes them through PassthroughCluster, the same trap as Failure 1 in the Auth Chain Saga.
Deleting the upstream hosts: ['*'] VS (the collision pattern).
A path-based AP in place of the principal-based ALLOW.
holdApplicationUntilProxyStarts: true on model-registry-deployment and model-catalog-server. Both use golang-migrate, which sets schema_migrations.dirty=true the moment a migration starts. If the pod gets killed mid-migration (say, istio-proxy isn't ready yet), the next start aborts with Dirty database version N. Fix and force version. and CrashLoops forever.

The UI is the hard part. In --deployment-mode=kubeflow it scopes service discovery to the user's Profile namespace via SubjectAccessReview, then filters Services by:

label component=model-registry;
a port named http-api (or https-api); port 8080 alone isn't enough;
a valid ClusterIP (ExternalName Services are rejected);
and it hardcodes the auth header to kubeflow-userid, so it isn't USERID_HEADER-configurable.

The hardcoded header is what made the EnvoyFilter Lua essential, and it's how Failure 8 in the Auth Chain Saga finally got noticed once MR was deployed.

Then the discovery filter. My Model Registry is one shared instance in the kubeflow namespace. The UI looks for MR Services in the user's own Profile namespace, finds none, and shows the "No registries found" empty state.

The fix is a Kyverno generate policy that creates, in every Profile namespace, a Service named model-registry-service (ClusterIP, no selector, port name http-api, the right labels) plus an Endpoints object pointing at the cluster-wide MR's ClusterIP. Kube-proxy chains it: user-ns ClusterIP, DNAT to the kubeflow-ns ClusterIP, DNAT to the real pod IP. The shared MR's ClusterIP survives pod restarts (pod IPs don't), so the chain holds.

The policy lives in kubernetes-addons/security-stack/kyverno/policies/kubeflow-user-namespace-governance.yaml, next to the quota, limit, network and PodDefault generators. Its apiCall context resolves the kubeflow-ns ClusterIP at evaluation time, so the Endpoints carries the actual IP rather than a placeholder.

Wiring this turned up one more Kyverno trap. Kyverno's validate-policy admission webhook rejects generate rules if the kyverno-background-controller lacks RBAC for the target resource. For Services and Endpoints, that meant aggregating a new ClusterRole into the background-controller's compiled role with the label rbac.kyverno.io/aggregate-to-background-controller: "true". The AuthorizationPolicy generator in the notebook routing section needed the same treatment.

End-to-end test from a notebook pod over REST:

MR=http://model-registry-service:8080
# 1. Register the model
curl -X POST "$MR/api/model_registry/v1alpha3/registered_models" \
  -H "Content-Type: application/json" \
  -d '{"name":"iris-classifier", "owner":"user@example.com"}'
# RegisteredModel id=1

# 2. Add a version (registeredModelId required in body too)
curl -X POST "$MR/api/model_registry/v1alpha3/registered_models/1/versions" \
  -H "Content-Type: application/json" \
  -d '{"registeredModelId":"1", "name":"v1.0.0", "author":"user@example.com"}'
# ModelVersion id=2

# 3. Add the artifact URI
curl -X POST "$MR/api/model_registry/v1alpha3/model_versions/2/artifacts" \
  -H "Content-Type: application/json" \
  -d '{"artifactType":"model-artifact", "name":"iris-classifier-onnx",
       "uri":"s3://kfp-artifacts/sample-models/iris/v1/model.onnx",
       "modelFormatName":"onnx"}'

All three POSTs return 200 from a notebook pod in the user namespace, and the model shows up in the UI straight away. The SDK path doesn't need an SSO round-trip.

Notebook routing: from bypass to ext_authz

This is where the architecture changed direction halfway through.

The bypass. Per-notebook VirtualServices (emitted by notebook-controller) bind to istio-system/kubeflow-gateway and route /notebook/<ns>/<name>/ straight to the user's notebook pod. They have to, because oauth2-proxy's --upstream is a static prefix-to-host map, while the notebook URL has a dynamic <ns> and <name>. There's no fixed backend, and no wildcard upstream entry can work out where to send the request.

The easy route was to send notebook traffic directly through Istio and skip oauth2-proxy entirely. The per-notebook VS already existed and only needed binding to the right gateway. So I did that, and added a Kyverno-generated path-based AP in user namespaces to let the traffic in.

The problem. With that bypass, /notebook/<ns>/<name>/ URLs are reachable with no auth check at all. Anyone on the LAN who guesses the URL pattern can open anyone's notebook. Tolerable for a single-user homelab, not for multi-tenant.

The fix is ext_authz, using Istio's extensionProviders MeshConfig feature:

Register a new extensionProvider named oauth2-proxy-kubeflow in MeshConfig, pointing at oauth2-proxy-kubeflow.kubeflow-sso.svc:80 with pathPrefix: /oauth2/auth.
Create a CUSTOM AuthorizationPolicy on istio-ingressgateway (in istio-system), scoped to /notebook/* paths and calling the oauth2-proxy-kubeflow provider.
Add an EnvoyFilter Lua that turns ext_authz 401s into a 302 with Location: /oauth2/start?rd=<original-url>, so unauthenticated users bounce through Keycloak login and back to their notebook.
Tighten the Kyverno-generated user-ns AP from path-only to requiring the X-Auth-Request-Email header. That header is in headersToUpstreamOnAllow on the provider, so its presence proves the request went through ext_authz.

Where the pieces live:

MeshConfig: kubernetes-addons/0-boostrap/istio-service-mesh/istiod/...-values.yaml
CUSTOM AP + redirect EnvoyFilter: kubernetes-addons/Data-platform-stacks/kubeflow/04-istio-resources/notebook-ext-authz.yaml
Method-normalising EnvoyFilter: envoyfilter-authz-path-normalize.yaml in the oauth2-proxy overlay
Kyverno tightening: the same kubeflow-user-namespace-governance.yaml as before.

Two follow-ups came after the first version shipped. The redirect Lua originally rewrote every 401 on the ingress gateway into a login redirect, which was fine until I found it had been breaking Harbor image pushes (the Docker registry protocol relies on a real 401 challenge). It's now scoped to /notebook/ only.

The second was terminals and kernels. Envoy builds the auth check by appending the original path to /oauth2/auth and keeping the original method, so opening a terminal sent POST /oauth2/auth/notebook/... to oauth2-proxy. That missed the auth endpoint, fell through to the dashboard upstream, and came back 404, which ext_authz treated as a deny. The notebook UI loaded, but every terminal and kernel start failed. The path-normalize EnvoyFilter rewrites the method of those checks to GET, which the authenticated dashboard answers with a 200, so POST and PUT now pass.

After the change, the flow looks like this:

the notebook path after the pivot, with the ext_authz side call to oauth2-proxy.

The browser asks for /notebook/<ns>/<name>/lab, and the notebook VS matches at the istio-ingressgateway.
The CUSTOM AuthorizationPolicy fires: the gateway makes an ext_authz call to oauth2-proxy's /oauth2/auth, forwarding the cookie.
oauth2-proxy answers 200 OK, and X-Auth-Request-Email is attached to the upstream request.
The request goes over mTLS to the notebook pod.
The user-namespace AP allows it only when X-Auth-Request-Email is present.
The notebook serves /lab.

So the build is hybrid: proxy mode for apps with static URLs, ext_authz for apps with dynamic ones. Going all-in on ext_authz would mean an extra hop on every request, losing cookie isolation and rebuilding the UX. A single uniform pattern isn't worth that.

Central Dashboard tiles: MLflow

The dashboard tiles live in a ConfigMap that centraldashboard reads at runtime through the K8s API (not a volume mount). At the time, two facts made updating it annoying:

Kyverno's make-configmaps-immutable policy marked every new CM immutable, so you couldn't kubectl edit or kubectl apply your way to new data.
The kubelet treats immutable as "cache forever", so even if you get past Kyverno somehow, pods that mount the CM (centraldashboard doesn't, other components do) won't see the update.

What I did instead:

Create a separate CM (centraldashboard-tiles) with the merged tile list (Kubeflow defaults plus the MLflow external link). A different name from centraldashboard-config means I can manage it independently.
Patch the centraldashboard Deployment's DASHBOARD_CONFIGMAP env to point at the new CM.
Generate the CM with disableNameSuffixHash: true (a stable name) and the argocd.argoproj.io/sync-options: Replace=true,Force=true annotation.

Replace=true,Force=true is the pair that makes it work. Replace=true alone runs kubectl replace -f, which is still an UPDATE and gets rejected on immutable resources. Replace=true,Force=true runs kubectl replace --force -f, which deletes and recreates, and that works. It took me an hour to find. The error (field is immutable when immutable is set) pointed clearly enough at the cause that I expected the obvious fix to work. It didn't.

The kubeflow namespace is now excluded from that Kyverno policy (Profile namespaces too, through a namespaceSelector), so new ConfigMaps there stay editable. ConfigMaps that were already made immutable stay frozen, which is why the replace-and-force pattern is still worth knowing.

MLflow is the only external link in the sidebar. There used to be an Airflow tile too, but Airflow was retired, so it went.

The Central Dashboard sidebar showing the MLflow external link.

How reliable is it?

The cluster is HA. Kubeflow on it mostly isn't, and I'd rather say that than imply otherwise. Here's what the live cluster shows:

Layer	What runs	What that means
Kubernetes control plane	Five masters, etcd on each, kube-vip for the API address	Survives losing a master
The front door	`oauth2-proxy-kubeflow` with 2 replicas, sessions in Redis	Survives losing one proxy pod; nobody gets logged out
Kubeflow itself	33 Deployments in `kubeflow`, one replica each, no PodDisruptionBudgets	A lost node means a reschedule and a short gap for the apps on it
GitOps	24 ArgoCD Applications, all Synced and Healthy	Drift is put back by ArgoCD's self-heal within seconds
KFP and Katib databases	The bundled MySQL, one pod each, dumped to Ceph RGW every day	The latest daily dump succeeded; a restore loses at most a day
The daily training run	A KFP RecurringRun every morning	The last three runs succeeded. The failures before them were the namespace quota and the cold-boot race, both fixed above
Cold boots	The whole lab, every morning	The scheduler now waits for its dependencies instead of firing into a half-started platform

The single replicas are a choice for a one-person homelab, not an accident: the Kubeflow controllers reconcile from Kubernetes state, so a restart costs a gap, not data. The databases are the part I'd fix first, which is why the CNPG move is at the top of the list below.

Kubeflow traps that will bite you

If you're putting Kubeflow on a cluster that isn't the reference one, these are the ones I'd check first:

Set --pass-host-header=false on oauth2-proxy. The default forwards the public Host, Istio's vhost lookup fails, traffic lands in PassthroughCluster, and you get blank pages with 200s.
Delete every upstream hosts: ['*'] VirtualService. They outrank your specific-host route and send requests around the proxy with no identity.
Replace principal-based AuthorizationPolicies with path-based ones under a PERMISSIVE mesh, or everything is RBAC: access denied.
Set the user header on all ten surfaces, and remember KFP's UI and API read KUBEFLOW_USERID_HEADER, not USERID_HEADER.
Keep the session out of the cookie. Keycloak tokens push it past the ~8 KB header limit of the Tornado-based UIs and you get HTTP 431.
Huge CRDs need per-CRD ServerSideApply and ClientSideApplyMigration=false, or the 256KB annotation limit stops the sync.
Every istio-injected namespace needs PSS privileged, Profile namespaces included.
Watch for overlapping wildcard Gateways on the same port. If another Gateway's *.domain server covers your host, bind your VirtualService to both or live with intermittent 404s.
Don't trust a TCP check inside the mesh. The pod's own sidecar accepts the connection, so "the port is open" proves nothing. Exclude those ports from the sidecar, or check at the HTTP level, and test the check against a Service with nothing behind it.
Size the Profile quota for notebooks and pipelines together. A Job the quota refuses never gets a pod and never errors; it just waits.

Conclusion

Here's what's running today:

Five master nodes, HA control plane, and three RTX A1000 GPUs available to notebooks and workloads.
Knative Serving + Eventing.
Kubeflow admission webhook, Trainer V2 (jobset-controller + trainer-controller), Katib, KFP (api-server + ui + persistenceagent + scheduledworkflow + viewer-crd + metadata-grpc + metadata-envoy + cache-server + workflow-controller), with a daily backup of KFP's and Katib's MySQL to RGW.
KServe controller + LocalModel controller + Models Web App, serving a real model: football-match-outcome in the terranetes namespace, retrained and redeployed by a daily XGBoost KFP RecurringRun.
Notebook controller + Jupyter Web App + Volumes Web App + TensorBoard controller + TensorBoards Web App + PVCViewer controller.
Central Dashboard + Profiles (kfam + manager containers) + per-user namespace governance and PodDefaults via Kyverno generate policies.
Model Registry + Model Registry UI + Model Catalog server + their two postgres Deployments.
oauth2-proxy-kubeflow in the kubeflow-sso ns, plus the per-component config (multi-upstream + pass-host-header=false + Redis sessions + EnvoyFilter Lua + ext_authz for /notebook/*).
Three Profiles: the upstream default, kubeflow-user-george from Git, and the hand-made terranetes.
The MLflow external link in the Central Dashboard sidebar.

How that served model gets trained, gated and promoted is its own story, in my separate article on the EPL match-prediction pipeline.

What isn't done yet:

KFP metadata on CNPG (still the bundled MySQL, backed up daily).
More than one replica for the Kubeflow components (see the reliability table).
Spark on Kubeflow (I use the mlops-spark-operator from the MLOps stack instead of Kubeflow's bundled overlay; they share infrastructure).
A real multi-tenant test. The Profiles exist, but I'm the only person using them, so isolation between two real users hasn't been exercised.
terranetes in Git. It works, but it's the one Profile that would vanish in a rebuild.
Network Policies on the Kubeflow control plane (user namespaces have them via Kyverno; the platform namespaces are open).

If I did this build again, I'd change five things:

Start with the auth chain, not finish with it. In the plan, Phase 6 was the front door. In practice it's where most of the debugging time went. Knowing the shape of the auth chain up front would have shaped every component overlay: the USERID_HEADER choices, the AP structure and the EnvoyFilter Lua would all have been settled in advance instead of fought one component at a time.
Build the EnvoyFilter Lua early. The Lua that aliases X-Forwarded-Email to kubeflow-userid is what holds cross-component auth together. I built it once, hit the wrong-header bug, fixed it, and now it works. Building it on day 1 with the right header would have saved half a dozen "why doesn't X have a user" sessions.
Write the trap notes as I go, not afterwards. Most of my notes on this build were written after the bug was fixed. Writing them during the debug, even as rough notes, would have made the next bug quicker to find.
Test with more than one user earlier. A single user (me) hides every cross-namespace issue. The Kyverno generate policies, the MR alias services and the user-ns governance have only ever been exercised by one person.
Lean on what I already knew about the substrate. At the start I had notes covering Vault, Keycloak, oauth2-proxy patterns, Istio STRICT-vs-PERMISSIVE and Kyverno, and none covering Kubeflow. Most of the surprises in this build came from the substrate, not from Kubeflow itself, so the old notes were more useful than I expected.

Practical takeaways if you're putting Kubeflow Platform 1.11 on a non-reference substrate:

Pin to v1.11.0, not master. A reproducible build beats the newest commit.
Design the auth chain first. Settle the OIDC provider, the proxy mode, the header names, the session store and the AP shape before you patch a single component overlay.
PSS=privileged on every Kubeflow namespace, user namespaces included. istio-init needs CAP_NET_ADMIN/NET_RAW wherever the sidecar is injected, and Profile namespaces are injected too. I enforce it with a Kyverno mutate policy so a new Profile can't come up restricted and fail every pod.
Disable the ClientSideApply migration at the AppSet level. Kubeflow CRDs are too big for the 256KB annotation limit.
Put Kyverno governance on Profile namespaces from the start. The generated quota, limits, network policies, PodDefaults, Model Registry alias and notebook AP all live in one governance file. Bolting them on afterwards is much harder than having them in place at Phase 5.
Write the traps down. The next platform with substrate swaps will hit a good share of the same ones.

What's Next?

The main thing I took from this build: a platform's reference architecture is a starting point, and you don't have to keep it. Kubeflow ships Dex, SeaweedFS and MySQL because it has to ship something. Each of them can be swapped out and wired to the stacks you already trust. Brownfield costs more up front, and in return the platform fits the cluster you already run.

The model-serving loop is already working, and it's covered in the EPL match-prediction pipeline article. What's left on this platform, roughly in order:

Move KFP metadata to CNPG. The database and credentials are waiting; the work is writing the Postgres patches upstream doesn't ship for multi-user mode.
Onboard a second tenant and test isolation properly. A second real user, with their own Profile in Git, trying to reach my namespace, my notebooks and my models, and failing.
Wire up OpenLineage so every pipeline run shows its inputs and outputs in a lineage graph, not just in MLflow tags.

If you're facing Kubeflow on a cluster that already has opinions of its own, the swap table at the top is a good place to start.

Follow me on LinkedIn

Author: George Ezejiofor

DEV Community