Our OKD clusters run on vSphere with IPI. Together with Puppet, that gives us one property we really like: if a cluster breaks badly, we throw it away and rebuild it within a few hours.
Now we want to move to Proxmox VE, and there is no IPI for Proxmox. No installer platform, no Machine API, no "just give it credentials and wait". So we built the missing piece ourselves and published it: okd-proxmox.
This post shows how it works and what we learned on the way, including a few traps that cost us real debugging time.
The idea: agent-based installer + OpenTofu
OKD ships the agent-based installer: you render an ISO from install-config.yaml and agent-config.yaml, boot every node from it, and the nodes install the cluster among themselves. It works on any hardware, but it does not create VMs, and it does not do anything after the installation.
OpenTofu with the bpg/proxmox provider is good at creating VMs, and bad at waiting for a cluster.
okd-pve glues the two together behind one CLI:
okd-pve init mycluster # commented configuration skeleton
okd-pve preflight # checks everything, changes nothing
okd-pve create # ISO, VMs, installation, labels, storage, hooks
okd-pve verify # cluster health
okd-pve node add --role worker --count 2
okd-pve destroy
A create run does the following:
- Preflight: Proxmox privileges of the API token, storage, bridge, free VMIDs and MACs, CPU/RAM capacity per host, DNS records, VIPs, NTP and the local clock. Everything that would make the installation fail an hour later fails here instead.
-
Render:
install-config.yamlandagent-config.yamlwith static IPs, fixed MACs and the root disk chosen by disk serial, plus "Pre-Boot" manifests that go into the ISO (chrony, kubelet garbage collection, CA trust, kernel arguments, infra machine pool, registry mirrors). - ISO and VMs: build the agent ISO, upload it, and create the VMs with OpenTofu. VMIDs and MACs are deterministic, and the control plane is spread over the Proxmox hosts.
-
Install: follow
wait-for bootstrap-completeandwait-for install-complete, with progress output that tells you which operator you are waiting for. - Day 1 that IPI would have done: detach the ISO, label and taint the infra nodes, verify the cluster.
- Storage: install a CSI driver (Proxmox CSI, Ceph CSI, LSO/ODF or NFS) and prove it works with a test PVC.
- Post-install hooks: ingress placement, monitoring, OperatorHub catalogs, operators, and your own hooks, followed by a report.
create is idempotent: interrupt it (Ctrl-C, timeout, failed hook) and the next run continues where it stopped. destroy removes everything again, including uploaded ISOs.
Configuration is split by confidentiality: install-config.yaml stays the standard OKD format, cluster.yaml holds everything Proxmox-specific (nodes, storage, node groups, IPs), and proxmox.conf holds only the API token. The token never shows up in logs, error messages, OpenTofu output or debug traces.
Things we learned the hard way
1. Registry mirrors can stop the bootstrap
We pull through an internal registry mirror, configured with ImageDigestMirrorSet and ImageTagMirrorSet, partly with mirrorSourcePolicy: NeverContactSource.
The agent installer creates its own ImageDigestMirrorSet from install-config.yaml, without the source policy. The Machine Config Operator then sees two different policies for the same source and refuses to render anything. The visible symptom is misleading: the bootstrap's Machine Config Server on port 22623 never comes up, and the masters wait forever for https://<api-vip>:22623/config/master. The real error was only in the kubelet journal of the bootstrap host (an init container in CrashLoopBackOff with "conflicting mirrorSourcePolicy").
The fix: the ISO only carries mirror sets without a policy, and okd-pve sets the policies once the cluster is installed.
2. Infra nodes without MachineSets
Without the Machine API there are no infra MachineSets, so okd-pve labels and taints infra nodes after they have joined. Two side effects:
- DaemonSet pods that started on a node before it got its taint keep running there, and the cluster reports
KubeDaemonSetMisScheduled.okd-pvegives the DNS operator the right tolerations and removes the stray pods. - CSI node plugins must tolerate the infra taint too. Otherwise a pod with a volume on an infra node (Prometheus, for example) fails with the not very helpful
no topology key found for node.
3. Proxmox CSI needs its own token and its own firewall rule
The Proxmox CSI plugin creates Proxmox disks and hot-plugs them into the node VMs. It talks to the Proxmox API from inside the cluster, so it needs a network path from the node network to port 8006, and it should get its own token with a narrow role, not the installer's. okd-pve preflight now checks that token and its privileges before anything is installed, and a failing storage test shows the events of the test PVC instead of just timing out.
With shared storage we migrated node VMs to other Proxmox hosts while a pod kept writing a numbered journal to its volume: no gaps in the sequence, no data loss.
4. ext4 instead of xfs
We started with xfs and switched the default to ext4, like the default classes of vSphere CSI, AWS EBS and ODF RBD:
- a restored copy of an xfs volume cannot be mounted next to the original (same UUID),
- recent
mkfs.xfsfeatures are refused by older kernels, for example on a rescue system, -
mkfs.xfsrefuses volumes below 300 MiB.
xfs is still available per StorageClass for the workloads that want it.
5. A root shell on the live ISO
When the installation hangs before SSH works, you want a shell on the node console. The CoreOS live ISO has a reserved area for kernel arguments, so okd-pve can add systemd.debug_shell to the ISO in place (install.debug_shell: true) and you get a root shell on tty9. That is how we found issue #1.
Limits
Being honest about what it does not do:
- no Machine API, MachineSets or autoscaling: nodes are plain VMs, you scale with
okd-pve node add/node remove, - no disconnected installation (registry mirrors work, an air-gapped setup does not),
- no cluster upgrades or etcd backup: use the regular OKD mechanisms for those,
- IPv4 only, one NIC per node, OKD 4.17 or newer.
Status
We currently run a test cluster with 3 masters, 3 dedicated infra nodes and 3 workers on OKD 4.22 and Proxmox VE 9.2, behind a corporate proxy, with Proxmox CSI on shared storage. We have rebuilt it many times. The project has CI with bats tests and shellcheck, releases with build attestations, and an OpenSSF Scorecard.
Try it, break it, help us
It is AGPL-3.0, written in Bash, and young, so it has only been tested in a few environments:
- Testers: other Proxmox setups (Ceph, NFS, local storage, SDN), other OKD versions, single-node clusters.
- Contributors: issues, pull requests, ideas, documentation.
👉 https://github.com/kellerkoenig/okd-proxmox
If you run OKD or OpenShift on Proxmox already, I would love to hear how you do it. Leave a comment or open a discussion on GitHub.

Top comments (0)