<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: kellerkoenig</title>
    <description>The latest articles on DEV Community by kellerkoenig (@kellerkoenig).</description>
    <link>https://dev.to/kellerkoenig</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4254125%2F96cb14f2-01be-48de-bbb9-85e9f7d04c08.png</url>
      <title>DEV Community: kellerkoenig</title>
      <link>https://dev.to/kellerkoenig</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/kellerkoenig"/>
    <language>en</language>
    <item>
      <title>How we install OKD on Proxmox VE without IPI</title>
      <dc:creator>kellerkoenig</dc:creator>
      <pubDate>Thu, 19 Nov 2026 15:48:37 +0000</pubDate>
      <link>https://dev.to/kellerkoenig/how-we-install-okd-on-proxmox-ve-without-ipi-3e9g</link>
      <guid>https://dev.to/kellerkoenig/how-we-install-okd-on-proxmox-ve-without-ipi-3e9g</guid>
      <description>&lt;p&gt;Our OKD clusters run on vSphere with IPI. Together with Puppet, that gives us one property we really like: if a cluster breaks badly, we throw it away and rebuild it within a few hours.&lt;/p&gt;

&lt;p&gt;Now we want to move to &lt;strong&gt;Proxmox VE&lt;/strong&gt;, and there is no IPI for Proxmox. No installer platform, no Machine API, no "just give it credentials and wait". So we built the missing piece ourselves and published it: &lt;strong&gt;&lt;a href="https://github.com/kellerkoenig/okd-proxmox" rel="noopener noreferrer"&gt;okd-proxmox&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This post shows how it works and what we learned on the way, including a few traps that cost us real debugging time.&lt;/p&gt;

&lt;h2&gt;
  
  
  The idea: agent-based installer + OpenTofu
&lt;/h2&gt;

&lt;p&gt;OKD ships the &lt;strong&gt;agent-based installer&lt;/strong&gt;: you render an ISO from &lt;code&gt;install-config.yaml&lt;/code&gt; and &lt;code&gt;agent-config.yaml&lt;/code&gt;, boot every node from it, and the nodes install the cluster among themselves. It works on any hardware, but it does not create VMs, and it does not do anything after the installation.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://opentofu.org" rel="noopener noreferrer"&gt;OpenTofu&lt;/a&gt; with the &lt;a href="https://github.com/bpg/terraform-provider-proxmox" rel="noopener noreferrer"&gt;bpg/proxmox&lt;/a&gt; provider is good at creating VMs, and bad at waiting for a cluster.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;okd-pve&lt;/code&gt; glues the two together behind one CLI:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;okd-pve init mycluster        # commented configuration skeleton
okd-pve preflight             # checks everything, changes nothing
okd-pve create                # ISO, VMs, installation, labels, storage, hooks
okd-pve verify                # cluster health
okd-pve node add --role worker --count 2
okd-pve destroy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F084oq3ucvzdvyd38chyp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F084oq3ucvzdvyd38chyp.png" alt=" " width="800" height="782"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A &lt;code&gt;create&lt;/code&gt; run does the following:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Preflight&lt;/strong&gt;: Proxmox privileges of the API token, storage, bridge, free VMIDs and MACs, CPU/RAM capacity per host, DNS records, VIPs, NTP and the local clock. Everything that would make the installation fail an hour later fails here instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render&lt;/strong&gt;: &lt;code&gt;install-config.yaml&lt;/code&gt; and &lt;code&gt;agent-config.yaml&lt;/code&gt; with static IPs, fixed MACs and the root disk chosen by disk serial, plus "Pre-Boot" manifests that go into the ISO (chrony, kubelet garbage collection, CA trust, kernel arguments, infra machine pool, registry mirrors).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ISO and VMs&lt;/strong&gt;: build the agent ISO, upload it, and create the VMs with OpenTofu. VMIDs and MACs are deterministic, and the control plane is spread over the Proxmox hosts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Install&lt;/strong&gt;: follow &lt;code&gt;wait-for bootstrap-complete&lt;/code&gt; and &lt;code&gt;wait-for install-complete&lt;/code&gt;, with progress output that tells you &lt;em&gt;which&lt;/em&gt; operator you are waiting for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Day 1 that IPI would have done&lt;/strong&gt;: detach the ISO, label and taint the infra nodes, verify the cluster.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage&lt;/strong&gt;: install a CSI driver (Proxmox CSI, Ceph CSI, LSO/ODF or NFS) and prove it works with a test PVC.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Post-install hooks&lt;/strong&gt;: ingress placement, monitoring, OperatorHub catalogs, operators, and your own hooks, followed by a report.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;code&gt;create&lt;/code&gt; is idempotent: interrupt it (Ctrl-C, timeout, failed hook) and the next run continues where it stopped. &lt;code&gt;destroy&lt;/code&gt; removes everything again, including uploaded ISOs.&lt;/p&gt;

&lt;p&gt;Configuration is split by confidentiality: &lt;code&gt;install-config.yaml&lt;/code&gt; stays the standard OKD format, &lt;code&gt;cluster.yaml&lt;/code&gt; holds everything Proxmox-specific (nodes, storage, node groups, IPs), and &lt;code&gt;proxmox.conf&lt;/code&gt; holds only the API token. The token never shows up in logs, error messages, OpenTofu output or debug traces.&lt;/p&gt;

&lt;h2&gt;
  
  
  Things we learned the hard way
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Registry mirrors can stop the bootstrap
&lt;/h3&gt;

&lt;p&gt;We pull through an internal registry mirror, configured with &lt;code&gt;ImageDigestMirrorSet&lt;/code&gt; and &lt;code&gt;ImageTagMirrorSet&lt;/code&gt;, partly with &lt;code&gt;mirrorSourcePolicy: NeverContactSource&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The agent installer creates its own &lt;code&gt;ImageDigestMirrorSet&lt;/code&gt; from &lt;code&gt;install-config.yaml&lt;/code&gt;, &lt;strong&gt;without&lt;/strong&gt; the source policy. The Machine Config Operator then sees two different policies for the same source and refuses to render anything. The visible symptom is misleading: the bootstrap's Machine Config Server on port 22623 never comes up, and the masters wait forever for &lt;code&gt;https://&amp;lt;api-vip&amp;gt;:22623/config/master&lt;/code&gt;. The real error was only in the kubelet journal of the bootstrap host (an init container in CrashLoopBackOff with "conflicting mirrorSourcePolicy").&lt;/p&gt;

&lt;p&gt;The fix: the ISO only carries mirror sets &lt;strong&gt;without&lt;/strong&gt; a policy, and &lt;code&gt;okd-pve&lt;/code&gt; sets the policies once the cluster is installed.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Infra nodes without MachineSets
&lt;/h3&gt;

&lt;p&gt;Without the Machine API there are no infra MachineSets, so &lt;code&gt;okd-pve&lt;/code&gt; labels and taints infra nodes after they have joined. Two side effects:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;DaemonSet pods that started on a node &lt;strong&gt;before&lt;/strong&gt; it got its taint keep running there, and the cluster reports &lt;code&gt;KubeDaemonSetMisScheduled&lt;/code&gt;. &lt;code&gt;okd-pve&lt;/code&gt; gives the DNS operator the right tolerations and removes the stray pods.&lt;/li&gt;
&lt;li&gt;CSI &lt;strong&gt;node plugins&lt;/strong&gt; must tolerate the infra taint too. Otherwise a pod with a volume on an infra node (Prometheus, for example) fails with the not very helpful &lt;code&gt;no topology key found for node&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Proxmox CSI needs its own token and its own firewall rule
&lt;/h3&gt;

&lt;p&gt;The &lt;a href="https://github.com/sergelogvinov/proxmox-csi-plugin" rel="noopener noreferrer"&gt;Proxmox CSI plugin&lt;/a&gt; creates Proxmox disks and hot-plugs them into the node VMs. It talks to the Proxmox API &lt;strong&gt;from inside the cluster&lt;/strong&gt;, so it needs a network path from the node network to port 8006, and it should get its own token with a narrow role, not the installer's. &lt;code&gt;okd-pve preflight&lt;/code&gt; now checks that token and its privileges before anything is installed, and a failing storage test shows the events of the test PVC instead of just timing out.&lt;/p&gt;

&lt;p&gt;With shared storage we migrated node VMs to other Proxmox hosts while a pod kept writing a numbered journal to its volume: no gaps in the sequence, no data loss.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. ext4 instead of xfs
&lt;/h3&gt;

&lt;p&gt;We started with xfs and switched the default to ext4, like the default classes of vSphere CSI, AWS EBS and ODF RBD:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a restored copy of an xfs volume cannot be mounted next to the original (same UUID),&lt;/li&gt;
&lt;li&gt;recent &lt;code&gt;mkfs.xfs&lt;/code&gt; features are refused by older kernels, for example on a rescue system,&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;mkfs.xfs&lt;/code&gt; refuses volumes below 300 MiB.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;xfs is still available per StorageClass for the workloads that want it.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. A root shell on the live ISO
&lt;/h3&gt;

&lt;p&gt;When the installation hangs before SSH works, you want a shell on the node console. The CoreOS live ISO has a reserved area for kernel arguments, so &lt;code&gt;okd-pve&lt;/code&gt; can add &lt;code&gt;systemd.debug_shell&lt;/code&gt; to the ISO in place (&lt;code&gt;install.debug_shell: true&lt;/code&gt;) and you get a root shell on tty9. That is how we found issue #1.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;Being honest about what it does &lt;strong&gt;not&lt;/strong&gt; do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;no Machine API, MachineSets or autoscaling: nodes are plain VMs, you scale with &lt;code&gt;okd-pve node add&lt;/code&gt; / &lt;code&gt;node remove&lt;/code&gt;,&lt;/li&gt;
&lt;li&gt;no disconnected installation (registry mirrors work, an air-gapped setup does not),&lt;/li&gt;
&lt;li&gt;no cluster upgrades or etcd backup: use the regular OKD mechanisms for those,&lt;/li&gt;
&lt;li&gt;IPv4 only, one NIC per node, OKD 4.17 or newer.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Status
&lt;/h2&gt;

&lt;p&gt;We currently run a test cluster with 3 masters, 3 dedicated infra nodes and 3 workers on OKD 4.22 and Proxmox VE 9.2, behind a corporate proxy, with Proxmox CSI on shared storage. We have rebuilt it many times. The project has CI with bats tests and shellcheck, releases with build attestations, and an OpenSSF Scorecard.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it, break it, help us
&lt;/h2&gt;

&lt;p&gt;It is AGPL-3.0, written in Bash, and young, so it has only been tested in a few environments:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Testers&lt;/strong&gt;: other Proxmox setups (Ceph, NFS, local storage, SDN), other OKD versions, single-node clusters.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contributors&lt;/strong&gt;: issues, pull requests, ideas, documentation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://github.com/kellerkoenig/okd-proxmox" rel="noopener noreferrer"&gt;https://github.com/kellerkoenig/okd-proxmox&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you run OKD or OpenShift on Proxmox already, I would love to hear how you do it. Leave a comment or open a discussion on GitHub.&lt;/p&gt;

</description>
      <category>okd</category>
      <category>proxmox</category>
      <category>openshift</category>
      <category>kubernetes</category>
    </item>
  </channel>
</rss>
