Why I Built kcore: Proxmox Broke My Automation One Too Many Times

#pensource #go #nixos #devops

After years running Proxmox in production, every automation attempt hit the same walls: an API bolted onto a web UI, mutable Debian hosts, Perl internals. So I built kcore — a modern, API-first hypervisor on NixOS.

I've been running Proxmox for years. On paper, it's great: open source, KVM-based, web UI included. In practice, every time I tried to automate anything beyond the basics, I hit a wall.

The API That Isn't Really an API

Proxmox has a REST API. Technically. But it's a thin wrapper over the web UI, not a first-class interface. Try doing something simple like attaching a cloud-init drive or configuring a network bridge programmatically — you'll end up shelling out to qm or pvesh from your Terraform or Ansible playbook. The Terraform provider for Proxmox is a community project that constantly fights against API limitations. Half the time you're writing provisioner "remote-exec" blocks to run CLI commands on the host because the API simply doesn't expose what you need.
That's not automation. That's scripting around an incomplete interface.

Perl, Corosync, and Legacy All the Way Down

Look under Proxmox's hood and you'll find Perl — lots of it. The core management framework, pve-manager, is written in Perl. The API server is Perl. The cluster stack depends on Corosync and pmxcfs, a custom clustered filesystem. It's engineering from a different era. Nobody is excited about debugging Perl stack traces in 2026. When something goes wrong — and in clustering, things go wrong — you're reading through code that feels like it belongs in a museum.

The Update Problem

Ask anyone running Proxmox in a serious deployment what version they're on. It's almost never the latest. Upgrading Proxmox is a manual, nerve-wracking process. You're apt upgrade-ing a mutable Debian system that's running your production VMs. There's no atomic rollback. If something breaks mid-upgrade — a kernel incompatibility, a Corosync version mismatch, a ZFS module that didn't rebuild — you're in for a long night. So people don't upgrade. They run Proxmox 7 when 8 has been out for years. They skip point releases. They treat their hypervisor like a legacy appliance: don't touch it, pray it keeps working.
This is the opposite of how modern infrastructure should work.

NixOS Changes Everything

kcore runs on NixOS. The entire host OS is defined declaratively in a single configuration. Every node built from the same flake is identical — no drift, no snowflakes, no "works on node 3 but not node 7." Updates produce a new system generation. If the update breaks something, you reboot into the previous generation. Done. No partial states, no manual rollback procedures, no downtime prayer sessions.
Updating a 10-node kcore cluster means pushing a new flake and rebuilding. Every node converges to the same state. If one node has a problem, it rolls back independently. Compare that to upgrading 10 Proxmox nodes one by one, crossing your fingers each time.

What I Built Instead

kcore is what I wanted Proxmox to be. The API is the product — every operation goes through gRPC with typed schemas. The CLI is a thin client. The Terraform provider talks to the same API. There's no web UI you have to click through to complete a workflow that the API forgot.
VMs are defined in YAML and applied declaratively. The control plane is written in Go — not Perl, not PHP, not a 20-year-old web framework. The codebase is small, readable, and modern.
Is it done? No. kcore is alpha software. It doesn't have HA clustering, live migration, or a web dashboard yet. Proxmox does, and if you need those things today, use Proxmox. I'm not pretending otherwise.
But if you've ever stared at a Proxmox Terraform run that failed because the API doesn't support what the UI does, or held your breath through a cluster upgrade, or wondered why your hypervisor's core is written in Perl — kcore is being built for you.