The Bare-Metal Integration Dilemma
When you run Kubernetes on traditional public clouds, a massive amount of infrastructure orchestration is taken for granted. The cloud provider hides the complexity of provisioning virtual machines, configuring VPCs, routing traffic through software-defined networks, and attaching block storage behind a clean, unified API. When you deploy a Kubernetes Service of type LoadBalancer on AWS or GCP, a Cloud Controller Manager (CCM) communicates with the provider's proprietary control plane to allocate an external IP, configure a load balancer, and update routing tables.
On bare metal, this elegant abstraction breaks down. Historically, platform engineers attempting to run bare-metal Kubernetes have been forced to stitch together disparate tools: BGP daemons like MetalLB for IP allocation, PXE booting infrastructure like Tinkerbell or MaaS for node provisioning, and complex CSI drivers that struggle to coordinate with physical SANs. The control loops of the physical hardware and the Kubernetes orchestration engine remain fundamentally decoupled, leading to fragile, hard-to-debug environments.
Traditional bare-metal deployments rely on IPMI/BMC interfaces and Redfish APIs that are notoriously slow, insecure, and prone to state desynchronization. When a node fails, the Kubernetes control plane has no reliable way to determine if the physical machine is dead, partitioned, or rebooting. This lack of a single source of truth leads to split-brain scenarios and delayed failovers.
Oxide Computer Company's approach to bare-metal cloud infrastructure offers a compelling case study in solving this friction. By co-designing hardware, hypervisors, and control planes, Oxide has introduced native Kubernetes integrations that rethink how bare-metal networking, compute provisioning, and storage control loops interact. In this article, I will analyze the architectural decisions behind these integrations, focusing on how decoupling the network control plane from the Cloud Controller Manager improves stability, how runtime filesystem boundaries impact node provisioning, and the engineering trade-offs of their emerging native storage integration.
An in-depth architectural analysis of bare-metal Kubernetes integrations, focusing on decoupled network control planes, immutable node provisioning, and the engineering trade-offs of co-designed hardw
🏗️ Decoupling the Network Control Plane from the Cloud Controller Manager
In a standard cloud deployment, the Cloud Controller Manager (CCM) is a daemon that runs inside the Kubernetes control plane. It is responsible for watching resources like Nodes and Services and translating changes into API calls to the underlying cloud provider. While this model works well for public clouds, relying solely on a CCM to manage bare-metal networking introduces a tight coupling that can compromise the availability of the entire rack. If the Kubernetes API server experiences a performance degradation or a control plane partition, the CCM can fail to update network routes, leaving external load balancers out of sync with the actual state of the workloads.
To mitigate this, the network control plane is decoupled from the CCM. Instead of the CCM directly manipulating physical switch configurations or routing tables, the CCM acts as a lightweight translator that writes desired state declarations to the Oxide API. The physical switches and internal networking fabric are managed by a separate, highly available control loop running on the rack's independent control plane. This separation of concerns ensures that even if the Kubernetes cluster completely loses its control plane, the existing network routing, IP allocations, and hardware-level packet forwarding remain entirely stable.
Let's look at how this works in practice when provisioning a LoadBalancer service. Instead of running a complex BGP daemon on every Kubernetes node, the Oxide CCM detects the creation of a LoadBalancer service and requests a virtual IP (VIP) from the Oxide network control plane. The Oxide control plane allocates the IP from a pre-configured subnet pool and configures its silicon-level virtual switches to route traffic for that VIP directly to the hypervisor hosts running the target pods.
This approach yields several architectural benefits:
- No Host-Level BGP : Nodes do not need to participate in BGP peering. This eliminates the CPU and memory overhead of running BGP daemons on every worker node and removes the risk of a misconfigured node poisoning the physical network's routing table.
- Hardware-Enforced Isolation : Because the routing is handled at the hypervisor and physical switch level, network isolation between different Kubernetes namespaces or tenant clusters is enforced by the hardware fabric, preventing container escape vectors from accessing the broader corporate network.
- Sub-Second Failover : If a physical node hosting a pod fails, the Oxide control plane detects the link loss at the hardware level and instantly reroutes the VIP traffic to healthy nodes, bypassing the slower Kubernetes endpoint reconciliation loop.
By moving the routing logic out of the guest operating system and into the hardware-controlled virtual switch (vswitch) layer, I have observed that the blast radius of a compromised or misconfigured Kubernetes node is significantly reduced. In traditional setups, a single node running a misconfigured BGP daemon can announce routes for the entire cluster, black-holing traffic and causing widespread outages. With a decoupled control plane, the physical switches only accept routing updates from the authenticated rack control plane, rendering host-level route hijacking impossible.
🏗️ Addressing Runtime Filesystem Boundaries and Node Provisioning
One of the most persistent pain points in bare-metal Kubernetes is node provisioning and OS lifecycle management. Traditional bare-metal deployments rely on mutable operating systems installed on local disks. Over time, these operating systems suffer from configuration drift, unpatched vulnerabilities, and corrupted filesystems.
Oxide addresses this by utilizing immutable, image-based operating systems for its compute instances. When a new Kubernetes node is provisioned, the Oxide control plane boots a clean, minimal virtual machine image running a specialized Linux distribution optimized for container runtimes. However, this immutable approach introduces a strict runtime filesystem boundary that complicates how Kubernetes agents (like the kubelet) and container runtimes (like containerd) operate. Specifically, the kubelet expects to have write access to several critical directories, such as /var/lib/kubelet for pod volumes, /var/lib/containerd for container images, and /etc/kubernetes for configuration files.
If these directories are located on a read-only root filesystem, the kubelet will fail to start. Conversely, if we simply mount these directories on a mutable, ephemeral RAM disk, we risk losing cached container images and local volumes whenever a node reboots, leading to slow startup times and potential data loss.
To resolve this runtime filesystem boundary issue, the Oxide integration utilizes a structured layout that separates the immutable OS image from mutable, persistent state. During the node boot sequence, the Oxide hypervisor attaches a dedicated, high-performance local NVMe block device to the instance. This block device is partitioned and mounted to handle the mutable state of the Kubernetes node.
I have summarized how these filesystem boundaries are structured in the table below:
| Directory | Filesystem Type | Purpose | Persistence Characteristics |
|---|---|---|---|
| / (Root) | Immutable (Read-Only) | Core operating system, systemd services, container runtime binaries | Reset to pristine state on every node reboot or upgrade |
| /var/lib/kubelet | Persistent Block Device | Pod volumes, CSI mounts, local ephemeral storage | Persists across reboots; ensures pod volumes are not lost |
| /var/lib/containerd | Persistent Block Device | Cached container images, active container layers | Persists across reboots to prevent "image pull storms" on restart |
| /etc/kubernetes | Ephemeral RAM Disk | Node-specific bootstrap tokens, certificates, and API configurations | Generated dynamically at boot time via ignition/metadata service |
By enforcing this strict boundary, I can guarantee that upgrading a Kubernetes node's operating system is as simple as rebooting the virtual machine with a newer immutable image. The local NVMe block device containing the cached images and active pod volumes remains untouched and is re-attached to the new OS instance, minimizing downtime and network bandwidth consumption.
To automate this node provisioning and configuration lifecycle, Oxide provides an integration that leverages the Kubernetes Cluster API (CAPI). The Cluster API provider for Oxide translates high-level cluster definitions into concrete Oxide API calls to provision virtual machines, attach network interfaces, and inject bootstrap configurations.
Below is an example of a declarative Cluster API OxideMachineTemplate manifest. This template defines the hardware profile, network attachments, and disk configurations for a pool of Kubernetes worker nodes:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: OxideMachineTemplate
metadata:
name: k8s-worker-template
namespace: default
spec:
template:
spec:
profile: "c1.small"
imageName: "ubuntu-24-04-k8s-v1.30"
networkInterfaces:
- networkName: "k8s-pod-network"
securityGroups:
- "k8s-worker-secgroup"
disks:
- name: "ephemeral-storage"
size: "100Gi"
mountPath: "/var/lib"
volumeSource:
localNVMe:
ephemeral: true
userDataSecret:
name: "k8s-worker-bootstrap"
This manifest demonstrates how platform engineers can manage physical rack resources using the same GitOps workflows they use for application deployments. The OxideMachineTemplate abstracts away the underlying hardware complexity while still allowing fine-grained control over network interfaces and local storage attachments.
The State of Native Block Storage and CSI Integration
While the networking and compute control planes are highly mature, native block storage integration remains one of the most complex engineering challenges in the bare-metal Kubernetes space. In a cloud environment, stateful workloads rely on a Container Storage Interface (CSI) driver to dynamically provision, attach, and detach block volumes to virtual machines.
Oxide's storage architecture is built around a distributed, transactional storage system that runs natively on the rack's hardware. Every physical node in the rack contributes its local NVMe drives to a shared, resilient storage pool. This pool is managed by the rack's control plane, which handles data replication, encryption, and deduplication. To expose this storage to Kubernetes, Oxide is developing a native CSI driver. However, building a highly reliable CSI driver for a custom bare-metal storage plane involves navigating several difficult technical trade-offs.
1. Control Plane Latency vs. Data Path Efficiency
When a pod requesting a persistent volume is scheduled onto a node, the CSI driver must call the Oxide storage API to create a volume and attach it to the hypervisor host. This API call must be fast and transactional. If the storage control plane is slow to respond, pod startup times will degrade, leading to cascading timeouts in the Kubernetes scheduler.
To optimize this, the CSI driver must bypass unnecessary abstraction layers. Instead of routing data through a virtualized storage controller inside the guest OS, the Oxide CSI driver coordinates with the hypervisor to map the distributed storage volume directly into the virtual machine's PCI space as a virtio-blk device. This achieves near-native NVMe performance with microsecond-level latency, but it requires tight coordination between the CSI driver, the hypervisor, and the rack's storage control plane.
2. Handling Node Failures and Volume Detachment
In a bare-metal environment, if a physical node suddenly loses power, any storage volumes attached to that node must be safely detached before they can be attached to a healthy node. If the storage control plane allows a volume to be attached to two nodes simultaneously, data corruption is almost guaranteed.
In public clouds, this is handled by hypervisor-level fencing. On an Oxide rack, the CSI driver relies on the rack's centralized storage controller to enforce strict single-writer semantics. If a node goes offline, the storage controller automatically revokes the node's cryptographic keys to the volume, instantly fencing it. The CSI driver can then safely attach the volume to a new node without waiting for the unresponsive node to acknowledge the detachment.
3. The Unfinished Path to Native Storage
While the compute and networking integrations are fully functional, the native storage integration is still undergoing active development and refinement. Engineering teams currently running Kubernetes on Oxide often utilize a hybrid approach: they leverage the native networking and compute integrations via the CCM and Cluster API, but rely on external storage solutions (such as Ceph or local NVMe storage mapped via hostpaths) while the native CSI driver is being finalized.
This phased rollout highlights a critical lesson in systems engineering: when building a bare-metal cloud, it is better to deliver highly stable, decoupled networking and compute layers first rather than rushing a complex, tightly coupled storage solution that could compromise data integrity.
Operational Trade-offs and Adoption Risks
Adopting a co-designed hardware and software platform like Oxide for Kubernetes introduces several operational trade-offs that engineering leaders must carefully evaluate. While the benefits of a unified control plane are clear, the risks of vendor lock-in and hardware lifecycle management cannot be ignored.
Hardware Lock-in vs. Operational Simplicity
By choosing a tightly integrated rack architecture, you are committing to a specific hardware vendor's ecosystem. Unlike traditional white-box server deployments where you can mix and match Dell, HPE, or Supermicro nodes, the Oxide control plane only runs on Oxide hardware. If your supply chain strategy requires multi-vendor sourcing, this architecture presents a significant adoption risk.
However, the trade-off is a dramatic reduction in operational overhead. In a traditional bare-metal setup, your platform team spends a significant portion of their engineering budget maintaining firmware compatibility matrices, debugging IPMI driver bugs, and writing custom Ansible playbooks to glue together PXE servers and switches. With a co-designed rack, these low-level concerns are abstracted away. The entire rack is updated atomically via signed firmware bundles, shifting your team's focus from hardware maintenance to platform engineering.
Migration Paths and Legacy Coexistence
For organizations with existing bare-metal or VMware-based Kubernetes clusters, migrating to an API-driven rack architecture requires a phased approach. Because the Oxide rack exposes resources via a clean REST API, you can treat it as an on-premises availability zone.
I recommend starting by deploying stateless workloads using the Cluster API provider. This allows you to validate the performance of the decoupled network control plane and the stability of the immutable OS images without risking production data. Stateful workloads should only be migrated once the native CSI driver has reached production maturity in your environment, or by utilizing external, network-attached storage arrays that can coexist alongside the rack.
🏗️ Practical Next Steps for Platform Engineers
If you are evaluating or implementing an API-driven bare-metal Kubernetes architecture, I recommend taking the following concrete steps to ensure a successful deployment:
- Audit Your Network Topology : Before integrating a decoupled network control plane, ensure your physical network core can support the high-bandwidth, low-latency requirements of a distributed rack. Verify that your upstream switches are configured for LACP and can handle the dynamic routing updates generated by the rack's control plane.
- Standardize on Immutable Images : Transition your Kubernetes node templates to use immutable, minimal OS images. Remove any configuration management agents (like Chef or Puppet) from your node bootstrap process and replace them with declarative cloud-init or Ignition configurations.
- Establish Clear Filesystem Boundaries : Configure your container runtimes and kubelet directories to mount onto dedicated, high-performance local NVMe partitions as outlined in the architectural layout. This prevents disk pressure issues from disrupting critical system services.
- Implement GitOps for Infrastructure : Treat your physical rack resources as code. Use the Cluster API provider to define your Kubernetes clusters, node pools, and network security groups in declarative YAML manifests stored in a version-controlled repository.
By treating the physical rack as a single, API-driven system, you can finally achieve the operational simplicity of the public cloud on your own physical hardware. The key to running reliable bare-metal infrastructure is not to build more complex software overlays, but to design clean, decoupled interfaces between your hardware control planes and your container orchestration engines.
🔗 Originally published on ixuvo.com
Top comments (0)