Staggered VM Boot: How I Prevented a Load Average of 147

Originally published at woitzik.dev

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

After a host reboot, every VM and LXC on the Proxmox host started simultaneously. Twelve containers and VMs, all booting at once, all hitting the same NVMe for root filesystem reads, service starts, and NFS mounts. The host load average hit 147.

For context: load average represents the number of processes in the run queue or waiting for I/O. On a 16-thread CPU, a load average of 147 means 147 processes are competing for CPU or disk time. Every service was slow to start, k3s took minutes to become ready, and DNS didn't resolve for the first 90 seconds because the Raspberry Pi DNS nodes were waiting for services that hadn't booted yet.

The fix wasn't more resources — it was boot order.

View the complete homelab infrastructure source on GitHub 🐙

The Boot Storm

When the Proxmox host starts, all VMs and LXCs configured with onboot=1 start simultaneously. The host's NVMe handles root filesystem reads for every container, plus the ZFS txg commits from the NFS server, plus etcd writes from the k3s control-plane.

The simultaneous startup creates a thundering herd: 12 processes all requesting I/O at the same time, the NVMe queue depth maxes out, I/O latency spikes, and services that depend on each other (k3s needs NFS, k3s apps need DNS, DNS needs k3s services) enter a cascading wait state.

The load average doesn't just spike and recover — it compounds. Services that fail to start within their timeout window retry, adding more processes to the queue. k3s control-plane tries to mount NFS volumes, NFS is slow because it's competing with 11 other containers for I/O, k3s retries the mount, adding more load.

The Fix: Staggered Boot Order

Proxmox supports startup order with delays. The configuration in Terraform:

# terraform/stacks/proxmox/lxc.tf

# NFS first — k3s depends on it
resource "proxmox_virtual_machine" "ct_srv_nfs_01" {
  # ...
  startup = {
    order = 1
    up    = 30  # wait 30s after boot before starting next
  }
}

# k3s control-plane — waits for NFS
resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  # ...
  startup = {
    order = 2
    up    = 30  # 30s after NFS is up
  }
}

# k3s workers — 30s apart from each other
resource "proxmox_virtual_machine" "vm_srv_k3s_12" {
  startup = {
    order = 3
    up    = 30
  }
}

resource "proxmox_virtual_machine" "vm_srv_k3s_13" {
  startup = {
    order = 4
    up    = 30
  }
}

The order:

1. NFS (order 1) — boots first, 30s head start. k3s PVCs mount from NFS, so NFS must be ready before k3s starts.
2. k3s-11 (order 2) — control-plane + etcd. Boots 30s after NFS. Needs NFS for system PVCs.
3. k3s-12 (order 3) — worker. Boots 30s after control-plane. Needs API server ready.
4. k3s-13 (order 4) — worker. Boots 30s after k3s-12.
5. LXCs (order 5+) — everything else. Docker workloads, media stack, DMZ.

Total boot sequence: ~3 minutes for everything to be online. Previously: all at once, 147 load average, 5+ minutes to stability.

Why NFS First

NFS is the foundation of the storage layer. Every k3s PVC (Authelia, Vaultwarden, Paperless, Nextcloud) mounts from the NFS server at 10.0.20.100. If NFS isn't ready when k3s starts, the pod mount attempts fail, Kubernetes retries with exponential backoff, and the pods sit in ContainerCreating for minutes.

NFS itself depends on ZFS — the NFS export directory lives on the ZFS pool. ZFS needs a few seconds after boot to complete any pending txg commits and mount the pool. The 30-second head start gives ZFS and NFS time to stabilize before k3s starts hammering them with mount requests.

The CPU Scheduling Priority

Beyond boot order, k3s VMs get CPU scheduling priority via cpu.units:

# k3s VMs get 2x scheduling priority
cpu {
  cores = 4
  units = 2048  # default is 1024
}

# LXCs stay at default priority
cpu {
  cores = 2
  units = 1024  # default
}

cpu.units tells the Proxmox scheduler how to weight CPU time when multiple VMs compete for the same physical cores. units=2048 means k3s VMs get twice the CPU scheduling priority over LXCs. When Ollama is running LLM inference on the AI LXC and k3s needs CPU for etcd fdatasync, etcd wins.

This matters during boot too: even with staggered starts, there's overlap between late-booting LXCs and already-running k3s workloads. The CPU priority ensures k3s gets scheduling preference.

The onboot Gotcha

Proxmox's bpg/proxmox Terraform provider doesn't reliably manage the onboot attribute. terraform plan always shows "No changes" regardless of the live value — a known limitation of the provider.

This means onboot must be set manually after any LXC recreate:

pct set <id> -onboot 1

I discovered this the hard way: after recreating a container via Terraform, onboot defaulted to 0. The next host reboot silently skipped that container. The k3s control-plane node came up without its NFS mount, and half the cluster was in CrashLoopBackOff until I noticed.

The workaround: a manual pct set step in the operations runbook, applied after every LXC creation. Not ideal, but documented and repeatable.

Before vs. After

Metric	Before (simultaneous)	After (staggered)
Peak load average	147	12
Time to k3s ready	5+ minutes	90 seconds
Time to DNS functional	90 seconds	30 seconds
I/O wait %	85%	15%
Failed mount attempts	12-15	0

The staggering didn't add total boot time — it redistributed the I/O load over 3 minutes instead of concentrating it in 30 seconds. Services come up later individually but the cluster as a whole is stable sooner because nothing is fighting for I/O. The same host also runs a memory overcommit guard for the RAM side of this problem — boot order fixes the I/O storm, the guard fixes the memory one.

---

Boot storm mitigation is the same problem in Azure: when you scale out a VMSS from 0 to 50 instances, all 50 hit the Azure fabric simultaneously. Azure handles this with staggered placement and shared disks, but the principle is identical — spread the I/O load over time instead of concentrating it. In a homelab, you do it yourself with startup.order and startup.up delays.

Designing Data-Intensive Applications* has a genuinely useful framing for this kind of thundering-herd problem, even though it's written about databases rather than hypervisors - the queueing math behind "everything wants the same resource at the same instant" doesn't care what the resource actually is.