The Myth of Invisible Virtualization: What Fly.io’s Fleet Patch Reminds Us About Hypervisors

Ed Stuurman — Sat, 14 Nov 2026 01:03:36 +0000

On November 6, 2026, the edge-cloud platform Fly.io published an incident post-mortem with a refreshingly candid title: "Well, That Wasn't Fun: October 16 '26".

The report detailed what happened three weeks earlier when an emergency Linux kernel zero-day landed on their desks. Fly runs bare-metal physical servers across dozens of colocation facilities worldwide, hosting customer workloads as lightweight microVMs on top of standard Linux hosts. They could not paper over this bug with a rolling configuration reload or a managed-cloud toggle.

In a single afternoon, they had to patch and reboot every physical hypervisor in their global fleet.

For a platform whose promise is effortless global deployment—type fly deploy and your code spins up in Tokyo, Amsterdam, and Virginia in seconds—watching bare-metal motherboards power-cycle across twenty time zones was a stark reminder of an uncomfortable reality:

Software cannot abstract away the physical machine.

Underneath every serverless worker and ephemeral microVM sits a physical motherboard with fans, memory DIMMs, a power supply, and a monolithic Linux host kernel. When ring 0 bleeds, the marketing illusion of weightless compute vanishes.

The Illusion of Live Migration

In enterprise IT, virtualization vendors spent twenty years conditioning architects to believe that physical host reboots should never disrupt running workloads. That promise had a name: live migration.

The pitch sounds magical: an active virtual machine, complete with open TCP connections and gigabytes of RAM, floats gracefully from Server A to Server B with zero downtime.

For enterprise clusters with deep pockets, shared SAN arrays, and homogeneous CPU families, live migration worked reasonably well for planned maintenance—swapping a failing power supply, upgrading host RAM, or flashing motherboard firmware on one node while the cluster hummed along. In the face of an emergency host kernel zero-day, however, live migration is a conceptual dead end.

First, modern edge platforms pair microVMs with fast local NVMe drives to deliver microsecond I/O that centralized SANs could never match. Copying tens of gigabytes of raw disk state per VM across top-of-rack switches takes hours, completely defeating the purpose.

More fundamentally: where do you migrate to?

When a critical vulnerability strikes the host Linux kernel, every hypervisor in the fleet is equally exposed. You cannot migrate workloads from one compromised machine to another and call it security. To make the destination host safe, you have to patch and reboot it first.

Attempting to juggle running workloads across unpatched hosts during a critical security event only drags an acute maintenance window into an agonizing, multi-day circus. A cold reboot is not an engineering failure; it is the only clean remediation.

The Litmus Test of an Infrastructure Engineer Is `reboot`

Years ago, my homelab ran a cluster of three 4U single-socket servers tied together with quad-port Intel 10GbE NICs and VMware vSAN, before I eventually consolidated down to a single Proxmox node to cut idle rack power.

Anyone who has operated a vSAN cluster recognizes the physical reality of that iron: the sharp whine of chassis fans spinning up, the heat rolling off the processor socket and banks of ECC RAM, amber drive LEDs chattering as storage pools balance, and the glacial wait while server UEFI firmware trains memory before the bootloader even displays on the IPMI console.

Rebooting a vSAN node was a choreographed, high-tension ritual:

# Evacuating storage before maintenance
esxcli system maintenanceMode set --enable true --vsan-mode ensureObjectAccessibility

You would wait for data to evacuate across the 10GbE links, watch surviving nodes absorb rebuild traffic, and pray an adjacent drive wouldn't throw an I/O fault while the first host was offline. The entire enterprise stack was engineered to avoid a reboot at all costs—which perversely turned every inevitable restart into an operational emergency.

Later, moving to native KVM on Debian stripped away the enterprise mysticism:

# Checking physical virtualization primitives directly on Linux
$ ls -lh /dev/kvm
crw-rw---- 1 root kvm 10, 232 Nov 13 10:15 /dev/kvm

A virtual machine is just an ordinary Linux process with a PID. A virtual disk is a block device. A network interface is a bridge. And that brings you face-to-face with an essential truth:

The ultimate litmus test of an infrastructure engineer is typing reboot.

If typing sudo reboot into a hypervisor shell induces panic, your infrastructure is not resilient. It is fragile state pretending to be high availability.

Hypervisors are mortal machines. Power supplies trip, memory modules throw uncorrectable ECC errors, and operating system kernels require zero-day patches. If your architecture relies on hypervisors running uninterrupted for five hundred days without a restart, you haven't engineered resilience—you have just been lucky.

Designing for Hypervisor Mortality

The reason Fly.io rebooted its entire global fleet in an afternoon without catastrophic downtime comes down to modern systems design. Resilience does not mean building an indestructible hypervisor; it means building workloads that treat host mortality as a routine, non-critical event.

flowchart LR
    Ingress["Edge Ingress Router"]
    subgraph HostA["Hypervisor A (Rebooting)"]
        Drain["Draining: SIGTERM"]
        uVM1["MicroVM A"]
    end
    subgraph HostB["Hypervisor B (Active)"]
        uVM2["MicroVM B (Cold Boot <300ms)"]
    end
    Storage[("Durable State\n(S3 / LiteFS WAL)")]

    Ingress -.->|"Drains"| HostA
    Ingress -->|"Reroutes"| HostB
    uVM1 -->|"Flushes In-Flight"| Storage
    uVM2 -->|"Pulls Fresh State"| Storage

Sub-second microVM boot times: Legacy virtual machines took minutes to initialize SCSI drivers, ACPI tables, and enterprise init scripts. By stripping emulated hardware down to bare essentials, modern KVM and Firecracker microVMs boot in 150 to 300 milliseconds. An orchestrator can spin up replacement compute on an adjacent node before an edge proxy's connection timeout has even elapsed.
Decoupling compute from durable state: Local NVMe should only ever be high-speed scratch space for ephemeral caches and temporary buffers. Real state belongs in durable, replicated tiers—object storage (S3) or replicated databases (like LiteFS or Raft WAL streams) distributed across independent physical machines. If a compute node carries zero irreplaceable state, power-cycling it carries zero emotional weight.
Disciplined SIGTERM handlers: A host reboot only drops traffic when software treats shutdown as an emergency. When a hypervisor drains traffic at the edge proxy, microVMs receive SIGTERM, complete in-flight HTTP requests, flush pending database transactions, and exit cleanly within seconds before the host reboots.

Respecting the Physical Machine

Fly.io’s incident report was a masterclass in operational honesty.

It stripped away the marketing veneer and reminded us of the physical reality underneath the modern cloud. You can wrap your systems in declarative YAML, invoke them through slick CLIs, and hide them behind global Anycast routing. But at the bottom of the stack, your code runs on silicon seated in sockets, powered by copper, and governed by an operating system kernel.

True engineering maturity is not about searching for an immortal hypervisor that never requires maintenance. It is about engineering humility:

Expect the host to reboot.
Make guest boot times instantaneous.
Decouple your state from your compute.
Handle shutdown signals gracefully.
And above all, never be afraid to type reboot.

Originally published at www.stuurman.com/blog/2026-11-14-the-myth-of-invisible-virtualization/ by Ed Stuurman.

Hello DEV! 👋 Systems, Homelabs, and Building from Japan

Ed Stuurman — Sat, 14 Nov 2026 00:19:21 +0000

Hi everyone! 👋

I figured it was overdue to introduce myself and join the conversation properly.

A bit about me

I'm Ed Stuurman, an infrastructure and reliability engineer based in Japan (originally from Europe).

My path in tech started in enterprise hardware escalations, debugging kernel dumps, and tracking down CPU microcode bugs at Hewlett Packard Enterprise. Since moving to Japan, my focus centered on carrier-grade messaging and distributed systems—scaling email security pipelines, MTAs, and observability systems protecting millions of domains.

After daily driving Linux for 15 years (and a Mac for the last 5), Linux and BSD remain the backbone of everything I run—both in production and across my homelab.

What I’m currently building & tinkering with

When I'm not in my terminal debugging Layer 1–7 networking quirks or pruning trees in the yard, I've been spending time on:

Minimalist Web Engineering: Building fast, static web experiences with Astro and Tailwind CSS, keeping client-side JavaScript to an absolute minimum (~5 KB).
The Homelab & Networking: Running a low-power Proxmox cluster, WireGuard edge relays, and navigating the weird and wonderful world of Japanese fiber ISP routing (MAP-E, IPv6 IPoE, and carrier NATs).
AI-Assisted Pair Programming: Treating modern AI coding agents not as magic autocomplete, but as high-leverage junior partners in a disciplined test-and-verify workflow.
The Physical Stack: M4/3 photography, DIY yard irrigation, and teaching my preteen how real computers and physical tools work.

What I’ll be writing about here

I enjoy writing about the gritty realities between software abstractions and physical hardware:

Post-mortems, hardware mysteries, and why systems fail in surprising ways
Homelab architecture, self-hosting, and practical networking
Lightweight web architectures and modern developer workflows
Musings on career longevity, international moves, and work-life balance

Let's connect!

What is your favorite piece of hardware or self-hosted service running in your homelab right now?
Anyone else navigating tech infrastructure from Japan or the APAC region?

Looking forward to reading your stories and sharing notes in the comments!

DEV Community: Ed Stuurman