What ioctl(KVM_RUN) Does: The VM Exit Loop Explained

The previous article established that guest code runs natively until it touches a device, at which point control has to reach QEMU.

ioctl(vcpu_fd, KVM_RUN) is the single most important line in the entire stack. Libvirt, the XML, the device models and VirtIO are just scaffolding around this one call, looped forever.

We'll build up to it step by step, because the this call only makes sense once you know what a file descriptor is and how ioctl works generally.

File descriptors, briefly

Before I explained what ioctl is let me refresh your memory about file descriptors. If you don't need a memory refresh, you can skip this section.

Simply put, a file descriptor (FD) is a non-negative integer that serves as a unique identifier for any resource a process has open.

The operating system kernel maintains a private lookup table for every running process. Let's say your program opens a resource. The kernel then creates a tracking entry for the underlying object and hands back the index of that slot. From that moment on, your program refers to that resource exclusively by its number.

By default, three descriptors are pre-allocated when a process spawns:

• 0 for standard input (stdin),
• 1 for standard output (stdout), and
• 2 for standard error (stderr).

The next resource you open grabs the lowest available integer, typically starting at 3. It doesn't matter whether you are interacting with a plain text file, a network socket, a hardware component, or a KVM virtual CPU, the kernel handles them all through file descriptors using uniform system calls like read(), write(), and close(). And the kernel automatically figures out how to route your data based on what that specific integer points to.

Let me show you the outputs from my raspberry pi 5:

home1@home1:~ $ ls -l /proc/self/fd
total 0
lrwx------ 1 home1 home1 64 Sep  7 07:32 0 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 1 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 2 -> /dev/pts/4
lr-x------ 1 home1 home1 64 Sep  7 07:32 3 -> /proc/3311761/fd

Look at the numbers on the far right. FDs 0, 1, and 2 are all pointing to /dev/pts/4. This is the virtual terminal window you are looking at. It says your input, output, and errors are all routed to your screen! Do you see the the number 3 at the end? That is the ls command itself reading the directory table, grabbing the lowest available integer.

ioctl: the escape hatch for everything read and write can't express

Standard read() and write() operations excel at shifting raw bytes back and forth but they fall short when you need to configure the underlying hardware itself.

For example, you cannot write() to an CD drive to force it to eject. Similarly, there's no meaningful way to read() a serial port's baud rate, or to express "change it to 115200" as a byte stream. This is because these are control operations on a device, not data transfers.

The Unix architecture solves this limitation with ioctl (Input/Output Control). ioctl is a highly generic system call designed specifically to send arbitrary commands to whatever resource a file descriptor references.

int ioctl(int fd, unsigned long request, ...);

   ioctl(fd, REQUEST, argument)
          │     │        │
          │     │        └─ Optional pointer to custom data structures
          │     └────────── The specific command ID (a unique integer)
          └──────────────── The file descriptor you are talking to

Under the hood, the driver handling the device interprets what each command integer means. This serves as a universal extension mechanism. And it allows any hardware driver to expose custom routines that the standard filesystem interface never planned for. For example, terminals rely on it to adjust window dimensions, network adapters use it to bind IP configurations, and KVM leverages it to orchestrate an entire virtual machine.

KVM's three-level ladder of descriptors

KVM exposes itself as /dev/kvm, a character device. QEMU opens it and gets a descriptor. That descriptor and a stream of ioctl calls is the entire QEMU-to-KVM interface.

To manage complex hardware setups better, KVM organizes this interface into a strict, three-tiered hierarchy.

When a VM is created, things descent down this ladder:

• System FD (KVM_CREATE_VM ): Allocates a blank, unconfigured VM instance and hands back a unique VM file descriptor.
• KVM_SET_USER_MEMORY_REGION on the VM fd — "here's memory I allocated; treat it as the guest's physical RAM." QEMU allocates guest memory as an ordinary mmap region in its own address space and registers it with KVM. This is the sandbox boundary: the guest can only address what was registered.
• VM FD (KVM_SET_USER_MEMORY_REGION): Defines the virtual sandbox. The Virtual Machine Monitor (VMM) reserves a normal block of userspace memory via mmap, then registers it here. The KVM kernel module transforms this space into the guest's physical RAM layout. It also ensures the guest can never read or write outside this boundary.
• VM FD (KVM_CREATE_VCPU): Spawns an execution core inside the VM structure and maps it to a fresh vCPU file descriptor.
• vCPU FD (KVM_RUN): Instructs the host processor to immediately start executing guest instructions on this specific core. "run this CPU now."

What KVM_RUN actually does

Let's zoom in. The KVM_RUN system call does not behave like a standard, predictable function call. Say a VMM thread(a QEMU vCPU thread) invokes ioctl(vcpu_fd, KVM_RUN). This thread immediately enters the guest. And it does not return to your user-space application until the guest does something KVM cannot handle alone.

The call blocks on purpose for extended periods. Just think about it. From QEMU's perspective, it called a function and that function didn't return for millions of guest instructions. And the entire time the guest is computing, the QEMU thread is parked inside a single ioctl. QEMU isn't polling or supervising anything. This is elegant, isn't it? It is asleep in a system call and wakes up only when there's work for it.

Context switching across the guest boundary is expensive. Entering the guest means loading the registers of the guest into the real core; exiting means saving them back out. This is not free and it's part of why an exit is costly!

Not every VM exit returns to QEMU. If the guest triggers an exit that the Linux kernel can settle internally, KVM resets the guest state and jumps right back into guest mode. For example, when servicing a hardware timer or adjusting a local interrupt controller the user-space program QEMU stays asleep, completely unaware that an exit ever occurred in the first place.

Cheap exits and expensive exits

There are two tiers of VM exits.

Cheap Exits (Kernel-Handled): KVM resolves these entirely within the kernel module and immediately re-enters the guest. The ioctl call never returns. And your VMM thread stays fast asleep. QEMU never learns it happened. Virtual timer ticks fall into this tier. So do accesses to devices KVM emulates in-kernel on ARM64, most importantly the GIC. GIC is the Generic Interrupt Controller. Interrupt handling is far too frequent to route through userspace, so KVM handles it directly.

Expensive Exits (User-Space Handled): Other exits KVM cannot resolve, because the guest touched a device that exists only as a QEMU device model. So, now the ioctl genuinely has to return. QEMU wakes up, runs the relevant C function, and calls KVM_RUN again.

When people say "VM exits are expensive," they mean the second kind. Because as I showed you, first is comparatively cheap.

This distinction is also why the in-kernel GIC exists at all. In the part 2 of the series the interrupt-related warning on Raspberry Pi from article 2 is worth understanding at this point. It generated more exits than a pristine hardware environment would, but the cheap ones.

How QEMU learns what happened: the shared kvm_run struct

When KVM_RUN does return, QEMU needs to know why. Passing that back through the simple numeric return value of ioctl would be far too limited, so KVM uses shared memory.

When a vCPU is created, KVM gives QEMU a small region mapped into both QEMU's address space and the kernel's. KVM writes the exit details into this struct kvm_run before returning. Then QEMU reads them immediately after. There is no need for additional system calls.

exit_reason is the dispatch key. KVM_EXIT_MMIO means the guest accessed a memory-mapped device region. This is the common case on ARM64, and we discussed the mechanism in article 4. QEMU looks up which device model owns that address, calls it, and loops.

Follow the example concretely. The guest writes the character H to the UART at 0x09000000:

1. The guest executes a store instruction. It has no idea anything unusual is happening.
2. The address is in a device region, so the CPU traps. VM exit.
3. KVM sees it's MMIO to an address it doesn't handle in-kernel. It fills in kvm_run and returns from ioctl.
4. QEMU wakes, reads exit_reason, dispatches to its PL011 UART model.
5. The UART model writes H to whatever the serial console is connected to — your terminal.
6. QEMU calls ioctl(KVM_RUN) again. The guest resumes at the next instruction.

Every character of output from a guest's serial console runs that loop. When you watch an OS installer scroll past over virsh console, you are watching this cycle execute thousands of times.

This is the foundation for everything that we will follow

When you can clear your head around this round trip, several later topics becomes easier to grasp.

VirtIO (article 6) is the architectural response to expensive exits. If each round trip costs, then we can make each one carry more work. VirtIO batches many operations behind a single notification, amortizing one exit across dozens of requests.

vCPU pinning Remember that the host thread driving our KVM_RUN cycle is treated by Linux as an ordinary, everyday thread. So, if you don't intervene, the Linux kernel scheduler will happily shift that thread from one physical core to another to balance host workloads, like it usually does. But, if you pinning your vCPU thread to a single core, you ensure the guest state stays perfectly warm inside that specific core's local cache lines between transitions. Now imagine you let that thread wander across your topology. Then every VM exit drops you onto a cold core. This forces the hardware to reload state variables from scratch and destroys your execution speeds.

QMP versus KVM_RUN Don't confuse QMP(QEMU Machine Protocol) with KVM_RUN. I know I did at first. These are two entirely separate communication paths. QMP is a standard Unix/TCP socket interface that management utilities like libvirt use to command QEMU from the outside. KVM_RUN is the internal system call (ioctl) that QEMU executes to hand control over to the Linux kernel module and physical hardware.

Keep this in mind: ioctl(vcpu_fd, KVM_RUN) commands execution, the guest computes directly on bare-metal hardware until a trap occurs, and the system call unblocks only when QEMU device emulation is required. That call, when looped forever, is what constitutes a running virtual machine.

Summary

• A file descriptor is an integer indexing a per-process table of open kernel objects.
• ioctl is the generic system call for device-specific commands that read and write can't express.
• KVM is driven entirely through ioctl on /dev/kvm, via a three-level ladder: system fd → VM fd → vCPU fd.
• KVM_RUN on a vCPU fd enters the guest and blocks for as long as the guest runs natively — potentially millions of instructions.
• Cheap exits are handled inside KVM (timers, the in-kernel GIC) and never wake QEMU. Expensive exits return to userspace so a QEMU device model can run.
• KVM reports why it exited through the shared kvm_run struct; exit_reason tells QEMU which device model to dispatch to.
• Every character on a guest's serial console is one full round trip through this loop.