DEV Community: Supun Sriyananda

What ioctl(KVM_RUN) Does: The VM Exit Loop Explained

Supun Sriyananda — Tue, 08 Sep 2026 09:10:18 +0000

The previous article established that guest code runs natively until it touches a device, at which point control has to reach QEMU.

ioctl(vcpu_fd, KVM_RUN) is the single most important line in the entire stack. Libvirt, the XML, the device models and VirtIO are just scaffolding around this one call, looped forever.

We'll build up to it step by step, because the this call only makes sense once you know what a file descriptor is and how ioctl works generally.

File descriptors, briefly

Before I explained what ioctl is let me refresh your memory about file descriptors. If you don't need a memory refresh, you can skip this section.

Simply put, a file descriptor (FD) is a non-negative integer that serves as a unique identifier for any resource a process has open.

The operating system kernel maintains a private lookup table for every running process. Let's say your program opens a resource. The kernel then creates a tracking entry for the underlying object and hands back the index of that slot. From that moment on, your program refers to that resource exclusively by its number.

By default, three descriptors are pre-allocated when a process spawns:

0 for standard input (stdin),
1 for standard output (stdout), and
2 for standard error (stderr).

The next resource you open grabs the lowest available integer, typically starting at 3. It doesn't matter whether you are interacting with a plain text file, a network socket, a hardware component, or a KVM virtual CPU, the kernel handles them all through file descriptors using uniform system calls like read(), write(), and close(). And the kernel automatically figures out how to route your data based on what that specific integer points to.

Let me show you the outputs from my raspberry pi 5:

home1@home1:~ $ ls -l /proc/self/fd
total 0
lrwx------ 1 home1 home1 64 Sep  7 07:32 0 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 1 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 2 -> /dev/pts/4
lr-x------ 1 home1 home1 64 Sep  7 07:32 3 -> /proc/3311761/fd

Look at the numbers on the far right. FDs 0, 1, and 2 are all pointing to /dev/pts/4. This is the virtual terminal window you are looking at. It says your input, output, and errors are all routed to your screen! Do you see the the number 3 at the end? That is the ls command itself reading the directory table, grabbing the lowest available integer.

ioctl: the escape hatch for everything read and write can't express

Standard read() and write() operations excel at shifting raw bytes back and forth but they fall short when you need to configure the underlying hardware itself.

For example, you cannot write() to an CD drive to force it to eject. Similarly, there's no meaningful way to read() a serial port's baud rate, or to express "change it to 115200" as a byte stream. This is because these are control operations on a device, not data transfers.

The Unix architecture solves this limitation with ioctl (Input/Output Control). ioctl is a highly generic system call designed specifically to send arbitrary commands to whatever resource a file descriptor references.

int ioctl(int fd, unsigned long request, ...);

   ioctl(fd, REQUEST, argument)
          │     │        │
          │     │        └─ Optional pointer to custom data structures
          │     └────────── The specific command ID (a unique integer)
          └──────────────── The file descriptor you are talking to

Under the hood, the driver handling the device interprets what each command integer means. This serves as a universal extension mechanism. And it allows any hardware driver to expose custom routines that the standard filesystem interface never planned for. For example, terminals rely on it to adjust window dimensions, network adapters use it to bind IP configurations, and KVM leverages it to orchestrate an entire virtual machine.

KVM's three-level ladder of descriptors

KVM exposes itself as /dev/kvm, a character device. QEMU opens it and gets a descriptor. That descriptor and a stream of ioctl calls is the entire QEMU-to-KVM interface.

To manage complex hardware setups better, KVM organizes this interface into a strict, three-tiered hierarchy.

When a VM is created, things descent down this ladder:

System FD (KVM_CREATE_VM ): Allocates a blank, unconfigured VM instance and hands back a unique VM file descriptor.
KVM_SET_USER_MEMORY_REGION on the VM fd — "here's memory I allocated; treat it as the guest's physical RAM." QEMU allocates guest memory as an ordinary mmap region in its own address space and registers it with KVM. This is the sandbox boundary: the guest can only address what was registered.
VM FD (KVM_SET_USER_MEMORY_REGION): Defines the virtual sandbox. The Virtual Machine Monitor (VMM) reserves a normal block of userspace memory via mmap, then registers it here. The KVM kernel module transforms this space into the guest's physical RAM layout. It also ensures the guest can never read or write outside this boundary.
VM FD (KVM_CREATE_VCPU): Spawns an execution core inside the VM structure and maps it to a fresh vCPU file descriptor.
vCPU FD (KVM_RUN): Instructs the host processor to immediately start executing guest instructions on this specific core. "run this CPU now."

What KVM_RUN actually does

Let's zoom in. The KVM_RUN system call does not behave like a standard, predictable function call. Say a VMM thread(a QEMU vCPU thread) invokes ioctl(vcpu_fd, KVM_RUN). This thread immediately enters the guest. And it does not return to your user-space application until the guest does something KVM cannot handle alone.

The call blocks on purpose for extended periods. Just think about it. From QEMU's perspective, it called a function and that function didn't return for millions of guest instructions. And the entire time the guest is computing, the QEMU thread is parked inside a single ioctl. QEMU isn't polling or supervising anything. This is elegant, isn't it? It is asleep in a system call and wakes up only when there's work for it.

Context switching across the guest boundary is expensive. Entering the guest means loading the registers of the guest into the real core; exiting means saving them back out. This is not free and it's part of why an exit is costly!

Not every VM exit returns to QEMU. If the guest triggers an exit that the Linux kernel can settle internally, KVM resets the guest state and jumps right back into guest mode. For example, when servicing a hardware timer or adjusting a local interrupt controller the user-space program QEMU stays asleep, completely unaware that an exit ever occurred in the first place.

Cheap exits and expensive exits

There are two tiers of VM exits.

Cheap Exits (Kernel-Handled): KVM resolves these entirely within the kernel module and immediately re-enters the guest. The ioctl call never returns. And your VMM thread stays fast asleep. QEMU never learns it happened. Virtual timer ticks fall into this tier. So do accesses to devices KVM emulates in-kernel on ARM64, most importantly the GIC. GIC is the Generic Interrupt Controller. Interrupt handling is far too frequent to route through userspace, so KVM handles it directly.

Expensive Exits (User-Space Handled): Other exits KVM cannot resolve, because the guest touched a device that exists only as a QEMU device model. So, now the ioctl genuinely has to return. QEMU wakes up, runs the relevant C function, and calls KVM_RUN again.

When people say "VM exits are expensive," they mean the second kind. Because as I showed you, first is comparatively cheap.

This distinction is also why the in-kernel GIC exists at all. In the part 2 of the series the interrupt-related warning on Raspberry Pi from article 2 is worth understanding at this point. It generated more exits than a pristine hardware environment would, but the cheap ones.

How QEMU learns what happened: the shared kvm_run struct

When KVM_RUN does return, QEMU needs to know why. Passing that back through the simple numeric return value of ioctl would be far too limited, so KVM uses shared memory.

When a vCPU is created, KVM gives QEMU a small region mapped into both QEMU's address space and the kernel's. KVM writes the exit details into this struct kvm_run before returning. Then QEMU reads them immediately after. There is no need for additional system calls.

exit_reason is the dispatch key. KVM_EXIT_MMIO means the guest accessed a memory-mapped device region. This is the common case on ARM64, and we discussed the mechanism in article 4. QEMU looks up which device model owns that address, calls it, and loops.

Follow the example concretely. The guest writes the character H to the UART at 0x09000000:

The guest executes a store instruction. It has no idea anything unusual is happening.
The address is in a device region, so the CPU traps. VM exit.
KVM sees it's MMIO to an address it doesn't handle in-kernel. It fills in kvm_run and returns from ioctl.
QEMU wakes, reads exit_reason, dispatches to its PL011 UART model.
The UART model writes H to whatever the serial console is connected to — your terminal.
QEMU calls ioctl(KVM_RUN) again. The guest resumes at the next instruction.

Every character of output from a guest's serial console runs that loop. When you watch an OS installer scroll past over virsh console, you are watching this cycle execute thousands of times.

This is the foundation for everything that we will follow

When you can clear your head around this round trip, several later topics becomes easier to grasp.

VirtIO (article 6) is the architectural response to expensive exits. If each round trip costs, then we can make each one carry more work. VirtIO batches many operations behind a single notification, amortizing one exit across dozens of requests.

vCPU pinning Remember that the host thread driving our KVM_RUN cycle is treated by Linux as an ordinary, everyday thread. So, if you don't intervene, the Linux kernel scheduler will happily shift that thread from one physical core to another to balance host workloads, like it usually does. But, if you pinning your vCPU thread to a single core, you ensure the guest state stays perfectly warm inside that specific core's local cache lines between transitions. Now imagine you let that thread wander across your topology. Then every VM exit drops you onto a cold core. This forces the hardware to reload state variables from scratch and destroys your execution speeds.

QMP versus KVM_RUN Don't confuse QMP(QEMU Machine Protocol) with KVM_RUN. I know I did at first. These are two entirely separate communication paths. QMP is a standard Unix/TCP socket interface that management utilities like libvirt use to command QEMU from the outside. KVM_RUN is the internal system call (ioctl) that QEMU executes to hand control over to the Linux kernel module and physical hardware.

Keep this in mind: ioctl(vcpu_fd, KVM_RUN) commands execution, the guest computes directly on bare-metal hardware until a trap occurs, and the system call unblocks only when QEMU device emulation is required. That call, when looped forever, is what constitutes a running virtual machine.

Summary

A file descriptor is an integer indexing a per-process table of open kernel objects.
ioctl is the generic system call for device-specific commands that read and write can't express.
KVM is driven entirely through ioctl on /dev/kvm, via a three-level ladder: system fd → VM fd → vCPU fd.
KVM_RUN on a vCPU fd enters the guest and blocks for as long as the guest runs natively — potentially millions of instructions.
Cheap exits are handled inside KVM (timers, the in-kernel GIC) and never wake QEMU. Expensive exits return to userspace so a QEMU device model can run.
KVM reports why it exited through the shared kvm_run struct; exit_reason tells QEMU which device model to dispatch to.
Every character on a guest's serial console is one full round trip through this loop.

How QEMU Emulates Hardware: Device Models, MMIO, and TCG

Supun Sriyananda — Fri, 04 Sep 2026 06:30:00 +0000

The previous article established that QEMU emulates the devices KVM deliberately doesn't.

We constantly hear the phrase, “QEMU emulates devices.” It is treated like a magic trick. A user-space program somehow conjures a virtual disk or a network card out of thin air, and a guest operating system is perfectly fooled.

But there is no black box. The entire illusion relies on a single mechanism which is applied consistently across every piece of virtual hardware. Once you see this mechanism, every performance characteristic of virtualization makes sense. Why does VirtIO exist, why certain operations feel catastrophically slower in a VM than on bare metal.

Before We Begin: QEMU is Two Different Programs

We need to clear up a common point of confusion before looking at the code or going further. QEMU actually does two entirely different jobs depending on how you launch it.

Full-system emulation (qemu-system-*) builds a virtual environment from scratch to boot a complete guest operating system. This is what Libvirt launches for every VM, and it is the sole focus of this series. For example, qemu-system-aarch64 constructs a complete, virtualized motherboard from scratch, including the CPU, RAM, disk controllers, network cards, and firmware—allowing a full guest OS to boot.

User-mode emulation (qemu-*) executes a single binary compiled for a foreign architecture directly on your host OS by translating its system calls on the fly. This engine doesn't build a machine or boot an OS. It's how you run an x86 binary on an ARM machine without a VM. And it is what Docker's binfmt_misc cross-architecture support uses under the hood. This is useful, but not virtualization.

Note that, this series is strictly about full-system emulation. If a command begins with qemu-system-, it is fabricating an entire computer.

The trick: devices live at memory addresses

The mechanism that makes emulation possible relies on a fundamental truth about real hardware:

When an operating system wants to send a byte to a serial port, it doesn't call a function. Instead, it writes to a specific memory address. This memory address is physically wired directly to the serial chip on the motherboard. As you can see no RAM is ever involved here. The motherboard's address decoder routes the signal straight to the hardware registers instead.

This is called memory-mapped I/O, or MMIO, and it's how the CPU talks to essentially every device on a modern ARM system. So, the devices are not special entities the CPU has a private channel to. The devices are just regions of the address space.

Which means the CPU's entire interface to a device comes down to this: writing to certain addresses and reading from certain addresses. That's it. Nothing else. And this is an interface software can easily impersonate!

QEMU maps out a guest address space where specific regions are intentionally flagged as "device space" rather than RAM. Now, let's imagine this guest attempts to access one of these coordinates. This operation is blocked from executing as a memory task. Instead, it triggers a trap. Then QEMU runs a dedicated software function.

That software function is the device. For example, a virtual UART is just a few hundred lines of C code that prints a character to your terminal when a write occurs. A virtual disk controller is C code that intercepts a guest read request and fetches bytes from a .qcow2 file on your host SSD.

Let's me show you this in code. The the following is a simplified, structural layout of what that C code of this software function looks like. Don't overwhelm yourself with all the details in the code. Just take a look at it. I just want show you that there is no magic here.

1. The Handoff Interface (MemoryRegionOps)

First of all, QEMU needs to map the memory region to specific C functions. It registers a structure that tells the engine exactly what code to execute when a guest touches the device's addresses:

/* This structure acts as the router for guest hardware accesses */
static const MemoryRegionOps uart_mmio_ops = {
    .read = uart_mmio_read,   /* Function to run when guest reads from an address */
    .write = uart_mmio_write, /* Function to run when guest writes to an address */
    .endianness = DEVICE_NATIVE_ENDIAN,
    .valid.min_access_size = 1,
    .valid.max_access_size = 4,
};

2. The Software Function (uart_mmio_write)

Let's imagine that the guest writes to 0x09000000. When it does, the hardware trap catches it, and QEMU executes this exact type of software function. You can see that it is just a standard C switch statement mapping address offsets to software behaviors.

/* The software function that IS the device hardware channel */
static void uart_mmio_write(void *opaque, hwaddr offset, 
                            uint64_t value, unsigned size)
{
    UARTState *s = (UARTState *)opaque;

    /* The offset is how far past the base address (0x09000000) the guest wrote */
    switch (offset) {
        case 0x00: // Data Register (TX Buffer)
            /* The guest wants to transmit a character. 
               Instead of triggering silicon, we print it to your host terminal. */
            putchar((char)value); 
            fflush(stdout);

            /* Tell our software model that an interrupt is ready (Data sent) */
            s->uart_status |= STATUS_TX_EMPTY; 
            break;

        case 0x04: // Interrupt Enable Register
            /* The guest is trying to toggle hardware interrupt lines */
            s->interrupt_mask = value;
            break;

        case 0x08: // Baud Rate Divisor
            /* A real chip would change electrical frequencies here.
               QEMU just updates an integer variable in RAM. */
            s->baud_rate = 115200 / (value ? value : 1);
            break;

        default:
            /* Guest tried to write to an unmapped register address */
            log_bad_hardware_access(offset);
            break;
    }
}

There is no magic anywhere in this. Every device your guest sees is a software model like this. A chunk of code pretending to be a chip. It does this well by responding to the same address accesses the real chip would respond to. And the guest's driver cannot tell the difference, because from the driver's perspective there is no difference.THe guest driver writes to an address and something happens.

Why the guest's firmware finds these devices at all

This architecture raises an obvious question: how does the guest OS know that a serial port is exactly at 0x09000000 in the first place?

On a physical ARM board, this layout is explicitly dictated by something called a Device Tree. A device tree is a structured data file compiled into the firmware that outlines exactly what hardware components exist and what memory addresses they occupy. x86 has legacy hardware discovery conventions, but ARM has no equivalent of this. I won't go into details about x86 architecture because the series is about ARM. Just remember that in ARM the entire motherboard topology must be spelled out line-by-line for the kernel at boot.

QEMU generates a this device tree describing the machine it's emulating, and hands it to the guest at boot. The guest kernel reads it and sees "there's a PL011 UART at 0x09000000," loads the PL011 driver, and starts writing to that address. The driver is the standard, unmodified Linux driver for real PL011 hardware. It has no idea it's talking to C code.

The device tree

Here is a highly simplified, annotated snippet of a real Device Tree Source (.dts). Again, just take a look only to recognize there is no magic.


dts/dts-v1/;

/ {
    interrupt-parent = <0x8001>;
    #address-cells = <0x02>;
    #size-cells = <0x02>;

    /* Tells the Linux kernel this is QEMU's standardized virtual board */
    model = "linux,dummy-virt";              /* Identifies this as the synthetic 'virt' board */
    compatible = "linux,dummy-virt";

    /* Core CPU Configuration (e.g., 2 vCPUs requested) */
    cpus {
        #address-cells = <0x01>;
        #size-cells = <0x00>;

        cpu@0 {
            device_type = "cpu";
            compatible = "arm,cortex-a57";
            reg = <0x00>;
        };

        cpu@1 {
            device_type = "cpu";
            compatible = "arm,cortex-a57";
            reg = <0x01>;
        };
    };

    /* System Memory Layout (e.g., -m 2048M starts at base address 0x40000000) */
    memory@40000000 {
        device_type = "memory";
        reg = <0x00 0x40000000 0x00 0x80000000>;
    };

    /* The Interrupt Controller (GIC) so the CPU can talk to peripherals */
    intc@8000000 {
        compatible = "arm,gic-v3";
        interrupt-controller;
        #interrupt-cells = <0x03>;
        reg = <0x00 0x08000000 0x00 0x010000>,   /* GIC Distributor physical address */
              <0x00 0x080A0000 0x00 0xF60000>;   /* GIC Redistributors physical address */
        phandle = <0x8001>;
    };

    /* A Virtual UART (Serial Port) for console output */
    pl011@9000000 {
        compatible = "arm,pl011", "arm,primecell"; /* Tells Linux which driver binary to load */
        reg = <0x00 0x09000000 0x00 0x1000>;      /* Base address (0x09000000) and memory size (4KB) */
        interrupts = <0x00 0x01 0x04>;            /* Bound to SPI interrupt 1 */
        clocks = <0x8000>;                         /* Tells the guest which hardware interrupt wire it uses */
        clock-names = "uartclk";
    };

    /* ... PCI host bridge, virtio devices, and flash memory nodes continue below ... */
};

At the very top, QEMU defines the basic structure of our synthetic motherboard. It explicitly states that this is not a real-world chip from Raspberry Pi or Apple, but QEMU’s custom engine.
Take a look at the pl011@9000000 block of the code at the end. This is a Virtual UART.

This is also why the virt machine type matters on ARM64. Because it is not emulating any real-world board. It is a synthetic machine defined by QEMU. On x86, machine types like pc and q35 emulate actual historical chipsets, complete with their quirks. ARM64 skipped that inheritance.

The Dual-Engine Architecture of QEMU

QEMU manages peripheral hardware. It has a device job. This should be clear to you by now. But, it also faces a separate challenge: how to execute the guest's CPU instructions. This is QEMU's CPU job.

Half 2 never changes. We discussed in the previous article, that QEMU even without a KVM still able to virtualize. But, it is just slower. The peripheral devices are emulated by QEMU in userspace even if the KVM is missing or inactive. KVM has no built-in device models and never acquires any previous article. (There is a minor exception: KVM implements a few latency-critical components, like the interrupt controller, inside the host kernel to boost performance—but the general rule stands, and that exception is simply an optimization worth remembering.)

Half 1 is the part with two implementations.

TCG — the Tiny Code Generator — This is QEMU's software CPU. It reads blocks of guest instructions, compiles them into equivalent host instructions on the fly, caches the resulting binaries, and runs them. It is a Just-In-Time (JIT) compiler operating between two processor architectures. This is true software emulation. And this is your only option when your guest and host architectures don't match. An x86 guest running on an ARM64 host must use TCG; no amount of hardware assistance can force an ARM core to natively parse x86 machine code.

KVM is the hardware path from article 1.2. Instead of translating instructions in software, QEMU asks KVM to orchestrate the execution. KVM runs the guest code directly on a real, physical core using hardware virtualization extensions. And after that KVM hands control back up to QEMU only when a device trap occurs.

The choice between them is what <domain type='kvm'> versus <domain type='qemu'> selects in libvirt's XML, and it's the single biggest performance factor in the entire stack. We will learn more about these in future posts.

The consequence: what a device access costs

When you combine QEMU's two halves, the central performance reality of all virtualization instantly falls into right place.

Under KVM, guest code runs natively at full speed. Arithmetic, conditional loops, and memory operations inside actual RAM happen directly on the bare metal without any intervention from KVM or QEMU. Millions of instructions can execute back-to-back without either host component ever waking up.

However, a device access can't complete natively, because there's no physical device. It has no choice but to trap out to QEMU's C code.

So the cost model of a virtual machine is: native speed for computation, and a relatively expensive round trip for every device interaction.

This single asymmetry explains an enormous amount of real-world behavior. If you have ever worked with VMs you will be able to relate immediately. It is the exact reason why CPU-bound workloads in a VM perform at near-bare-metal speeds while I/O-heavy workloads lag. It is why a virtual serial console is perfectly fine for login shell. Because a human types slowly. So a trap per character goes unnoticed. However, it would be absolutely catastrophic for a virtual network card processing a hundred thousand packets per second.

And it's the reason VirtIO exists. If device access is the expensive operation, the winning strategy is to make each one carry more work. In other words: you batch. That's article 6.

But you would understand the batching argument only if you know what the round trip actually involves: what "control leaves the guest" means mechanically, what data crosses the boundary, and why some traps are far cheaper than others. That's the next article.

Summary

qemu-system-* emulates a whole machine and boots an OS. qemu-<arch> runs a single foreign binary. Only the first is virtualization.
Devices on real hardware are reached through memory-mapped I/O — the CPU writes to addresses that are wired to chips rather than RAM.
QEMU exploits this by marking regions of the guest address space as device regions. An access there traps and runs a C function. That function is the device.
On ARM64, the guest discovers what exists and where via a device tree QEMU generates. Guest drivers are the standard, unmodified Linux drivers.
QEMU has two halves. Devices are always emulated in QEMU userspace. The CPU is either translated in software (TCG) or run natively on real hardware (KVM).
Under KVM, computation runs at native speed but every device access costs a trap out to QEMU. That asymmetry is the foundation of every performance decision in virtualization.

KVM vs QEMU vs libvirt: Who Does What

Supun Sriyananda — Wed, 02 Sep 2026 06:30:00 +0000

The previous article covered the CPU virtualization extension and how KVM drives it.

"What’s the difference between KVM and QEMU?" is one of the most-asked questions in Linux virtualization. If you look around most answers just stack definitions on top of each other. For me, definitions are easily forgotten. I think the best way is to build the causal chain—understanding that each component exists specifically to solve a limitation left behind by the layer beneath it. So, rather than defining everything upfront and hoping you connect the dots, I will start with KVM’s deliberate boundaries and follows the architectural consequences all the way up.

KVM only virtualizes the CPU and Memory. Nothing else.

The previous article established what KVM(Kernel-based Virtual Machine) contributes. KVM drives the CPU's virtualization extension so guest code executes natively on real cores, trapping only on privileged operations. It also manages guest memory by mapping the guest's idea of physical addresses onto host pages, and keeping each guest confined to its own.

KVM is incredibly efficient at executing CPU instructions and managing guest memory at near-native speed. But it only virtualizes the CPU and Memory. Nothing else.

A naked CPU and raw RAM do not make a computer. A virtual machine needs a motherboard, a PCI bus, a keyboard, a mouse, a graphics card, serial ports, and disk controllers. And KVM deliberately includes none of these hardware simulations. So, if you only had KVM, your virtual machine couldn't boot because it wouldn't have a virtual hard drive to read an OS from, or a virtual screen to display it.

You need to keep this in mind. This is NOT an oversight. It is a design decision, and article 1.1 explained why: because the hypervisor lives inside a full Linux kernel, it inherits everything Linux already does well. Building device emulation into KVM would mean duplicating work the kernel and userspace already handle.

So KVM leaves an obvious gap. A guest kernel boots, probes for a disk, and finds nothing. We need something that can answer.

QEMU Fills the Hardware Gap: It pretends to be all the devices

Because KVM leaves the virtual machine without a "body," it needs a partner. QEMU(Quick Emulator) is a userspace program that emulates an entire machine's worth of devices. When the guest probes for a disk controller, QEMU responds. When the guest writes a byte to a serial port, QEMU receives it. When the guest sends a network packet, QEMU hands it to the host's network stack.

This reveals the exact handoff between KVM and QEMU. When the guest tries to talk to a hardware device, KVM intercepts the request because KVM doesn't know how to handle hardware. KVM pauses the VM and hands the request up to QEMU. Inside QEMU, a piece of C code pretends to be the chip the guest is looking for, handles the data, and tells KVM to let the guest resume. Every piece of "hardware" the guest sees is just an illusion written in software. Article 1.4 covers how that actually works, since "emulates devices" is doing a lot of quiet work in that sentence.

QEMU can also emulate a CPU. On its own it's a complete machine emulator. So, even if you don't give it a KVM, it will translate guest instructions in software and boot an OS anyway, slowly. But if you pair it with a KVM, it hands the CPU and memory job to KVM and keeps only the devices.

That pairing is what a running VM is:

This architecture leads to two structural realities that govern how virtual machines behave on a live Linux system:

One VM is one QEMU process. A virtual machine is not a mysterious container or a cluster of workers. It is a single, standard Linux process (like qemu-system-x86_64 or qemu-system-aarch64) running in user space. qemu-system-x86_64 and qemu-system-aarch64 are examples. qemu-system-x86_64 process emulates a 64-bit Intel / AMD (x86_64) architecture and qemu-system-aarch64 emulates 64-bit ARM (ARM64 / AArch64) architecture. They are executable programs that run as processes on your operating system. So, in your task manager (like top or htop on Linux/macOS, or Task Manager on Windows), you will see qemu-system-x86_64 or qemu-system-aarch64 listed as an active process. If you kill that process, the VM instantly dies. If a guest has four vCPUs, that single process simply spins up four ordinary Linux threads. Each thread runs a continuous loop: execute guest code, halt when something traps, handle the intercept, and resume. Because these are standard host threads, the Linux scheduler manages them normally, which is exactly why you can pin specific vCPUs to physical CPU cores using standard Linux tools.

The two halves exist in a constant state of handoff Because KVM only handles CPU and memory, it runs guest code at native hardware speed right up until the guest tries to interact with a device (like writing to a disk or sending a network packet). KVM cannot handle this, so it pauses the vCPU thread and yields control back up to QEMU. QEMU’s C-coded device model simulates the hardware response, updates the virtual state, and tells KVM to resume execution. That handoff is article 1.5.

So KVM and QEMU together are a complete machine. Which raises the next question: how do you actually start one?

libvirt: Turning a Messy Command Into a Permanent Document

There is the problem with QEMU as a user interface. Describing a virtual machine to it means you have to to pass every single hardware detail as a raw command-line flag.

To launch a standard VM, you have to manually specify each disk's format and bus type, every network interface's model and host backend, the exact firmware paths, the CPU microarchitecture, the serial console configurations, and the explicit memory layout. A realistic, production-ready QEMU invocation easily runs to dozens of flags across multiple lines of unreadable text.

Nobody wants to type that. More importantly, nobody wants to remember it.

Libvirt solves this by making the VM a document rather than a command.

Instead of writing a transient shell script, you describe the machine’s desired state once in a structured XML file. The Libvirt daemon (libvirtd) reads that XML, dynamically constructs the corresponding, massive QEMU command line behind the scenes, launches the process, and tracks its lifecycle from that moment forward.

The Broader Scope: Managing the Ecosystem

libvirt's scope is broader than launching processes. It defines and manages virtual networks, storage pools, and snapshots. It exposes a stable API, so tooling written against libvirt keeps working across QEMU versions. And it maintains a control channel to each running QEMU — the QMP socket — which is how a running VM can have a disk hot-plugged or a snapshot taken without restarting.

The tools you type are all front-ends to that daemon. virsh is the command-line client and the one this series lives in. virt-manager is a desktop GUI. virt-install creates new VMs. Cockpit provides a web interface. None of them talk to QEMU directly — they all send API calls to libvirtd, which does the work.

The payoff is that the XML is the VM. There's no hidden state elsewhere. Change a line in the document and you've changed the machine. That's why the domain XML is the central artifact of libvirt work, and it's the subject that opens the next phase of this series.

Libvirt’s responsibility doesn't end once the QEMU process is running. Its scope is much broader than launching processes. Libvirt defines and manages the entire virtualization ecosystem, including virtual networks, storage pools, and snapshots.

By exposing a stable, unified API, Libvirt ensures that any tooling written against it keeps working seamlessly across different QEMU versions. This shields you from breaking changes in upstream software.

The Frontends: Just Messengers for the Daemon

Because Libvirt acts as the central brain, the tools you actually type or click are just thin frontends to the daemon:

virsh – The powerful command-line client (and the core focus of this series).
virt-manager – A traditional desktop GUI for visual management.
virt-install – A specialized CLI tool dedicated to provisioning new VMs.
Cockpit – A modern, browser-based web interface for server management.

None of these tools talk to QEMU directly. They all send standard API calls to libvirtd, the Libvirt daemon. It executes the heavy lifting behind the scenes.

The ultimate payoff of this architecture is simple: the XML is the VM. There's no hidden state elsewhere. If you change a line in that document, you change the machine. Because the domain XML is the absolute central artifact of Libvirt, it is the exact subject that will open the next phase of this series.

The three sentences

Compressed to their essentials:

KVM runs the guest's CPU and memory on real hardware at native speed.
QEMU pretends to be all the devices the guest thinks it has.
libvirt stores your VM as XML and drives QEMU so you don't have to.

Definitions are handy, but the causal version is far more useful because it explains exactly why the Linux virtualization stack has this specific shape:

KVM is lightweight because Linux already provides everything except raw CPU virtualization. That leaves peripheral hardware unhandled. So, QEMU steps in to simulate devices in userspace. However, QEMU’s resulting command-line interface is completely unusable by hand, so Libvirt abstracts it into a clean, permanent document and manages it for you.

If you want to test your understanding of this stack, imagine pulling a single brick out of the wall. And see what happens:

Without Libvirt: Everything still runs fast and has hardware, but you are stuck hand-writing, troubleshooting, and maintaining enormous, brittle command lines.
Without QEMU: Your virtual machine has a hyper-fast brain but no body. The guest has no virtual motherboard, network card, or disk to boot from.
Without KVM: Everything still works perfectly, and your guest has all its devices. But, it just runs painfully slow completely in software emulation.

What this means for the rest of the series

The division of labour is a map for everything that follows.

Storage work — disk formats, snapshots, thin provisioning — is about what QEMU's virtual disk points at on the host.

Networking work is about what QEMU's virtual NIC plugs into: a NAT bridge, a LAN bridge, a TAP device.

Performance work splits across the boundary. Replacing emulated devices with VirtIO ones is QEMU-side. Pinning vCPU threads to physical cores is host-scheduler-side, and possible only because those threads are ordinary Linux threads.

Passthrough is the exception that proves the rule: it bypasses QEMU's emulation entirely and hands a guest real hardware.

Every one of those is a modification to one part of a machine whose shape you now know.

Summary

KVM handles guest CPU and memory only. It has no device emulation at all, by design, because Linux already provides everything else.
QEMU fills that gap: a userspace program that emulates disks, NICs, consoles, and firmware. It can emulate a CPU too, but hands that job to KVM when available.
One running VM is one QEMU process, with one host thread per vCPU.
libvirt exists because QEMU's command line is impractical to write by hand. It stores each VM as XML, builds the command line, launches QEMU, and manages it via a control socket.
virsh, virt-manager, virt-install, and Cockpit are all front-ends to the libvirtd daemon.
The domain XML is the VM — there's no hidden state elsewhere.

Why KVM Needs the CPU: ARM64 EL2, VHE, and the Virtualization Extension

Supun Sriyananda — Tue, 01 Sep 2026 09:06:04 +0000

The previous article established that KVM is a kernel module that turns the running Linux kernel into a hypervisor.

That description is accurate but incomplete, and the gap is where a lot of otherwise-solid mental models come apart. A kernel can't simply decide to run guest operating systems at native speed. It needs the CPU's cooperation — a hardware feature that sits dormant in the silicon until software switches it on.

This article is about that feature: what problem it solves, what it looks like on ARM64, and how it relates to the kernel module. The relationship between those last two is the part worth getting right, because "is virtualization hardware or software?" has an answer that isn't either.

The problem: a guest kernel wants to run privileged instructions

Recall the lie from the previous article; "every guest OS believes it owns the hardware". That belief has teeth. A kernel doesn't just think it's in charge; it executes instructions that only something in charge is allowed to execute. It reconfigures page tables. It writes to device registers. It masks and unmasks interrupts. It sets up the memory management unit.

Now put that kernel in a VM. It's going to attempt all of those things, because that's what kernels do at boot. Two options present themselves, and both are bad.

Option one: let it run them on the real CPU. The guest reconfigures the actual MMU, writes to actual device registers, and takes down the host and every other guest with it.

Option two: catch every instruction in software. Inspect each one before it executes, and when it's privileged, emulate its effect against fake state instead of letting it touch hardware. This works. And it is how software-only virtualization operates. However, the cost is enormous. Software is now involved in every single instruction the guest runs, including the overwhelming majority that are perfectly harmless arithmetic and branches.

The insight that makes modern virtualization viable is that neither option is necessary if the CPU itself can tell the difference.

But, think about it for a second. What if the CPU can tell the difference?

The fix is in the silicon

Modern CPUs ship with a virtualization extension: a hardware feature that adds a privilege level above the normal kernel level, designed specifically for a hypervisor to occupy.

With it, the guest kernel runs at its usual privilege level, fully believing it's in charge. Its ordinary instructions — the arithmetic, the branches, the memory accesses — execute directly on a real core at full speed, with no software involvement whatsoever. But the moment it attempts something that would affect real hardware, the CPU traps. It freezes the guest and transfers control up to the hypervisor.

This is the crucial economic property. Now the expensive interventions happen only on privileged operations, which are rare. Everything else runs native. The software now can stop babysitting and start handling exceptions.

That trap is the VM exit. A VM Exit is a hardware-driven transition where control shifts from the guest virtual machine back to the host operating system hypervisor. This article establishes what causes a VM exit and why it exists; the mechanics of what happens during one — how control actually returns to userspace, what data crosses the boundary, and why some exits are far cheaper than others — are the subject of article 5.

ARM64's privilege levels, and the x86 equivalents

ARM calls its privilege levels Exception Levels, abbreviated EL. Higher numbers mean more privilege, which is the opposite of x86's ring numbering.

A guest's applications run at EL0. The guest kernel runs at EL1, where it has always expected to run, and where it can do everything a kernel normally does. Above it sits EL2, the hypervisor level, which the guest cannot see or reach.

On x86 the same concept is called VT-x on Intel and AMD-V on AMD, and the additional level is described as "root mode" rather than given a number. The naming differs but the structure is the same. If you know x86 virtualization, EL2 is your root mode.

There's a fourth level on ARM — EL3, for secure-world firmware. This is outside the scope of virtualization and not something KVM touches.

VHE: why EL2 alone wasn't enough

The original ARM virtualization design assumed the hypervisor would be a small, purpose-built piece of software living at EL2. EL2 was designed for that: a lean environment with its own register set, deliberately unlike EL1.

Linux is not that. Linux is a large general-purpose kernel written to run at EL1, and KVM is a module inside it. Under the original design, running KVM meant splitting the kernel. So most of Linux will be at EL1, with a small stub at EL2 handling world switches. It worked, but every transition between the two carried overhead. And this split was structurally unpleasant.

VHE — Virtualization Host Extensions — fixed this. Introduced in ARMv8.1, VHE makes EL2 capable of running an ordinary EL1-style kernel directly, by remapping registers so that a kernel written for EL1 can execute at EL2 without modification. The host kernel, KVM included can run entirely at EL2. Guests run at EL1 below it. Neither split nor stub is needed now.

The practical upshot: on VHE-capable hardware, KVM runs in its efficient configuration. On older ARM cores without it, KVM still works, using the split approach. The Cortex-A76 in the Raspberry Pi 5 supports VHE. If we check the kernel boot log we can confirm this.

Seeing it on real hardware

Here's the boot log from a Raspberry Pi 5 running Raspberry Pi OS Bookworm, kernel 6.12 aarch64:

$ dmesg | grep -i kvm
[    0.046684] kvm [1]: nv: 554 coarse grained trap handlers
[    0.046799] kvm [1]: IPA Size Limit: 40 bits
[    0.046811] kvm [1]: GICV region size/alignment is unsafe, using trapping (reduced performance)
[    0.046835] kvm [1]: vgic interrupt IRQ9
[    0.046846] kvm [1]: VHE mode initialized successfully

Every concept in this article is visible in those five lines.

VHE mode initialized successfully It confirms not just that KVM found a usable virtualization extension, but that it's running in the modern configuration described above. The host kernel is at EL2 without a split. This is the exact moment the dormant silicon feature was switched on, recorded in the boot log.

IPA Size Limit: 40 bits refers to the Intermediate Physical Address space. This is the guest's view of physical memory. When a guest accesses what it believes is a physical address, that's an IPA(Intermediate Physical Address), which hardware then translates to a real host physical address. A 40-bit tracking system can create (2^{40}) unique address combinations(which is 1 Terabyte (TB) of RAM). The hardware allows the virtual machine to hold drastically more memory than your physical computer actually possesses.

vgic interrupt IRQ9 GIC (Generic Interrupt Controller): This is the physical ARM hardware chip responsible for delivering interrupts from hardware devices directly to the CPU. Guests need interrupts too. Because multiple virtual machines cannot safely share the same physical interrupt chip, KVM creates a simulated, virtual version (Virtual GIC) for each VM.

The GICV region size/alignment is unsafe, using trapping (reduced performance) line is a Raspberry Pi–specific quirk. One optimization for injecting interrupts directly into guests isn't safely available on this hardware, so KVM falls back to trapping them. It does not affect whether acceleration works. it makes one specific interrupt path slightly slower, that is all. This is a physical trait of the Raspberry Pi chip that cannot be changed. So we ignore this warning and move on.

nv: 554 coarse grained trap handlers refers to nested virtualization support — running a hypervisor inside a guest. Not relevant to in this series.

Is virtualization silicon or software?

Now we can take on the question this article exists to answer. If virtualization is a CPU feature, why is KVM a kernel module? And if KVM is a kernel module, what is the CPU actually contributing?

They are two halves of one mechanism, and neither accomplishes anything alone.

The CPU extension is a dormant hardware capability. EL2 exists in the Cortex-A76 the moment it leaves the factory. It is baked into the silicon and does nothing on its own. It's a feature waiting for software to claim it. Is is basically an engine with no driver.

The KVM module is the software that claims and drives it. When KVM initializes, this is the sequence:

So the answer to "is virtualization hardware or software" is that it's a partnership with a strict dependency. The extension without KVM is an unused silicon feature. KVM without the extension has no job. It exists solely to drive that hardware, and it will refuse to initialize on a CPU that lacks it.

This dependency has a useful consequence: the existence of /dev/kvm is proof the whole chain worked. Beacause, that file is created only when KVM has successfully initialized against a real virtualization extension. If it is there, the silicon feature is present, the software found it, and the door is open. That's why every KVM troubleshooting guide starts by checking for it, and it's the first command in article 7.

What happens when the extension is absent

This is worth stating explicitly, because it clarifies the boundary. If a CPU has no virtualization extension, you are not stuck. You can virtualize but you're just slow.

QEMU(Quick Emulator) is a complete machine emulator independent of KVM. It is a software program that emulates and virtualizes computers.

QEMU can mimic an entire computer hardware system in software. It lets you run an operating system made for one processor (like an ARM chip) on a completely different processor (like an Intel laptop chip) by translating the code step-by-step instruction by instruction. And QEMU can do this without any hardware support. It works. It's simply far slower and you can guess why. Because software is translating guest instructions rather than letting them run on a real core.

But, if you pair QEMU with a kernel module like KVM, it drops the slow software translation. Instead, it passes instructions directly to your real CPU cores, letting the virtual machine run at near-native hardware speed.

   QEMU alone   = full machine in software    → works, slow
   QEMU + KVM   = QEMU does devices,          → works, fast
                  KVM rides the CPU extension
                  for CPU and memory
   KVM alone    = impossible — KVM only does CPU and memory.
                  It can't be a disk or a NIC. It always needs QEMU.

That third line is the setup for the next article, and there's a consequence of the second because it's specific to ARM64 hosts.

KVM can only accelerate guests whose instruction set matches the host's. An aarch64 guest on an ARM64 host runs natively. The Cortex-A76(like in a Raspberry pi 5) executes the guest's instructions directly, so KVM applies and it's fast. An x86 guest on the same host cannot. Because, those aren't ARM instructions, and no extension makes a real ARM core execute them. QEMU falls back to full software translation.

This isn't a limitation of KVM or of the Raspberry Pi. It's the boundary of what hardware acceleration means. Acceleration requires the guest and host architectures to match; when they don't, you're emulating, and emulation is slow. Anything in this series involving KVM assumes aarch64 guests.

Remember: Hardware acceleration always requires the guest and host architectures to match, regardless of whether you are using ARM or x86.

Summary

A guest kernel executes privileged instructions. Letting them touch real hardware is unsafe; catching every one in software is slow.
CPU virtualization extensions solve this by adding a privilege level above the kernel. Guest code runs natively until it does something privileged, which traps to the hypervisor. That trap is a VM exit.
On ARM64 the hypervisor level is EL2, above EL1 (kernel) and EL0 (user). The x86 equivalent is VT-x/AMD-V root mode.
VHE lets a full Linux kernel run at EL2 unmodified, which is the efficient configuration KVM uses on modern ARM cores. dmesg reports it at boot.
The silicon extension and the KVM module are partners: the extension is dormant hardware, KVM is the software that switches it on. Neither works alone, and KVM refuses to initialize without it.
/dev/kvm existing is proof the partnership succeeded.
KVM only accelerates guests matching the host architecture. x86 guests on ARM64 fall back to slow software emulation.

I smashed a bug so hard and came out with a Python library

Supun Sriyananda — Fri, 21 Aug 2026 15:01:47 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This is the story of my bug spray.

I build sensor systems that live outdoors. Things like Raspberry Pi units on 4G cellular, embedded gateways, battery monitoring hardware in places nobody visits twice a year. On those networks, connection reliability isn't a simple nice-to-have.

I was a junior back then, and I lost data for weeks before I understood why. When I finally went looking for answers, all I found were scattered fixes. Someone had commented on a years-old thread, "this worked for me". Without any consistency anywhere.

So I patched. Patch after patch, bug after bug, year after year. At some point I thought: why not smash this thing once and for all, for everyone?

And so I did. Smashed, and smashed, and smashed. The final result? An open source python library.

The device that went deaf

Let me tell you about the bug first. So that you know what I smashed and you don't feel pity for it.

Picture a busy restaurant. You're working the pass out front; the kitchen is through the back, and the two of you talk over a radio link. Every thirty seconds the kitchen shouts, "All dishes on time!" Then the food stops coming out. You check the channel, the power light and the microphone. And Everything is perfect. The kitchen keeps cheerfully reporting they're doing great, and ignores every order you scream back.(Sounds annoying, right?)

What happened is that the signal hissed for half a second earlier that night. The link dropped and came back instantly. But the radio system had been set up with a rule: when a link drops, wipe everything about it. The kitchen's standing request to be told about incoming orders was part of that everything. So it went ahead and wiped everything!. The kitchen isn't muted and isn't broken. Its request simply isn't on the list anymore, and it has no way of knowing that. So there are no warining lights or warning beeps beacuse there is no failure to report, since nothing failed.

The system expects the kitchen to re-announce "we're listening on channel four" itself, every single reconnect, forever, unprompted. If it doesn't, it wakes up half alive: still broadcasting to the dining room, deaf to every order coming in. And the only channel it has for telling you something is wrong is the half that still works.

The radio wasn't defective. It did exactly what it had been told to do. And who was the person told it? Me!. I told it months earlier, when I set it up. It was just never designed for a kitchen four hours away with nobody standing next to it.

MQTT is a simple messaging system used by smart gadgets to talk to servers, acting like a two-way radio channel where devices must specifically "subscribe" to a frequency to hear messages. By default, if a device loses its connection even for a second, the system resets and forgets what channel it was listening to, requiring the device to manually tune back in. Paho MQTT is an open-source library that gives you this MQTT messaging protocol.

Once I started looking, it wasn't the only bug. A bug spray is required!

The deaf device showed me a buch of other bugs and issues with this whole thing.

Reconnection is your problem, not the library's.

paho has reconnect_delay_set, but only loop_forever() honours it. With loop_start() — which is what you want the moment the client isn't the main thread — you're writing the retry loop yourself. I wrote the naive one. Fixed a five-second retry, like everybody's first attempt.

Now, having fixed the reconnection bug, I wanted to try it on a device. It worked. I was happy. Wasn't so hard, I thought. So a while later I proudly rolled it out to ten devices.

It died right away. Ten devices, one outage, and every one of them waiting exactly five seconds before knocking again — in perfect unison, forever. The broker came up, took ten simultaneous connections in the same instant, and went straight back down. I waited. It never resurrected itself. It couldn't: each restart just called the stampede back.

Ten devices were enough to demostrate this bug. So, yes. I fell to a bug from a bug. The fix was exponential backoff with a jitter, so they don't re-synchronise. This is the difference between recovery and a self-inflicted DoS.

#Backoff needs a cap and jitter.
# And without the jitter, ten devices that dropped together retry together. That's what killed my broker.
delay = min(1.0 * 2 ** attempt, 300)   # cap: never wait over 5 min
time.sleep(random.uniform(0, delay))   # jitter: don't come back in unison

The bug came back!

Smash a bug hard enough and it comes back as a phoenix. Karma, probably. My new backoff loop retried forever, which is what I wanted, because a device four hours away has to keep trying. But it had no idea why a connection ended. So when I hit Ctrl-C, my shutdown handler closed the connection, my own retry loop saw a dead connection, and it immediately opened a new one. The process wouldn't exit. I had made it unkillable by making it reliable.

There is a simple fix, though. When paho tells you the connection dropped, it hands you a number. Zero means you closed it yourself. Anything else means it broke. Only reconnect on "anything else". Nothing does this for you.

#The bug: any disconnect triggers a reconnect.
def on_disconnect(client, userdata, rc):
    reconnect()   # including the disconnect you asked for

#The fix is one number.
def on_disconnect(client, userdata, rc):
    if rc == 0:
        return        # you closed it. leave it closed.
    schedule_reconnect()

The first connection is a different path, and it usually isn't retried at all.

All that backoff logic I just wrote lives in on_disconnect — the callback that fires when a connection we already had goes away. But a device that boots before its 4G modem is ready never had one. connect() raises, main() dies. Because the modem is not ready to communicate. And the process exits before any of my retry code ever runs. When systemd restarts the service if the modem still isn't up, it dies again. That's not a retry loop I designed; it's just a process crashing over and over.

And the final boss, which is the worst: messages just vanish.

My five-minute ceiling had a consequence I hadn't thought about. After a bad drop, the device deliberately just sits there doing nothing for up to five minutes. Which means every reading it takes during that window has nowhere to go. publish() doesn't complain about this. It hands back a result code saying the message never left. But that is all. Paho can queue messages for you, but that queue lives in memory, so it dies with the process. A power cycle in the field takes your data with it. Gone.

I tried a few of my own patches over the years for this without ever fixing it properly. Sensor data is the entire reason the device is out there. That's when I stopped patching and started writing a queue that saves messages to disk, so they survive a restart.

All these four different failure modes had one thing in common: all of them fail quietly.

So I made the thing I'd been looking for: The bug spray!

Everything above became robmqtt — a resilient MQTT client for edge devices.

Offline queue in SQLite — unreachable broker means messages get written to disk, not dropped. They survive restarts and power cycles.
Inflight tracking — sent-but-unacknowledged messages are tracked separately and re-sent on reconnect.
Priority eviction — when the queue fills, low-priority telemetry goes before critical alerts.
Exponential backoff — no stampede.
Subscription registry — subscriptions are restored automatically on every reconnect. No on_connect bookkeeping, no deaf devices. It also means you can call subscribe() before connect(), which is how it should have worked in the first place.
Structured logging throughout — so the next person doesn't lose days to a problem that leaves no trace.

The application code never has to know whether the broker is reachable. That was the entire goal.

from robmqtt import ProductionMQTTClient
import json, time

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,
    db_path="./device.db",
)
client.connect()
client.start()

while True:
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(read_sensor()),
        qos=1,
        priority=5,
    )
    time.sleep(30)

That's it. The bug spray!

What I'd tell myself at the start

Silence is not success. A green dashboard measures the things you thought to measure. My device was reporting perfect health while it was completely deaf to commands, because I'd never built anything that could notice the difference.

If a library leaves it to you, it will not warn you. Reconnection, re-subscription, retry and queueing in paho isn't broken for leaving these open. It's a protocol client, not a delivery guarantee. But every gap it leaves is a gap that fails in actual production.

Write the log line you'd want at 2am. Tests and structured logging exist in robmqtt for one reason: I spent days chasing something that left no trace, and I refuse to do that to whoever picks this up next.

The thing that cost me weeks is now one pip install away. That's all I wanted.

pip install robmqtt

PyPI: https://pypi.org/project/robmqtt/
GitHub: https://github.com/ranaweerasupun/resilient-edge-mqtt-client

Built by Supun Sriyananda — R&D Engineer working on embedded and IoT systems.

What a Hypervisor Is, and Where KVM Fits (Type-1 vs Type-2)

Supun Sriyananda — Sat, 15 Aug 2026 11:36:51 +0000

This series builds the KVM/QEMU/libvirt stack starting from the silicon on ARM64. Commands are from a Raspberry Pi 5 (8 GB) running Raspberry Pi OS Bookworm, kernel 6.12 aarch64. Note that, Every command is aarch64-native, and the terminal output is real. The concepts, however, apply anywhere Linux runs. I chose ARM64 deliberately, because almost every KVM tutorial out there is written for x86.

The first question to start with is: what is a hypervisor, and which kind is KVM?

Virtualization works on a lie!

A normal computer runs one operating system, and that OS believes it owns the hardware. It believes all the RAM, every CPU core, the disks, and the network interface belong to it. This belief is not a simplification for beginners. It is baked into how kernels are written. A kernel executes privileged instructions, sets up memory maps, talks directly to devices, and assumes nothing exists above it.

A hypervisor is what lets several of those operating systems, each believing "I own everything", run on one machine. Each of them keeps believing the lie. The hypervisor's job is to keep that lie convincing enough and quietly share the real hardware underneath.

Everything else I will be writing about in this series, like VM exits, VirtIO, virtqueues, and passthrough, is a technique for maintaining that lie either more convincingly or more cheaply.

The types of hypervisors

The textbook taxonomy divides hypervisors in two:

Type 1 hypervisors run directly on the hardware. There is no general-purpose OS underneath them, which means the hypervisor is the thing the machine boots into. ESXi and Xen are the standard examples. What you get in return is performance and isolation. Because, the hypervisor is the sole layer separating a guest from the hardware, and there is nothing else in the way.

Type 2 hypervisors run as applications on top of a normal operating system. We boot Linux, macOS, or Windows on our computers, then install a program like VirtualBox and launch it the same way we'd launch a browser. Here, what you get in return is convenience. Your usual desktop environment carries on exactly as before, and a virtual machine also ends up being just another application running alongside everything else.

People usually summarise the difference as a choice between speed or convenience.

Where is KVM located. And is it type-1 or type-2 ?

KVM is a kernel module. Once it is loaded, the Linux kernel itself becomes the hypervisor. There is no separate hypervisor layer above or below it. The kernel you are already running gains the ability to execute guests directly on the hardware. I meant literally, not as an analogy. There is only one kernel here. It runs the ordinary applications, and it runs virtual machines too.

Now let's try to place that on the Type-1/Type-2 chart.

It has Type-1's defining property: the hypervisor communicates directly to the silicon, without an intermediary OS between it and the hardware. Guest code runs on real cores at native speed. There is no translation layer and no host kernel being asked to relay requests on the guest's behalf. The Type 1 label was invented to describe hypervisors that let guests run at full speed. KVM does exactly that, so by that reasoning, it belongs in the category.

But it is inside a general-purpose operating system that is simultaneously running SSH sessions, your browser, and a few dozen daemons. Your VMs appear in ps output. They're scheduled by the ordinary Linux scheduler, they are killable with kill, and constrained by cgroups like any other process. So, by the structural argument, that's unmistakably Type 2.

This is why the argument never gets settled. People usually call KVM, "Type-1-ish", and that hedge is fair. Because, the two categories were named before anything like KVM existed.

The practical consequence of this categorization

Because the hypervisor is part of a full Linux kernel, it inherits everything that kernel already knows how to do. It doesn't need its own scheduler, because Linux has one, and vCPUs become ordinary threads on it. It doesn't need its own memory manager, driver model, filesystem layer, or a network stack. Because linux has all of them. A dedicated Type-1 hypervisor has to build or port every one of those.

This inheritance is going to shape this whole series. When we pin vCPU threads to physical cores, we'll be using the ordinary Linux scheduler affinity mechanism, because a vCPU is just a thread!. When guest disks turn out to be files in a directory, that's the host filesystem doing its normal job. Later when we bridge guest networking, we'll use the same Linux bridge that would connect any two interfaces.

The more important point is the flip side: KVM is intentionally minimal. Its responsibilities begin and end with the CPU and memory. It does not know how to present itself as a disk, a network card, a display, or a USB port, and it was never designed to. The reason is that it does not need to. Inside the Linux kernel, mature subsystems already exist for storage, networking, and device management, and above the kernel, in userspace, QEMU takes care of whatever remains.

That naturally leads to the question of how the work is actually split between these pieces. Article 1.3 in this series answers it. Before we get there, though, one thing needs to be cleared up, because saying that "the kernel becomes the hypervisor" ignores something important. A kernel cannot simply choose to run guest operating systems at native speed. It requires the processor to cooperate, and that cooperation comes from a hardware capability built into the silicon that stays inactive until software explicitly enables it.

On ARM64 that feature is EL2, and it is where I will start in [the next article].

Summary

A hypervisor lets multiple operating systems each believe they own the hardware, and shares the real hardware underneath.
Type 1 runs directly on hardware (ESXi, Xen). Type 2 runs as an application on a host OS (VirtualBox).
KVM is a kernel module that turns the running Linux kernel into a hypervisor. Guest code executes directly on real cores, but guests are also ordinary Linux processes.
It's Type-1 by the performance argument and Type-2 by the structural one, hence "Type-1-ish". The taxonomy predates the design.
Because the hypervisor is part of Linux, it reuses Linux's scheduler, memory manager, and drivers. Which is why KVM itself only handles CPU and memory.

MQTT vs HTTP for IoT: What I Learned Using Both in Production

Supun Sriyananda — Thu, 30 Jul 2026 14:00:26 +0000

Every IoT project eventually reaches the same decision point:

Should the device communicate with the cloud over MQTT or HTTP?

Both protocols are mature, widely supported, and capable of moving device data reliably. The more useful question is not which protocol is better, but which communication model fits the device.

I have used MQTT for production edge clients running on deployed robot units and warehouse sensors. I have also used HTTP for a building-management monitoring backend and a smart-entry system built with Flask.

Both worked. They simply created very different systems.

This article explains the trade-offs I saw in practice: bandwidth overhead, connection behavior, command latency, power consumption, infrastructure, fan-out, and operational complexity.

The architectural difference that drives everything else

HTTP is built around a request-response model. A client sends a request, and a server returns a response.

MQTT is built around persistent connections and publish-subscribe messaging. A device connects to a broker, publishes messages to topics, and subscribes to topics it wants to receive.

That difference shapes almost every other trade-off.

With HTTP, a device usually addresses a specific API endpoint:

POST /api/readings
GET /devices/robot-01/commands

With MQTT, the device communicates through topics:

warehouse/sensor-01/temperature
robots/robot-01/commands

The publisher does not need to know which services consume a message. The broker handles routing.

Bandwidth and message overhead

The difference becomes easy to see when you compare a small sensor reading.

An HTTP request might look like this:

POST /api/sensors/temperature HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer <token>
Content-Length: 52

{"device_id": "sensor_01", "temperature": 23.5}

The payload is small, but the request also carries headers. Authentication tokens can make those headers much larger than the sensor reading itself.

An MQTT PUBLISH packet carries:

Fixed header
Topic name
Packet identifier, when required by QoS
Properties, in MQTT 5
Payload

For a short topic and a small payload, the MQTT packet can be substantially smaller than an equivalent HTTP request. The MQTT fixed header starts at two bytes, although the complete packet is larger once the topic, payload, QoS fields, and any MQTT 5 properties are included.

The comparison also depends on connection reuse.

HTTP/1.1 supports persistent connections by default, so a well-configured client does not need to open a new TCP and TLS connection for every request. HTTP/2 can also multiplex many requests over one connection. Poorly configured clients, short-lived processes, network timeouts, and reconnects can still cause repeated handshakes, but “one request equals one new connection” is not an accurate general rule.

MQTT still has an advantage for frequent, small messages because its application-layer framing is compact and the connection is normally kept open for ongoing messaging.

For one device publishing every few minutes, the absolute difference may not matter. Across hundreds of devices publishing several times per second, it can become significant.

Receiving commands: polling versus subscriptions

The biggest practical difference I encountered was server-to-device communication.

A basic HTTP API is client-initiated. The device sends a request before the server can return anything. A simple implementation therefore polls for commands:

import time
import requests


def poll_for_commands(device_id: str, token: str, interval_seconds: int = 5) -> None:
    while True:
        response = requests.get(
            f"https://api.example.com/devices/{device_id}/commands",
            headers={"Authorization": f"Bearer {token}"},
            timeout=10,
        )
        response.raise_for_status()

        for command in response.json():
            handle_command(command)

        time.sleep(interval_seconds)

At a five-second interval, that is 17,280 requests per device per day, including requests that return no commands. A newly created command may also wait almost five seconds before the next poll.

HTTP-based systems can avoid simple polling with techniques such as long polling, Server-Sent Events, or WebSockets. Those are valid designs, but they add a different connection-management model to the application.

MQTT supports this pattern directly. The device subscribes once, and the broker forwards matching messages over the existing connection:

import json
import paho.mqtt.client as mqtt

DEVICE_ID = "robot-01"
BROKER_HOST = "mqtt.example.com"


def on_connect(client, userdata, flags, reason_code, properties=None):
    client.subscribe(f"devices/{DEVICE_ID}/commands", qos=1)


def on_message(client, userdata, message):
    command = json.loads(message.payload.decode("utf-8"))
    handle_command(command)


client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=DEVICE_ID)
client.on_connect = on_connect
client.on_message = on_message
client.connect(BROKER_HOST, 1883, keepalive=60)
client.loop_forever()

For the telepresence robot, this mattered. Camera and movement-related commands needed low and predictable latency. A five-second polling loop was not acceptable, while an MQTT subscription matched the communication pattern naturally.

For truly hard real-time control, however, neither MQTT nor a general-purpose cloud connection should be treated as a deterministic control bus. Network delay, broker load, retransmission, and connectivity loss still need to be considered.

Reliability and QoS

MQTT offers three protocol-level Quality of Service levels:

QoS 0: delivered at most once
QoS 1: delivered at least once
QoS 2: delivered exactly once between MQTT protocol peers

QoS 1 is useful when losing a message is worse than processing a duplicate. The application must therefore be able to handle duplicate delivery safely.

QoS 2 reduces duplicate delivery at the MQTT protocol layer, but it adds more packet exchanges and does not automatically make the entire application workflow exactly once. A message can still be processed twice if application state, database writes, retries, or downstream integrations are not idempotent.

HTTP can also be reliable, but the retry and deduplication behavior is usually designed at the application layer. For example, a device can retry a failed POST with an idempotency key, while the API stores the key to avoid inserting the same reading twice.

The important point is that neither protocol removes the need to design failure handling.

Power consumption is workload-dependent

It is tempting to say that MQTT always uses less power because its packets are smaller. The real answer is more nuanced.

Radio state often matters more than payload size. Establishing a cellular or Wi-Fi connection, negotiating TCP and TLS, and waiting for network timers can consume more energy than transmitting a short reading.

A continuously connected MQTT client can avoid repeated connection setup and receive commands immediately. That can be efficient for devices that communicate frequently or must remain reachable.

However, maintaining a live MQTT connection requires periodic keepalive traffic and may prevent some devices or modems from entering their deepest sleep states. MQTT PINGREQ and PINGRESP packets maintain the connection; they do not allow a device to turn its radio fully off while keeping the same TCP connection alive.

For a battery-powered sensor that wakes once an hour, uploads one reading, and never receives commands, a short HTTPS request may be simpler and just as efficient. The device must reconnect either way after deep sleep.

MQTT persistent sessions are still valuable for intermittently connected devices. A broker can preserve subscriptions and queue eligible QoS messages while the client is offline, subject to the broker configuration, protocol version, session settings, and retention limits. When the device reconnects and resumes the session, it can receive the stored messages.

So the useful rule is:

Frequent bidirectional communication often favors MQTT.
Rare uplink-only communication may favor HTTP.
Measure on the actual radio, modem, network, and sleep schedule before making a battery-life claim.

Infrastructure and operational complexity

HTTP is easy to start with because almost every platform already supports it.

A small ingestion API can be only a few lines:

import sqlite3
import time

from flask import Flask, jsonify, request

app = Flask(__name__)


@app.post("/api/readings")
def receive_reading():
    data = request.get_json()

    with sqlite3.connect("readings.db") as connection:
        connection.execute(
            """
            INSERT INTO readings (timestamp, device_id, value)
            VALUES (?, ?, ?)
            """,
            (time.time(), data["device_id"], data["value"]),
        )

    return jsonify({"status": "ok"}), 201

The tooling is familiar: curl, Postman, browser developer tools, reverse proxies, API gateways, serverless functions, status codes, and standard observability platforms.

MQTT requires a broker. Mosquitto is a popular lightweight open-source option, while managed services such as AWS IoT Core and HiveMQ Cloud reduce the operational burden.

Running a broker means thinking about:

Device authentication
TLS certificates
Topic-level authorization
Session persistence
Message expiry and retained messages
Broker storage
Monitoring and capacity
Reconnection storms
High availability

A single broker instance can become a single point of failure, but MQTT itself does not require a single-instance deployment. Production systems can use clustered brokers, load-balanced endpoints, replicated state, and managed broker services.

HTTP is not automatically immune to failure either. A single Flask process is also a single point of failure. HTTP infrastructure is often easier to scale horizontally because stateless APIs and load balancers are so widely supported, but both architectures require deliberate redundancy.

Fan-out is where MQTT feels fundamentally different

MQTT is especially useful when one message has several consumers.

A sensor can publish once:

sensors/warehouse/temp01

Several services can subscribe independently:

Dashboard
Alerting service
Time-series database
Analytics pipeline
Anomaly detector

The sensor does not need to know that these consumers exist. A new consumer can be added without changing device firmware.

With HTTP, the device can still send one request to an ingestion service, but that service must then distribute the event. This is commonly done with a queue, event bus, stream, or webhook system.

That architecture can be excellent. It simply means that HTTP handles device ingestion while another system provides asynchronous fan-out.

Debugging and developer experience

HTTP is usually easier to inspect manually.

You can reproduce a request with curl, view status codes, inspect headers, and test an endpoint without maintaining a long-lived client session.

MQTT debugging is still manageable, but it requires different tools and concepts:

mosquitto_sub -h broker.example.com -t 'sensors/#' -v

mosquitto_pub -h broker.example.com \
  -t 'devices/robot-01/commands' \
  -m '{"action":"stop"}' \
  -q 1

You also need to understand wildcard subscriptions, retained messages, session state, QoS acknowledgements, duplicate delivery, and topic permissions.

In my experience, HTTP produces a faster first prototype. MQTT often produces a cleaner final design once the system becomes continuously connected, bidirectional, or event-driven.

A practical decision framework

Choose HTTP when:

The device integrates with an existing REST API.
Communication is infrequent and mostly device-to-cloud.
The device uploads large payloads, images, logs, or firmware files.
Simple debugging and minimal infrastructure matter most.
Server-to-device messaging is unnecessary or can tolerate polling.

Choose MQTT when:

Devices must receive commands or configuration changes quickly.
Messages are small and frequent.
Network links are unreliable or intermittent.
Multiple services need the same device events.
Session state, topic subscriptions, and QoS are useful.
Bandwidth efficiency matters across a large fleet.

Use both when the workloads differ.

That is the design I now prefer for many IoT systems:

MQTT for telemetry, commands, device status, and real-time events
HTTP for firmware downloads, image uploads, administration, and third-party integrations

The broker handles operational messaging. The API handles resource-oriented workflows and large transfers.

Final takeaway

The decision is not really “MQTT or HTTP?”

It is:

Does this device behave more like an API client, or more like a participant in a live event system?

For an occasional one-way upload, HTTP is often the simplest answer.

For a connected fleet that publishes continuously, receives commands, survives intermittent networks, and feeds several downstream services, MQTT usually fits better.

In production, the strongest architecture is often not choosing one protocol everywhere. It is giving each protocol the job it was designed to do.

Have you used MQTT and HTTP in the same IoT system? I would be interested to hear where you drew the boundary between them.

Python vs C++ for Embedded Systems: When to Use Each

Supun Sriyananda — Fri, 10 Jul 2026 07:03:04 +0000

When you first step into the world of embedded systems, one of the earliest and most consequential decisions you will face is choosing a programming language. Two names come up more than any others: Python and C++. Both are powerful, both have passionate communities, and both are genuinely useful — but for very different reasons and in very different contexts. This article is not about declaring a winner. It is about understanding why each language exists in this space, what trade-offs you are actually making, and how to make a confident, informed decision for your next project.

Understanding the Fundamental Difference

Before comparing features, it helps to understand why these two languages feel so different at a deeper level.

C++ is a compiled, statically-typed, systems-level language. When you write C++, you are writing code that gets translated directly into machine instructions. You manage memory manually. You control exactly when objects are created and destroyed. The hardware does precisely what you tell it to, nothing more and nothing less. This directness is both its superpower and its source of complexity.

Python, by contrast, is an interpreted, dynamically-typed, high-level language. A Python runtime sits between your code and the hardware, managing memory automatically through garbage collection, resolving types at runtime, and handling a lot of bookkeeping so you don't have to. This makes Python wonderfully expressive and fast to write, but it introduces overhead that matters enormously on constrained hardware.

The mental model to hold onto is this: C++ gives you control, Python gives you speed of development. Both are valuable. The question is which one your project needs more.

Where C++ Shines in Embedded Systems

1. Bare-Metal and Resource-Constrained Environments

If you are programming a microcontroller like an STM32, an AVR ATmega, or an ESP32 running its native SDK, C++ is almost always your primary language. These devices often have kilobytes — not megabytes — of RAM, and they have no operating system to fall back on. Python's runtime alone would consume more memory than the entire chip provides.

// C++ on a bare-metal microcontroller (e.g., STM32 HAL)
// Direct register manipulation gives you precise, deterministic control
// over hardware peripherals with zero overhead

#include "stm32f4xx_hal.h"

// A simple GPIO toggle to blink an LED — every cycle counts here
void blink_led(GPIO_TypeDef* port, uint16_t pin, uint32_t delay_ms) {
    while (true) {
        HAL_GPIO_TogglePin(port, pin);   // Direct hardware write
        HAL_Delay(delay_ms);              // Blocking delay — acceptable on bare metal
    }
}

The key insight here is determinism. In safety-critical or real-time systems — think motor controllers, medical devices, or industrial sensors — you need to guarantee that a particular piece of code executes within a precise time window. Python's garbage collector can pause your program at unpredictable moments. C++ has no such hidden pauses.

2. Real-Time Operating Systems (RTOS)

When you step up to an RTOS like FreeRTOS, Zephyr, or ThreadX, C++ remains the dominant choice. RTOS kernels are themselves written in C/C++, and the APIs they expose are designed for direct use from those languages. You get preemptive multitasking, precise interrupt handling, and microsecond-level timing — all with the kind of control that C++ makes natural.

// FreeRTOS task written in C++ — structured, type-safe, and efficient
// Each task runs as an independent thread with its own stack allocation

#include "FreeRTOS.h"
#include "task.h"

class SensorTask {
public:
    // Static wrapper needed because FreeRTOS expects a plain C function pointer
    static void taskEntry(void* param) {
        // Cast back to our class instance — a common C++/RTOS pattern
        static_cast<SensorTask*>(param)->run();
    }

    void start() {
        // Stack size (512 words) and priority are explicit — you own these decisions
        xTaskCreate(taskEntry, "Sensor", 512, this, tskIDLE_PRIORITY + 1, &handle_);
    }

private:
    TaskHandle_t handle_;

    void run() {
        while (true) {
            readSensor();
            vTaskDelay(pdMS_TO_TICKS(100)); // Yield for exactly 100ms
        }
    }

    void readSensor() {
        // Hardware-specific sensor reading logic lives here
    }
};

3. Performance-Critical Signal Processing

Digital signal processing (DSP), motor control algorithms, PID loops, and image processing pipelines all benefit enormously from C++. The language lets you write tightly optimized loops, use SIMD intrinsics, and avoid any dynamic allocation in hot paths. When you need to process a sensor stream at 10 kHz with a 50-microsecond deadline, Python is not a practical option.

4. Production Firmware

If you are shipping a product — a consumer device, an industrial controller, a wearable — your firmware will almost certainly be C or C++. It compiles to a fixed binary, has predictable resource usage, can be certified against safety standards, and does not require a runtime environment on the target device.

Where Python Shines in Embedded Systems

1. Rapid Prototyping on Linux-Based Boards

Single-board computers like the Raspberry Pi, BeagleBone, or NVIDIA Jetson run full Linux, which means they can run a complete Python interpreter. On these platforms, Python becomes extraordinarily powerful for prototyping. You can wire up a sensor, write twenty lines of Python, and have a working proof-of-concept in under an hour.

# Python on Raspberry Pi using the RPi.GPIO library
# This simplicity is Python's greatest asset for hardware prototyping

import RPi.GPIO as GPIO
import time

LED_PIN = 18  # BCM numbering

GPIO.setmode(GPIO.BCM)
GPIO.setup(LED_PIN, GPIO.OUT)

# A PWM-based fade effect — would take far more boilerplate in C++
pwm = GPIO.PWM(LED_PIN, 100)  # 100 Hz frequency
pwm.start(0)

try:
    while True:
        # Gradually increase and decrease brightness
        for duty_cycle in range(0, 101, 5):
            pwm.ChangeDutyCycle(duty_cycle)
            time.sleep(0.05)
        for duty_cycle in range(100, -1, -5):
            pwm.ChangeDutyCycle(duty_cycle)
            time.sleep(0.05)
finally:
    pwm.stop()
    GPIO.cleanup()  # Always clean up GPIO state on exit

2. MicroPython and CircuitPython on Microcontrollers

This is where the story gets more nuanced. MicroPython and Adafruit's CircuitPython are lean Python implementations designed specifically to run on microcontrollers like the RP2040, ESP32, and STM32. They bring Python's expressiveness to hardware with just a few hundred kilobytes of flash.

These environments are excellent for education, hobbyist projects, and rapid iteration. The trade-off is that you are running an interpreter with managed memory, so real-time guarantees are off the table and performance is a fraction of native C++.

# MicroPython on an RP2040 (Raspberry Pi Pico)
# The simplicity here is remarkable — I2C in just a few lines

from machine import I2C, Pin
import time

# Initialize I2C bus on pins 4 (SDA) and 5 (SCL)
i2c = I2C(0, sda=Pin(4), scl=Pin(5), freq=400_000)

# Scan for connected devices — invaluable during prototyping
devices = i2c.scan()
print(f"Found I2C devices at addresses: {[hex(d) for d in devices]}")

# Read 6 bytes from a hypothetical sensor at address 0x68
if 0x68 in devices:
    data = i2c.readfrom(0x68, 6)
    # Parse raw bytes — still manageable in Python
    accel_x = int.from_bytes(data[0:2], 'big')
    print(f"Accelerometer X: {accel_x}")

3. Data Pipelines, ML Inference, and High-Level Logic

On Linux-based edge devices, Python's ecosystem is unmatched. Libraries like NumPy, OpenCV, TensorFlow Lite, and PySerial let you build sophisticated data pipelines, run machine learning inference, and communicate with sensors — all without writing a single line of C++. This is where Python earns its place even in serious embedded projects.

4. Testing, Tooling, and Automation

Even on projects where the final firmware is C++, Python plays a huge role in the surrounding ecosystem. Test frameworks, hardware-in-the-loop test scripts, build automation, log parsers, and configuration generators are all commonly written in Python. It is the glue language of the embedded world.

A Direct Comparison Across Key Dimensions

Performance

C++ wins decisively here. A well-written C++ program will typically run 10x to 100x faster than equivalent Python, and in tight real-time loops the gap is even larger. On microcontrollers without a Python runtime, C++ is often the only option.

Development Speed

Python wins just as decisively. The combination of dynamic typing, interactive REPL, extensive libraries, and concise syntax means you can explore ideas, test hypotheses, and build prototypes dramatically faster in Python than in C++.

Memory Usage

C++ gives you granular control. You can preallocate fixed-size buffers, use stack memory exclusively in critical sections, and tune every byte of usage. Python's runtime itself consumes significant memory before your application even starts — typically several megabytes for CPython, several hundred kilobytes for MicroPython.

Ecosystem and Libraries

For general-purpose computing tasks (networking, data science, ML, web APIs), Python's ecosystem is simply enormous. For low-level hardware interaction, peripheral drivers, and embedded frameworks, the C/C++ ecosystem has decades of depth.

Debugging and Observability

Python is easier to debug interactively. You can print, inspect, and modify state at runtime with minimal friction. C++ debugging on embedded targets requires more tooling — JTAG/SWD debuggers, GDB, logic analyzers — but that tooling is very mature and gives you deep visibility into the hardware.

Safety and Reliability

C++ makes it possible (with discipline) to write code with no dynamic allocation, bounded execution time, and no hidden runtime behavior. This is essential for safety-critical systems. Python's garbage collector and dynamic nature make such guarantees essentially impossible.

The Hybrid Approach: Using Both

In practice, many serious embedded projects use both languages in complementary roles. A common architecture looks like this:

The C++ layer handles all time-critical operations: sensor interrupt handlers, motor control loops, communication protocol drivers, and safety watchdogs. This layer is compiled to a binary that runs deterministically on the target hardware.

The Python layer sits above it, handling high-level orchestration: reading data from the C++ layer over a serial or USB interface, applying machine learning models, sending data to cloud services, and providing a configuration interface. On a Raspberry Pi, this layer might coordinate with a microcontroller over UART while simultaneously pushing data to an MQTT broker.

# Python orchestration layer on a Raspberry Pi
# Talking to a C++ firmware over serial — a very common real-world pattern

import serial
import struct
import json

class FirmwareBridge:
    """
    High-level Python wrapper around low-level C++ firmware.
    The firmware handles real-time control; we handle logic and connectivity.
    """
    def __init__(self, port: str, baud: int = 115200):
        self.serial = serial.Serial(port, baud, timeout=1.0)

    def read_sensor_packet(self) -> dict | None:
        """
        Read a binary packet from firmware.
        The C++ side sends fixed-width structs for efficiency.
        """
        header = self.serial.read(1)
        if not header or header[0] != 0xAA:  # Sync byte
            return None

        # Unpack a 3-float struct: temperature, humidity, pressure
        raw = self.serial.read(12)
        if len(raw) < 12:
            return None

        temp, humidity, pressure = struct.unpack('<fff', raw)
        return {"temperature": temp, "humidity": humidity, "pressure": pressure}

    def send_command(self, command_id: int, payload: bytes = b'') -> None:
        """Send a control command down to the C++ firmware."""
        packet = bytes([0xBB, command_id, len(payload)]) + payload
        self.serial.write(packet)

This hybrid model lets you play to each language's strengths: C++ for the parts where performance and determinism are non-negotiable, Python for the parts where developer velocity and ecosystem richness matter most.

Decision Framework: How to Choose

Rather than a rigid rule, think of it as a set of questions to work through for any given project or module.

Ask: Does this code have hard real-time requirements? If yes, use C++. Real-time means a missed deadline is a failure, not just a slowdown.

Ask: Does this run on a microcontroller without an OS? If yes, use C++. Python runtimes need significantly more resources than bare-metal MCUs typically provide.

Ask: Am I building a proof-of-concept or exploring hardware behavior? If yes, start with Python (on a suitable platform). Validate your idea fast before optimizing.

Ask: Does this code interact with complex libraries — ML, computer vision, networking? If yes, Python is likely more practical. The C++ equivalents exist but are far harder to integrate.

Ask: Will this code ship in a product with safety or reliability requirements? If yes, lean heavily toward C++, potentially with a formal coding standard like MISRA C++.

Ask: Is this part of a larger system's tooling, testing, or configuration? If yes, Python is almost certainly the right choice regardless of what the firmware itself uses.

Conclusion

The Python vs C++ question in embedded systems is not a rivalry — it is a spectrum. C++ is the language of determinism, performance, and direct hardware control. Python is the language of expressiveness, rapid iteration, and ecosystem richness. Understanding why each language works the way it does is far more valuable than memorizing a comparison table.

The most effective embedded engineers are fluent in both. They reach for C++ when they need to talk directly to hardware with precision and speed, and they reach for Python when they need to move fast, analyze data, or connect their hardware to the broader software world. Knowing which tool to pick up, and why, is the skill worth developing.

Implementing Exponential Backoff: Preventing Thundering Herd Problems

Supun Sriyananda — Wed, 08 Jul 2026 08:47:10 +0000

When a broker goes down and a thousand edge devices all try to reconnect every second, the moment the broker comes back online it gets hit with a thousand simultaneous connection requests and dies again. This is the thundering herd problem, and it is entirely self-inflicted. The fix is exponential backoff with jitter — a reconnection strategy every networked embedded application should implement from day one.

🔗 See project on GitHub: Resilient Edge MQTT Client

The Wrong Way

# Every device retries every second. 1000 devices = 1000 req/s on reconnect.
while True:
    try:
        client.connect(broker_host, broker_port)
        break
    except Exception:
        time.sleep(1)  # Fixed delay — all devices retry in lockstep

Fixed-delay retry is the problem. Every device that disconnected at the same time will retry at exactly the same time, forever, until one of them happens to succeed. At scale, this makes outages self-perpetuating.

The Right Way

import time
import random

def connect_with_backoff(client, broker_host, broker_port,
                         min_delay=1, max_delay=300):
    """
    Reconnect with exponential backoff and jitter.

    Delay sequence (approximate): 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s...
    Jitter adds ±10% randomness so devices that disconnected together
    do not retry together.
    """
    delay = min_delay

    while True:
        try:
            # Jitter is proportional to the current delay, not fixed.
            # At delay=1 it adds up to 0.1s. At delay=60 it adds up to 6s.
            # This keeps devices spread out even after long outages.
            jitter = random.uniform(0, delay * 0.1)
            time.sleep(delay + jitter)

            client.connect(broker_host, broker_port)
            return  # Success — caller resets delay to min_delay

        except Exception as e:
            print(f"Connection failed: {e}. Retrying in {delay:.1f}s")
            delay = min(delay * 2, max_delay)

The three decisions baked into this function are worth understanding individually.

Doubling the delay (the exponential part) means the load on the broker drops by half with each retry cycle. After a few rounds, devices are spread so far apart in time that the broker recovers comfortably before the next wave arrives.

Capping at max_delay prevents devices from backing off indefinitely. A cap of 60–300 seconds is usually right for embedded systems — long enough to give the broker real recovery time, short enough that you do not lose hours of data during a brief outage.

Proportional jitter is the part most implementations get wrong. Adding a small fixed jitter like random.uniform(0, 0.5) works when delay is 1 second but is meaningless when delay is 60 seconds — all devices are still retrying within the same half-second window. Making jitter proportional to the current delay (10% here) keeps devices spread out at every stage of backoff.

Three Ways to Add Jitter

The 10% proportional jitter above is deliberately conservative — it keeps retries close to the intended backoff curve while still desynchronizing devices. But it's not the only approach, and for larger fleets more aggressive strategies spread load better. These are the three established patterns, all built on the same base calculation:

temp = min(cap, base * (2 ** attempt))  # uncapped exponential value

Full Jitter picks the delay uniformly between zero and the full exponential value. This maximizes spread but sacrifices any guaranteed minimum wait — some devices will retry almost immediately.

sleep = random.uniform(0, temp)

Equal Jitter keeps half the delay fixed and randomizes the other half. This guarantees a minimum wait while still breaking synchronization — a good middle ground when you don't want devices hammering back instantly.

sleep = temp / 2 + random.uniform(0, temp / 2)

Decorrelated Jitter bases each delay on the previous sleep time rather than the attempt count, letting the backoff wander upward more smoothly. AWS's analysis found this among the most effective at minimizing total work against a recovering service.

sleep = min(cap, random.uniform(base, prev_sleep * 3))

For a small edge fleet, proportional or equal jitter is plenty. For thousands of clients hitting a shared API, full or decorrelated jitter spreads the recovery load more evenly.

Reset the Delay on Success

This is the step that gets forgotten. After a successful connection, the delay must reset to min_delay. If it stays at whatever value it reached during the outage, the next disconnection — even a brief one — will start with a 60-second wait instead of a 1-second wait.

class ResilientMQTTClient:
    def __init__(self, broker_host, broker_port):
        self.broker_host = broker_host
        self.broker_port = broker_port
        self.min_delay = 1
        self.max_delay = 300
        self.current_delay = self.min_delay  # Reset this on every successful connect

    def on_connect(self, client, userdata, flags, rc):
        if rc == 0:
            self.current_delay = self.min_delay  # Always reset here
            print("Connected — backoff delay reset to minimum")

    def reconnect(self):
        jitter = random.uniform(0, self.current_delay * 0.1)
        time.sleep(self.current_delay + jitter)

        try:
            self.client.reconnect()
        except Exception:
            self.current_delay = min(self.current_delay * 2, self.max_delay)

With Paho MQTT, the right place to reset is inside the on_connect callback, which fires on the background network thread whenever a connection is established. Resetting in the reconnection loop itself is too early — the connection might still fail after the reset.

Add a Restart Rate Limit at the System Level

Backoff handles network-level reconnection. But if your application itself is crashing and restarting — not just reconnecting — you need a second line of defence. systemd's StartLimitBurst and StartLimitIntervalSec directives cap how many times a service can restart within a given window before systemd stops trying and marks it failed.

[Service]
Restart=on-failure
RestartSec=5                  # Base delay before first restart
StartLimitIntervalSec=120     # Window for counting restarts
StartLimitBurst=5             # Give up after 5 restarts in 120 seconds

This prevents a crashing service from hammering a broker with rapid reconnections that bypass your application-level backoff entirely.

Don't Retry Blindly

Backoff controls when you retry. It says nothing about whether you should — and retrying the wrong things causes worse problems than the outage itself.

Not every error is retryable. Retry only on transient failures: connection timeouts and 5xx responses like 500 and 503, which mean the service is temporarily unwell and may recover. Do not retry client errors like 401 or 404 — a request that's unauthorized or points at something that doesn't exist will fail identically no matter how many times you send it. Retrying it just wastes cycles and pollutes your logs.

Non-idempotent operations need protection. If a retry can duplicate a side effect — charging a card, creating a record, sending a message — then a retry that succeeds after the original silently went through will corrupt your data. The standard fix is an idempotency key: the client attaches a unique token to the request, and the server uses it to recognize and deduplicate a retry of an operation it already completed. Reconnecting to a broker is naturally idempotent; posting a payment is not.

Always set an attempt limit. The max_delay cap stops any single wait from growing unbounded, but a client can still retry a capped delay forever. For request/response operations, give up after a fixed number of attempts (typically 5–7) and surface the error to the caller rather than retrying into the void. Persistent connections like MQTT are the exception — there, retrying indefinitely against max_delay is usually what you want.

Quick Reference

Enable exponential backoff on every service that connects to an external broker, API, or database — not just MQTT. Start with min_delay=1, max_delay=60 for most edge applications. Use proportional jitter (10% of current delay) for small fleets, or full/decorrelated jitter when thousands of clients share a service — never fixed jitter. Always reset the delay counter inside the successful-connection callback. Retry only transient errors (timeouts, 5xx), never client errors (4xx). Protect non-idempotent operations with an idempotency key, and cap request/response retries at 5–7 attempts. Pair application-level backoff with systemd restart rate limiting for complete coverage.

If you found this useful, drop a ❤️ or a comment — I'd love to hear how you handle reconnection in your own systems.

Why SQLite is Perfect for Edge Device Data Logging

Supun Sriyananda — Thu, 02 Jul 2026 08:57:42 +0000

Most embedded Linux projects do not need a database server. They need something that stores data reliably when the network is down, survives a power cut, does not eat RAM, and requires zero administration. SQLite does all of that. It is a single .db file on disk, it has been in production use for over twenty years, and it is already installed on virtually every Linux system you will ever deploy to. If you are currently writing sensor data to a flat file or a custom binary format, this article will show you why SQLite is almost always the better choice — and exactly how to use it correctly.

See the project on GitHub: Resilient Edge MQTT client

Enable WAL Mode Before Anything Else

The single most impactful thing you can do is switch from the default journal mode to Write-Ahead Logging. In the default mode, SQLite locks the entire database file during a write, which means a reader has to wait. WAL mode allows reads and writes to happen simultaneously, which matters enormously when you have one thread logging sensor data and another thread reading it for transmission. It also makes crash recovery safer — incomplete writes never corrupt the database.

import sqlite3

conn = sqlite3.connect('sensor_data.db')

# Do this immediately after opening — before any other operations
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")  # Safe with WAL, much faster than FULL

conn.commit()

synchronous=NORMAL is safe to use in combination with WAL mode. The default FULL mode flushes to disk on every transaction, which is slow on SD cards and eMMC storage. NORMAL flushes at critical checkpoints instead, giving you durability against crashes while being significantly faster on the kinds of storage you find in embedded devices.

Schema Design: Keep It Flat, Add the Right Indexes

Resist the urge to normalise aggressively. On an edge device you are almost always inserting one kind of data repeatedly and querying it by time range. A flat table with a proper index on the timestamp column handles both operations efficiently and keeps queries simple.

conn.execute("""
    CREATE TABLE IF NOT EXISTS sensor_readings (
        id        INTEGER PRIMARY KEY AUTOINCREMENT,
        timestamp REAL    NOT NULL,  -- Unix epoch, fractional seconds
        topic     TEXT    NOT NULL,  -- e.g. 'sensors/warehouse/temperature'
        payload   TEXT    NOT NULL,  -- JSON string
        synced    INTEGER NOT NULL DEFAULT 0  -- 0 = pending upload, 1 = done
    )
""")

# This index makes time-range queries fast and powers the sync queue
conn.execute("""
    CREATE INDEX IF NOT EXISTS idx_timestamp_synced
    ON sensor_readings(timestamp, synced)
""")

The synced column is a pattern worth adopting from the start. Edge devices often need to buffer data locally and then upload it in batches when connectivity returns. Tracking sync state in the same table keeps everything in one place and makes the query for "give me everything not yet sent" trivially simple:

SELECT * FROM sensor_readings WHERE synced = 0 ORDER BY timestamp ASC LIMIT 100

Always Use Batch Inserts

Never insert one row per commit in a logging loop. Each commit flushes a transaction to disk, and on embedded storage that is slow enough to miss readings at any meaningful sample rate. Batch your inserts inside a single transaction — the difference in throughput is dramatic.

import time
import json
from contextlib import contextmanager

@contextmanager
def batch_insert(conn, batch_size=100):
    """Context manager that accumulates rows and commits in one transaction."""
    rows = []

    def add(topic, payload):
        rows.append((time.time(), topic, json.dumps(payload)))
        if len(rows) >= batch_size:
            _flush(conn, rows)
            rows.clear()

    yield add

    if rows:  # Flush any remaining rows on exit
        _flush(conn, rows)

def _flush(conn, rows):
    conn.executemany(
        "INSERT INTO sensor_readings (timestamp, topic, payload) VALUES (?, ?, ?)",
        rows
    )
    conn.commit()


# Usage in a sensor loop
with batch_insert(conn, batch_size=50) as log:
    for reading in sensor_stream():
        log('sensors/warehouse/temperature', {
            'value': reading.temperature,
            'unit': 'celsius'
        })

executemany with a list of tuples is the correct tool here. It prepares the statement once and executes it repeatedly within a single transaction, which is both faster and safer than building SQL strings manually. Never concatenate user data or variable values into SQL strings directly — always use parameterised queries with ? placeholders.

Thread Safety: One Connection Per Thread

If your application is multi-threaded — and most real embedded applications are — do not share a single SQLite connection across threads. SQLite connections are not thread-safe by default, and sharing one between a logging thread and a sync thread will give you intermittent errors that are hard to reproduce. The cleanest approach is to use a connection pool or simply open a dedicated connection per thread.

import threading

# Thread-local storage: each thread gets its own connection automatically
_local = threading.local()

def get_connection(db_path: str) -> sqlite3.Connection:
    """Return a per-thread SQLite connection, creating it if needed."""
    if not hasattr(_local, 'conn') or _local.conn is None:
        _local.conn = sqlite3.connect(db_path, check_same_thread=False)
        _local.conn.row_factory = sqlite3.Row  # Access columns by name
        _local.conn.execute("PRAGMA journal_mode=WAL")
        _local.conn.execute("PRAGMA synchronous=NORMAL")
    return _local.conn

row_factory = sqlite3.Row is small but worth enabling. It makes query results accessible by column name (row['timestamp']) rather than index (row[0]), which makes your code substantially easier to read and maintain.

Manage Storage: Prevent the Database from Growing Forever

On a device with a 16 GB SD card, an unbounded database will eventually fill the storage and crash your application. Build a retention policy in from the start. The simplest approach is a periodic cleanup that deletes synced rows older than your retention window.

import time

def cleanup_old_records(conn, retain_days=7):
    """
    Delete synced records older than retain_days.
    Only deletes synced rows — unsynced data is preserved regardless of age,
    because it hasn't been uploaded yet and may still be needed.
    """
    cutoff = time.time() - (retain_days * 86400)

    cursor = conn.execute(
        "DELETE FROM sensor_readings WHERE synced = 1 AND timestamp < ?",
        (cutoff,)
    )
    conn.commit()

    deleted = cursor.rowcount
    if deleted > 0:
        # VACUUM reclaims the disk space freed by deletions
        # Run this infrequently — it rewrites the entire database file
        conn.execute("VACUUM")

    return deleted

Run cleanup_old_records once per day from a background thread or a systemd timer. The VACUUM command rewrites the database to reclaim the disk space freed by deletions — SQLite does not shrink the file automatically. VACUUM is expensive, so do not call it every cleanup cycle; once a day or once a week is usually fine depending on your write volume.

The Complete Pattern

Here is what the full setup looks like when these practices are combined:

import sqlite3
import threading
import time
import json

DB_PATH = '/var/lib/myapp/sensor_data.db'
_local = threading.local()

def get_db() -> sqlite3.Connection:
    if not hasattr(_local, 'conn') or _local.conn is None:
        conn = sqlite3.connect(DB_PATH, check_same_thread=False)
        conn.row_factory = sqlite3.Row
        conn.execute("PRAGMA journal_mode=WAL")
        conn.execute("PRAGMA synchronous=NORMAL")
        _local.conn = conn
    return _local.conn

def init_schema():
    conn = get_db()
    conn.execute("""
        CREATE TABLE IF NOT EXISTS sensor_readings (
            id        INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp REAL    NOT NULL,
            topic     TEXT    NOT NULL,
            payload   TEXT    NOT NULL,
            synced    INTEGER NOT NULL DEFAULT 0
        )
    """)
    conn.execute("""
        CREATE INDEX IF NOT EXISTS idx_timestamp_synced
        ON sensor_readings(timestamp, synced)
    """)
    conn.commit()

def log_reading(topic: str, payload: dict):
    conn = get_db()
    conn.execute(
        "INSERT INTO sensor_readings (timestamp, topic, payload) VALUES (?, ?, ?)",
        (time.time(), topic, json.dumps(payload))
    )
    conn.commit()

def get_pending(limit: int = 100) -> list:
    """Return unsynced readings, oldest first."""
    conn = get_db()
    return conn.execute(
        "SELECT * FROM sensor_readings WHERE synced = 0 ORDER BY timestamp ASC LIMIT ?",
        (limit,)
    ).fetchall()

def mark_synced(row_ids: list[int]):
    """Mark a batch of rows as successfully uploaded."""
    conn = get_db()
    conn.executemany(
        "UPDATE sensor_readings SET synced = 1 WHERE id = ?",
        [(rid,) for rid in row_ids]
    )
    conn.commit()

Quick Reference

Do these from the start. Enable WAL mode and synchronous=NORMAL immediately after opening each connection. Use parameterised queries — never string concatenation. Add a timestamp index and a synced column if you are buffering for later upload. Set a retention policy before your first deployment.

Avoid these. Do not commit per row in a high-frequency logging loop — batch your inserts. Do not share a connection across threads. Do not call VACUUM on every cleanup. Do not use the default journal mode if you have concurrent readers and writers.

Use when you need it. executemany for bulk inserts. row_factory = sqlite3.Row for readable query results. VACUUM once a day or week to reclaim space. PRAGMA integrity_check after a power loss if you want to verify the database is undamaged.

If you found this useful, drop a comment or a ❤️ — and let me know what you'd like to see covered next.

DuckDB vs SQLite: Which one is better?

Supun Sriyananda — Tue, 09 Jun 2026 09:18:52 +0000

I spend most of my time somewhere between microcontrollers and dashboards. One week I'm squeezing firmware onto a device with barely any memory to spare, the next I'm staring at a few million sensor readings trying to work out why a gateway in the field keeps misbehaving. So when people line up "DuckDB vs SQLite" as a fight to the death, I always want to gently jump in.

They're both small. They both run inside your own program instead of off on some server. They both let you work with a single file using SQL. On paper that makes them sound like rivals. But the more I've used them — across embedded work, edge devices, and plain old data crunching — the more they feel like two neighbours who happen to do completely different jobs.

Let me walk through what I mean, starting from the very beginning.

First, what "embedded" actually means here

When people hear the word "database," they often picture something big and separate: a program running on its own server somewhere that your app has to connect to over the network, with a username and password, before it can read or write anything. PostgreSQL and MySQL work like that. There's nothing wrong with it — it's how most large web apps run — but it's a lot of moving parts.

SQLite and DuckDB throw all of that out. There's no separate program to start. Nothing to log into. No server quietly running in the background. The whole database is just an ordinary file sitting on your disk, and the database engine is a small library your code loads in directly. You point it at the file, you run your queries, and that's the whole setup. Nothing else to install or babysit.

That shared simplicity is the lovely part, and it's exactly why you find these two in places a big server-based database could never go — phones, web browsers, tiny sensors, a quick script on your laptop. But the moment you look at how each one stores your data and works through it, they head off in opposite directions. That difference is the whole story, so let's go there next.

SQLite: the one that's already everywhere

SQLite stores data in rows. Picture a spreadsheet where each row is one complete record, and all the values for that record sit together: the id, the name, the temperature reading, all in one place. That layout is perfect when you're constantly poking at individual records — add this new reading, update that setting, grab the latest entry for device 42. You want the whole record at once, and SQLite hands it to you fast.

A few things I love about it:

It's tiny. The whole engine is around a megabyte. On a small device, that matters enormously.
It's everywhere, and I mean everywhere. There are tens of billions of SQLite files in active use — it's running on basically every phone, and your web browser uses it right now to store your history and settings. It has earned the right to be boring.
It's astonishingly reliable. It's one of the most thoroughly tested pieces of software on the planet. The US Library of Congress even recommends SQLite as a format for preserving digital files long-term, because they trust it'll still open decades from now. When you're shipping a device that has to run untouched in a cabinet for years, that track record buys you a lot of peace of mind.

In my embedded work, SQLite is the default for anything that holds state — the current situation the device needs to remember. Its settings. A queue of readings waiting to be uploaded. A small log of recent events. It handles a steady stream of small writes gracefully and it doesn't ask for resources the hardware doesn't have.

Where it starts to struggle is heavy number-crunching across huge piles of data. Ask SQLite to add up and average ten million rows and it'll get there — but slowly, because it reads through the data row by row, dragging along every column even when your question only touches one of them. That's not a flaw. It simply wasn't built for that job.

DuckDB: the one that makes big analysis feel easy

DuckDB flips the storage layout around. Instead of keeping each row together, it keeps each column together — all the temperatures in one place, all the device names in another. So when you ask "what's the average temperature across ten million readings," it reads just the temperature column and skips everything else. On top of that it processes data in big batches rather than one row at a time, which modern processors are very good at. The result is that the heavy questions that make SQLite sweat come back from DuckDB before you've finished a sip of coffee.

There's another part that genuinely changed how I work, and it has nothing to do with speed. DuckDB will read your data files directly, right where they sit:

SELECT device_id, avg(temperature)
FROM 'readings/*.parquet'
GROUP BY device_id;

No loading step. No importing anything into a table first. You point it at a folder of files — CSV, JSON, or Parquet (a compact file format that's common for this kind of data) — and it just reads them. For someone who regularly gets handed a pile of sensor dumps, that's the difference between "give me an afternoon to set up a pipeline" and "give me thirty seconds."

It also handles data bigger than your computer's memory by spilling the overflow to disk, so a dataset that would crash a normal in-memory tool just... works.

It's newer than SQLite and the engine is a bit chunkier, but it's still the same idea at heart: a file and a small library, nothing to run separately.

Are these actually used in the real world?

Both, heavily — this isn't a case of betting on something obscure.

SQLite is the most widely deployed database engine that exists, full stop. The tens of billions of copies in daily use make it more common than every other database combined. It's not going anywhere.

DuckDB is younger but its adoption has shot up. It's pulling in around 37 million downloads a month on Python's package index, it's MIT licensed and free, and it has real commercial backing behind it (a company called MotherDuck builds a cloud service on top while keeping the core engine open and free), which answers the usual worry about whether an open-source tool will still be maintained in five years. It also fits neatly with where the industry is heading: open file formats, modern multi-core processors, and even AI coding assistants, which tend to be good at writing SQL and so reach for DuckDB naturally.

And SQLite isn't standing still either. Newer spin-offs like Turso/libSQL are adding things like replication and edge support on top of the classic engine. Both of these tools are safe bets.

So when does each one actually win?

Here's how it shakes out across the three places I work.

On small, constrained hardware, SQLite, almost every time. DuckDB's appetite for memory and processing power during a big query is more than a tiny chip wants to give. On a larger edge device running Linux it's a different story, but down at the small end, SQLite's tiny size and long history are hard to argue with.

At the edge — meaning the gateway boxes that sit between your sensors and the cloud — they actually team up. A pattern I keep coming back to: let SQLite handle the incoming readings on the device, quietly buffering them as they arrive, then let DuckDB do the local number-crunching before anything gets sent upstream. You ship neat summaries instead of the raw firehose, which is kinder to both your bandwidth bill and your cloud costs. They're not competing here. They're a relay team.

For sitting down and analysing data, DuckDB is the one I reach for. Being able to throw SQL at a folder of files without setting up any heavy machinery is exactly the kind of low-fuss tool the job usually calls for. It has quietly become a go-to for local analysis, and for good reason.

The one line I keep in my head

If I had to shrink all of this down to a fridge magnet:

SQLite is for managing data while it's being created and changed. DuckDB is for analysing it once it's all piled up.

Same small, no-server spirit — opposite ends of the data's life. SQLite looks after the data as it's coming in and changing. DuckDB shows up later to make sense of the whole pile. Once that clicked for me, the "versus" framing kind of fell apart.

So if you're choosing between them, the honest answer is often both, for different things. Work out whether the job in front of you is about handling data as it changes or making sense of a big pile of it, and the choice mostly makes itself.

If you've wired these two together in your own projects, I'd love to hear how you split the work. I'm always tinkering with my own setup, and the edge folks always seem to have the most interesting war stories.

Deploying Production Systems on Raspberry Pi: Lessons from the Field

Supun Sriyananda — Sun, 07 Jun 2026 05:38:31 +0000

Deploying Production Systems on Raspberry Pi: Lessons from the Field

These are the things I wish I had known before deploying Pis in production.

SD Cards Will Kill You

The first production Pi I deployed used a generic microSD card. It failed after four months. The second one used a "name brand" card. It failed after six months. The pattern remained always the same: the filesystem corrupts during a power loss, the Pi boots into read-only mode, and whatever the system was supposed to be doing silently stops working.

SD card corruption under power loss is not a bug you can fix in software. It is a fundamental characteristic of flash storage that was designed for cameras, not servers. The cells wear out, write operations are not atomic, and a sudden power cut mid-write leaves the filesystem in a state that fsck (File System Consistency Check) sometimes cannot recover.

If you choose SD cards: Switch to a Pi-rated industrial SD card (SanDisk MAX Endurance, Samsung Pro Endurance) or eliminate the SD card entirely by booting from a USB SSD. USB boot on Pi 4 and Pi 5 is stable and the endurance difference is enormous — a decent SSD handles orders of magnitude more write cycles than any SD card.

For systems that must use SD, mount the filesystem read-only and put all writable state on a tmpfs or a separate partition with journaling.

tmpfs is a special type of temporary file storage facility in Linux and Unix-like systems that stores files directly in volatile memory (RAM) instead of on a persistent drive like an SD card or SSD.

When you mount a folder as tmpfs, any files written to that folder behave like regular files, but they consume RAM and exist purely in memory.

# /etc/fstab — mount root read-only, put logs and data elsewhere
/dev/mmcblk0p2  /        ext4  ro,defaults  0  1
tmpfs           /tmp     tmpfs defaults      0  0
tmpfs           /var/log tmpfs defaults      0  0

# Separate partition for application data with journaling
/dev/mmcblk0p3  /var/lib/myapp  ext4  defaults,data=journal  0  2

For a warehouse sensor deployment, the application writes data to SQLite on a separate ext4 partition with journaling enabled. If the Pi loses power mid-write, fsck can recover the journal. The root partition is read-only and survives power loss cleanly every time.

Thermal Throttling Is Silent and Intermittent

The thing is Pi will not tell you that it is throttling. It will not log a warning. That means, your video stream will just start dropping frames, your serial latency will increase, and your MQTT reconnects will take longer. And as you can see, all these symptoms look like software bugs. But if you check:

vcgencmd get_throttled

vcgencmd get_throttled is a command line tool unique to the Raspberry Pi that checks whether the computer has lowered its CPU speed.

0x0 means everything is fine. Anything else means the Pi is throttling now or has throttled since last boot. The common values:

0x50005 — The danger zone. You are currently under-volted and throttled right now, and it has happened before.
0x50000 — This means your system is physically running fine right now, but under-voltage (0x10000) and throttling (0x40000) have occurred in the past. Your current power supply is dropping voltage under load, making your SD card highly vulnerable to corruption.
0x4 — soft temperature limit active

I was building a telepresence robot, encoding 720p30 H.264 while running the aiohttp server and serial communication was pushing the Pi 4 to 80°C without a heatsink. The encoder started dropping frames randomly. Adding a heatsink brought idle temperature to 45°C and load temperature to 62°C. Managed to get the throttling under control.

On a Pi 5, the situation is better but not solved. The Pi 5 has an active cooler as an official accessory and it is worth using in any deployment where the Pi is in an enclosure. But, enclosures trap heat. A Pi in a plastic project box with no airflow will throttle faster than a bare board.

Also it is a best practice to add temperature monitoring to your health check endpoint so you find out before users do:

import subprocess

def get_pi_health():
    temp = subprocess.run(
        ['vcgencmd', 'measure_temp'],
        capture_output=True, text=True
    ).stdout.strip()  # "temp=58.0'C"

    throttled = subprocess.run(
        ['vcgencmd', 'get_throttled'],
        capture_output=True, text=True
    ).stdout.strip()  # "throttled=0x0"

    return {
        'temperature': temp,
        'throttled': throttled,
        'throttled_ok': throttled == 'throttled=0x0'
    }

Power Supply Quality Matters More Than Rated Current

A power supply rated for 3A is not always a 3A power supply. Cheap USB-C supplies have poor voltage regulation. So, under a load they sag below the 5.1V the Pi needs and trigger under-voltage throttling. The symptom is identical to thermal throttling: random slowdowns, occasional reboots, SD card corruption on shutdown.

The official Raspberry Pi power supply is not a premium product — it is a specification-compliant one. Use it, or use a bench power supply for development and a known-good supply for deployment. The Pi 5 draws up to 5A under full load; a 3A supply will cause under-voltage events when the CPU is fully loaded.

For any deployment running off mains power, a UPS hat (Geekworm UPS hat, Waveshare UPS hat) is worth the £20. The Pi gets notified of incoming power loss and can initiate a clean shutdown before the battery dies, which eliminates the entire class of "power cut during SD write" corruption events.

Network time synchronization errors can cause unexpected system failures

A Pi that has been offline for an extended period will have a wrong system clock when it boots — sometimes wrong by days if the RTC battery is dead or there is no RTC at all. Applications that timestamp log entries, certificate validity checks, and SQLite timestamp comparisons all behave unexpectedly when the system time is wrong.

A specific failure case I encountered: the MQTT client was writing timestamps using datetime.now().isoformat(). After a boot without internet, the system clock was set to 2023-01-01 (the default). All queued messages got timestamps in 2023. When the clock corrected to 2024 via NTP after network connection, the retention policy deleted those messages as being "older than 7 days" — because relative to the current time they appeared to be a year old.

Fix 1: Use a hardware RTC. The DS3231 costs about £3 and keeps accurate time across power cycles without network. Enable it with a device tree overlay:

# /boot/firmware/config.txt
dtoverlay=i2c-rtc,ds3231

Fix 2: For timestamps that survive offline periods, use monotonic time for intervals and NTP-synced time only for absolute timestamps. Do not mix them.

Fix 3: Configure chrony or systemd-timesyncd to be aggressive about syncing on boot and to accept large time jumps:

# /etc/chrony.conf
makestep 1 -1   # Accept any step size, any number of times

Watchdog Timers Are Not Optional

If your Pi is in a location where you cannot physically reach it — mounted on a robot, installed in a warehouse, bolted inside a wall panel — a software crash or infinite loop that freezes the application is effectively a permanent failure until someone intervenes.

The Linux kernel watchdog kicks the hardware watchdog timer while the kernel is running. If the kernel hangs, the watchdog expires and forces a reboot. But it does not know whether your application is running correctly. For that, you need an application-level watchdog.

systemd's built-in watchdog support requires almost no code:

# In your main loop, notify systemd you're still alive
import os
import time

def notify_watchdog():
    """Tell systemd the application is healthy."""
    os.system("systemd-notify WATCHDOG=1")

# Call this from your main loop — if you stop calling it,
# systemd will restart the service
while True:
    do_work()
    notify_watchdog()
    time.sleep(5)

# /etc/systemd/system/myapp.service
[Service]
WatchdogSec=30        # Restart if no heartbeat for 30 seconds
Restart=always
RestartSec=5
StartLimitIntervalSec=120
StartLimitBurst=5

For applications where notify_watchdog() cannot be called from the main loop (async applications, multi-threaded servers), run it from a background thread that monitors the health of the main thread.

Remote Access Must Work Before Anything Breaks

The time to set up remote access is before you deploy, not after. I use Tailscale for most Pi deployments because it takes five minutes to configure, works through NAT, does not require port forwarding, and uses WireGuard under the hood. Once it is running, you have a reliable backdoor to your hardware. However, if you need more control, complete data sovereignty, and zero third-party dependencies, use vanilla WireGuard instead. While WireGuard requires you to manually configure routing rules and host a central server with an open port for NAT traversal, it gives you total ownership over your network topology without device or account limitations:

# Install
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

# Enable SSH in your tailnet policy and you can reach the Pi from anywhere
ssh user@cyrobot.turkey-trench.ts.net

Tailscale also provides valid TLS certificates for your device's hostname in the tailnet — which is how the WebRTC server serves HTTPS without a domain name or public CA.

Set up mosh alongside SSH for unreliable connections. Regular SSH sessions die when the network hiccups. Mosh sessions survive.

Logs Fill Up the Filesystem

/var/log on a production Pi will fill up over months of continuous operation. When it does, your application cannot write logs, SQLite cannot open its WAL file, and things fail in confusing ways that do not obviously point to "disk full."

Set up log rotation from day one:

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    size 10M
}

And add disk usage to your health check:

import shutil

def check_disk():
    usage = shutil.disk_usage('/')
    free_percent = (usage.free / usage.total) * 100
    return {
        'free_gb': round(usage.free / 1e9, 2),
        'used_percent': round(100 - free_percent, 1),
        'warning': free_percent < 15
    }

The Summary

Before deploying any Pi to a location you cannot easily reach:

Boot storage is an industrial SD card or USB SSD. The root filesystem is read-only or has journaling on writable partitions. A hardware RTC is installed. A heatsink or active cooler is fitted. A quality power supply is used and a UPS hat is fitted if on mains power. Tailscale is installed and tested from outside the local network. systemd service has Restart=always, WatchdogSec, and StartLimitBurst set. Log rotation is configured. A health endpoint exposes temperature, throttle status, and disk usage. You have confirmed you can SSH in and restart the service from the office before going to the deployment site.

Written by Supun Akalanka | Category: Lessons Learned | Tags: Raspberry Pi, Production, Reliability, Embedded Linux, Hardware