DEV Community: Supun Sriyananda

How to cap the journal so it does not fill or wear out an SD card: Logging with journald

Supun Sriyananda — Tue, 13 Oct 2026 23:30:00 +0000

One of the most underappreciated benefits of running your application as a systemd service is that you get a powerful, structured logging system essentially for free. Any text your application prints to stdout or stderr is automatically captured, timestamped, and stored in something called the journal. journal is systemd's centralised log database. This article covers how to read, filter, and manage those logs, and how to configure the journal to be a responsible citizen on embedded devices with limited storage. Yes, journal works well with embedded devices. But, it works well only if you configure it properly!

How systemd Captures Your Logs

When your service is running, systemd intercepts everything written to stdout and stderr and passes it to systemd-journald. So, you don't need to change anything in your application code. Just a plain print("sensor reading: 42.3") in Python becomes a structured journal entry with a precise timestamp. This entry comes with the service name, the process ID, and the message text all stored together.

The only service file configuration needed to make this explicit is:

StandardOutput=journal
StandardError=journal
SyslogIdentifier=my-app

StandardOutput=journal and StandardError=journal are actually the defaults in modern systemd, so you don't strictly need to specify them. But being explicit makes the service file self-documenting. Anyone reading it can immediately see where the logs go. SyslogIdentifier sets a short tag attached to every journal entry from this service. This makes it easier to filter for your specific application even if multiple services share similar names.

Reading Logs with journalctl

The journalctl command is our primary interface to the journal. Its most important option for service debugging is -u. It filters entries to a specific unit:

# Show all historical logs for your service
journalctl -u my-app.service

# Follow the log in real time — like 'tail -f' on a log file
journalctl -u my-app.service -f

# Show only logs from the current boot
journalctl -u my-app.service -b

# Show only logs from the previous boot (useful after a crash)
journalctl -u my-app.service -b -1

# Show logs from the last hour
journalctl -u my-app.service --since "1 hour ago"

# Show logs between two specific timestamps
journalctl -u my-app.service --since "2025-12-16 10:00" --until "2025-12-16 11:00"

# Show only error-level and above (useful to find crashes quickly)
journalctl -u my-app.service -p err

# Show the last 50 lines, most recent first
journalctl -u my-app.service -n 50 -r

The option I use frequently is -b -1. It is particularly valuable for embedded systems. If your device rebooted unexpectedly due to a power cut, a kernel panic, you can look at the logs from the previous boot to understand what was happening right before it went down. Magical isn't it!

A note:

Simply put, the Pi keeps a diary of what your programs do. That diary is the journal. The question is where the diary is kept, and there are two choices.

Storage=volatile means the diary is kept in RAM, like writing on a whiteboard. It's fast and doesn't wear out the SD card, but when the power goes off, the whiteboard is wiped. This is how my Pi in the appendix is set up.

Storage=persistent means the diary is kept on the SD card, like writing in a notebook. It survives reboots and power cuts, but every write slightly wears out the card, and the notebook can fill up.

journalctl -b -1 only works with persistent storage, because a volatile journal is wiped on reboot. Likewise, SystemMaxUse only limits persistent storage; for volatile storage, the matching setting is RuntimeMaxUse. I'll cover how to choose between the two in an upcoming post in this series.

What are Log Priority Levels

systemd's journal understands syslog-compatible priority levels. If your application logs at different severity levels, you can filter journal output to show only entries at or above a certain severity. The levels are written from most to least severe. These are emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), and debug (7).

For most embedded projects, simplest and most readable approach is Python's standard logging module:

import logging
import sys

logging.basicConfig(
    level=logging.INFO,
    format="%(levelname)s: %(message)s",
    stream=sys.stdout   # systemd captures stdout to the journal
)

logger = logging.getLogger(__name__)
logger.info("Service started successfully")
logger.warning("Sensor reading out of expected range")
logger.error("Failed to connect to broker — will retry")

If you use this pattern, your log messages stay clean and consistent. Not only that, you can filter them in the journal by searching for ERROR or WARNING in the message text with journalctl -u my-app.service -g "ERROR".

How to Manage Journal Size on Embedded Devices

This is an important practical concern that most tutorials skip. The Raspberry Pi typically uses an SD card for storage, and SD cards have two relevant limitations. They are finite in capacity, and a limited number of write cycles before they start to fail. By default, systemd's journal will grow to fill a significant fraction of available disk space. For a small SD card, this will eventually cause problems. If you leave the journal unconfigured on an embedded device, it becomes a recipe for running out of disk space or unnecessarily wearing out the SD card.

The journal daemon is configured in /etc/systemd/journald.conf. Open it with:

sudo nano /etc/systemd/journald.conf

Add or uncomment these lines in the [Journal] section:

[Journal]
# Maximum total size of all journal files on disk
SystemMaxUse=50M

# Maximum size of a single journal file
SystemMaxFileSize=10M

# Maximum time to store journal entries (optional, for low-activity devices)
MaxRetentionSec=1month

After making changes, restart the journal daemon:

sudo systemctl restart systemd-journald

50 megabytes is a reasonable upper bound for most edge devices. It is large enough to retain several days of logs at typical verbosity, and small enough to not strain a 16GB or 32GB SD card. Let's say your application is particularly chatty, then you may want to reduce this further, or adjust your application's log verbosity before reaching for the journal size limit. This is something you will learn to adjust by doing.

Check the current journal disk usage at any time with:

journalctl --disk-usage

A Side Note For Curious You - A .conf is a configuration file, not a service unit file (.service). journald.conf is a plain text INI file located in /etc/systemd/journald.conf. Its only job is to store settings—like size limits, retention times, and storage behavior—that tell the logging daemon how to behave. The systemd-journald daemon does have its own service file behind the scenes called systemd-journald.service, but you rarely edit that. Instead, you edit journald.conf to tune how that service handles log data.

You might have already guessed by now. We can write a .conf for our myapp.service as well. Instead of modifying our myapp.service file directly, we can create a .conf file to override or add specific settings. This is standard practice in production environments because it keeps the core service file clean. More on this should be in a future article but not this one.

Viewing Logs Without Paging

By default, journalctl opens output in a pager (like less) which requires you to press q to exit. When running quick diagnostic commands over SSH, this can be inconvenient. Add --no-pager to get plain terminal output you can scroll or pipe to grep:

# Print all logs without opening a pager
journalctl -u my-app.service --no-pager

# Find all lines containing "error" (case-insensitive)
journalctl -u my-app.service --no-pager | grep -i error

# Count how many warning lines appeared in the last hour
journalctl -u my-app.service --since "1 hour ago" --no-pager | grep -c WARNING

The combination of journalctl filtering and standard Unix tools like grep, awk, and wc gives you a surprisingly capable log analysis workflow without needing any additional tooling.

Appendix

Here is the real output from my Raspberry pi.

admin@sbrsl:~ $ journalctl --disk-usage
Archived and active journals take up 8.8M in the file system.

sudo nano /etc/systemd/journald.conf

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the journald.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config.
#
# See journald.conf(5) for details.

[Journal]
Storage=volatile
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=no

How to keep secrets out of the service file in systemd

Supun Sriyananda — Mon, 12 Oct 2026 14:51:00 +0000

Two practical issues come up immediately when you move a real application into a systemd service. The first is that relative file path problem. The code works fine when you run it from your terminal but it stops finding its own files when systemd runs it. The second is that your application needs configuration: server addresses, API keys, database credentials. But hardcoding those values into your service file is both inflexible and a security risk. This artilce is about how to solve these two.

If you want a concrete working example to follow along with, use the script from examples/env-service/app_with_config.py in the GitHub repository. It is a Python service that reads all its configuration from environment variables — both sensitive secrets (via EnvironmentFile) and non-sensitive settings (via inline Environment directives).

The Working Directory Problem

When you run a Python script from your terminal, your shell sets the working directory to wherever you currently are. If you're in /home/pi/my-app/ and your script opens config.json without a full path, Python looks for config.json in /home/pi/my-app/ and finds it. This is so natural that most developers don't even think about it.

But when systemd runs your service things are different. The working directory is not set to your project folder. By default it's /, the root of the filesystem. That same open("config.json") call will look for /config.json. I don't need to tell you what happens now. It fails with a FileNotFoundError, and crash your service before it has done anything useful.

The fix is the WorkingDirectory directive in the [Service] section:

WorkingDirectory=/home/pi/my-app

When you set this, any relative file paths your application uses will resolve relative to that directory, just as they would if you ran the script from there in your terminal. While you're at it, specify which user the service should run as. By default, systemd runs services as root. This is more privilege than most applications need and it can create a security risk if the application is ever compromised.

User=pi
Group=pi
WorkingDirectory=/home/pi/my-app

So running as a non-root user is the principle of least privilege. It gives a process only the permissions it actually needs. If your MQTT client or sensor pipeline has a bug that an attacker could exploit, running as pi means the attacker gets the capabilities of the pi user and not root access to the entire system. However, the tradeoff is that you need to ensure the pi user has read/write permission to the files your application needs, but that's almost always already the case for files in /home/pi/.

To find your 'User' and 'Group', run the command id. This returns the active User ID (UID), Primary Group ID (GID), and all Supplementary Groups.

Inline Environment Variables

The simplest way to pass configuration to a service is with the Environment directive. It sets individual environment variables directly inside the service file:

[Service]
Environment="APP_PORT=8080"
Environment="LOG_LEVEL=INFO"
Environment="BROKER_HOST=localhost"

Your Python code reads these with os.environ.get("APP_PORT") or os.getenv("APP_PORT", "8080") (the second argument being a default if the variable isn't set). This approach is fine for non-sensitive configuration. For example, server addresses, port numbers, feature flags, log verbosity. It makes debugging easy because it is clear, explicit, and the values are visible in systemctl status output.

However, you should never use Environment for secrets. API keys, database passwords and authentication tokens are sensitive values that should never be visible in service file contents, which can be read by anyone with filesystem access. For secrets, what you need is an environment file.

Environment Files: The Right Way to Handle Secrets

An EnvironmentFile is a separate text file, usually stored in /etc/ with restricted permissions. It contains key-value pairs which systemd loads and injects into your service's environment before starting it. The service file references the path to this file rather than containing the values themselves.

The good practice is to create the directory and the environment file first:

sudo mkdir -p /etc/my-app
sudo nano /etc/my-app/config.env

The file format is simple. It has a one variable per line, like the same KEY=VALUE syntax we use in a shell script. Comments starting with # are allowed and considered good practice. Always use short but clear comments:

# /etc/my-app/config.env
# Server configuration
BROKER_HOST=mqtt.example.com
BROKER_PORT=8883

# Authentication. Keep this file private
API_KEY=secret-api-put-here
DB_PASSWORD=database-password

Now restrict the file's permissions so only root can read it. Most people skips this step. This is the crucial step that makes the whole thing secure:

sudo chmod 600 /etc/my-app/config.env
sudo chown root:root /etc/my-app/config.env

Finally, reference the file in your service with the EnvironmentFile directive:

EnvironmentFile=/etc/my-app/config.env

When systemd starts your service, it reads this file and sets all the listed variables in the service's environment. Our Python code reads them with os.environ.get("API_KEY") exactly as it would read any other environment variable and doesn't care where they came from.

Here, is a small but useful syntax trick: if you prefix the path with a minus sign (-), systemd will ignore a missing file rather than failing to start the service. I always use this in development where the config file does not exist yet:

EnvironmentFile=-/etc/my-app/config.env

The Complete Service File So Far

Combining everything from the last three articles, here's the service file in its current form:

[Unit]
Description=My Application
After=network.target
Wants=network.target

[Service]
# Run as a non-root user for security
User=pi
Group=pi

# Set the working directory so relative paths work correctly
WorkingDirectory=/home/pi/my-app

# Load secrets and configuration from a separate, restricted file
EnvironmentFile=/etc/my-app/config.env

# Non-sensitive configuration can go inline
Environment="LOG_LEVEL=INFO"

# The command that launches your application
ExecStart=/usr/bin/python3 /home/pi/my-app/main.py

# Restart automatically on unexpected crashes
Restart=on-failure
RestartSec=5

# Give up after 5 crashes within 60 seconds signals a real problem
StartLimitIntervalSec=60
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

After making changes, reload and restart as always:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

How to Access Environment Variables in Python

Here is how your application can read the environment variables set by the service file and environment file:

import os

# Read from EnvironmentFile (secrets)
broker_host = os.environ.get("BROKER_HOST", "localhost")
broker_port = int(os.environ.get("BROKER_PORT", "1883"))
api_key     = os.environ.get("API_KEY")

# Read from inline Environment directive
log_level   = os.environ.get("LOG_LEVEL", "INFO")

if api_key is None:
    # Using print() works fine since systemd captures stdout to the journal
    print("ERROR: API_KEY is not set. Check /etc/my-app/config.env")
    raise SystemExit(1)

print(f"Connecting to {broker_host}:{broker_port} with log level {log_level}")

# And your code ...

When systemd captures that error from stdout and logs it to the journal, you'll see exactly why the service failed. This is why I always use the pattern of failing fast with a clear, specific error message if a required variable is missing. It is much better than a cryptic exception from deep inside a library.

In the next article, we will look at how to read, filter, and manage logs, and how to configure the journal so it does not fill or wear out an SD card. This is 101 on embedded devices with limited storage!

How to make a systemd service recover from crashes without looping forever: Restart policies and Dependencies

Supun Sriyananda — Fri, 09 Oct 2026 12:30:00 +0000

The minimal service from the previous document starts your application at boot, but it has a significant weakness: if the application crashes, systemd simply leaves it in a failed state. I don't have to remind that for a device running unattended in the field, something you might not physically access for weeks or months, a silent crash that's never recovered from is unacceptable. This document covers the two additions that transform a basic service into a resilient one: restart policies and dependency declarations.

If you want a concrete working example to follow along with, use the script from examples/python-service/hello_service.py in the GitHub repository. It's a simple loop that logs a heartbeat message every ten seconds. It is straightforward enough to understand at a glance, but long-running enough to behave like a real service.

Restart Policies: For Surviving Crashes

The Restart directive in the [Service] section tells systemd what to do when your application's process exits. The default is Restart=no. In this case systemd does nothing and your service just stays dead.

There are several possible values, but in practice you'll almost always choose between two of them based on this simple question: " Should this service restart if you deliberately stop it with systemctl stop? ".

Restart=on-failure means: restart the service if the process exits with a non-zero exit code. It restarts if it is killed by a signal, or hits a timeout, but not if it exits cleanly with exit code 0. The practical consequence is that systemctl stop is a clean exist. So it doesn't trigger a restart, because systemd sends a clean termination signal which your application handles gracefully. For most production services this is the right choice because it means you can stop a service for maintenance without a restart loop annoying you.

Restart=always means: restart no matter what, even if the process exits with code 0. Obviously, this is appropriate for services that should never be down or where even a clean exit is considered abnormal. It is a less common case but useful for things like a watchdog process or a service whose restart you want explicit control over from systemd alone.

For the vast majority of embedded applications, on-failure is the correct choice.

The RestartSec Pairing

Always pair Restart with RestartSec. RestartSec sets the delay in seconds before systemd attempts to restart the process. Without it, systemd restarts immediately. This may sound desirable, but think about why this creates a problem. Your service is crashing for a reason. That reason could be a dependency it needs isn't yet ready. For example, the network stack is still initialising, or a database is still starting up. So an immediate restart will just crash again, and again, and again. Even a small delay of five seconds gives the rest of the system time to settle into a stable state.

Restart=on-failure
RestartSec=5

How to Prevent Infinite Crash Loops

Now consider a more serious scenario: your service has a real bug that makes it crash immediately on every startup. Perhaps it could be a malformed configuration file or a missing Python dependency. If don't set a limit, systemd will restart it, watch it crash, restart it, watch it crash, forever. This consumes CPU, floods your logs with crash reports, and masks the real problem.

The StartLimitIntervalSec and StartLimitBurst directives work together to prevent this. StartLimitBurst defines the maximum number of restart attempts allowed, and StartLimitIntervalSec defines the time window those attempts are counted in. To put simply, if the service restarts more than StartLimitBurst times within StartLimitIntervalSec seconds, systemd declares the service as failed and stops trying.

StartLimitIntervalSec=60
StartLimitBurst=5

With these settings, systemd makes up to five restart attempts within any 60-second window. If the sixth crash happens within that window, it marks the service as failed and stops. At that point systemctl status clearly shows that the restart limit was reached. In my personal experience, I have found this to be a very valuble signal. It means something is fundamentally broken and needs human attention, not more restart attempts.

I used to imagine StartLimitBurst as a circuit breaker. In electrical engineering, a circuit breaker trips when it detects a fault and it doesn't keep trying to send power through a short circuit. systemd's start limit is the same idea: try a reasonable number of times, then stop and let a human investigate rather than hammering a broken process indefinitely. How beautiful is that!

Dependency Declarations: Starting things in the Right Order

Most real applications don't run in isolation. An MQTT client is useless if it tries to connect before the network interface exists. A database-backed service will crash if the database process hasn't finished starting. A camera streaming service first needs the camera hardware to be initialised. There is a way to express these dependencies explicitly in systemd. This way your service starts at the right moment in the boot sequence rather than just racing against everything else.

The two most important dependency directives live in the [Unit] section.

After: Ordering Without Hard Requirements

After=some.target tells systemd not to start your service until some.target has been reached. This is purely about ordering and it doesn't create a hard requirement. If some.target fails entirely, your service still starts after it in the sequence. It just may find that what it was waiting for isn't actually available.

The target you'll use most often is network.target, which represents the point where the network subsystem has been initialised. On Raspberry Pi OS, this typically means the kernel network stack is ready and configured interfaces have been brought up.

After=network.target

A more reliable option for network-dependent services is network-online.target. This is particularly valuble for those that connect to remote hosts. It doesn't just wait for the network subsystem to initialise but waits until at least one network interface has an active connection and is reachable. This is more conservative and also means a slightly longer boot time. But think about why it prevents some of the common failure modes. Imagine i senario where your service starts, tries to open a TCP connection to a remote server but fails because the network is still negotiating a DHCP address, and crashes before the restart delay kicks in.

After=network-online.target

Wants and Requires: What are Soft and Hard Dependencies

Wants=some.target and Requires=some.target declare what your service needs, not just what it should start after. The difference is what happens when the dependency isn't available.

Wants is what is called a soft dependency. If some.target isn't available or fails to start, systemd will still attempt to start your service. Wants is like saying "I'd prefer this to be available, but I can try to run without it." This is appropriate when your application has internal handling for the dependency not being there. Which means you have thought about these cases and written you application code to handle them. For example, you write an MQTT client that queues messages if the broker is ever unreachable. Now you can use a soft dependency because you know that your application handles it.

Requires is a hard dependency. If some.target fails to start, systemd won't start your service either. Furthermore, if some.target is stopped while your service is running, systemd will also stop your service. This is appropriate when your service cannot function at all without the dependency. For example, a service that reads from a database where the database is also managed by systemd.

In practice, the combination you'll use most often for network-dependent services is:

After=network.target
Wants=network.target

This says "start after the network is ready, and express a preference for it to be available but if the network fails, keep the service running, because I handle network loss internally."

The Updated Service File

Putting the restart policy and dependency declarations together, here's the service file updated from the minimal version:

[Unit]
Description=My Application
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/my-app/main.py
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

After editing the file, reload systemd and restart the service to apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart my-app.service

You can test the restart policy right now without rebooting. Find the process ID from systemctl status, then kill it:

sudo kill -9 <PID>

Wait five seconds and run systemctl status again. You should see the service back in active (running) state with a new PID and a note that it was restarted. That automatic recovery is exactly what you want from a service running unattended in the field.

Common Dependency Targets Reference

Beyond network.target, a handful of other targets come up regularly in embedded service files.

local-fs.target ensures all local filesystems are mounted. This is useful if your service reads or writes files on a specific mount point.
time-sync.target ensures the system clock has been synchronised via NTP, which matters for any service that timestamps data or signs tokens.
bluetooth.target ensures the Bluetooth subsystem is available, relevant for Raspberry Pi projects that communicate with BLE devices.

You can also use your services written by yourself. If a service depends on another service you've written yourself, you can declare After=my-other-app.service and Requires=my-other-app.service directly by service name. Clever isn't it?

In this next article explains how to fix the relative-path failures that appear under systemd, run as a non-root user, and how to keep secrets out of the service file.

Write a minimal linux service file

Supun Sriyananda — Thu, 08 Oct 2026 13:12:49 +0000

Let's create a real, working service file from scratch. By the end you'll have a Python script registered with systemd, starting at boot, and showing up cleanly in systemctl status. Every line of the service file is explained so you understand not just what to type but why it's there.

The Application We're Registering

For this example, suppose you have a Python script at /home/admin/resources-tools/systemd/examples/python-service. The specific content doesn't matter much for learning systemd — what matters is that it's a long-running process (something that loops, listens for connections, or monitors sensors) rather than a script that runs once and exits.

Where Service Files are Located

Service files belong in /etc/systemd/system/. This directory is specifically reserved for service files you write and manage yourself. It is separate from the directories where systemd's own built-in services are, and separate from the directories package managers use. Putting your files here means they won't be overwritten by a system update, and they'll take precedence over any lower-priority defaults.

The file must have a .service extension. The name you give it becomes the name you use with systemctl. So the convention is to choose something short and descriptive. In this tutorial I will be using hello_python.service.

The Minimal Service File

Create the file with:

sudo nano /etc/systemd/system/hello_python.service

Type or paste the following:

[Unit]
Description=My Application

[Service]
ExecStart=/usr/bin/python3 /home/pi/hello_python/main.py

[Install]
WantedBy=multi-user.target

This is the absolute minimum. Let me explain what each line is doing.

The [Unit] Section

Description=My Application is a plain-English label. It appears whenever you inspect this service in systemctl status, in journal logs or in any system monitoring tool. If you write something vague, you or anyone else using the system won't be able to identify the service at a glance after some time. Always write descriptive messages. For now I will be using "My Application", but in a real project write something like "Temperature Sensor Data Pipeline" or "MQTT Edge Client".

The [Service] Section

ExecStart=/usr/bin/python3 /home/admin/resources-tools/systemd/examples/python-service is the command that launches your program. A critical rule: you must use the full, absolute path to every executable. You can't write python3 hello_service.py because systemd doesn't use your shell's PATH environment. It needs to know the exact location of the Python interpreter. You can find it with which python3 in your terminal. On most Raspberry Pi systems it'll be /usr/bin/python3. The path to your script must also be absolute.

The [Install] Section

Systemd organises the boot process into targets. Targets are milestones that represent specific states of the system. multi-user.target represents the state where the system is fully booted, all essential hardware is initialised, networking is available, and the system is ready for normal use but before any graphical desktop starts. It essentially means "everything is ready, start normal services" point in the boot sequence. By declaring WantedBy=multi-user.target, you're saying "start my service when the system reaches this state." This is the correct target for the vast majority of application services.

A note

If you are using my example, in the Install section you need to fill 'User' and 'Group'. To find yours run the command id. This returns the active User ID (UID), Primary Group ID (GID), and all Supplementary Groups.

Activating the Service

Writing the file isn't enough on its own. Now we need to tell systemd the file exists and then enable it.

Step 1: Reload systemd's Configuration

sudo systemctl daemon-reload

systemd reads unit files when it starts and caches their contents. When we create or modify a service file, systemd doesn't automatically notice the change. When you run daemon-reload, it re-scans all unit file directories and updates its internal cache. Remember, you must run this every time you create or edit a service file, or your changes will be ignored.

Step 2: Enable the Service

sudo systemctl enable hello_python.service

This creates a symbolic link inside the multi-user.target.wants/ directory that points to your service file. This symlink is what causes systemd to start our service at boot. If you are following this tutorial, go ahead and run this command. You can actually see the symlink after running enable.

ls -la /etc/systemd/system/multi-user.target.wants/ | grep hello_python

Here is the output from my Pi:

admin@sbrsl:~ $ ls -la /etc/systemd/system/multi-user.target.wants/ | grep hello_python
lrwxrwxrwx  1 root root   40 Oct  5 14:51 hello_python.service -> /etc/systemd/system/hello_python.service

Step 3: Start the Service Now

Enabling only affects future boots. If you want to start the service in the current session without rebooting:

sudo systemctl start hello_python.service

In practice, when deploying a new service, I almost always run all three commands in sequence:

sudo systemctl daemon-reload
sudo systemctl enable hello_python.service
sudo systemctl start hello_python.service

Verifying it's Running

sudo systemctl status hello_python.service

If everything is working, the output will look something like this:

admin@sbrsl:~ $ sudo systemctl status hello_python.service
● hello_python.service - Python Heartbeat Service (Tutorial Example)
     Loaded: loaded (/etc/systemd/system/hello_python.service; enabled; preset:>
     Active: active (running) since Thu 2025-10-08 12:27:30 +0530; 4min 56s ago
   Main PID: 31967 (python3)
      Tasks: 1 (limit: 9563)
        CPU: 43ms
     CGroup: /system.slice/hello_python.service
             └─31967 /usr/bin/python3 hello_service.py

Oct 08 12:30:51 sbrsl hello-python[31967]: 2025-10-08 12:30:51,061 INFO: Heartb>
Oct 08 12:31:01 sbrsl hello-python[31967]: 2025-10-08 12:31:01,062 INFO: Heartb>
Oct 08 12:31:11 sbrsl hello-python[31967]: 2025-10-08 12:31:11,063 INFO: Heartb>
Oct 08 12:31:21 sbrsl hello-python[31967]: 2025-10-08 12:31:21,064 INFO: Heartb>
Oct 08 12:31:31 sbrsl hello-python[31967]: 2025-10-08 12:31:31,065 INFO: Heartb>
Oct 08 12:31:41 sbrsl hello-python[31967]: 2025-10-08 12:31:41,068 INFO: Heartb>
Oct 08 12:31:51 sbrsl hello-python[31967]: 2025-10-08 12:31:51,069 INFO: Heartb>
Oct 08 12:32:01 sbrsl hello-python[31967]: 2025-10-08 12:32:01,070 INFO: Heartb>
Oct 08 12:32:11 sbrsl hello-python[31967]: 2025-10-08 12:32:11,070 INFO: Heartb>
Oct 08 12:32:21 sbrsl hello-python[31967]: 2025-10-08 12:32:21,071 INFO: Heartb>
lines 1-19/19 (END)

The above is the real output from my pi.

The most important line here is Active: active (running). The word enabled next to Loaded confirms it'll also start at boot. If you see active (dead), the service ran and exited. In this case, if you expected your script exits cleanly this is the right thing you should see. But if you expected it to keep running, this is a problem. If you see failed, the status output usually includes the last few log lines pointing to the cause. I will show you how to inspect them.

Testing Reboot Persistence

The real test of a service is whether it survives a reboot:

sudo reboot

After the Pi/Linux machine comes back up, check the status again:

sudo systemctl status hello_python.service

If it shows active (running) with a start time matching the current boot, our service is working correctly as a persistent, auto-starting system service.

One-Line Enable + Start Shortcut

On newer versions of systemd (v220 and later, which includes all current Raspberry Pi OS versions), we can combine enable and start with the --now flag:

sudo systemctl enable --now hello_python.service

Functionally this is identical to running enable and start separately, but just more convenient.

This Minimal Service is Still Missing Something

The service you have right now works, but it's missing several things that matter for real-world use. If the application crashes, systemd will leave it dead and never restart it. If you're writing relative file paths in your Python code, they'll likely fail because systemd starts services from the root directory, not your project folder. And if you have any configuration like API keys, database passwords, and server addresses, there's no good place to put them yet. In the next two articles I will address all of these.

Deliberately stopping or forcefully killing a service isn't just about handling an absolute emergency. In day-to-day administration, there are several practical reasons you will need to step in and shut a process down:

Applying Software Updates: When you modify your code, update a Python package, or download a security patch, the old version of the program keeps running in the Pi's RAM. You must stop the service so systemd can load the fresh code.
Conserving System Resources: The Raspberry Pi has limited RAM and CPU power. If you are about to run a heavy task like compiling code or processing video, you may need to temporarily kill background services (like a database or web server) to free up hardware memory.
Testing and Debugging: If your script is misbehaving, it is much easier to stop the systemd service completely and run the script manually in your terminal. This allows us to see live print statements and interactive error messages in real-time.
Changing Configuration Settings: Many applications only read their configuration or environment files exactly once when they launch. So, if you change a setting, the running service won't notice until you stop and start it again.
Preventing Data Corruption: If you need to back up a database or copy a live log file from your SD card, stopping the service first ensures that no new data is being written mid-copy. This prevents your backup from becoming corrupted.

Force-kill the Python process with kill <PID> and then run systemctl status to see what systemd does. Observe this and the restart policies we will be discussing in the next article would become much more concrete.

What is systemd? And why should you care?

Supun Sriyananda — Mon, 21 Sep 2026 14:48:05 +0000

testedOn:

hardware: Raspberry Pi 5
os: Raspberry Pi OS Bookworm

May it be a sensor pipeline, an MQTT client or a web server you have engineered something that works perfectly. And then on one day, the device loses power or you reboot it. You get the device powered again but there is only silence. Your MQTT client or the webserver doesn't start. You curse, SSH in, run your script manually, and it all comes back to life. "Well, this is eveyday life!" you say. But it doesn't have to be. You just need to learn how to laverage systemd to make them run at boot.

The Boot Process in Plain English

When you power on a Raspberry Pi (or any Linux computer), a very specific sequence of events unfolds. The processor runs a small piece of firmware stored in ROM, which locates the bootloader on your SD card. The bootloader then loads the Linux kernel into memory and starts it. Now that the kernel is running, it initialises memory, filesystems and device drivers. At the very end of this setup phase, it needs to hand control over to the rest of the operating system.

The kernel does this by starting exactly one process and giving it process ID 1. This process is the init system and it becomes the parent of everything else that runs on the machine. On virtually every modern Linux distribution such as Raspberry Pi OS, Ubuntu, Debian or Fedora that process is systemd.

This is a common point of confusion for many people at first. I was one of them. The init process is the special PID 1 process that starts and manages the rest of the userspace processes. On modern Linux systems, systemd is the program that serves as the init process, so systemd is PID 1. In other words, “init” describes the role, while “systemd” is the program performing that role.

So, yes. You thought right! There are other programs out there other than systemd which performs 'init' role and become PID 1. Some of them are SysVinit, Upstart, OpenRC and runit. Systemd became dominant because it provided a broader and more integrated way to manage a Linux system. It ended up establishing itself as the common standard after major distributions adopted it.

From the moment systemd starts, it is in charge. It reads configuration files called unit files. 'unit files' describe everything the system needs to do such as mounting filesystems, configuring the network, starting background services and bringing up the desktop environment. It does all of this in a carefully ordered, parallelised way that's much faster and more reliable than the older shell-script-based init systems it replaced.

This is the key insight you need as a developer: if you want your application to be part of that startup sequence and be managed, monitored, and restarted by the system itself, you just need to write a unit file that describes it. Once systemd knows about your application, it treats it like any other first-class system component.

Unit Files: Configuration for Everything

Now you know that systemd manages everything through unit files. They are small, structured text files that follow a consistent format. 'unit files' let us tell systemd these things:

What program should start?
When should it start?
What does it depend on?
Should it restart if it crashes?
Which user should run it as?
Where should its output/logs go?

The one you'll work with most is the service unit, which describes a long-running process (a daemon). Other types include timer units (for scheduled tasks, covered in the examples section), socket units (for socket-activated services), and mount units (for filesystem mounts).

All service unit files share the same three-section structure:

[Unit]
# Metadata, description, and dependency declarations

[Service]
# How to start, stop, and manage the actual process

[Install]
# How this service integrates into the system boot targets

You'll become familiar with each of these sections throughout this tutorial. The important thing to know now is that systemd reads these files from /etc/systemd/system/. So, this is where you'll put your own service files. Files in this directory take precedence over systemd's built-in defaults. And files stored here are completely immune to system upgrades.

The Two Essential Tools: systemctl and journalctl

You interact with systemd through two command-line tools, and you will end up using them often. So, let's learn them first.

systemctl is the control interface for systemd allowing you to start, stop, enable, disable, and inspect services. The most important thing to understand is the difference between these two pairs of concepts: start vs. enable and stop vs. disable.

The first pair is start vs enable. systemctl start launches a service right now, in the current session. If you reboot, it won't start again unless you've also enabled it. systemctl enable creates a symbolic link that tells systemd to start the service at boot. However, it doesn't start it right now. In everyday use you almost always want to run both commands together after creating a new service.

The second pair is stop vs disable. systemctl stop stops a running service immediately, but it'll start again at the next reboot if it's enabled. systemctl disable removes the boot-time symlink so it won't start at the next reboot, but it doesn't stop the currently running instance. In practice you often want both when removing a service.

journalctl is how you can look at the logs that systemd collects from every service it manages. Systemd captures everything your service prints to stdout and stderr and stores it in a structured binary database called the journal. So, it makes filtering, searching, and correlating logs across services much more powerful than traditional log files.

The command you'll end up using most often is journalctl -u my-app.service. It shows all historical logs for a specific service. Adding -f follows the log in real time, like tail -f. Adding -b shows only logs from the current boot.

Why should you care as an Embedded Systems engineer?

If you're running a Raspberry Pi as a headless embedded device such as a sensor node, a home automation hub or an edge computing device, the consequences of your application crashing and never restarting are dire. Which is even worse if the device is in a remote location or inside a piece of equipment, you might not notice for hours, days, or weeks.

There are alternatives people often reach for first. Putting commands in /etc/rc.local, using cron @reboot, or writing shell scripts in /etc/init.d/ are some of them. They all work after a fashion, but they give you almost none of the reliability features systemd provides out of the box. They don't automatically restart crashed processes. They don't provide ordered startup relative to hardware and network dependencies. They don't capture logs in a queryable, structured way. And they don't give you a clean interface for checking whether your application is actually running and healthy.

systemd gives you all of that, and all it costs is learning how to write a service file. In the rest of this tutorial I will teach you how to write service files building from the simplest possible example up to a production-grade configuration.

What ioctl(KVM_RUN) Does: The VM Exit Loop Explained

Supun Sriyananda — Tue, 08 Sep 2026 09:10:18 +0000

The previous article established that guest code runs natively until it touches a device, at which point control has to reach QEMU.

ioctl(vcpu_fd, KVM_RUN) is the single most important line in the entire stack. Libvirt, the XML, the device models and VirtIO are just scaffolding around this one call, looped forever.

We'll build up to it step by step, because the this call only makes sense once you know what a file descriptor is and how ioctl works generally.

File descriptors, briefly

Before I explained what ioctl is let me refresh your memory about file descriptors. If you don't need a memory refresh, you can skip this section.

Simply put, a file descriptor (FD) is a non-negative integer that serves as a unique identifier for any resource a process has open.

The operating system kernel maintains a private lookup table for every running process. Let's say your program opens a resource. The kernel then creates a tracking entry for the underlying object and hands back the index of that slot. From that moment on, your program refers to that resource exclusively by its number.

By default, three descriptors are pre-allocated when a process spawns:

0 for standard input (stdin),
1 for standard output (stdout), and
2 for standard error (stderr).

The next resource you open grabs the lowest available integer, typically starting at 3. It doesn't matter whether you are interacting with a plain text file, a network socket, a hardware component, or a KVM virtual CPU, the kernel handles them all through file descriptors using uniform system calls like read(), write(), and close(). And the kernel automatically figures out how to route your data based on what that specific integer points to.

Let me show you the outputs from my raspberry pi 5:

home1@home1:~ $ ls -l /proc/self/fd
total 0
lrwx------ 1 home1 home1 64 Sep  7 07:32 0 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 1 -> /dev/pts/4
lrwx------ 1 home1 home1 64 Sep  7 07:32 2 -> /dev/pts/4
lr-x------ 1 home1 home1 64 Sep  7 07:32 3 -> /proc/3311761/fd

Look at the numbers on the far right. FDs 0, 1, and 2 are all pointing to /dev/pts/4. This is the virtual terminal window you are looking at. It says your input, output, and errors are all routed to your screen! Do you see the the number 3 at the end? That is the ls command itself reading the directory table, grabbing the lowest available integer.

ioctl: the escape hatch for everything read and write can't express

Standard read() and write() operations excel at shifting raw bytes back and forth but they fall short when you need to configure the underlying hardware itself.

For example, you cannot write() to an CD drive to force it to eject. Similarly, there's no meaningful way to read() a serial port's baud rate, or to express "change it to 115200" as a byte stream. This is because these are control operations on a device, not data transfers.

The Unix architecture solves this limitation with ioctl (Input/Output Control). ioctl is a highly generic system call designed specifically to send arbitrary commands to whatever resource a file descriptor references.

int ioctl(int fd, unsigned long request, ...);

   ioctl(fd, REQUEST, argument)
          │     │        │
          │     │        └─ Optional pointer to custom data structures
          │     └────────── The specific command ID (a unique integer)
          └──────────────── The file descriptor you are talking to

Under the hood, the driver handling the device interprets what each command integer means. This serves as a universal extension mechanism. And it allows any hardware driver to expose custom routines that the standard filesystem interface never planned for. For example, terminals rely on it to adjust window dimensions, network adapters use it to bind IP configurations, and KVM leverages it to orchestrate an entire virtual machine.

KVM's three-level ladder of descriptors

KVM exposes itself as /dev/kvm, a character device. QEMU opens it and gets a descriptor. That descriptor and a stream of ioctl calls is the entire QEMU-to-KVM interface.

To manage complex hardware setups better, KVM organizes this interface into a strict, three-tiered hierarchy.

When a VM is created, things descent down this ladder:

System FD (KVM_CREATE_VM ): Allocates a blank, unconfigured VM instance and hands back a unique VM file descriptor.
KVM_SET_USER_MEMORY_REGION on the VM fd — "here's memory I allocated; treat it as the guest's physical RAM." QEMU allocates guest memory as an ordinary mmap region in its own address space and registers it with KVM. This is the sandbox boundary: the guest can only address what was registered.
VM FD (KVM_SET_USER_MEMORY_REGION): Defines the virtual sandbox. The Virtual Machine Monitor (VMM) reserves a normal block of userspace memory via mmap, then registers it here. The KVM kernel module transforms this space into the guest's physical RAM layout. It also ensures the guest can never read or write outside this boundary.
VM FD (KVM_CREATE_VCPU): Spawns an execution core inside the VM structure and maps it to a fresh vCPU file descriptor.
vCPU FD (KVM_RUN): Instructs the host processor to immediately start executing guest instructions on this specific core. "run this CPU now."

What KVM_RUN actually does

Let's zoom in. The KVM_RUN system call does not behave like a standard, predictable function call. Say a VMM thread(a QEMU vCPU thread) invokes ioctl(vcpu_fd, KVM_RUN). This thread immediately enters the guest. And it does not return to your user-space application until the guest does something KVM cannot handle alone.

The call blocks on purpose for extended periods. Just think about it. From QEMU's perspective, it called a function and that function didn't return for millions of guest instructions. And the entire time the guest is computing, the QEMU thread is parked inside a single ioctl. QEMU isn't polling or supervising anything. This is elegant, isn't it? It is asleep in a system call and wakes up only when there's work for it.

Context switching across the guest boundary is expensive. Entering the guest means loading the registers of the guest into the real core; exiting means saving them back out. This is not free and it's part of why an exit is costly!

Not every VM exit returns to QEMU. If the guest triggers an exit that the Linux kernel can settle internally, KVM resets the guest state and jumps right back into guest mode. For example, when servicing a hardware timer or adjusting a local interrupt controller the user-space program QEMU stays asleep, completely unaware that an exit ever occurred in the first place.

Cheap exits and expensive exits

There are two tiers of VM exits.

Cheap Exits (Kernel-Handled): KVM resolves these entirely within the kernel module and immediately re-enters the guest. The ioctl call never returns. And your VMM thread stays fast asleep. QEMU never learns it happened. Virtual timer ticks fall into this tier. So do accesses to devices KVM emulates in-kernel on ARM64, most importantly the GIC. GIC is the Generic Interrupt Controller. Interrupt handling is far too frequent to route through userspace, so KVM handles it directly.

Expensive Exits (User-Space Handled): Other exits KVM cannot resolve, because the guest touched a device that exists only as a QEMU device model. So, now the ioctl genuinely has to return. QEMU wakes up, runs the relevant C function, and calls KVM_RUN again.

When people say "VM exits are expensive," they mean the second kind. Because as I showed you, first is comparatively cheap.

This distinction is also why the in-kernel GIC exists at all. In the part 2 of the series the interrupt-related warning on Raspberry Pi from article 2 is worth understanding at this point. It generated more exits than a pristine hardware environment would, but the cheap ones.

How QEMU learns what happened: the shared kvm_run struct

When KVM_RUN does return, QEMU needs to know why. Passing that back through the simple numeric return value of ioctl would be far too limited, so KVM uses shared memory.

When a vCPU is created, KVM gives QEMU a small region mapped into both QEMU's address space and the kernel's. KVM writes the exit details into this struct kvm_run before returning. Then QEMU reads them immediately after. There is no need for additional system calls.

exit_reason is the dispatch key. KVM_EXIT_MMIO means the guest accessed a memory-mapped device region. This is the common case on ARM64, and we discussed the mechanism in article 4. QEMU looks up which device model owns that address, calls it, and loops.

Follow the example concretely. The guest writes the character H to the UART at 0x09000000:

The guest executes a store instruction. It has no idea anything unusual is happening.
The address is in a device region, so the CPU traps. VM exit.
KVM sees it's MMIO to an address it doesn't handle in-kernel. It fills in kvm_run and returns from ioctl.
QEMU wakes, reads exit_reason, dispatches to its PL011 UART model.
The UART model writes H to whatever the serial console is connected to — your terminal.
QEMU calls ioctl(KVM_RUN) again. The guest resumes at the next instruction.

Every character of output from a guest's serial console runs that loop. When you watch an OS installer scroll past over virsh console, you are watching this cycle execute thousands of times.

This is the foundation for everything that we will follow

When you can clear your head around this round trip, several later topics becomes easier to grasp.

VirtIO (article 6) is the architectural response to expensive exits. If each round trip costs, then we can make each one carry more work. VirtIO batches many operations behind a single notification, amortizing one exit across dozens of requests.

vCPU pinning Remember that the host thread driving our KVM_RUN cycle is treated by Linux as an ordinary, everyday thread. So, if you don't intervene, the Linux kernel scheduler will happily shift that thread from one physical core to another to balance host workloads, like it usually does. But, if you pinning your vCPU thread to a single core, you ensure the guest state stays perfectly warm inside that specific core's local cache lines between transitions. Now imagine you let that thread wander across your topology. Then every VM exit drops you onto a cold core. This forces the hardware to reload state variables from scratch and destroys your execution speeds.

QMP versus KVM_RUN Don't confuse QMP(QEMU Machine Protocol) with KVM_RUN. I know I did at first. These are two entirely separate communication paths. QMP is a standard Unix/TCP socket interface that management utilities like libvirt use to command QEMU from the outside. KVM_RUN is the internal system call (ioctl) that QEMU executes to hand control over to the Linux kernel module and physical hardware.

Keep this in mind: ioctl(vcpu_fd, KVM_RUN) commands execution, the guest computes directly on bare-metal hardware until a trap occurs, and the system call unblocks only when QEMU device emulation is required. That call, when looped forever, is what constitutes a running virtual machine.

Summary

A file descriptor is an integer indexing a per-process table of open kernel objects.
ioctl is the generic system call for device-specific commands that read and write can't express.
KVM is driven entirely through ioctl on /dev/kvm, via a three-level ladder: system fd → VM fd → vCPU fd.
KVM_RUN on a vCPU fd enters the guest and blocks for as long as the guest runs natively — potentially millions of instructions.
Cheap exits are handled inside KVM (timers, the in-kernel GIC) and never wake QEMU. Expensive exits return to userspace so a QEMU device model can run.
KVM reports why it exited through the shared kvm_run struct; exit_reason tells QEMU which device model to dispatch to.
Every character on a guest's serial console is one full round trip through this loop.

How QEMU Emulates Hardware: Device Models, MMIO, and TCG

Supun Sriyananda — Fri, 04 Sep 2026 06:30:00 +0000

The previous article established that QEMU emulates the devices KVM deliberately doesn't.

We constantly hear the phrase, “QEMU emulates devices.” It is treated like a magic trick. A user-space program somehow conjures a virtual disk or a network card out of thin air, and a guest operating system is perfectly fooled.

But there is no black box. The entire illusion relies on a single mechanism which is applied consistently across every piece of virtual hardware. Once you see this mechanism, every performance characteristic of virtualization makes sense. Why does VirtIO exist, why certain operations feel catastrophically slower in a VM than on bare metal.

Before We Begin: QEMU is Two Different Programs

We need to clear up a common point of confusion before looking at the code or going further. QEMU actually does two entirely different jobs depending on how you launch it.

Full-system emulation (qemu-system-*) builds a virtual environment from scratch to boot a complete guest operating system. This is what Libvirt launches for every VM, and it is the sole focus of this series. For example, qemu-system-aarch64 constructs a complete, virtualized motherboard from scratch, including the CPU, RAM, disk controllers, network cards, and firmware—allowing a full guest OS to boot.

User-mode emulation (qemu-*) executes a single binary compiled for a foreign architecture directly on your host OS by translating its system calls on the fly. This engine doesn't build a machine or boot an OS. It's how you run an x86 binary on an ARM machine without a VM. And it is what Docker's binfmt_misc cross-architecture support uses under the hood. This is useful, but not virtualization.

Note that, this series is strictly about full-system emulation. If a command begins with qemu-system-, it is fabricating an entire computer.

The trick: devices live at memory addresses

The mechanism that makes emulation possible relies on a fundamental truth about real hardware:

When an operating system wants to send a byte to a serial port, it doesn't call a function. Instead, it writes to a specific memory address. This memory address is physically wired directly to the serial chip on the motherboard. As you can see no RAM is ever involved here. The motherboard's address decoder routes the signal straight to the hardware registers instead.

This is called memory-mapped I/O, or MMIO, and it's how the CPU talks to essentially every device on a modern ARM system. So, the devices are not special entities the CPU has a private channel to. The devices are just regions of the address space.

Which means the CPU's entire interface to a device comes down to this: writing to certain addresses and reading from certain addresses. That's it. Nothing else. And this is an interface software can easily impersonate!

QEMU maps out a guest address space where specific regions are intentionally flagged as "device space" rather than RAM. Now, let's imagine this guest attempts to access one of these coordinates. This operation is blocked from executing as a memory task. Instead, it triggers a trap. Then QEMU runs a dedicated software function.

That software function is the device. For example, a virtual UART is just a few hundred lines of C code that prints a character to your terminal when a write occurs. A virtual disk controller is C code that intercepts a guest read request and fetches bytes from a .qcow2 file on your host SSD.

Let's me show you this in code. The the following is a simplified, structural layout of what that C code of this software function looks like. Don't overwhelm yourself with all the details in the code. Just take a look at it. I just want show you that there is no magic here.

1. The Handoff Interface (MemoryRegionOps)

First of all, QEMU needs to map the memory region to specific C functions. It registers a structure that tells the engine exactly what code to execute when a guest touches the device's addresses:

/* This structure acts as the router for guest hardware accesses */
static const MemoryRegionOps uart_mmio_ops = {
    .read = uart_mmio_read,   /* Function to run when guest reads from an address */
    .write = uart_mmio_write, /* Function to run when guest writes to an address */
    .endianness = DEVICE_NATIVE_ENDIAN,
    .valid.min_access_size = 1,
    .valid.max_access_size = 4,
};

2. The Software Function (uart_mmio_write)

Let's imagine that the guest writes to 0x09000000. When it does, the hardware trap catches it, and QEMU executes this exact type of software function. You can see that it is just a standard C switch statement mapping address offsets to software behaviors.

/* The software function that IS the device hardware channel */
static void uart_mmio_write(void *opaque, hwaddr offset, 
                            uint64_t value, unsigned size)
{
    UARTState *s = (UARTState *)opaque;

    /* The offset is how far past the base address (0x09000000) the guest wrote */
    switch (offset) {
        case 0x00: // Data Register (TX Buffer)
            /* The guest wants to transmit a character. 
               Instead of triggering silicon, we print it to your host terminal. */
            putchar((char)value); 
            fflush(stdout);

            /* Tell our software model that an interrupt is ready (Data sent) */
            s->uart_status |= STATUS_TX_EMPTY; 
            break;

        case 0x04: // Interrupt Enable Register
            /* The guest is trying to toggle hardware interrupt lines */
            s->interrupt_mask = value;
            break;

        case 0x08: // Baud Rate Divisor
            /* A real chip would change electrical frequencies here.
               QEMU just updates an integer variable in RAM. */
            s->baud_rate = 115200 / (value ? value : 1);
            break;

        default:
            /* Guest tried to write to an unmapped register address */
            log_bad_hardware_access(offset);
            break;
    }
}

There is no magic anywhere in this. Every device your guest sees is a software model like this. A chunk of code pretending to be a chip. It does this well by responding to the same address accesses the real chip would respond to. And the guest's driver cannot tell the difference, because from the driver's perspective there is no difference.THe guest driver writes to an address and something happens.

Why the guest's firmware finds these devices at all

This architecture raises an obvious question: how does the guest OS know that a serial port is exactly at 0x09000000 in the first place?

On a physical ARM board, this layout is explicitly dictated by something called a Device Tree. A device tree is a structured data file compiled into the firmware that outlines exactly what hardware components exist and what memory addresses they occupy. x86 has legacy hardware discovery conventions, but ARM has no equivalent of this. I won't go into details about x86 architecture because the series is about ARM. Just remember that in ARM the entire motherboard topology must be spelled out line-by-line for the kernel at boot.

QEMU generates a this device tree describing the machine it's emulating, and hands it to the guest at boot. The guest kernel reads it and sees "there's a PL011 UART at 0x09000000," loads the PL011 driver, and starts writing to that address. The driver is the standard, unmodified Linux driver for real PL011 hardware. It has no idea it's talking to C code.

The device tree

Here is a highly simplified, annotated snippet of a real Device Tree Source (.dts). Again, just take a look only to recognize there is no magic.


dts/dts-v1/;

/ {
    interrupt-parent = <0x8001>;
    #address-cells = <0x02>;
    #size-cells = <0x02>;

    /* Tells the Linux kernel this is QEMU's standardized virtual board */
    model = "linux,dummy-virt";              /* Identifies this as the synthetic 'virt' board */
    compatible = "linux,dummy-virt";

    /* Core CPU Configuration (e.g., 2 vCPUs requested) */
    cpus {
        #address-cells = <0x01>;
        #size-cells = <0x00>;

        cpu@0 {
            device_type = "cpu";
            compatible = "arm,cortex-a57";
            reg = <0x00>;
        };

        cpu@1 {
            device_type = "cpu";
            compatible = "arm,cortex-a57";
            reg = <0x01>;
        };
    };

    /* System Memory Layout (e.g., -m 2048M starts at base address 0x40000000) */
    memory@40000000 {
        device_type = "memory";
        reg = <0x00 0x40000000 0x00 0x80000000>;
    };

    /* The Interrupt Controller (GIC) so the CPU can talk to peripherals */
    intc@8000000 {
        compatible = "arm,gic-v3";
        interrupt-controller;
        #interrupt-cells = <0x03>;
        reg = <0x00 0x08000000 0x00 0x010000>,   /* GIC Distributor physical address */
              <0x00 0x080A0000 0x00 0xF60000>;   /* GIC Redistributors physical address */
        phandle = <0x8001>;
    };

    /* A Virtual UART (Serial Port) for console output */
    pl011@9000000 {
        compatible = "arm,pl011", "arm,primecell"; /* Tells Linux which driver binary to load */
        reg = <0x00 0x09000000 0x00 0x1000>;      /* Base address (0x09000000) and memory size (4KB) */
        interrupts = <0x00 0x01 0x04>;            /* Bound to SPI interrupt 1 */
        clocks = <0x8000>;                         /* Tells the guest which hardware interrupt wire it uses */
        clock-names = "uartclk";
    };

    /* ... PCI host bridge, virtio devices, and flash memory nodes continue below ... */
};

At the very top, QEMU defines the basic structure of our synthetic motherboard. It explicitly states that this is not a real-world chip from Raspberry Pi or Apple, but QEMU’s custom engine.
Take a look at the pl011@9000000 block of the code at the end. This is a Virtual UART.

This is also why the virt machine type matters on ARM64. Because it is not emulating any real-world board. It is a synthetic machine defined by QEMU. On x86, machine types like pc and q35 emulate actual historical chipsets, complete with their quirks. ARM64 skipped that inheritance.

The Dual-Engine Architecture of QEMU

QEMU manages peripheral hardware. It has a device job. This should be clear to you by now. But, it also faces a separate challenge: how to execute the guest's CPU instructions. This is QEMU's CPU job.

Half 2 never changes. We discussed in the previous article, that QEMU even without a KVM still able to virtualize. But, it is just slower. The peripheral devices are emulated by QEMU in userspace even if the KVM is missing or inactive. KVM has no built-in device models and never acquires any previous article. (There is a minor exception: KVM implements a few latency-critical components, like the interrupt controller, inside the host kernel to boost performance—but the general rule stands, and that exception is simply an optimization worth remembering.)

Half 1 is the part with two implementations.

TCG — the Tiny Code Generator — This is QEMU's software CPU. It reads blocks of guest instructions, compiles them into equivalent host instructions on the fly, caches the resulting binaries, and runs them. It is a Just-In-Time (JIT) compiler operating between two processor architectures. This is true software emulation. And this is your only option when your guest and host architectures don't match. An x86 guest running on an ARM64 host must use TCG; no amount of hardware assistance can force an ARM core to natively parse x86 machine code.

KVM is the hardware path from article 1.2. Instead of translating instructions in software, QEMU asks KVM to orchestrate the execution. KVM runs the guest code directly on a real, physical core using hardware virtualization extensions. And after that KVM hands control back up to QEMU only when a device trap occurs.

The choice between them is what <domain type='kvm'> versus <domain type='qemu'> selects in libvirt's XML, and it's the single biggest performance factor in the entire stack. We will learn more about these in future posts.

The consequence: what a device access costs

When you combine QEMU's two halves, the central performance reality of all virtualization instantly falls into right place.

Under KVM, guest code runs natively at full speed. Arithmetic, conditional loops, and memory operations inside actual RAM happen directly on the bare metal without any intervention from KVM or QEMU. Millions of instructions can execute back-to-back without either host component ever waking up.

However, a device access can't complete natively, because there's no physical device. It has no choice but to trap out to QEMU's C code.

So the cost model of a virtual machine is: native speed for computation, and a relatively expensive round trip for every device interaction.

This single asymmetry explains an enormous amount of real-world behavior. If you have ever worked with VMs you will be able to relate immediately. It is the exact reason why CPU-bound workloads in a VM perform at near-bare-metal speeds while I/O-heavy workloads lag. It is why a virtual serial console is perfectly fine for login shell. Because a human types slowly. So a trap per character goes unnoticed. However, it would be absolutely catastrophic for a virtual network card processing a hundred thousand packets per second.

And it's the reason VirtIO exists. If device access is the expensive operation, the winning strategy is to make each one carry more work. In other words: you batch. That's article 6.

But you would understand the batching argument only if you know what the round trip actually involves: what "control leaves the guest" means mechanically, what data crosses the boundary, and why some traps are far cheaper than others. That's the next article.

Summary

qemu-system-* emulates a whole machine and boots an OS. qemu-<arch> runs a single foreign binary. Only the first is virtualization.
Devices on real hardware are reached through memory-mapped I/O — the CPU writes to addresses that are wired to chips rather than RAM.
QEMU exploits this by marking regions of the guest address space as device regions. An access there traps and runs a C function. That function is the device.
On ARM64, the guest discovers what exists and where via a device tree QEMU generates. Guest drivers are the standard, unmodified Linux drivers.
QEMU has two halves. Devices are always emulated in QEMU userspace. The CPU is either translated in software (TCG) or run natively on real hardware (KVM).
Under KVM, computation runs at native speed but every device access costs a trap out to QEMU. That asymmetry is the foundation of every performance decision in virtualization.

KVM vs QEMU vs libvirt: Who Does What

Supun Sriyananda — Wed, 02 Sep 2026 06:30:00 +0000

The previous article covered the CPU virtualization extension and how KVM drives it.

"What’s the difference between KVM and QEMU?" is one of the most-asked questions in Linux virtualization. If you look around most answers just stack definitions on top of each other. For me, definitions are easily forgotten. I think the best way is to build the causal chain—understanding that each component exists specifically to solve a limitation left behind by the layer beneath it. So, rather than defining everything upfront and hoping you connect the dots, I will start with KVM’s deliberate boundaries and follows the architectural consequences all the way up.

KVM only virtualizes the CPU and Memory. Nothing else.

The previous article established what KVM(Kernel-based Virtual Machine) contributes. KVM drives the CPU's virtualization extension so guest code executes natively on real cores, trapping only on privileged operations. It also manages guest memory by mapping the guest's idea of physical addresses onto host pages, and keeping each guest confined to its own.

KVM is incredibly efficient at executing CPU instructions and managing guest memory at near-native speed. But it only virtualizes the CPU and Memory. Nothing else.

A naked CPU and raw RAM do not make a computer. A virtual machine needs a motherboard, a PCI bus, a keyboard, a mouse, a graphics card, serial ports, and disk controllers. And KVM deliberately includes none of these hardware simulations. So, if you only had KVM, your virtual machine couldn't boot because it wouldn't have a virtual hard drive to read an OS from, or a virtual screen to display it.

You need to keep this in mind. This is NOT an oversight. It is a design decision, and article 1.1 explained why: because the hypervisor lives inside a full Linux kernel, it inherits everything Linux already does well. Building device emulation into KVM would mean duplicating work the kernel and userspace already handle.

So KVM leaves an obvious gap. A guest kernel boots, probes for a disk, and finds nothing. We need something that can answer.

QEMU Fills the Hardware Gap: It pretends to be all the devices

Because KVM leaves the virtual machine without a "body," it needs a partner. QEMU(Quick Emulator) is a userspace program that emulates an entire machine's worth of devices. When the guest probes for a disk controller, QEMU responds. When the guest writes a byte to a serial port, QEMU receives it. When the guest sends a network packet, QEMU hands it to the host's network stack.

This reveals the exact handoff between KVM and QEMU. When the guest tries to talk to a hardware device, KVM intercepts the request because KVM doesn't know how to handle hardware. KVM pauses the VM and hands the request up to QEMU. Inside QEMU, a piece of C code pretends to be the chip the guest is looking for, handles the data, and tells KVM to let the guest resume. Every piece of "hardware" the guest sees is just an illusion written in software. Article 1.4 covers how that actually works, since "emulates devices" is doing a lot of quiet work in that sentence.

QEMU can also emulate a CPU. On its own it's a complete machine emulator. So, even if you don't give it a KVM, it will translate guest instructions in software and boot an OS anyway, slowly. But if you pair it with a KVM, it hands the CPU and memory job to KVM and keeps only the devices.

That pairing is what a running VM is:

This architecture leads to two structural realities that govern how virtual machines behave on a live Linux system:

One VM is one QEMU process. A virtual machine is not a mysterious container or a cluster of workers. It is a single, standard Linux process (like qemu-system-x86_64 or qemu-system-aarch64) running in user space. qemu-system-x86_64 and qemu-system-aarch64 are examples. qemu-system-x86_64 process emulates a 64-bit Intel / AMD (x86_64) architecture and qemu-system-aarch64 emulates 64-bit ARM (ARM64 / AArch64) architecture. They are executable programs that run as processes on your operating system. So, in your task manager (like top or htop on Linux/macOS, or Task Manager on Windows), you will see qemu-system-x86_64 or qemu-system-aarch64 listed as an active process. If you kill that process, the VM instantly dies. If a guest has four vCPUs, that single process simply spins up four ordinary Linux threads. Each thread runs a continuous loop: execute guest code, halt when something traps, handle the intercept, and resume. Because these are standard host threads, the Linux scheduler manages them normally, which is exactly why you can pin specific vCPUs to physical CPU cores using standard Linux tools.

The two halves exist in a constant state of handoff Because KVM only handles CPU and memory, it runs guest code at native hardware speed right up until the guest tries to interact with a device (like writing to a disk or sending a network packet). KVM cannot handle this, so it pauses the vCPU thread and yields control back up to QEMU. QEMU’s C-coded device model simulates the hardware response, updates the virtual state, and tells KVM to resume execution. That handoff is article 1.5.

So KVM and QEMU together are a complete machine. Which raises the next question: how do you actually start one?

libvirt: Turning a Messy Command Into a Permanent Document

There is the problem with QEMU as a user interface. Describing a virtual machine to it means you have to to pass every single hardware detail as a raw command-line flag.

To launch a standard VM, you have to manually specify each disk's format and bus type, every network interface's model and host backend, the exact firmware paths, the CPU microarchitecture, the serial console configurations, and the explicit memory layout. A realistic, production-ready QEMU invocation easily runs to dozens of flags across multiple lines of unreadable text.

Nobody wants to type that. More importantly, nobody wants to remember it.

Libvirt solves this by making the VM a document rather than a command.

Instead of writing a transient shell script, you describe the machine’s desired state once in a structured XML file. The Libvirt daemon (libvirtd) reads that XML, dynamically constructs the corresponding, massive QEMU command line behind the scenes, launches the process, and tracks its lifecycle from that moment forward.

The Broader Scope: Managing the Ecosystem

libvirt's scope is broader than launching processes. It defines and manages virtual networks, storage pools, and snapshots. It exposes a stable API, so tooling written against libvirt keeps working across QEMU versions. And it maintains a control channel to each running QEMU — the QMP socket — which is how a running VM can have a disk hot-plugged or a snapshot taken without restarting.

The tools you type are all front-ends to that daemon. virsh is the command-line client and the one this series lives in. virt-manager is a desktop GUI. virt-install creates new VMs. Cockpit provides a web interface. None of them talk to QEMU directly — they all send API calls to libvirtd, which does the work.

The payoff is that the XML is the VM. There's no hidden state elsewhere. Change a line in the document and you've changed the machine. That's why the domain XML is the central artifact of libvirt work, and it's the subject that opens the next phase of this series.

Libvirt’s responsibility doesn't end once the QEMU process is running. Its scope is much broader than launching processes. Libvirt defines and manages the entire virtualization ecosystem, including virtual networks, storage pools, and snapshots.

By exposing a stable, unified API, Libvirt ensures that any tooling written against it keeps working seamlessly across different QEMU versions. This shields you from breaking changes in upstream software.

The Frontends: Just Messengers for the Daemon

Because Libvirt acts as the central brain, the tools you actually type or click are just thin frontends to the daemon:

virsh – The powerful command-line client (and the core focus of this series).
virt-manager – A traditional desktop GUI for visual management.
virt-install – A specialized CLI tool dedicated to provisioning new VMs.
Cockpit – A modern, browser-based web interface for server management.

None of these tools talk to QEMU directly. They all send standard API calls to libvirtd, the Libvirt daemon. It executes the heavy lifting behind the scenes.

The ultimate payoff of this architecture is simple: the XML is the VM. There's no hidden state elsewhere. If you change a line in that document, you change the machine. Because the domain XML is the absolute central artifact of Libvirt, it is the exact subject that will open the next phase of this series.

The three sentences

Compressed to their essentials:

KVM runs the guest's CPU and memory on real hardware at native speed.
QEMU pretends to be all the devices the guest thinks it has.
libvirt stores your VM as XML and drives QEMU so you don't have to.

Definitions are handy, but the causal version is far more useful because it explains exactly why the Linux virtualization stack has this specific shape:

KVM is lightweight because Linux already provides everything except raw CPU virtualization. That leaves peripheral hardware unhandled. So, QEMU steps in to simulate devices in userspace. However, QEMU’s resulting command-line interface is completely unusable by hand, so Libvirt abstracts it into a clean, permanent document and manages it for you.

If you want to test your understanding of this stack, imagine pulling a single brick out of the wall. And see what happens:

Without Libvirt: Everything still runs fast and has hardware, but you are stuck hand-writing, troubleshooting, and maintaining enormous, brittle command lines.
Without QEMU: Your virtual machine has a hyper-fast brain but no body. The guest has no virtual motherboard, network card, or disk to boot from.
Without KVM: Everything still works perfectly, and your guest has all its devices. But, it just runs painfully slow completely in software emulation.

What this means for the rest of the series

The division of labour is a map for everything that follows.

Storage work — disk formats, snapshots, thin provisioning — is about what QEMU's virtual disk points at on the host.

Networking work is about what QEMU's virtual NIC plugs into: a NAT bridge, a LAN bridge, a TAP device.

Performance work splits across the boundary. Replacing emulated devices with VirtIO ones is QEMU-side. Pinning vCPU threads to physical cores is host-scheduler-side, and possible only because those threads are ordinary Linux threads.

Passthrough is the exception that proves the rule: it bypasses QEMU's emulation entirely and hands a guest real hardware.

Every one of those is a modification to one part of a machine whose shape you now know.

Summary

KVM handles guest CPU and memory only. It has no device emulation at all, by design, because Linux already provides everything else.
QEMU fills that gap: a userspace program that emulates disks, NICs, consoles, and firmware. It can emulate a CPU too, but hands that job to KVM when available.
One running VM is one QEMU process, with one host thread per vCPU.
libvirt exists because QEMU's command line is impractical to write by hand. It stores each VM as XML, builds the command line, launches QEMU, and manages it via a control socket.
virsh, virt-manager, virt-install, and Cockpit are all front-ends to the libvirtd daemon.
The domain XML is the VM — there's no hidden state elsewhere.

Why KVM Needs the CPU: ARM64 EL2, VHE, and the Virtualization Extension

Supun Sriyananda — Tue, 01 Sep 2026 09:06:04 +0000

The previous article established that KVM is a kernel module that turns the running Linux kernel into a hypervisor.

That description is accurate but incomplete, and the gap is where a lot of otherwise-solid mental models come apart. A kernel can't simply decide to run guest operating systems at native speed. It needs the CPU's cooperation — a hardware feature that sits dormant in the silicon until software switches it on.

This article is about that feature: what problem it solves, what it looks like on ARM64, and how it relates to the kernel module. The relationship between those last two is the part worth getting right, because "is virtualization hardware or software?" has an answer that isn't either.

The problem: a guest kernel wants to run privileged instructions

Recall the lie from the previous article; "every guest OS believes it owns the hardware". That belief has teeth. A kernel doesn't just think it's in charge; it executes instructions that only something in charge is allowed to execute. It reconfigures page tables. It writes to device registers. It masks and unmasks interrupts. It sets up the memory management unit.

Now put that kernel in a VM. It's going to attempt all of those things, because that's what kernels do at boot. Two options present themselves, and both are bad.

Option one: let it run them on the real CPU. The guest reconfigures the actual MMU, writes to actual device registers, and takes down the host and every other guest with it.

Option two: catch every instruction in software. Inspect each one before it executes, and when it's privileged, emulate its effect against fake state instead of letting it touch hardware. This works. And it is how software-only virtualization operates. However, the cost is enormous. Software is now involved in every single instruction the guest runs, including the overwhelming majority that are perfectly harmless arithmetic and branches.

The insight that makes modern virtualization viable is that neither option is necessary if the CPU itself can tell the difference.

But, think about it for a second. What if the CPU can tell the difference?

The fix is in the silicon

Modern CPUs ship with a virtualization extension: a hardware feature that adds a privilege level above the normal kernel level, designed specifically for a hypervisor to occupy.

With it, the guest kernel runs at its usual privilege level, fully believing it's in charge. Its ordinary instructions — the arithmetic, the branches, the memory accesses — execute directly on a real core at full speed, with no software involvement whatsoever. But the moment it attempts something that would affect real hardware, the CPU traps. It freezes the guest and transfers control up to the hypervisor.

This is the crucial economic property. Now the expensive interventions happen only on privileged operations, which are rare. Everything else runs native. The software now can stop babysitting and start handling exceptions.

That trap is the VM exit. A VM Exit is a hardware-driven transition where control shifts from the guest virtual machine back to the host operating system hypervisor. This article establishes what causes a VM exit and why it exists; the mechanics of what happens during one — how control actually returns to userspace, what data crosses the boundary, and why some exits are far cheaper than others — are the subject of article 5.

ARM64's privilege levels, and the x86 equivalents

ARM calls its privilege levels Exception Levels, abbreviated EL. Higher numbers mean more privilege, which is the opposite of x86's ring numbering.

A guest's applications run at EL0. The guest kernel runs at EL1, where it has always expected to run, and where it can do everything a kernel normally does. Above it sits EL2, the hypervisor level, which the guest cannot see or reach.

On x86 the same concept is called VT-x on Intel and AMD-V on AMD, and the additional level is described as "root mode" rather than given a number. The naming differs but the structure is the same. If you know x86 virtualization, EL2 is your root mode.

There's a fourth level on ARM — EL3, for secure-world firmware. This is outside the scope of virtualization and not something KVM touches.

VHE: why EL2 alone wasn't enough

The original ARM virtualization design assumed the hypervisor would be a small, purpose-built piece of software living at EL2. EL2 was designed for that: a lean environment with its own register set, deliberately unlike EL1.

Linux is not that. Linux is a large general-purpose kernel written to run at EL1, and KVM is a module inside it. Under the original design, running KVM meant splitting the kernel. So most of Linux will be at EL1, with a small stub at EL2 handling world switches. It worked, but every transition between the two carried overhead. And this split was structurally unpleasant.

VHE — Virtualization Host Extensions — fixed this. Introduced in ARMv8.1, VHE makes EL2 capable of running an ordinary EL1-style kernel directly, by remapping registers so that a kernel written for EL1 can execute at EL2 without modification. The host kernel, KVM included can run entirely at EL2. Guests run at EL1 below it. Neither split nor stub is needed now.

The practical upshot: on VHE-capable hardware, KVM runs in its efficient configuration. On older ARM cores without it, KVM still works, using the split approach. The Cortex-A76 in the Raspberry Pi 5 supports VHE. If we check the kernel boot log we can confirm this.

Seeing it on real hardware

Here's the boot log from a Raspberry Pi 5 running Raspberry Pi OS Bookworm, kernel 6.12 aarch64:

$ dmesg | grep -i kvm
[    0.046684] kvm [1]: nv: 554 coarse grained trap handlers
[    0.046799] kvm [1]: IPA Size Limit: 40 bits
[    0.046811] kvm [1]: GICV region size/alignment is unsafe, using trapping (reduced performance)
[    0.046835] kvm [1]: vgic interrupt IRQ9
[    0.046846] kvm [1]: VHE mode initialized successfully

Every concept in this article is visible in those five lines.

VHE mode initialized successfully It confirms not just that KVM found a usable virtualization extension, but that it's running in the modern configuration described above. The host kernel is at EL2 without a split. This is the exact moment the dormant silicon feature was switched on, recorded in the boot log.

IPA Size Limit: 40 bits refers to the Intermediate Physical Address space. This is the guest's view of physical memory. When a guest accesses what it believes is a physical address, that's an IPA(Intermediate Physical Address), which hardware then translates to a real host physical address. A 40-bit tracking system can create (2^{40}) unique address combinations(which is 1 Terabyte (TB) of RAM). The hardware allows the virtual machine to hold drastically more memory than your physical computer actually possesses.

vgic interrupt IRQ9 GIC (Generic Interrupt Controller): This is the physical ARM hardware chip responsible for delivering interrupts from hardware devices directly to the CPU. Guests need interrupts too. Because multiple virtual machines cannot safely share the same physical interrupt chip, KVM creates a simulated, virtual version (Virtual GIC) for each VM.

The GICV region size/alignment is unsafe, using trapping (reduced performance) line is a Raspberry Pi–specific quirk. One optimization for injecting interrupts directly into guests isn't safely available on this hardware, so KVM falls back to trapping them. It does not affect whether acceleration works. it makes one specific interrupt path slightly slower, that is all. This is a physical trait of the Raspberry Pi chip that cannot be changed. So we ignore this warning and move on.

nv: 554 coarse grained trap handlers refers to nested virtualization support — running a hypervisor inside a guest. Not relevant to in this series.

Is virtualization silicon or software?

Now we can take on the question this article exists to answer. If virtualization is a CPU feature, why is KVM a kernel module? And if KVM is a kernel module, what is the CPU actually contributing?

They are two halves of one mechanism, and neither accomplishes anything alone.

The CPU extension is a dormant hardware capability. EL2 exists in the Cortex-A76 the moment it leaves the factory. It is baked into the silicon and does nothing on its own. It's a feature waiting for software to claim it. Is is basically an engine with no driver.

The KVM module is the software that claims and drives it. When KVM initializes, this is the sequence:

So the answer to "is virtualization hardware or software" is that it's a partnership with a strict dependency. The extension without KVM is an unused silicon feature. KVM without the extension has no job. It exists solely to drive that hardware, and it will refuse to initialize on a CPU that lacks it.

This dependency has a useful consequence: the existence of /dev/kvm is proof the whole chain worked. Beacause, that file is created only when KVM has successfully initialized against a real virtualization extension. If it is there, the silicon feature is present, the software found it, and the door is open. That's why every KVM troubleshooting guide starts by checking for it, and it's the first command in article 7.

What happens when the extension is absent

This is worth stating explicitly, because it clarifies the boundary. If a CPU has no virtualization extension, you are not stuck. You can virtualize but you're just slow.

QEMU(Quick Emulator) is a complete machine emulator independent of KVM. It is a software program that emulates and virtualizes computers.

QEMU can mimic an entire computer hardware system in software. It lets you run an operating system made for one processor (like an ARM chip) on a completely different processor (like an Intel laptop chip) by translating the code step-by-step instruction by instruction. And QEMU can do this without any hardware support. It works. It's simply far slower and you can guess why. Because software is translating guest instructions rather than letting them run on a real core.

But, if you pair QEMU with a kernel module like KVM, it drops the slow software translation. Instead, it passes instructions directly to your real CPU cores, letting the virtual machine run at near-native hardware speed.

   QEMU alone   = full machine in software    → works, slow
   QEMU + KVM   = QEMU does devices,          → works, fast
                  KVM rides the CPU extension
                  for CPU and memory
   KVM alone    = impossible — KVM only does CPU and memory.
                  It can't be a disk or a NIC. It always needs QEMU.

That third line is the setup for the next article, and there's a consequence of the second because it's specific to ARM64 hosts.

KVM can only accelerate guests whose instruction set matches the host's. An aarch64 guest on an ARM64 host runs natively. The Cortex-A76(like in a Raspberry pi 5) executes the guest's instructions directly, so KVM applies and it's fast. An x86 guest on the same host cannot. Because, those aren't ARM instructions, and no extension makes a real ARM core execute them. QEMU falls back to full software translation.

This isn't a limitation of KVM or of the Raspberry Pi. It's the boundary of what hardware acceleration means. Acceleration requires the guest and host architectures to match; when they don't, you're emulating, and emulation is slow. Anything in this series involving KVM assumes aarch64 guests.

Remember: Hardware acceleration always requires the guest and host architectures to match, regardless of whether you are using ARM or x86.

Summary

A guest kernel executes privileged instructions. Letting them touch real hardware is unsafe; catching every one in software is slow.
CPU virtualization extensions solve this by adding a privilege level above the kernel. Guest code runs natively until it does something privileged, which traps to the hypervisor. That trap is a VM exit.
On ARM64 the hypervisor level is EL2, above EL1 (kernel) and EL0 (user). The x86 equivalent is VT-x/AMD-V root mode.
VHE lets a full Linux kernel run at EL2 unmodified, which is the efficient configuration KVM uses on modern ARM cores. dmesg reports it at boot.
The silicon extension and the KVM module are partners: the extension is dormant hardware, KVM is the software that switches it on. Neither works alone, and KVM refuses to initialize without it.
/dev/kvm existing is proof the partnership succeeded.
KVM only accelerates guests matching the host architecture. x86 guests on ARM64 fall back to slow software emulation.

I smashed a bug so hard and came out with a Python library

Supun Sriyananda — Fri, 21 Aug 2026 15:01:47 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This is the story of my bug spray.

I build sensor systems that live outdoors. Things like Raspberry Pi units on 4G cellular, embedded gateways, battery monitoring hardware in places nobody visits twice a year. On those networks, connection reliability isn't a simple nice-to-have.

I was a junior back then, and I lost data for weeks before I understood why. When I finally went looking for answers, all I found were scattered fixes. Someone had commented on a years-old thread, "this worked for me". Without any consistency anywhere.

So I patched. Patch after patch, bug after bug, year after year. At some point I thought: why not smash this thing once and for all, for everyone?

And so I did. Smashed, and smashed, and smashed. The final result? An open source python library.

The device that went deaf

Let me tell you about the bug first. So that you know what I smashed and you don't feel pity for it.

Picture a busy restaurant. You're working the pass out front; the kitchen is through the back, and the two of you talk over a radio link. Every thirty seconds the kitchen shouts, "All dishes on time!" Then the food stops coming out. You check the channel, the power light and the microphone. And Everything is perfect. The kitchen keeps cheerfully reporting they're doing great, and ignores every order you scream back.(Sounds annoying, right?)

What happened is that the signal hissed for half a second earlier that night. The link dropped and came back instantly. But the radio system had been set up with a rule: when a link drops, wipe everything about it. The kitchen's standing request to be told about incoming orders was part of that everything. So it went ahead and wiped everything!. The kitchen isn't muted and isn't broken. Its request simply isn't on the list anymore, and it has no way of knowing that. So there are no warining lights or warning beeps beacuse there is no failure to report, since nothing failed.

The system expects the kitchen to re-announce "we're listening on channel four" itself, every single reconnect, forever, unprompted. If it doesn't, it wakes up half alive: still broadcasting to the dining room, deaf to every order coming in. And the only channel it has for telling you something is wrong is the half that still works.

The radio wasn't defective. It did exactly what it had been told to do. And who was the person told it? Me!. I told it months earlier, when I set it up. It was just never designed for a kitchen four hours away with nobody standing next to it.

MQTT is a simple messaging system used by smart gadgets to talk to servers, acting like a two-way radio channel where devices must specifically "subscribe" to a frequency to hear messages. By default, if a device loses its connection even for a second, the system resets and forgets what channel it was listening to, requiring the device to manually tune back in. Paho MQTT is an open-source library that gives you this MQTT messaging protocol.

Once I started looking, it wasn't the only bug. A bug spray is required!

The deaf device showed me a buch of other bugs and issues with this whole thing.

Reconnection is your problem, not the library's.

paho has reconnect_delay_set, but only loop_forever() honours it. With loop_start() — which is what you want the moment the client isn't the main thread — you're writing the retry loop yourself. I wrote the naive one. Fixed a five-second retry, like everybody's first attempt.

Now, having fixed the reconnection bug, I wanted to try it on a device. It worked. I was happy. Wasn't so hard, I thought. So a while later I proudly rolled it out to ten devices.

It died right away. Ten devices, one outage, and every one of them waiting exactly five seconds before knocking again — in perfect unison, forever. The broker came up, took ten simultaneous connections in the same instant, and went straight back down. I waited. It never resurrected itself. It couldn't: each restart just called the stampede back.

Ten devices were enough to demostrate this bug. So, yes. I fell to a bug from a bug. The fix was exponential backoff with a jitter, so they don't re-synchronise. This is the difference between recovery and a self-inflicted DoS.

#Backoff needs a cap and jitter.
# And without the jitter, ten devices that dropped together retry together. That's what killed my broker.
delay = min(1.0 * 2 ** attempt, 300)   # cap: never wait over 5 min
time.sleep(random.uniform(0, delay))   # jitter: don't come back in unison

The bug came back!

Smash a bug hard enough and it comes back as a phoenix. Karma, probably. My new backoff loop retried forever, which is what I wanted, because a device four hours away has to keep trying. But it had no idea why a connection ended. So when I hit Ctrl-C, my shutdown handler closed the connection, my own retry loop saw a dead connection, and it immediately opened a new one. The process wouldn't exit. I had made it unkillable by making it reliable.

There is a simple fix, though. When paho tells you the connection dropped, it hands you a number. Zero means you closed it yourself. Anything else means it broke. Only reconnect on "anything else". Nothing does this for you.

#The bug: any disconnect triggers a reconnect.
def on_disconnect(client, userdata, rc):
    reconnect()   # including the disconnect you asked for

#The fix is one number.
def on_disconnect(client, userdata, rc):
    if rc == 0:
        return        # you closed it. leave it closed.
    schedule_reconnect()

The first connection is a different path, and it usually isn't retried at all.

All that backoff logic I just wrote lives in on_disconnect — the callback that fires when a connection we already had goes away. But a device that boots before its 4G modem is ready never had one. connect() raises, main() dies. Because the modem is not ready to communicate. And the process exits before any of my retry code ever runs. When systemd restarts the service if the modem still isn't up, it dies again. That's not a retry loop I designed; it's just a process crashing over and over.

And the final boss, which is the worst: messages just vanish.

My five-minute ceiling had a consequence I hadn't thought about. After a bad drop, the device deliberately just sits there doing nothing for up to five minutes. Which means every reading it takes during that window has nowhere to go. publish() doesn't complain about this. It hands back a result code saying the message never left. But that is all. Paho can queue messages for you, but that queue lives in memory, so it dies with the process. A power cycle in the field takes your data with it. Gone.

I tried a few of my own patches over the years for this without ever fixing it properly. Sensor data is the entire reason the device is out there. That's when I stopped patching and started writing a queue that saves messages to disk, so they survive a restart.

All these four different failure modes had one thing in common: all of them fail quietly.

So I made the thing I'd been looking for: The bug spray!

Everything above became robmqtt — a resilient MQTT client for edge devices.

Offline queue in SQLite — unreachable broker means messages get written to disk, not dropped. They survive restarts and power cycles.
Inflight tracking — sent-but-unacknowledged messages are tracked separately and re-sent on reconnect.
Priority eviction — when the queue fills, low-priority telemetry goes before critical alerts.
Exponential backoff — no stampede.
Subscription registry — subscriptions are restored automatically on every reconnect. No on_connect bookkeeping, no deaf devices. It also means you can call subscribe() before connect(), which is how it should have worked in the first place.
Structured logging throughout — so the next person doesn't lose days to a problem that leaves no trace.

The application code never has to know whether the broker is reachable. That was the entire goal.

from robmqtt import ProductionMQTTClient
import json, time

client = ProductionMQTTClient(
    client_id="field_device_001",
    broker_host="mqtt.yourdomain.com",
    broker_port=1883,
    max_queue_size=5000,
    db_path="./device.db",
)
client.connect()
client.start()

while True:
    client.publish(
        topic="sensors/temperature",
        payload=json.dumps(read_sensor()),
        qos=1,
        priority=5,
    )
    time.sleep(30)

That's it. The bug spray!

What I'd tell myself at the start

Silence is not success. A green dashboard measures the things you thought to measure. My device was reporting perfect health while it was completely deaf to commands, because I'd never built anything that could notice the difference.

If a library leaves it to you, it will not warn you. Reconnection, re-subscription, retry and queueing in paho isn't broken for leaving these open. It's a protocol client, not a delivery guarantee. But every gap it leaves is a gap that fails in actual production.

Write the log line you'd want at 2am. Tests and structured logging exist in robmqtt for one reason: I spent days chasing something that left no trace, and I refuse to do that to whoever picks this up next.

The thing that cost me weeks is now one pip install away. That's all I wanted.

pip install robmqtt

PyPI: https://pypi.org/project/robmqtt/
GitHub: https://github.com/ranaweerasupun/resilient-edge-mqtt-client

Built by Supun Sriyananda — R&D Engineer working on embedded and IoT systems.

What a Hypervisor Is, and Where KVM Fits (Type-1 vs Type-2)

Supun Sriyananda — Sat, 15 Aug 2026 11:36:51 +0000

This series builds the KVM/QEMU/libvirt stack starting from the silicon on ARM64. Commands are from a Raspberry Pi 5 (8 GB) running Raspberry Pi OS Bookworm, kernel 6.12 aarch64. Note that, Every command is aarch64-native, and the terminal output is real. The concepts, however, apply anywhere Linux runs. I chose ARM64 deliberately, because almost every KVM tutorial out there is written for x86.

The first question to start with is: what is a hypervisor, and which kind is KVM?

Virtualization works on a lie!

A normal computer runs one operating system, and that OS believes it owns the hardware. It believes all the RAM, every CPU core, the disks, and the network interface belong to it. This belief is not a simplification for beginners. It is baked into how kernels are written. A kernel executes privileged instructions, sets up memory maps, talks directly to devices, and assumes nothing exists above it.

A hypervisor is what lets several of those operating systems, each believing "I own everything", run on one machine. Each of them keeps believing the lie. The hypervisor's job is to keep that lie convincing enough and quietly share the real hardware underneath.

Everything else I will be writing about in this series, like VM exits, VirtIO, virtqueues, and passthrough, is a technique for maintaining that lie either more convincingly or more cheaply.

The types of hypervisors

The textbook taxonomy divides hypervisors in two:

Type 1 hypervisors run directly on the hardware. There is no general-purpose OS underneath them, which means the hypervisor is the thing the machine boots into. ESXi and Xen are the standard examples. What you get in return is performance and isolation. Because, the hypervisor is the sole layer separating a guest from the hardware, and there is nothing else in the way.

Type 2 hypervisors run as applications on top of a normal operating system. We boot Linux, macOS, or Windows on our computers, then install a program like VirtualBox and launch it the same way we'd launch a browser. Here, what you get in return is convenience. Your usual desktop environment carries on exactly as before, and a virtual machine also ends up being just another application running alongside everything else.

People usually summarise the difference as a choice between speed or convenience.

Where is KVM located. And is it type-1 or type-2 ?

KVM is a kernel module. Once it is loaded, the Linux kernel itself becomes the hypervisor. There is no separate hypervisor layer above or below it. The kernel you are already running gains the ability to execute guests directly on the hardware. I meant literally, not as an analogy. There is only one kernel here. It runs the ordinary applications, and it runs virtual machines too.

Now let's try to place that on the Type-1/Type-2 chart.

It has Type-1's defining property: the hypervisor communicates directly to the silicon, without an intermediary OS between it and the hardware. Guest code runs on real cores at native speed. There is no translation layer and no host kernel being asked to relay requests on the guest's behalf. The Type 1 label was invented to describe hypervisors that let guests run at full speed. KVM does exactly that, so by that reasoning, it belongs in the category.

But it is inside a general-purpose operating system that is simultaneously running SSH sessions, your browser, and a few dozen daemons. Your VMs appear in ps output. They're scheduled by the ordinary Linux scheduler, they are killable with kill, and constrained by cgroups like any other process. So, by the structural argument, that's unmistakably Type 2.

This is why the argument never gets settled. People usually call KVM, "Type-1-ish", and that hedge is fair. Because, the two categories were named before anything like KVM existed.

The practical consequence of this categorization

Because the hypervisor is part of a full Linux kernel, it inherits everything that kernel already knows how to do. It doesn't need its own scheduler, because Linux has one, and vCPUs become ordinary threads on it. It doesn't need its own memory manager, driver model, filesystem layer, or a network stack. Because linux has all of them. A dedicated Type-1 hypervisor has to build or port every one of those.

This inheritance is going to shape this whole series. When we pin vCPU threads to physical cores, we'll be using the ordinary Linux scheduler affinity mechanism, because a vCPU is just a thread!. When guest disks turn out to be files in a directory, that's the host filesystem doing its normal job. Later when we bridge guest networking, we'll use the same Linux bridge that would connect any two interfaces.

The more important point is the flip side: KVM is intentionally minimal. Its responsibilities begin and end with the CPU and memory. It does not know how to present itself as a disk, a network card, a display, or a USB port, and it was never designed to. The reason is that it does not need to. Inside the Linux kernel, mature subsystems already exist for storage, networking, and device management, and above the kernel, in userspace, QEMU takes care of whatever remains.

That naturally leads to the question of how the work is actually split between these pieces. Article 1.3 in this series answers it. Before we get there, though, one thing needs to be cleared up, because saying that "the kernel becomes the hypervisor" ignores something important. A kernel cannot simply choose to run guest operating systems at native speed. It requires the processor to cooperate, and that cooperation comes from a hardware capability built into the silicon that stays inactive until software explicitly enables it.

On ARM64 that feature is EL2, and it is where I will start in [the next article].

Summary

A hypervisor lets multiple operating systems each believe they own the hardware, and shares the real hardware underneath.
Type 1 runs directly on hardware (ESXi, Xen). Type 2 runs as an application on a host OS (VirtualBox).
KVM is a kernel module that turns the running Linux kernel into a hypervisor. Guest code executes directly on real cores, but guests are also ordinary Linux processes.
It's Type-1 by the performance argument and Type-2 by the structural one, hence "Type-1-ish". The taxonomy predates the design.
Because the hypervisor is part of Linux, it reuses Linux's scheduler, memory manager, and drivers. Which is why KVM itself only handles CPU and memory.

MQTT vs HTTP for IoT: What I Learned Using Both in Production

Supun Sriyananda — Thu, 30 Jul 2026 14:00:26 +0000

Every IoT project eventually reaches the same decision point:

Should the device communicate with the cloud over MQTT or HTTP?

Both protocols are mature, widely supported, and capable of moving device data reliably. The more useful question is not which protocol is better, but which communication model fits the device.

I have used MQTT for production edge clients running on deployed robot units and warehouse sensors. I have also used HTTP for a building-management monitoring backend and a smart-entry system built with Flask.

Both worked. They simply created very different systems.

This article explains the trade-offs I saw in practice: bandwidth overhead, connection behavior, command latency, power consumption, infrastructure, fan-out, and operational complexity.

The architectural difference that drives everything else

HTTP is built around a request-response model. A client sends a request, and a server returns a response.

MQTT is built around persistent connections and publish-subscribe messaging. A device connects to a broker, publishes messages to topics, and subscribes to topics it wants to receive.

That difference shapes almost every other trade-off.

With HTTP, a device usually addresses a specific API endpoint:

POST /api/readings
GET /devices/robot-01/commands

With MQTT, the device communicates through topics:

warehouse/sensor-01/temperature
robots/robot-01/commands

The publisher does not need to know which services consume a message. The broker handles routing.

Bandwidth and message overhead

The difference becomes easy to see when you compare a small sensor reading.

An HTTP request might look like this:

POST /api/sensors/temperature HTTP/1.1
Host: api.example.com
Content-Type: application/json
Authorization: Bearer <token>
Content-Length: 52

{"device_id": "sensor_01", "temperature": 23.5}

The payload is small, but the request also carries headers. Authentication tokens can make those headers much larger than the sensor reading itself.

An MQTT PUBLISH packet carries:

Fixed header
Topic name
Packet identifier, when required by QoS
Properties, in MQTT 5
Payload

For a short topic and a small payload, the MQTT packet can be substantially smaller than an equivalent HTTP request. The MQTT fixed header starts at two bytes, although the complete packet is larger once the topic, payload, QoS fields, and any MQTT 5 properties are included.

The comparison also depends on connection reuse.

HTTP/1.1 supports persistent connections by default, so a well-configured client does not need to open a new TCP and TLS connection for every request. HTTP/2 can also multiplex many requests over one connection. Poorly configured clients, short-lived processes, network timeouts, and reconnects can still cause repeated handshakes, but “one request equals one new connection” is not an accurate general rule.

MQTT still has an advantage for frequent, small messages because its application-layer framing is compact and the connection is normally kept open for ongoing messaging.

For one device publishing every few minutes, the absolute difference may not matter. Across hundreds of devices publishing several times per second, it can become significant.

Receiving commands: polling versus subscriptions

The biggest practical difference I encountered was server-to-device communication.

A basic HTTP API is client-initiated. The device sends a request before the server can return anything. A simple implementation therefore polls for commands:

import time
import requests


def poll_for_commands(device_id: str, token: str, interval_seconds: int = 5) -> None:
    while True:
        response = requests.get(
            f"https://api.example.com/devices/{device_id}/commands",
            headers={"Authorization": f"Bearer {token}"},
            timeout=10,
        )
        response.raise_for_status()

        for command in response.json():
            handle_command(command)

        time.sleep(interval_seconds)

At a five-second interval, that is 17,280 requests per device per day, including requests that return no commands. A newly created command may also wait almost five seconds before the next poll.

HTTP-based systems can avoid simple polling with techniques such as long polling, Server-Sent Events, or WebSockets. Those are valid designs, but they add a different connection-management model to the application.

MQTT supports this pattern directly. The device subscribes once, and the broker forwards matching messages over the existing connection:

import json
import paho.mqtt.client as mqtt

DEVICE_ID = "robot-01"
BROKER_HOST = "mqtt.example.com"


def on_connect(client, userdata, flags, reason_code, properties=None):
    client.subscribe(f"devices/{DEVICE_ID}/commands", qos=1)


def on_message(client, userdata, message):
    command = json.loads(message.payload.decode("utf-8"))
    handle_command(command)


client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=DEVICE_ID)
client.on_connect = on_connect
client.on_message = on_message
client.connect(BROKER_HOST, 1883, keepalive=60)
client.loop_forever()

For the telepresence robot, this mattered. Camera and movement-related commands needed low and predictable latency. A five-second polling loop was not acceptable, while an MQTT subscription matched the communication pattern naturally.

For truly hard real-time control, however, neither MQTT nor a general-purpose cloud connection should be treated as a deterministic control bus. Network delay, broker load, retransmission, and connectivity loss still need to be considered.

Reliability and QoS

MQTT offers three protocol-level Quality of Service levels:

QoS 0: delivered at most once
QoS 1: delivered at least once
QoS 2: delivered exactly once between MQTT protocol peers

QoS 1 is useful when losing a message is worse than processing a duplicate. The application must therefore be able to handle duplicate delivery safely.

QoS 2 reduces duplicate delivery at the MQTT protocol layer, but it adds more packet exchanges and does not automatically make the entire application workflow exactly once. A message can still be processed twice if application state, database writes, retries, or downstream integrations are not idempotent.

HTTP can also be reliable, but the retry and deduplication behavior is usually designed at the application layer. For example, a device can retry a failed POST with an idempotency key, while the API stores the key to avoid inserting the same reading twice.

The important point is that neither protocol removes the need to design failure handling.

Power consumption is workload-dependent

It is tempting to say that MQTT always uses less power because its packets are smaller. The real answer is more nuanced.

Radio state often matters more than payload size. Establishing a cellular or Wi-Fi connection, negotiating TCP and TLS, and waiting for network timers can consume more energy than transmitting a short reading.

A continuously connected MQTT client can avoid repeated connection setup and receive commands immediately. That can be efficient for devices that communicate frequently or must remain reachable.

However, maintaining a live MQTT connection requires periodic keepalive traffic and may prevent some devices or modems from entering their deepest sleep states. MQTT PINGREQ and PINGRESP packets maintain the connection; they do not allow a device to turn its radio fully off while keeping the same TCP connection alive.

For a battery-powered sensor that wakes once an hour, uploads one reading, and never receives commands, a short HTTPS request may be simpler and just as efficient. The device must reconnect either way after deep sleep.

MQTT persistent sessions are still valuable for intermittently connected devices. A broker can preserve subscriptions and queue eligible QoS messages while the client is offline, subject to the broker configuration, protocol version, session settings, and retention limits. When the device reconnects and resumes the session, it can receive the stored messages.

So the useful rule is:

Frequent bidirectional communication often favors MQTT.
Rare uplink-only communication may favor HTTP.
Measure on the actual radio, modem, network, and sleep schedule before making a battery-life claim.

Infrastructure and operational complexity

HTTP is easy to start with because almost every platform already supports it.

A small ingestion API can be only a few lines:

import sqlite3
import time

from flask import Flask, jsonify, request

app = Flask(__name__)


@app.post("/api/readings")
def receive_reading():
    data = request.get_json()

    with sqlite3.connect("readings.db") as connection:
        connection.execute(
            """
            INSERT INTO readings (timestamp, device_id, value)
            VALUES (?, ?, ?)
            """,
            (time.time(), data["device_id"], data["value"]),
        )

    return jsonify({"status": "ok"}), 201

The tooling is familiar: curl, Postman, browser developer tools, reverse proxies, API gateways, serverless functions, status codes, and standard observability platforms.

MQTT requires a broker. Mosquitto is a popular lightweight open-source option, while managed services such as AWS IoT Core and HiveMQ Cloud reduce the operational burden.

Running a broker means thinking about:

Device authentication
TLS certificates
Topic-level authorization
Session persistence
Message expiry and retained messages
Broker storage
Monitoring and capacity
Reconnection storms
High availability

A single broker instance can become a single point of failure, but MQTT itself does not require a single-instance deployment. Production systems can use clustered brokers, load-balanced endpoints, replicated state, and managed broker services.

HTTP is not automatically immune to failure either. A single Flask process is also a single point of failure. HTTP infrastructure is often easier to scale horizontally because stateless APIs and load balancers are so widely supported, but both architectures require deliberate redundancy.

Fan-out is where MQTT feels fundamentally different

MQTT is especially useful when one message has several consumers.

A sensor can publish once:

sensors/warehouse/temp01

Several services can subscribe independently:

Dashboard
Alerting service
Time-series database
Analytics pipeline
Anomaly detector

The sensor does not need to know that these consumers exist. A new consumer can be added without changing device firmware.

With HTTP, the device can still send one request to an ingestion service, but that service must then distribute the event. This is commonly done with a queue, event bus, stream, or webhook system.

That architecture can be excellent. It simply means that HTTP handles device ingestion while another system provides asynchronous fan-out.

Debugging and developer experience

HTTP is usually easier to inspect manually.

You can reproduce a request with curl, view status codes, inspect headers, and test an endpoint without maintaining a long-lived client session.

MQTT debugging is still manageable, but it requires different tools and concepts:

mosquitto_sub -h broker.example.com -t 'sensors/#' -v

mosquitto_pub -h broker.example.com \
  -t 'devices/robot-01/commands' \
  -m '{"action":"stop"}' \
  -q 1

You also need to understand wildcard subscriptions, retained messages, session state, QoS acknowledgements, duplicate delivery, and topic permissions.

In my experience, HTTP produces a faster first prototype. MQTT often produces a cleaner final design once the system becomes continuously connected, bidirectional, or event-driven.

A practical decision framework

Choose HTTP when:

The device integrates with an existing REST API.
Communication is infrequent and mostly device-to-cloud.
The device uploads large payloads, images, logs, or firmware files.
Simple debugging and minimal infrastructure matter most.
Server-to-device messaging is unnecessary or can tolerate polling.

Choose MQTT when:

Devices must receive commands or configuration changes quickly.
Messages are small and frequent.
Network links are unreliable or intermittent.
Multiple services need the same device events.
Session state, topic subscriptions, and QoS are useful.
Bandwidth efficiency matters across a large fleet.

Use both when the workloads differ.

That is the design I now prefer for many IoT systems:

MQTT for telemetry, commands, device status, and real-time events
HTTP for firmware downloads, image uploads, administration, and third-party integrations

The broker handles operational messaging. The API handles resource-oriented workflows and large transfers.

Final takeaway

The decision is not really “MQTT or HTTP?”

It is:

Does this device behave more like an API client, or more like a participant in a live event system?

For an occasional one-way upload, HTTP is often the simplest answer.

For a connected fleet that publishes continuously, receives commands, survives intermittent networks, and feeds several downstream services, MQTT usually fits better.

In production, the strongest architecture is often not choosing one protocol everywhere. It is giving each protocol the job it was designed to do.

Have you used MQTT and HTTP in the same IoT system? I would be interested to hear where you drew the boundary between them.